How do I get query results from QuestDB into a Pandas dataframe?

Question

I'm connecting to QuestDB using psycopg2 from Python and running select * from my_table but the column names are sequential numbers:

import psycopg2
import pandas as pd

dataframe = pd.DataFrame()
try:
    connection = psycopg2.connect(user="myuser",
                                  password="mypassword",
                                  host="127.0.0.1",
                                  port="8812",
                                  database="qdb")
    cursor = connection.cursor()
    cursor.execute("SELECT * FROM my_table")

    dataframe = pd.DataFrame(cursor.fetchall())

except (Exception, psycopg2.Error) as error:
    print("Error while connecting to QuestDB", error)
finally:
    if (connection):
        cursor.close()
        connection.close()
        print("QuestDB connection closed")
print(dataframe)

The rows look like the following without column names:

                           0    1  2
0 2021-08-23 12:15:43.582771  100  1
1 2021-08-23 12:15:46.529379    1  2
2 2021-08-23 12:15:46.656823    1  2
3 2021-08-23 12:15:46.662040    1  2
4 2021-08-23 12:15:46.805505    1  2
5 2021-08-23 12:15:46.807359    1  2
6 2021-08-23 12:15:48.631560    1  2
7 2021-08-23 12:16:08.120285    6  3

Answer 1

The problem is that fetchall gets the results for each row in the cursor, to properly return the table results, use read_sql_query:

import psycopg2
import pandas as pd

dataframe = pd.DataFrame()
try:
    connection = psycopg2.connect(user="admin",
                                  password="quest",
                                  host="127.0.0.1",
                                  port="8812",
                                  database="qdb")
    cursor = connection.cursor()
    dataframe = pd.read_sql_query("select * from my_table",connection)

except (Exception, psycopg2.Error) as error:
    print("Error while connecting to QuestDB", error)
finally:
    if (connection):
        cursor.close()
        connection.close()
        print("QuestDB connection closed")
print(dataframe)

This returns:

                    timestamp  event  origin
0  2021-08-23 12:15:43.582771    100       1
1  2021-08-23 12:15:46.529379      1       2
2  2021-08-23 12:15:46.656823      1       2
3  2021-08-23 12:15:46.662040      1       2
4  2021-08-23 12:15:46.805505      1       2
5  2021-08-23 12:15:46.807359      1       2
6  2021-08-23 12:15:48.631560      1       2
7  2021-08-23 12:16:08.120285      6       3
8  2021-08-23 12:16:58.080873      6       3
9  2021-08-23 12:16:58.081986      6       3
10 2021-08-23 12:16:58.084083      1       3