Python streaming API data output, how to save it to csv?

Question

I am trying something new with streaming data API and see all this data in the output as a dictionary but now what? How do I do a dump into a csv file so I can do something with it?

# Initialize the `StreamingApiClient` service.
streaming_api_service = td_client.streaming_api_client()

# Stream equity bars
streaming_services.chart(
    service=ChartServices.ChartEquity,
    symbols=['QQQ'],
    fields=ChartEquity.All
)

# Start Streaming.
streaming_api_service.open_stream()

Live output to the terminal:

{'data': [{'command': 'SUBS',
           'content': [{'1': 309.29,
                        '2': 309.29,
                        '3': 309.27,
                        '4': 309.27,
                        '5': 2.0,
                        '6': 611,
                        '7': 1654636260000,
                        '8': 19150,
                        'key': 'QQQ',
                        'seq': 538}],
           'service': 'CHART_EQUITY',
           'timestamp': 1654636324096}]}

This output is updated every minute until termination of the script. How do I capture this so I can append to a CSV?

Answer 1

You could use Python's csv.DictWriter() to help with this. This takes a dictionary and writes it as a row. You appear to only want certain fields, this can be done by ignoring fields not present in the fieldnames parameter. For example:

import csv

data = {'data': [{'command': 'SUBS',
           'content': [{'1': 309.29,
                        '2': 309.29,
                        '3': 309.27,
                        '4': 309.27,
                        '5': 2.0,
                        '6': 611,
                        '7': 1654636260000,
                        '8': 19150,
                        'key': 'QQQ',
                        'seq': 538}],
           'service': 'CHART_EQUITY',
           'timestamp': 1654636324096}]}
           
req_fields = ["7", "1", "2", "3", "4", "5"]

with open("output.csv", "a", newline="") as f_output:
    csv_output = csv.DictWriter(f_output, fieldnames=req_fields, extrasaction="ignore")
    csv_output.writeheader()        # only needed once
    csv_output.writerow(data['data'][0]['content'][0])

This would give you output.csv containing:

7,1,2,3,4,5
1654636260000,309.29,309.29,309.27,309.27,2.0