dynamodb pipe object to pandas dataframe

Question

I am really struggling figuring out how to move my database values from AWS Dynamodb to a pandas dataframe. My data isn't very large (100,000 rows). I got the data into s3 then downloaded it for simplicity. I tried using boto3 and S3Fs to get the data into a pandas frame without success, so for now I have given into just downloading the file directly after creating a pipe to S3. The output comes in the following format which pd.read_JSON, pd.read_csv, and pd.read_table don't seem to comprehend.

 {"id":{"s":"3115136104"},"created":{"s":"Wed Mar 25 15:15:35 +0000 2015"},"location":{"s":"Dover, Kent"},"description":{"s":"#TrafficandWeather information from the #PortofDover - follow for regular round the clock updates. NOTE: we are not always able to respond to queries"},"friends_count":{"n":"66"},"name":{"s":"Port of Dover Travel"},"URL":{"s":"doverport.co.uk/weather/"},"statuses_count":{"n":"11062"},"lang":{"s":"en"},"followers_count":{"n":"11517"}}

This is clearly in a JSON format, but because it has broken each line item into a dictionary without a key. I am completely baffled as to how to get pandas to read this.

My question is thus, what is the easiest way to get an entire dynamodb table into Pandas, assuming there are some empty values? Note i have attempted this many different ways without getting solid forward movement in any direction which is why I have not included any general code.

Answer 1

I have answered the question myself.

columns = ['id', 'created', 'description', 'followers_count', 'friends_count', 'lang', 'location', 'name', 'statuses_count', 'URL']
df = pd.DataFrame()
with open(r'C:\dynamodb-in-s3-file-that-was-downloaded') as s3:
    for item in s3:
        newdf = pd.read_json(item)
        newdf.fillna(method='ffill', inplace=True)
        newdf = newdf.loc['s']
        df = df.append(newdf, ignore_index=True)