Adding row to dataframe with time stamp index

Question

I'm trying to dynamically add a new row to a Pandas data frame. The index is a timestamp and I cannot figure out how to insert a new row without messing up the index. First part of the code creates the data frame:

data = {'time_stamp': ['2014-05-01 18:47:05.069', '2014-05-01 18:47:05.119', '2014-05-02 18:47:05.230',],
        'col_a': [34, 25, 26],
        'col_b' : [21,32,43]}
df = pd.DataFrame(data, columns = ['time_stamp', 'col_a', 'col_b'])
df['time_stamp'] = pd.to_datetime(df['time_stamp'], format="%Y-%m-%d %H:%M:%S.%f")
df.index = df['time_stamp'] # Make time_stamp the index
del df['time_stamp']        # Drop the initial time_stamp column
print df

Result:

                             col_a  col_b
time_stamp                           
2014-05-01 18:47:05.069     34     21
2014-05-01 18:47:05.119     25     32
2014-05-02 18:47:05.230     26     43

Trying to add a row with concat (same problem with append):

#Insert new row (corresponding to an incoming update message with a time stamp an a new value on col_a
ts = pd.to_datetime("2014-05-04 18:47:05.487", format="%Y-%m-%d %H:%M:%S.%f")
new_row = pd.DataFrame([[11]], columns = ["col_a"])
df = pd.concat([df, pd.DataFrame(new_row)], ignore_index=False)
print df

Result:

                            col_a  col_b
2014-05-01 18:47:05.069000     34   21.0
2014-05-01 18:47:05.119000     25   32.0
2014-05-02 18:47:05.230000     26   43.0
0                              11    NaN

If I extend "new_row" with a column called "time_frame" and a corresponding time stamp, it will create a new column called "time_stamp", rather than inserting a new value in the index column.

                            col_a  col_b              time_stamp
2014-05-01 18:47:05.069000     34   21.0                     NaT
2014-05-01 18:47:05.119000     25   32.0                     NaT
2014-05-02 18:47:05.230000     26   43.0                     NaT
0                              11    NaN 2014-05-04 18:47:05.487

Any ideas would be greatly appreciated.

Answer 1

Let's try using the index paramater in the pd.DataFrame construct.

ts = pd.to_datetime("2014-05-04 18:47:05.487", format="%Y-%m-%d %H:%M:%S.%f")
new_row = pd.DataFrame([[11]], columns = ["col_a"], index=[ts])
df1 = pd.concat([df, pd.DataFrame(new_row)], ignore_index=False)
print(df1)

Output:

                        col_a  col_b
2014-05-01 18:47:05.069     34   21.0
2014-05-01 18:47:05.119     25   32.0
2014-05-02 18:47:05.230     26   43.0
2014-05-04 18:47:05.487     11    NaN