Real time data plotting from a high throughput source

Question

I want to plot Real time in a way that updates fast.

The data I have:

• arrives via serial port at 62.5 Hz
• data corresponds to 32 sensors (so plot 32 lines vs time).
• 32points *62.5Hz = 2000 points/sec

The problem with my current plotting loop is that it runs slower than 62.5[Hz], meaning I miss some data coming in from serial port.

I am looking for any solution to this problem that allows for:

• All data from serial port to be saved.
• Plots the data (even skipping a few points/using averages/eliminating old points and only keeping the most recent)

Here is my code, I am using random data to simulate the serial port data.

import numpy as np 
import time 
import matplotlib.pyplot as plt

#extra plot debugging
hz_ = [] #list of speed
time_=[] #list for time vs Hz plot


 #store all data generated
store_data = np.zeros((1, 33))
 #only data to plot 
to_plot = np.zeros((1, 33)) 
 #color each line 
colours = [f"C{i}" for i in range (1,33)]

fig,ax = plt.subplots(1,1, figsize=(10,8))
ax.set_xlabel('time(s)')
ax.set_ylabel('y')
ax.set_ylim([0, 300])
ax.set_xlim([0, 200])

start_time = time.time()
for i in range (100):
    loop_time = time.time()
     #make data with col0=time and col[1:11] = y values
    data = np.random.randint(1,255,(1,32)).astype(float) #simulated data, usually comes in at 62.5 [Hz]
    data =  np.insert(data, 0, time.time()-start_time).reshape(1,33) #adding time for first column
    store_data = np.append(store_data, data , axis=0)
    to_plot = store_data[-100:,]
    
    for i in range(1, to_plot.shape[1]):
        ax.plot(to_plot[:,0], to_plot[:,i],c = colours[i-1], marker=(5, 2), linewidth=0, label=i)
        #ax.lines = ax.lines[-33:] #This soluition speeds it up, to clear old code. 

    fig.canvas.draw()  
    fig.canvas.flush_events()
    Hz = 1/(time.time()-loop_time)
     #for time vs Hz plot
    hz_.append(Hz)
    time_.append( time.time()-start_time)
    print(1/(time.time()-loop_time), "Hz - frequncy program loops at")
   
 #extra fig showing how speed drops off vs time
fig,ax = plt.subplots(1,1, figsize=(10,8))
fig.suptitle('Decreasingn Speed vs Time', fontsize=20)
ax.set_xlabel('time(s)')
ax.set_ylabel('Hz')

ax.plot(time_, hz_)
fig.show()

I also tried while using

ax.lines = ax.lines[-33:]

to remove older points, and this speed up the plotting, but still slower than the speed i aquire data.

Any library/solution to make sure I collect all data and plot the general trendlines (so even not all points) is ok. Maybe something that runs acquiring data and plotting in parallel?

Answer 1

You could try to have two separate processes:

• one for acquiring and storing the data
• one for plotting the data

---

Below there are two basic scripts to get the idea. You first run gen.py which starts to generate numbers and save them in a file. Then, in the same directory, you can run plot.py which will read the last part of the file and will update the a Matplotlib plot.

Here is the gen.py script to generate data:

#!/usr/bin/env python3

import time
import random

LIMIT_TIME = 100  # s
DATA_FILENAME = "data.txt"


def gen_data(filename, limit_time):
    start_time = time.time()
    elapsed_time = time.time() - start_time
    with open(filename, "w") as f:
        while elapsed_time < limit_time:
            f.write(f"{time.time():30.12f} {random.random():30.12f}\n")  # produces 64 bytes
            f.flush()
            elapsed = time.time() - start_time
            

gen_data(DATA_FILENAME, LIMIT_TIME)

and here is the plot.py script to plot the data (reworked from this one):

#!/usr/bin/env python3


import io
import time
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.animation


BUFFER_LEN = 64
DATA_FILENAME = "data.txt"
PLOT_LIMIT = 20
ANIM_FILENAME = "video.gif"


fig, ax = plt.subplots(1, 1, figsize=(10,8))
ax.set_title("Plot of random numbers from `gen.py`")
ax.set_xlabel("time / s")
ax.set_ylabel("random number / #")
ax.set_ylim([0, 1])


def get_data(filename, buffer_len, delay=0.0):
    with open(filename, "r") as f:
        f.seek(0, io.SEEK_END)
        data = f.read(buffer_len)
        if delay:
            time.sleep(delay)
    return data


def animate(i, xs, ys, limit=PLOT_LIMIT, verbose=False):
    # grab the data
    try:
        data = get_data(DATA_FILENAME, BUFFER_LEN)
        if verbose:
            print(data)
        x, y = map(float, data.split())
        if x > xs[-1]:
            # Add x and y to lists
            xs.append(x)
            ys.append(y)
            # Limit x and y lists to 10 items
            xs = xs[-limit:]
            ys = ys[-limit:]
        else:
            print(f"W: {time.time()} :: STALE!")
    except ValueError:
        print(f"W: {time.time()} :: EXCEPTION!")
    else:
        # Draw x and y lists
        ax.clear()
        ax.set_ylim([0, 1])
        ax.plot(xs, ys)


# save video (only to attach here) 
#anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1, frames=3 * PLOT_LIMIT, repeat=False)
#anim.save(ANIM_FILENAME, writer='imagemagick', fps=10)
#print(f"I: Saved to `{ANIM_FILENAME}`")

# show interactively
anim = mpl.animation.FuncAnimation(fig, animate, fargs=([time.time()], [None]), interval=1)
plt.show()
plt.close()

Note that I have also included and commented out the portion of code that I used to generate the animated GIF above.

I believe this should be enough to get you going.