python EOF error when reading csv data with pylab (matplotlib)

Question

I am trying to plot data from a csv file using matplotlib (in Python 2.6), but I am having some trouble with reading in the data from csv:

import pylab

# works fine - manually output data for debug
with open(datafile,'rb') as f:
    for row in f:
        print row

# fails - "no data" error
a = pylab.loadtxt(datafile, comments='#', delimiter=',', skiprows=1)

Manually reading the data works fine (the with open part). The pylab.loadtxt code throws an error:

raise IOError('End-of-file reached before encountering data.')
    IOError: End-of-file reached before encountering data.

I originally thought it was a problem with newline characters in the data file (i.e. maybe everything is on one line and is skipped by skiprows=1), but I ruled this out by manually creating a test file in notepad and seeing the same error. Here is the data in the test file:

time,temperature
193,23.1
4040,23.2
4357,23.3
4423,23.4

I also tried deleting the header row and omitting the skiplines=1 part of the code. This also failed, but with a different error:

ValueError: invalid literal for float(): 23.1

At least this showed it "sees" the numerical data.

What am I doing wrong here?

Answer 1

On Windows the line separator is \r\n. On Unix the line separator is \n. Your data file is not following either of these conventions, which is why pylab (err, numpy) is failing to parse the file correctly.

To fix the file:

import os
outfile = datafile+'-fixed'
with open(datafile, 'rb') as f, open(outfile, 'wb') as g:
    content = f.read()
    g.write(content.replace('\r', '\r\n'))
os.rename(outfile, datafile)