Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Specify dtype option on import or set low_memory=False

I am using the following code:

df = pd.read_csv('/Python Test/AcquirerRussell3000.csv')

I have the following type of data:

18.07.2000  27.1875         0 08.08.2000  25.3125       0.1 05.09.2000  \
0   19.07.00  26.6250 -0.020690   09.08.00  25.2344 -0.003085   06.09.00   
1   20.07.00  26.6250  0.000000   10.08.00  25.1406 -0.003717   07.09.00   
2   21.07.00  25.6875 -0.035211   11.08.00  25.5781  0.017402   08.09.00   
3   24.07.00  26.2500  0.021898   14.08.00  25.4375 -0.005497   11.09.00   
4   25.07.00  26.6875  0.016667   15.08.00  25.5625  0.004914   12.09.00 

I am getting the following error:

Pythone Test/untitled0.py:1: DtypeWarning: Columns (long list of numbers) have mixed types. 
Specify dtype option on import or set low_memory=False.

So every 3rd column is a date the rest are numbers. I guess there is no single dtype since dates are strings and the rest is a float or int? I have about 5000 columns or more and around 400 rows.

I have seen similar questions to this but dont quite know how to apply this to my data. Furthermore I want to run the following code after to stack the data frame.

a = np.arange(len(df.columns))
df.columns = [a % 3, a // 3]
df = df.stack().reset_index(drop=True)

df.to_csv('AcquirerRussell3000stacked.csv', sep=',')

What dtype should I use? Or should I just set low_memory to false?

like image 342
Elias K. Avatar asked Oct 22 '25 08:10

Elias K.


2 Answers

This solved my problem from here

dashboard_df = pd.read_csv(p_file, sep=',', error_bad_lines=False, index_col=False, dtype='unicode')

Could anyone explain this answer to me tough?

like image 122
Elias K. Avatar answered Oct 23 '25 20:10

Elias K.


df = pd.read_csv('/Python Test/AcquirerRussell3000.csv', engine='python')

or

df = pd.read_csv('/Python Test/AcquirerRussell3000.csv', low_memory=False)

does the trick for me.

like image 29
Ka-Wa Yip Avatar answered Oct 23 '25 22:10

Ka-Wa Yip



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!