DtypeWarning: Columns (15,16,18,24) have mixed types. Columns get removed if it has mixed type

Question

I am trying to read a csv file

pd.set_option('display.max_columns', None)
inventory = pd.read_csv('inventory-new.csv', sep=";", names=columns)

it says:

DtypeWarning: Columns (15,16,18,24) have mixed types. Specify dtype option on import or set low_memory=False.
interactivity=interactivity, compiler=compiler, result=result)

and column numbers 15,16,18,24 gets completely removed

I tried:

inventory = pd.read_csv('inventory-new.csv', sep=";", names=columns, dtype=object)

also

inventory = pd.read_csv('inventory-new.csv', sep=";", names=columns, low_memory=False)

but the result is still the same. Why is this happening?

Answer 1

You need to set a dtype for each column.

From the doc:

dtype : Type name or dict of column -> type, default None

Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32} Use str or object to preserve and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.

Why it is happening ?

Most of the time, pandas try to figure out the dtype before processing rows. But if it happens that a value is not of the selected dtype, it will raise an error. Thus you will need to either correct the original data or choose a more permissive dtype to import (like you did with object).