Replicating Anaconda Environment From Windows Command Prompt

Question

I want to run Python code from a Windows command prompt. Some of the features in scripts that I run in an Anaconda environment, i.e. from Anaconda Prompt, Spyder, or Jupyter Notebook, are not available when I run the same script from a Windows command prompt. For instance, the read.csv method in Pandas can read a 2.8 GB tab-delimited text file when the script is run in an Anaconda environment, but I get the following message when the same script is run from a Windows command prompt within the environment where I've installed a copy of Python: "pandas.errors.ParserError: Error tokenizing data. C error: out of memory". I assume the Anaconda environment has a package installed for reading large text files. I've tried running the script from my home directory, where Anaconda stores my Python scripts by default, and I still receive the same "out of memory" message. I've also run the script from C:[my home directory]\AppData\Local\Continuum\anaconda3, where Anaconda is apparently installed, and I experience the same memory problem. Does anybody know how I can run a Python script in the same environment that Anaconda Prompt, Spyder, and Jupyter Notebooks use?

Answer 1

This might be resulting from using different Python interpreters. For example, the 32-bit version of Python in windows only gets 2GB of memory to use. There's a chance that you might be using that on the command line vs a 64-bit interpreter in your notebook.

Compare the environments using the sys package. From the command line, try the following command.

python -c "from __future__ import print_function; import sys; print(sys.version); print(sys.executable)"

I shared an example below. In this case, I'm running these commands from inside Anaconda Prompt, which comes installed with (all?) versions of Anaconda.

(base) C:\Users\my.user.name>python -c "from __future__ import print_function; import sys; print(sys.version); print(sys.executable)"
3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
C:\ProgramData\Anaconda3\python.exe

(base) C:\Users\my.user.name>conda env list
# conda environments:
#
base                  *  C:\ProgramData\Anaconda3
colors                   C:\Users\my.user.name\.conda\envs\colors
someotherenv             C:\Users\my.user.name\.conda\envs\someotherenv
hellostevep              C:\Users\my.user.name\.conda\envs\hellostevep

(base) C:\Users\my.user.name>conda activate colors

(colors) C:\Users\my.user.name>python -c "from __future__ import print_function; import sys; print(sys.version); print(sys.executable)"
3.7.2 (default, Feb 21 2019, 17:35:59) [MSC v.1915 64 bit (AMD64)]
C:\Users\my.user.name\.conda\envs\colors\python.exe

Notice that python.exe is different depending on my environment. You may have additional Python environments on your OS that are separate from Anaconda. For example, you may have a Python environment setup for Windows PowerShell. If so, you could run the command python -c "from __future__ import print_function; import sys; print(sys.version); print(sys.executable)" from your PowerShell prompt as well and then compare the output.

Inside of the Jupyter Notebook where the Pandas method is working, try running similar code inside a cell (example below).

import sys
print(sys.version)
print(sys.executable)

Do you get the same output?

If the output is identical between your notebook and the command line, this may not be the issue you're experiencing. If so, could you share the code that you're executing from inside the notebook as well as the code from the script that you're running on the command line?