How do I have multiple versions of numpy installed on Amazon EMR and how to I delete the early versions?

Question

I do not understand how Python can have multiple versions of a single package installed, or why, when I have multiple versions installed, import package does not give me the most recent one.

I am using AWS linux and the AWS repo in AWS EMR. When I install Python 3.6 it gives me Numpy version 1.14.5 by default. I want to use version 1.19.1 and have put it in a directory.

$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/packages/ numpy==1.19.1
Requirement already up-to-date: numpy==1.19.1 in /usr/local/lib64/python3.6/site-packages
$

But notice:

 $ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.14.5'
>>>

Because I have both version 1.14.5 and version 1.19.1 installed.

I can uninstall version 1.19.1 and then I can uninstall version 1.14.5, although it seems that I have to uninstall it twice (see below for the extended remix).

So how do I make it so that import numpy gets version 1.19.1 and not the older version?

$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib64/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib64/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/__version__.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/compat.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/environment.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/f2py/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/fft/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/tests/test_fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/lib/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/linalg/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/ma/version.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/randomkit.h
  /usr/local/lib64/python3.6/site-packages/numpy/testing/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/__init__.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/nosetester.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/parameterized.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/utils.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nosetester.py
Proceed (y/n)? y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib/python3.6/site-packages/numpy/LICENSE.txt
  /usr/local/lib/python3.6/site-packages/numpy/__config__.py
  /usr/local/lib/python3.6/site-packages/numpy/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/_distributor_init.py
  /usr/local/lib/python3.6/site-packages/numpy/_globals.py
  /usr/local/lib/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/_inspect.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/py3k.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/conftest.py
  /usr/local/lib/python3.6/site-packages/numpy/core/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib/python3.6/site-packages/numpy/core/_internal.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_methods.py
  /usr/local/lib/python3.6/site-packages/numpy/core/arrayprint.py
  /usr/local/lib/python3.6/site-packages/numpy/core/cversions.py
  /usr/local/lib/python3.6/site-packages/numpy/core/defchararray.py
  /usr/local/lib/python3.6/site-packages/numpy/core/einsumfunc.py
  /usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py
  /usr/local/lib/python3.6/site-packages/numpy/core/function_base.py
  /usr/local/lib/python3.6/site-packages/numpy/core/generate_numpy_api.py
  /usr/local/lib/python3.6/site-packages/numpy/core/getlimits.py
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__multiarray_api.h
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__ufunc_api.h
...
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/lahey.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/mips.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/nag.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/none.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pathf95.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pg.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/sun.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/vast.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/from_template.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/intelccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/lib2def.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/line_endings.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/log.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw/gfortran_vs2003_hack.c
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw32ccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvc9compiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/numpy_distribution.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/pathccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_exec_command.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_gnu.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_nagfor.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/unixccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/basics.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/broadcasting.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/byteswapping.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/constants.py
...
  /usr/local/lib/python3.6/site-packages/numpy/testing/print_coercion_tables.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_decorators.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_doctesting.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_utils.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/utils.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_ctypeslib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_matlib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_numpy_version.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_reloading.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_scripts.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_warnings.py
  /usr/local/lib/python3.6/site-packages/numpy/version.py
Proceed (y/n)? Y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/gits/das-vm-config/dist/python numpy==1.19.1
Collecting numpy==1.19.1
Installing collected packages: numpy
Successfully installed numpy-1.19.1
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.19.1'
>>>

Answer 1

Our solution is to do the following in the EMR bootstrap script (which gets run on every node):

1. Use yum to install python36 and python36-pip
2. use pip-3.6 with the --target option to install numpy-1.19.1 in /mnt/python_additions. We do this in the Bootstrap action so that it happens on every node.
3. Let Amazon's script install python36-numpy, which installs numpy-1.14.4 in the site library location.
4. In the program that gets called from spark-submit, add /mnt/python_additions to sys.path.

It's a hack, but it works.