python virtual environments in aws EMR

Question

How to have mutiple python virtual environments in an aws EMR cluster . The users will be using Zeppelin or Jupyter for each projects and each project will have different set of python libraries or python versions

Answer 1

You can write a bash script to loop over a map-like structure containing the name of your virtual environment and its libraries (it'll make things easier to maintain if you have multiple projects). The body of the loop will consist of:

1. Create a virtual environment: virtualenv project_foo
2. Activate the virtual environment: source project_foo/bin/activate
3. Install ipykernel which provides you with an Ipython kernel for your Jupyter notebook
4. Install your python libraries
5. Add the virtual environment to Jupyter: python -m ipykernel install --user --name=project_foo
6. Exit the virtual environment: deactivate

Now use Bootstrap Actions to run the script on startup. You should be able to see your virtual environments from the Jupyter's Launcher.