Spark Cell magic not found

Question

I have python2 env. on windows 10 with jupyter notebook.

after following instructions in this tutorial I managed to install spark on windows 10: https://medium.com/@GalarnykMichael/install-spark-on-windows-pyspark-4498a5d8d66c

but when I try to run cell magic for SQL I get the following error :

ERROR:root:Cell magic %%sql not found.

when I used %lsmagic I could not find sql cell magic among them.

also I noticed there was no option for pyspark kernel when starting new notebook in Jupyter.

Answer 1

Are you trying to use SQL or Spark-SQL? I've used iPython-SQL which was great, and there's also SparkMagic which sounds like what you're looking for. Try installing SparkMagic, which does use %%sql magic.