how to set checkpiont dir PySpark Data Science Experience

Question

Could you help me with instructions on how to set the checkpoint dir for a PySpark session on IBM's Data Science Experience?.

The need came because i have to run connectedComponents() from GraphFrames and it raises the following error

Py4JJavaError: An error occurred while calling o221.run.
: java.io.IOException: Checkpoint directory is not set. Please set it first using sc.setCheckpointDir(). 

Answer 1

The main issue is to get the directory that the notebook has as working directory to set the checkpoit dir with sc.setCheckpointDir(). this can be done easily with

!pwd

Then, a directory for checkpoints should be created on that route

!mkdir <pwd_output>/checkpoints

Finally set the checkpoint

spark.sparkContext.setCheckpointDir('<pwd_output>/checkpoints')