Starting Jupyter Notebook with Apache Spark
Required Variable Setup
Open a terminal and enter command to edit the profile variable
$ gedit ~/.bashrc
Once the window opens, enter the following two lines
export PYSPARK_DRIVER_PYTHON=ipython
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
Once done hit save and exit the terminal.
Running the Notebook with Spark Cluster
Assuming that its a local standalone cluster, we can start it using the following commands
$ pyspark --master local[2]
