Building spark-jobserver Using SBT and Scala

Question

Can anyone suggest me a better documentation about spark-jobserver. I have gone through the url spark-jobserver but unable to follow the same. It will be great if some one explain step by step instruction on how to use spark-jobserver.

Tools used in building the project.

• sbt launcher version 0.13.5
• Scala code runner version 2.11.6

With the above mentioned tools I am getting errors while building the spark-jobserver.

Answer 1

The documentation provided in the jobserver repo is indeed confusing.

Here's the steps I followed to manually build and run Spark Job Server on a local machine.

1. git clone https://github.com/spark-jobserver/spark-jobserver
2. sudo mkdir -p /var/log/job-server
3. sudo chown user1:user1 /var/log/job-server
4. cd spark-jobserver
5. sbt job-server/assembly
6. cd config
7. cp local.sh.template abc.sh  # Note that the same name 'abc' is used in steps 8 and 10 as well
8. cp ec2.conf.template abc.conf
9. cd .. # The jobserver root dir
10. ./bin/server_package.sh abc # This script copies the files and packages necessary to run job server into a single dir [ default - /tmp/job-server]
11. cd /tmp/job-server [This is where the files and packages necessary to run job server are published by default]
12. ./server_start.sh
13. Run ./server_stop.sh to stop the server

Hope this helps

Answer 2

Here are the steps that I used to install:

1. Clone the jobserver repo.
2. Get sbt using wget https://dl.bintray.com/sbt/native-packages/sbt/0.13.8/sbt-0.13.8.tgz
3. Move "sbt-launch.jar" in sbt/bin to /bin
4. Create a script /bin/sbt, contents found here, making sure to change the pointer to java if necessary
5. Make the above script executable
6. Now cd into the spark jobserver directory, and run sbt publish-local
7. Assuming the above was successful, run sbt in the same directory
8. Finally, use the command re-start, and if it succeeds the server is now running!