This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
spark [2019/02/27 10:55] mantis [Providing spark jars] |
spark [2020/05/12 14:26] (current) mantis [Providing spark jars] |
||
---|---|---|---|
Line 1: | Line 1: | ||
Unless noted otherwise, code is tested with Spark **2.2** | Unless noted otherwise, code is tested with Spark **2.2** | ||
+ | |||
+ | ====== Non-committal testdrive ====== | ||
+ | |||
+ | Minimum-effort way to test-drive Spark with a | ||
+ | [[https://databricks.com/spark/getting-started-with-apache-spark/quick-start#overview|Databricks tutorial]] (no local setup required) | ||
Line 122: | Line 127: | ||
==== Providing spark jars ==== | ==== Providing spark jars ==== | ||
https://spark.apache.org/docs/latest/running-on-yarn.html#preparations | https://spark.apache.org/docs/latest/running-on-yarn.html#preparations | ||
+ | |||
+ | Download the required version [https://spark.apache.org/downloads.html|here]. | ||
How to setup provided jars (found [[https://mapr.com/docs/60/Spark/ConfigureSparkJARLocation_2.0.1.html|here]]): | How to setup provided jars (found [[https://mapr.com/docs/60/Spark/ConfigureSparkJARLocation_2.0.1.html|here]]): | ||
Line 127: | Line 134: | ||
<code bash> | <code bash> | ||
cd /opt/spark-2.2.0-bin-hadoop2.7/jars | cd /opt/spark-2.2.0-bin-hadoop2.7/jars | ||
- | zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./*</code> | + | zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./* |
- | + | # and then copy the archive to your HDFS | |
- | and then copy the archive to your HDFS | + | hdfs dfs -put /tmp/spark220-jars.zip /user/hdfs/</code> |
- | <code bash>hdfs dfs -put /tmp/spark220-jars.zip /user/hdfs/</code> | + | |
- | add to spark-submit | + | Then you can make use of the provided archive by adding to spark-submit |
<code> --conf spark.yarn.archive=hdfs:///user/hdfs/spark220-jars.zip </code> | <code> --conf spark.yarn.archive=hdfs:///user/hdfs/spark220-jars.zip </code> | ||
Line 145: | Line 151: | ||
+ | ====== Tuning ====== | ||