Differences

This shows you the differences between two versions of the page.

--- spark [2019/02/27 10:55]
mantis [Providing spark jars]
+++ spark [2020/05/12 14:26] (current)
mantis [Providing spark jars]
@@ Line 1: / Line 1: @@
 Unless noted otherwise, code is tested with Spark **2.2**
+====== Non-committal testdrive ======
+Minimum-effort way to test-drive Spark with a
+[[https://databricks.com/spark/getting-started-with-apache-spark/quick-start#overview|Databricks tutorial]] (no local setup required)
@@ Line 122: / Line 127: @@
 ==== Providing spark jars ====
 https://spark.apache.org/docs/latest/running-on-yarn.html#preparations
+Download the required version [https://spark.apache.org/downloads.html|here].
 How to setup provided jars (found [[https://mapr.com/docs/60/Spark/ConfigureSparkJARLocation_2.0.1.html|here]]):
@@ Line 127: / Line 134: @@
 <code bash>
 cd /opt/spark-2.2.0-bin-hadoop2.7/jars
-zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./*</code>
+zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./*
+# and then copy the archive to your HDFS
-and then copy the archive to your HDFS
+hdfs dfs -put /tmp/spark220-jars.zip  /user/hdfs/</code>
-<code bash>hdfs dfs -put /tmp/spark220-jars.zip  /user/hdfs/</code>
-add to spark-submit
+Then you can make use of the provided archive by adding to spark-submit
 <code>    --conf spark.yarn.archive=hdfs:///user/hdfs/spark220-jars.zip </code>
@@ Line 145: / Line 151: @@
+====== Tuning ======

Small heaps of code