spark
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
spark [2019/02/27 10:55] mantis [Providing spark jars] |
spark [2019/02/27 10:56] mantis [Providing spark jars] |
||
|---|---|---|---|
| Line 127: | Line 127: | ||
| <code bash> | <code bash> | ||
| cd /opt/spark-2.2.0-bin-hadoop2.7/jars | cd /opt/spark-2.2.0-bin-hadoop2.7/jars | ||
| - | zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./*</code> | + | zip /opt/spark-2.2.0-bin-hadoop2.7/spark220-jars.zip ./* |
| + | # and then copy the archive to your HDFS | ||
| + | hdfs dfs -put /tmp/spark220-jars.zip /user/hdfs/</code> | ||
| - | and then copy the archive to your HDFS | ||
| - | <code bash>hdfs dfs -put /tmp/spark220-jars.zip /user/hdfs/</code> | ||
| - | + | Then you can make use of the provided archive by adding to spark-submit | |
| - | add to spark-submit | + | |
| <code> --conf spark.yarn.archive=hdfs:///user/hdfs/spark220-jars.zip </code> | <code> --conf spark.yarn.archive=hdfs:///user/hdfs/spark220-jars.zip </code> | ||
spark.txt · Last modified: 2020/05/12 14:26 by mantis
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC0 1.0 Universal
