ubuntu安装sbt,编译运行scala文件 安装sbt 本篇文章建在在hadoop安装搭建成功的基础上
我使用的是1.3.8版本的sbt 2.4.0版本的spark 2.11.12版本的scala
我使用的是ubuntu1604版本
最开始参考的是这篇文章
linux ubuntu1604桌面版安装sbt
仿照文章的操作:
# 建立sbt安装目录 sudo mkdir /usr/local/sbt sudo chown -R wh /usr/local/sbt # 解压 tar -zxvf ~/Downloads/sbt-1.3.8.tgz /usr/local/sbt # 将位于sbt/bin下面的sbt-launch.jar文件放在sbt目录下 cd /usr/local/sbt cp ./bin/sbt-launch.jar ./ # 在/usr/local/sbt下创建sbt脚本,该脚本与bin目录同级 vim ./sbt # 添加如下内容 # !/bin/bash SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M" java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@" # 为sbt脚本添加可执行权限 chmod u+x ./sbt # 配置环境变量 vim ~/.bashrc # 在文件最后添加以下内容: export SBT_HOME=/usr/local/sbt export PATH=SBT_HOME:$PATH # 使配置文件生效 source ~/.bashrc
查看sbt版本:
根据错误提示:
[error] Expected ';' [error] Not a valid command: sbt-version (similar: writeSbtVersion, session) [error] Not a valid project ID: sbt-version [error] Expected ':' [error] Not a valid key: sbt-version (similar: sbtVersion, version, sbtBinaryVersion) [error] sbt-version [error]
可得:sbt-version在此时是错误的,sbtVersion, version, sbtBinaryVersion
这些是正确的。
就像这样:
sbt编译打包 1.创建文件夹sparkapp1作为应用程序根目录
cd ~ mkdir sparkapp1 mkdir -p ./sparkapp1/src/main/scala vim ./sparkapp1/src/main/scala/hello.scala # 输入以下代码 object hello{ def main(args: Array[String]){ println("hello") } }
2.在应用程序根目录下创建sbt文件
vim ./sparkapp1/hello.sbt # 输入以下内容 name := "Hello Project" version := "1.0" scalaVersion :="2.11.12" libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
这些版本号在博客第一张图片中有体现,也可以spark-shell查看:
(这里是因为已经搭建了spark集群,故而冗长)
hello.sbt
hello.scala
代码截图:
3.编译打包:
wh@master:~/sparkapp1$ /usr/local/sbt/sbt package Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 [info] Updated file /home/wh/sparkapp1/project/build.properties: set sbt.version to 1.3.8 [info] Loading project definition from /home/wh/sparkapp1/project [info] Loading settings for project sparkapp1 from hello.sbt ... [info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/) [info] Updating https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.4.… 100.0% [##########] 30.3 KiB (13.3 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-parent_2.11/2.… 100.0% [##########] 100.4 KiB (104.3 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/18/apache-18.pom 100.0% [##########] 15.3 KiB (15.8 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.… 100.0% [##########] 846 B (2.5 KiB / s) https://repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/… 100.0% [##########] 2.3 KiB (6.8 KiB / s) https://repo1.maven.org/maven2/com/twitter/chill-java/0.9.3/chill-ja… 100.0% [##########] 2.0 KiB (2.2 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2… 100.0% [##########] 6.9 KiB (7.6 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/2.… 100.0% [##########] 2.3 KiB (2.3 KiB / s) https://repo1.maven.org/maven2/org/apache/xbean/xbean-asm6-shaded/4.… 100.0% [##########] 4.0 KiB (12.5 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-clie… 100.0% [##########] 7.4 KiB (21.5 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.4.… 100.0% [##########] 26.6 KiB (72.2 KiB / s) https://repo1.maven.org/maven2/io/netty/netty-all/4.1.17.Final/netty… 100.0% [##########] 24.8 KiB (26.7 KiB / s) https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.pom 100.0% [##########] 21.2 KiB (13.7 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-da… 100.0% [##########] 5.9 KiB (18.0 KiB / s) https://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0… 100.0% [##########] 2.0 KiB (5.8 KiB / s) https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.pom 100.0% [##########] 6.1 KiB (20.6 KiB / s) https://repo1.maven.org/maven2/javax/activation/activation/1.1.1/act… 100.0% [##########] 644 B (2.2 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-… 100.0% [##########] 4.2 KiB (14.6 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-network-shuffl… 100.0% [##########] 4.9 KiB (15.2 KiB / s) https://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.pom 100.0% [##########] 140 B (571 B / s) https://repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.11/2.… 100.0% [##########] 4.7 KiB (2.3 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-kvstore_2.11/2… 100.0% [##########] 4.6 KiB (16.9 KiB / s) https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.7.1… 100.0% [##########] 3.9 KiB (12.4 KiB / s) https://repo1.maven.org/maven2/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.p… 100.0% [##########] 2.1 KiB (5.9 KiB / s) https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.5.1… 100.0% [##########] 6.2 KiB (20.3 KiB / s) https://repo1.maven.org/maven2/javax/servlet/javax.servlet-api/3.1.0… 100.0% [##########] 13.2 KiB (15.2 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-graphit… 100.0% [##########] 1.3 KiB (4.7 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-jackson_2.11/3.5.3/… 100.0% [##########] 2.0 KiB (7.5 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.5/… 100.0% [##########] 22.8 KiB (59.5 KiB / s) https://repo1.maven.org/maven2/com/ning/compress-lzf/1.0.3/compress-… 100.0% [##########] 7.7 KiB (24.0 KiB / s) https://repo1.maven.org/maven2/org/slf4j/jul-to-slf4j/1.7.16/jul-to-… 100.0% [##########] 986 B (4.0 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.6.5… 100.0% [##########] 10.8 KiB (10.2 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.8.2/avr… 100.0% [##########] 7.6 KiB (25.1 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-crypto/1.0… 100.0% [##########] 21.3 KiB (61.5 KiB / s) https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.16/slf4j-… 100.0% [##########] 909 B (930 B / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.… 100.0% [##########] 1.4 KiB (5.4 KiB / s) https://repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.… 100.0% [##########] 6.5 KiB (26.2 KiB / s) https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.3.2-2/zst… 100.0% [##########] 1.9 KiB (6.1 KiB / s) https://repo1.maven.org/maven2/io/netty/netty/3.9.9.Final/netty-3.9.… 100.0% [##########] 28.2 KiB (81.6 KiB / s) https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9… 100.0% [##########] 965 B (3.0 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1… 100.0% [##########] 967 B (2.6 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-comm… 100.0% [##########] 10.2 KiB (16.4 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse… 100.0% [##########] 4.5 KiB (14.5 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse… 100.0% [##########] 4.6 KiB (15.7 KiB / s) https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-… 100.0% [##########] 2.7 KiB (9.1 KiB / s) https://repo1.maven.org/maven2/org/slf4j/jcl-over-slf4j/1.7.16/jcl-o… 100.0% [##########] 963 B (2.8 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-serv… 100.0% [##########] 12.0 KiB (19.3 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-network-common… 100.0% [##########] 6.3 KiB (3.3 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-tags_2.11/2.4.… 100.0% [##########] 2.6 KiB (10.6 KiB / s) https://repo1.maven.org/maven2/com/twitter/chill_2.11/0.9.3/chill_2.… 100.0% [##########] 2.3 KiB (7.1 KiB / s) https://repo1.maven.org/maven2/commons-net/commons-net/3.1/commons-n… 100.0% [##########] 18.0 KiB (26.6 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-launcher_2.11/… 100.0% [##########] 6.7 KiB (23.4 KiB / s) https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/… 100.0% [##########] 3.3 KiB (10.8 KiB / s) https://repo1.maven.org/maven2/net/razorvine/pyrolite/4.13/pyrolite-… 100.0% [##########] 3.5 KiB (6.1 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/containers/proje… 100.0% [##########] 3.7 KiB (11.1 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-parent/… 100.0% [##########] 11.8 KiB (33.6 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project-dist… 100.0% [##########] 18.1 KiB (51.0 KiB / s) https://repo1.maven.org/maven2/org/apache/xbean/xbean/4.8/xbean-4.8.… 100.0% [##########] 17.5 KiB (28.6 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/… 100.0% [##########] 7.6 KiB (22.3 KiB / s) https://repo1.maven.org/maven2/net/java/jvnet-parent/3/jvnet-parent-… 100.0% [##########] 4.7 KiB (14.3 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-parent/1.8.2/avr… 100.0% [##########] 20.9 KiB (21.5 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/project/2.22.2/p… 100.0% [##########] 82.3 KiB (55.2 KiB / s) https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.16/slf4j-p… 100.0% [##########] 13.2 KiB (20.1 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/7/apache-7.pom 100.0% [##########] 14.1 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/apache-curator/2.6… 100.0% [##########] 29.6 KiB (12.5 KiB / s) https://repo1.maven.org/maven2/io/netty/netty-parent/4.1.17.Final/ne… 100.0% [##########] 50.2 KiB (18.4 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/41/… 100.0% [##########] 63.3 KiB (24.1 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/40/… 100.0% [##########] 62.6 KiB (16.3 KiB / s) https://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-par… 100.0% [##########] 4.0 KiB (6.6 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/34/… 100.0% [##########] 54.6 KiB (11.6 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/23/… 100.0% [##########] 43.4 KiB (19.5 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/oss-parent/24/oss-paren… 100.0% [##########] 19.0 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/17/apache-17.pom 100.0% [##########] 15.7 KiB (15.3 KiB / s) https://repo1.maven.org/maven2/net/java/jvnet-parent/4/jvnet-parent-… 100.0% [##########] 7.6 KiB (3.9 KiB / s) https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis-j… 100.0% [##########] 5.4 KiB (2.6 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-toplevel/1.8.2/a… 100.0% [##########] 15.3 KiB (5.0 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.6.… 100.0% [##########] 42.8 KiB (10.7 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom 100.0% [##########] 14.3 KiB (8.9 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom 100.0% [##########] 13.6 KiB (6.9 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/9/apache-9.pom 100.0% [##########] 14.8 KiB (7.2 KiB / s) https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis-d… 100.0% [##########] 15.7 KiB (20.1 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/10/apache-10.pom 100.0% [##########] 14.5 KiB (13.2 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-main/2.6.5/h… 100.0% [##########] 17.7 KiB (8.7 KiB / s) https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis/2… 100.0% [##########] 13.9 KiB (22.1 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-bom/2.4.0-b34/h… 100.0% [##########] 20.4 KiB (38.0 KiB / s) https://repo1.maven.org/maven2/net/java/jvnet-parent/5/jvnet-parent-… 100.0% [##########] 8.7 KiB (26.2 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-core_2.11/3.5.3/jso… 100.0% [##########] 2.3 KiB (7.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 4.6 KiB (11.6 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 3.4 KiB (13.1 KiB / s) https://repo1.maven.org/maven2/javax/validation/validation-api/1.1.0… 100.0% [##########] 7.7 KiB (9.3 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-api/2.4.0-b34/h… 100.0% [##########] 4.5 KiB (16.9 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/media/jersey-med… 100.0% [##########] 7.0 KiB (6.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h… 100.0% [##########] 28.7 KiB (25.9 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag… 100.0% [##########] 13.0 KiB (17.7 KiB / s) https://repo1.maven.org/maven2/com/google/guava/guava/16.0.1/guava-1… 100.0% [##########] 6.0 KiB (18.7 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-framework/… 100.0% [##########] 2.1 KiB (7.7 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6… 100.0% [##########] 4.5 KiB (17.9 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-co… 100.0% [##########] 4.8 KiB (12.4 KiB / s) https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/… 100.0% [##########] 5.1 KiB (17.0 KiB / s) https://repo1.maven.org/maven2/org/tukaani/xz/1.5/xz-1.5.pom 100.0% [##########] 1.9 KiB (5.5 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-an… 100.0% [##########] 1.2 KiB (4.2 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-locator/2.4.0-b… 100.0% [##########] 5.4 KiB (17.0 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/… 100.0% [##########] 2.3 KiB (3.6 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1… 100.0% [##########] 11.2 KiB (42.0 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.8.2/avro-i… 100.0% [##########] 5.0 KiB (2.1 KiB / s) https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-… 100.0% [##########] 7.1 KiB (27.1 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-a… 100.0% [##########] 1.5 KiB (5.9 KiB / s) https://repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.… 100.0% [##########] 3.4 KiB (12.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 5.5 KiB (8.6 KiB / s) https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0.1/jav… 100.0% [##########] 20.9 KiB (11.6 KiB / s) https://repo1.maven.org/maven2/commons-codec/commons-codec/1.9/commo… 100.0% [##########] 11.3 KiB (40.0 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/osgi-resource-locat… 100.0% [##########] 6.0 KiB (16.6 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-… 100.0% [##########] 3.9 KiB (3.0 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/external/javax.inje… 100.0% [##########] 6.3 KiB (10.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/… 100.0% [##########] 1.5 KiB (5.0 KiB / s) https://repo1.maven.org/maven2/javax/annotation/javax.annotation-api… 100.0% [##########] 13.1 KiB (4.8 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.5… 100.0% [##########] 29.9 KiB (20.6 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl… 100.0% [##########] 1.2 KiB (525 B / s) https://repo1.maven.org/maven2/org/glassfish/jersey/media/project/2.… 100.0% [##########] 2.9 KiB (1.2 KiB / s) https://repo1.maven.org/maven2/com/esotericsoftware/kryo-parent/4.0.… 100.0% [##########] 6.9 KiB (2.9 KiB / s) https://repo1.maven.org/maven2/com/google/guava/guava-parent/16.0.1/… 100.0% [##########] 7.2 KiB (22.8 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/external/2.4.0-b34/… 100.0% [##########] 2.9 KiB (8.7 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/… 100.0% [##########] 7.6 KiB (2.6 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn/2.6.5/h… 100.0% [##########] 3.5 KiB (10.8 KiB / s) https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-… 100.0% [##########] 9.4 KiB (27.3 KiB / s) https://repo1.maven.org/maven2/org/glassfish/pom/8/pom-8.pom 100.0% [##########] 10.3 KiB (3.3 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-… 100.0% [##########] 2.4 KiB (7.6 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag… 100.0% [##########] 2.7 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 6.6 KiB (1.8 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-parent/2.4.0-b3… 100.0% [##########] 46.7 KiB (49.0 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/33/… 100.0% [##########] 51.8 KiB (12.5 KiB / s) https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer-… 100.0% [##########] 11.2 KiB (9.2 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/32/… 100.0% [##########] 51.5 KiB (26.6 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/project/… 100.0% [##########] 3.0 KiB (9.9 KiB / s) https://repo1.maven.org/maven2/org/codehaus/codehaus-parent/4/codeha… 100.0% [##########] 4.7 KiB (13.5 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/… 100.0% [##########] 8.0 KiB (20.7 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/oss-parent/23/oss-paren… 100.0% [##########] 18.8 KiB (27.3 KiB / s) https://repo1.maven.org/maven2/org/fusesource/fusesource-pom/1.9/fus… 100.0% [##########] 14.4 KiB (15.2 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/oss-parent/25/oss-paren… 100.0% [##########] 18.9 KiB (17.7 KiB / s) https://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/min… 100.0% [##########] 3.2 KiB (11.1 KiB / s) https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/j… 100.0% [##########] 3.8 KiB (13.1 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-utils/2.4.0-b34… 100.0% [##########] 5.0 KiB (15.2 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/external/aopallianc… 100.0% [##########] 6.6 KiB (23.5 KiB / s) https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.… 100.0% [##########] 1.3 KiB (2.3 KiB / s) https://repo1.maven.org/maven2/org/javassist/javassist/3.18.1-GA/jav… 100.0% [##########] 9.3 KiB (27.4 KiB / s) https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons… 100.0% [##########] 17.1 KiB (25.5 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-ast_2.11/3.5.3/json… 100.0% [##########] 1.6 KiB (5.2 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.6.5/h… 100.0% [##########] 7.5 KiB (23.0 KiB / s) https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient… 100.0% [##########] 7.6 KiB (6.8 KiB / s) https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-… 100.0% [##########] 7.8 KiB (22.7 KiB / s) https://repo1.maven.org/maven2/commons-configuration/commons-configu… 100.0% [##########] 13.0 KiB (22.3 KiB / s) https://repo1.maven.org/maven2/org/objenesis/objenesis/2.5.1/objenes… 100.0% [##########] 2.7 KiB (2.7 KiB / s) https://repo1.maven.org/maven2/commons-collections/commons-collectio… 100.0% [##########] 12.1 KiB (17.4 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-client/2.6… 100.0% [##########] 2.2 KiB (6.8 KiB / s) https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.pom 100.0% [##########] 623 B (2.2 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 3.0 KiB (9.7 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-scalap_2.11/3.5.3/j… 100.0% [##########] 1.6 KiB (6.4 KiB / s) https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/… 100.0% [##########] 5.2 KiB (6.1 KiB / s) https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-… 100.0% [##########] 9.9 KiB (25.7 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11… 100.0% [##########] 2.5 KiB (7.9 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 3.2 KiB (11.9 KiB / s) https://repo1.maven.org/maven2/org/htrace/htrace-core/3.0.4/htrace-c… 100.0% [##########] 2.8 KiB (8.9 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/… 100.0% [##########] 10.3 KiB (14.8 KiB / s) https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-c… 100.0% [##########] 7.8 KiB (7.8 KiB / s) https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5… 100.0% [##########] 8.2 KiB (7.6 KiB / s) https://repo1.maven.org/maven2/com/google/google/1/google-1.pom 100.0% [##########] 1.5 KiB (4.8 KiB / s) https://repo1.maven.org/maven2/org/mortbay/jetty/project/6.1.26/proj… 100.0% [##########] 9.0 KiB (26.5 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/39/… 100.0% [##########] 60.5 KiB (77.8 KiB / s) https://repo1.maven.org/maven2/org/htrace/htrace/3.0.4/htrace-3.0.4.… 100.0% [##########] 6.4 KiB (8.2 KiB / s) https://repo1.maven.org/maven2/org/objenesis/objenesis-parent/2.5.1/… 100.0% [##########] 16.1 KiB (13.6 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/4/apache-4.pom 100.0% [##########] 4.4 KiB (4.8 KiB / s) https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer-… 100.0% [##########] 11.3 KiB (7.5 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/25/… 100.0% [##########] 47.2 KiB (13.0 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/17/… 100.0% [##########] 30.5 KiB (7.1 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-parent/11/… 100.0% [##########] 24.1 KiB (13.2 KiB / s) https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-parent/10/jet… 100.0% [##########] 3.2 KiB (7.3 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/16/apache-16.pom 100.0% [##########] 15.0 KiB (12.6 KiB / s) https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-parent/14/jet… 100.0% [##########] 16.1 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-… 100.0% [##########] 5.6 KiB (7.5 KiB / s) https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8… 100.0% [##########] 6.8 KiB (8.8 KiB / s) https://repo1.maven.org/maven2/xml-apis/xml-apis/1.3.04/xml-apis-1.3… 100.0% [##########] 1.8 KiB (4.7 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/… 100.0% [##########] 6.2 KiB (13.2 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-… 100.0% [##########] 3.8 KiB (6.2 KiB / s) https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-c… 100.0% [##########] 1.6 KiB (796 B / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.… 100.0% [##########] 1.9 KiB (924 B / s) https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inj… 100.0% [##########] 612 B (1.8 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-… 100.0% [##########] 11.8 KiB (12.0 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/… 100.0% [##########] 5.8 KiB (2.0 KiB / s) https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-ap… 100.0% [##########] 5.4 KiB (3.7 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcompone… 100.0% [##########] 14.4 KiB (12.8 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/3/apache-3.pom 100.0% [##########] 3.3 KiB (2.7 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-… 100.0% [##########] 41.6 KiB (18.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server/… 100.0% [##########] 1.7 KiB (231 B / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/project/6/p… 100.0% [##########] 23.4 KiB (13.3 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/project/project/… 100.0% [##########] 26.1 KiB (14.0 KiB / s) https://repo1.maven.org/maven2/org/apache/apache/11/apache-11.pom 100.0% [##########] 14.5 KiB (21.1 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api… 100.0% [##########] 2.6 KiB (8.2 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0… 100.0% [##########] 2.3 KiB (6.1 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.… 100.0% [##########] 5.6 KiB (14.6 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/je… 100.0% [##########] 3.3 KiB (8.6 KiB / s) https://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom 100.0% [##########] 6.2 KiB (16.4 KiB / s) https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0… 100.0% [##########] 7.1 KiB (6.0 KiB / s) https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-… 100.0% [##########] 962 B (779 B / s) https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1… 100.0% [##########] 357 B (168 B / s) https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-… 100.0% [##########] 2.3 KiB (857 B / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-par… 100.0% [##########] 1.5 KiB (5.5 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-parent/1… 100.0% [##########] 26.7 KiB (70.6 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcompone… 100.0% [##########] 11.4 KiB (26.5 KiB / s) https://repo1.maven.org/maven2/com/google/inject/guice-parent/3.0/gu… 100.0% [##########] 12.7 KiB (19.5 KiB / s) https://repo1.maven.org/maven2/com/google/google/5/google-5.pom 100.0% [##########] 2.4 KiB (7.2 KiB / s) https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopallian… 100.0% [##########] 363 B (1.4 KiB / s) [info] Resolved dependencies [warn] [warn] Note: Unresolved dependencies path: [error] sbt.librarymanagement.ResolveException: Error downloading org.scala-lang:scala-library:2.11.12 [error] Not found [error] Not found [error] not found: /home/wh/.ivy2/local/org.scala-lang/scala-library/2.11.12/ivys/ivy.xml [error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.pom [error] Error downloading org.codehaus.jackson:jackson-xc:1.9.13 [error] Not found [error] Not found [error] not found: /home/wh/.ivy2/local/org.codehaus.jackson/jackson-xc/1.9.13/ivys/ivy.xml [error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.pom [error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:246) [error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$34(CoursierDependencyResolution.scala:215) [error] at scala.util.Either$LeftProjection.map(Either.scala:573) [error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:215) [error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60) [error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:52) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:102) [error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:69) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$20(LibraryManagement.scala:115) [error] at scala.util.control.Exception$Catch.apply(Exception.scala:228) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:115) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:96) [error] at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:150) [error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:129) [error] at sbt.Classpaths$.$anonfun$updateTask0$5(Defaults.scala:2947) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:67) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19) [error] at sbt.Execute.work(Execute.scala:290) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [error] at java.lang.Thread.run(Thread.java:748) [error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scala-lang:scala-library:2.11.12 [error] Not found [error] Not found [error] not found: /home/wh/.ivy2/local/org.scala-lang/scala-library/2.11.12/ivys/ivy.xml [error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.pom [error] Error downloading org.codehaus.jackson:jackson-xc:1.9.13 [error] Not found [error] Not found [error] not found: /home/wh/.ivy2/local/org.codehaus.jackson/jackson-xc/1.9.13/ivys/ivy.xml [error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.pom [error] Total time: 125 s (02:05), completed Apr 29, 2022 12:12:47 PM wh@master:~/sparkapp1$ /usr/local/sbt/sbt package Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 [info] Loading project definition from /home/wh/sparkapp1/project [info] Loading settings for project sparkapp1 from hello.sbt ... [info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/) [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/… 100.0% [##########] 1.4 KiB (719 B / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.1… 100.0% [##########] 1.9 KiB (5.4 KiB / s) [info] Resolved dependencies [info] Updating https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.12… 100.0% [##########] 2.3 KiB (9.3 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-c… 100.0% [##########] 2.1 KiB (3.3 KiB / s) https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.pom 100.0% [##########] 19.4 KiB (18.6 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11… 100.0% [##########] 2.5 KiB (2.3 KiB / s) [info] Resolved dependencies [info] Fetching artifacts of https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8… 100.0% [##########] 140.2 KiB (125.1 KiB / s) https://repo1.maven.org/maven2/org/objenesis/objenesis/2.5.1/objenes… 100.0% [##########] 53.1 KiB (42.1 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-… 100.0% [##########] 43.9 KiB (32.0 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.… 100.0% [##########] 117.6 KiB (78.7 KiB / s) https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-… 100.0% [##########] 22.8 KiB (10.7 KiB / s) https://repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.… 100.0% [##########] 170.3 KiB (97.2 KiB / s) https://repo1.maven.org/maven2/commons-net/commons-net/3.1/commons-n… 100.0% [##########] 267.0 KiB (61.4 KiB / s) https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9… 100.0% [##########] 32.2 KiB (34.5 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl… 100.0% [##########] 226.8 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 59.9 KiB (28.3 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse… 100.0% [##########] 17.7 KiB (24.8 KiB / s) https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-c… 100.0% [##########] 201.2 KiB (16.0 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-core_2.11/3.5.3/jso… 100.0% [##########] 673.8 KiB (17.2 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-crypto/1.0… 100.0% [##########] 131.4 KiB (10.8 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.6.5… 100.0% [##########] 25.4 KiB (10.5 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/external/aopallianc… 100.0% [##########] 14.4 KiB (9.4 KiB / s) https://repo1.maven.org/maven2/com/ning/compress-lzf/1.0.3/compress-… 100.0% [##########] 78.0 KiB (11.0 KiB / s) https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-… 100.0% [##########] 180.8 KiB (8.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 1.5 MiB (23.2 KiB / s) https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-… 100.0% [##########] 39.6 KiB (10.9 KiB / s) https://repo1.maven.org/maven2/org/apache/xbean/xbean-asm6-shaded/4.… 100.0% [##########] 276.3 KiB (18.5 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/… 100.0% [##########] 1.6 MiB (18.1 KiB / s) https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5… 100.0% [##########] 521.0 KiB (21.6 KiB / s) https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar 100.0% [##########] 1.2 MiB (15.7 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-jackson_2.11/3.5.3/… 100.0% [##########] 47.1 KiB (8.0 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11… 100.0% [##########] 655.4 KiB (13.7 KiB / s) https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar 100.0% [##########] 14.7 KiB (13.4 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-tags_2.11/2.4.… 100.0% [##########] 15.1 KiB (6.4 KiB / s) https://repo1.maven.org/maven2/javax/servlet/javax.servlet-api/3.1.0… 100.0% [##########] 93.6 KiB (14.0 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1… 100.0% [##########] 357.0 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/com/twitter/chill_2.11/0.9.3/chill_2.… 100.0% [##########] 228.1 KiB (16.0 KiB / s) https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.16/slf4j-… 100.0% [##########] 9.7 KiB (13.9 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/… 100.0% [##########] 148.3 KiB (16.0 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-scalap_2.11/3.5.3/j… 100.0% [##########] 600.4 KiB (12.3 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/external/javax.inje… 100.0% [##########] 5.8 KiB (3.8 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-kvstore_2.11/2… 100.0% [##########] 51.8 KiB (10.1 KiB / s) https://repo1.maven.org/maven2/io/netty/netty-all/4.1.17.Final/netty… 100.0% [##########] 3.6 MiB (17.0 KiB / s) https://repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.… 100.0% [##########] 401.2 KiB (18.3 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.1… 100.0% [##########] 26.4 KiB (15.4 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-framework/… 100.0% [##########] 180.9 KiB (14.1 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 66.2 KiB (21.9 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.… 100.0% [##########] 17.9 KiB (23.4 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-api/2.4.0-b34/h… 100.0% [##########] 174.8 KiB (19.8 KiB / s) https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-… 100.0% [##########] 1021.2 KiB (15.4 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.5… 100.0% [##########] 3.2 MiB (14.3 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/osgi-resource-locat… 100.0% [##########] 19.8 KiB (6.3 KiB / s) https://repo1.maven.org/maven2/org/json4s/json4s-ast_2.11/3.5.3/json… 100.0% [##########] 88.5 KiB (19.4 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-da… 100.0% [##########] 1.1 MiB (17.2 KiB / s) https://repo1.maven.org/maven2/javax/activation/activation/1.1.1/act… 100.0% [##########] 67.8 KiB (29.7 KiB / s) https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-… 100.0% [##########] 186.0 KiB (16.4 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-c… 100.0% [##########] 413.8 KiB (13.6 KiB / s) https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopallian… 100.0% [##########] 4.4 KiB (3.3 KiB / s) https://repo1.maven.org/maven2/javax/annotation/javax.annotation-api… 100.0% [##########] 25.7 KiB (28.4 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.… 100.0% [##########] 221.9 KiB (16.4 KiB / s) https://repo1.maven.org/maven2/net/razorvine/pyrolite/4.13/pyrolite-… 100.0% [##########] 92.6 KiB (12.8 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-locator/2.4.0-b… 100.0% [##########] 177.0 KiB (10.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-… 100.0% [##########] 651.9 KiB (11.7 KiB / s) https://repo1.maven.org/maven2/org/slf4j/jcl-over-slf4j/1.7.16/jcl-o… 100.0% [##########] 16.0 KiB (15.3 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-co… 100.0% [##########] 252.9 KiB (20.8 KiB / s) https://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/min… 100.0% [##########] 5.6 KiB (13.3 KiB / s) https://repo1.maven.org/maven2/com/google/guava/guava/16.0.1/guava-1… 100.0% [##########] 2.1 MiB (4.0 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 516.2 KiB (18.0 KiB / s) https://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.jar 100.0% [##########] 63.7 KiB (13.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h… 92.5% [######### ] 7.0 MiB (8.7 KiB / s) https://repo1.maven.org/maven2/xml-apis/xml-apis/1.3.04/xml-apis-1.3… 100.0% [##########] 189.8 KiB (13.9 KiB / s) https://repo1.maven.org/maven2/commons-configuration/commons-configu… 100.0% [##########] 291.8 KiB (17.4 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/… 75.1% [####### ] 3.3 MiB (5.5 KiB / s) https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/j… 100.0% [##########] 173.0 KiB (16.7 KiB / s) https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/… 100.0% [##########] 423.2 KiB (1.6 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.5/… 100.0% [##########] 468.6 KiB (12.4 KiB / s) https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/… 100.0% [##########] 33.8 KiB (6.0 KiB / s) https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/… 100.0% [##########] 774.4 KiB (2.5 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-… 100.0% [##########] 503.6 KiB (13.7 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-client/2.6… 100.0% [##########] 67.3 KiB (14.7 KiB / s) https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-utils/2.4.0-b34… 100.0% [##########] 116.2 KiB (15.3 KiB / s) https://repo1.maven.org/maven2/org/slf4j/jul-to-slf4j/1.7.16/jul-to-… 100.0% [##########] 4.5 KiB (4.5 KiB / s) https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0.1/jav… 100.0% [##########] 112.8 KiB (302 B / s) https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-a… 100.0% [##########] 762.4 KiB (17.8 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2… 100.0% [##########] 1.5 MiB (11.8 KiB / s) https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar 100.0% [##########] 478.4 KiB (11.1 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/media/jersey-med… 100.0% [##########] 71.0 KiB (6.4 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse… 100.0% [##########] 64.7 KiB (11.0 KiB / s) https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.3.2-2/zst… 100.0% [##########] 2.2 MiB (15.6 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl… 100.0% [##########] 672.1 KiB (12.4 KiB / s) https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-ap… 100.0% [##########] 102.7 KiB (20.0 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.8.2/avro-i… 100.0% [##########] 129.9 KiB (21.4 KiB / s) https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.… 100.0% [##########] 1.2 MiB (15.4 KiB / s) https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.8.2/avr… 100.0% [##########] 182.7 KiB (30.3 KiB / s) https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.4.… 100.0% [##########] 1.9 MiB (27.4 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-graphit… 100.0% [##########] 20.7 KiB (24.6 KiB / s) https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons… 100.0% [##########] 277.6 KiB (26.6 KiB / s) https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0… 100.0% [##########] 693.8 KiB (22.7 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-… 100.0% [##########] 312.5 KiB (21.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.6.5/h… 100.0% [##########] 88.5 KiB (27.4 KiB / s) https://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar 100.0% [##########] 85.3 KiB (34.4 KiB / s) https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/je… 100.0% [##########] 66.2 KiB (31.8 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1… 100.0% [##########] 38.4 KiB (20.9 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0… 100.0% [##########] 78.0 KiB (14.2 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-serv… 100.0% [##########] 929.4 KiB (18.5 KiB / s) https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inj… 100.0% [##########] 2.4 KiB (1004 B / s) https://repo1.maven.org/maven2/org/apache/spark/spark-launcher_2.11/… 100.0% [##########] 74.2 KiB (15.4 KiB / s) https://repo1.maven.org/maven2/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.j… 100.0% [##########] 119.9 KiB (13.5 KiB / s) https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient… 100.0% [##########] 297.9 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/… 100.0% [##########] 5.5 MiB (33.2 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-an… 100.0% [##########] 45.9 KiB (35.0 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.4.… 100.0% [##########] 12.8 MiB (30.7 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-clie… 100.0% [##########] 163.5 KiB (58.7 KiB / s) https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.5.1… 100.0% [##########] 197.2 KiB (32.0 KiB / s) https://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0… 100.0% [##########] 361.4 KiB (40.7 KiB / s) https://repo1.maven.org/maven2/javax/validation/validation-api/1.1.0… 100.0% [##########] 62.3 KiB (36.9 KiB / s) https://repo1.maven.org/maven2/commons-collections/commons-collectio… 100.0% [##########] 574.5 KiB (31.7 KiB / s) https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.… 100.0% [##########] 15.5 KiB (10.6 KiB / s) https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-c… 100.0% [##########] 40.2 KiB (20.5 KiB / s) https://repo1.maven.org/maven2/io/netty/netty/3.9.9.Final/netty-3.9.… 100.0% [##########] 1.3 MiB (21.6 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-comm… 100.0% [##########] 682.0 KiB (29.4 KiB / s) https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-… 100.0% [##########] 41.9 KiB (35.9 KiB / s) https://repo1.maven.org/maven2/com/twitter/chill-java/0.9.3/chill-ja… 100.0% [##########] 57.3 KiB (24.9 KiB / s) https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/2.… 100.0% [##########] 242.4 KiB (23.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/… 100.0% [##########] 39.5 KiB (37.8 KiB / s) https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag… 100.0% [##########] 948.5 KiB (31.7 KiB / s) https://repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/… 100.0% [##########] 2.7 KiB (9.5 KiB / s) https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1… 100.0% [##########] 184.2 KiB (39.6 KiB / s) https://repo1.maven.org/maven2/org/javassist/javassist/3.18.1-GA/jav… 100.0% [##########] 697.5 KiB (24.9 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.11/2.… 100.0% [##########] 48.0 KiB (36.5 KiB / s) https://repo1.maven.org/maven2/org/tukaani/xz/1.5/xz-1.5.jar 100.0% [##########] 97.2 KiB (48.7 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-network-shuffl… 100.0% [##########] 69.2 KiB (59.5 KiB / s) https://repo1.maven.org/maven2/org/htrace/htrace-core/3.0.4/htrace-c… 100.0% [##########] 30.5 KiB (38.1 KiB / s) https://repo1.maven.org/maven2/org/apache/spark/spark-network-common… 100.0% [##########] 2.3 MiB (29.0 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api… 100.0% [##########] 16.2 KiB (5.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6… 100.0% [##########] 1.8 MiB (24.1 KiB / s) https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.7.1… 100.0% [##########] 1.9 MiB (13.7 KiB / s) https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar 100.0% [##########] 262.2 KiB (16.0 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/… 100.0% [##########] 4.4 MiB (26.3 KiB / s) https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h… 100.0% [##########] 7.6 MiB (27.3 KiB / s) [warn] Canceling execution... [error] Total time: 6131 s (01:42:11), completed Apr 29, 2022 1:56:34 PM wh@master:~/sparkapp1$ /usr/local/sbt/sbt package Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 [info] Loading project definition from /home/wh/sparkapp1/project [info] Loading settings for project sparkapp1 from hello.sbt ... [info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/) [info] Fetching artifacts of https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11… 100.0% [##########] 654.7 KiB (152.1 KiB / s) https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-… 100.0% [##########] 675.3 KiB (106.0 KiB / s) https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.12… 100.0% [##########] 14.9 MiB (879.2 KiB / s) [info] Fetched artifacts of [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings. [info] Compiling 1 Scala source to /home/wh/sparkapp1/target/scala-2.11/classes ... https://repo1.maven.org/maven2/org/scala-sbt/compiler-bridge_2.11/1.… 100.0% [##########] 2.8 KiB (10.6 KiB / s) https://repo1.maven.org/maven2/org/scala-sbt/compiler-bridge_2.11/1.… 100.0% [##########] 47.7 KiB (158.0 KiB / s) [info] Non-compiled module 'compiler-bridge_2.11' for Scala 2.11.12. Compiling... [info] Compilation completed in 16.01s. [success] Total time: 38 s, completed Apr 29, 2022 1:57:29 PM wh@master:~/sparkapp1$
cd ~/sparkapp1 /usr/local/sbt/sbt package # 第一次需要下载文件,很费时间,后面再打包就很快了
运行 $ SPARK_HOME/bin/spark-submit --class "hello" ~/sparkapp1/target/scala-2.11/hello-project_2.11-1.0.jar
输出了hello
为了更加清晰输出,替换hello.scala文件:
import org.apache.spark.SparkContext import org.apache.spark.SparkContext ._import org.apache.spark.SparkConf object SimpleApp {def main (args: Array [String ]) { val logFile = "file:///usr/local/spark/README.md" val conf = new SparkConf ().setAppName("Simple Application" ) val sc = new SparkContext (conf) val logData = sc.textFile(logFile, 2 ).cache() val numAs = logData.filter(line => line.contains("a" )).count() val numBs = logData.filter(line => line.contains("b" )).count() println("Lines with a: %s, Lines with b: %s" .format(numAs, numBs)) } }
删除~/sparkapp1下的project和target
重新编译打包:
参考链接 https://blog.csdn.net/m0_51195997/article/details/124094278
https://www.cnblogs.com/zjj999/p/12710702.html
https://yusure.cn/backend/308.html
https://www.cnblogs.com/midworld/p/15647016.html
http://dblab.xmu.edu.cn/blog/2137-2/
https://blog.csdn.net/J_oshua/article/details/109403952