ubuntu安装sbt,编译运行scala文件

安装sbt

本篇文章建在在hadoop安装搭建成功的基础上

我使用的是1.3.8版本的sbt 2.4.0版本的spark 2.11.12版本的scala

1

我使用的是ubuntu1604版本

最开始参考的是这篇文章

linux ubuntu1604桌面版安装sbt

仿照文章的操作:

   #建立sbt安装目录
sudo mkdir /usr/local/sbt
sudo chown -R wh /usr/local/sbt
#解压
tar -zxvf ~/Downloads/sbt-1.3.8.tgz /usr/local/sbt
#将位于sbt/bin下面的sbt-launch.jar文件放在sbt目录下
cd /usr/local/sbt
cp ./bin/sbt-launch.jar ./
#在/usr/local/sbt下创建sbt脚本,该脚本与bin目录同级
vim ./sbt
# 添加如下内容
#!/bin/bash
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M" java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"
#为sbt脚本添加可执行权限
chmod u+x ./sbt
#配置环境变量
vim ~/.bashrc
#在文件最后添加以下内容:
export SBT_HOME=/usr/local/sbt
export PATH=SBT_HOME:$PATH
# 使配置文件生效
source ~/.bashrc

查看sbt版本:

2

3

根据错误提示:

[error] Expected ';'
[error] Not a valid command: sbt-version (similar: writeSbtVersion, session)
[error] Not a valid project ID: sbt-version
[error] Expected ':'
[error] Not a valid key: sbt-version (similar: sbtVersion, version, sbtBinaryVersion)
[error] sbt-version
[error]

可得:sbt-version在此时是错误的,sbtVersion, version, sbtBinaryVersion这些是正确的。

就像这样:

4

sbt编译打包

1.创建文件夹sparkapp1作为应用程序根目录

cd ~
mkdir sparkapp1
mkdir -p ./sparkapp1/src/main/scala
vim ./sparkapp1/src/main/scala/hello.scala
#输入以下代码
object hello{
def main(args: Array[String]){
println("hello")
}
}

2.在应用程序根目录下创建sbt文件

vim ./sparkapp1/hello.sbt
#输入以下内容
name := "Hello Project"
version := "1.0"
scalaVersion :="2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"

这些版本号在博客第一张图片中有体现,也可以spark-shell查看:

(这里是因为已经搭建了spark集群,故而冗长)

5

hello.sbt

6

hello.scala

7

代码截图:

8

3.编译打包:

wh@master:~/sparkapp1$ /usr/local/sbt/sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[info] Updated file /home/wh/sparkapp1/project/build.properties: set sbt.version to 1.3.8
[info] Loading project definition from /home/wh/sparkapp1/project
[info] Loading settings for project sparkapp1 from hello.sbt ...
[info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/)
[info] Updating
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.4.…
100.0% [##########] 30.3 KiB (13.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-parent_2.11/2.…
100.0% [##########] 100.4 KiB (104.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/18/apache-18.pom
100.0% [##########] 15.3 KiB (15.8 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.…
100.0% [##########] 846 B (2.5 KiB / s)
https://repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/…
100.0% [##########] 2.3 KiB (6.8 KiB / s)
https://repo1.maven.org/maven2/com/twitter/chill-java/0.9.3/chill-ja…
100.0% [##########] 2.0 KiB (2.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2…
100.0% [##########] 6.9 KiB (7.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/2.…
100.0% [##########] 2.3 KiB (2.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/xbean/xbean-asm6-shaded/4.…
100.0% [##########] 4.0 KiB (12.5 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-clie…
100.0% [##########] 7.4 KiB (21.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.4.…
100.0% [##########] 26.6 KiB (72.2 KiB / s)
https://repo1.maven.org/maven2/io/netty/netty-all/4.1.17.Final/netty…
100.0% [##########] 24.8 KiB (26.7 KiB / s)
https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.pom
100.0% [##########] 21.2 KiB (13.7 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-da…
100.0% [##########] 5.9 KiB (18.0 KiB / s)
https://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0…
100.0% [##########] 2.0 KiB (5.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.pom
100.0% [##########] 6.1 KiB (20.6 KiB / s)
https://repo1.maven.org/maven2/javax/activation/activation/1.1.1/act…
100.0% [##########] 644 B (2.2 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-…
100.0% [##########] 4.2 KiB (14.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-network-shuffl…
100.0% [##########] 4.9 KiB (15.2 KiB / s)
https://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.pom
100.0% [##########] 140 B (571 B / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.11/2.…
100.0% [##########] 4.7 KiB (2.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-kvstore_2.11/2…
100.0% [##########] 4.6 KiB (16.9 KiB / s)
https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.7.1…
100.0% [##########] 3.9 KiB (12.4 KiB / s)
https://repo1.maven.org/maven2/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.p…
100.0% [##########] 2.1 KiB (5.9 KiB / s)
https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.5.1…
100.0% [##########] 6.2 KiB (20.3 KiB / s)
https://repo1.maven.org/maven2/javax/servlet/javax.servlet-api/3.1.0…
100.0% [##########] 13.2 KiB (15.2 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-graphit…
100.0% [##########] 1.3 KiB (4.7 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-jackson_2.11/3.5.3/…
100.0% [##########] 2.0 KiB (7.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.5/…
100.0% [##########] 22.8 KiB (59.5 KiB / s)
https://repo1.maven.org/maven2/com/ning/compress-lzf/1.0.3/compress-…
100.0% [##########] 7.7 KiB (24.0 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/jul-to-slf4j/1.7.16/jul-to-…
100.0% [##########] 986 B (4.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.6.5…
100.0% [##########] 10.8 KiB (10.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.8.2/avr…
100.0% [##########] 7.6 KiB (25.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-crypto/1.0…
100.0% [##########] 21.3 KiB (61.5 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.16/slf4j-…
100.0% [##########] 909 B (930 B / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.…
100.0% [##########] 1.4 KiB (5.4 KiB / s)
https://repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.…
100.0% [##########] 6.5 KiB (26.2 KiB / s)
https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.3.2-2/zst…
100.0% [##########] 1.9 KiB (6.1 KiB / s)
https://repo1.maven.org/maven2/io/netty/netty/3.9.9.Final/netty-3.9.…
100.0% [##########] 28.2 KiB (81.6 KiB / s)
https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9…
100.0% [##########] 965 B (3.0 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1…
100.0% [##########] 967 B (2.6 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-comm…
100.0% [##########] 10.2 KiB (16.4 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse…
100.0% [##########] 4.5 KiB (14.5 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse…
100.0% [##########] 4.6 KiB (15.7 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-…
100.0% [##########] 2.7 KiB (9.1 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/jcl-over-slf4j/1.7.16/jcl-o…
100.0% [##########] 963 B (2.8 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-serv…
100.0% [##########] 12.0 KiB (19.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-network-common…
100.0% [##########] 6.3 KiB (3.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-tags_2.11/2.4.…
100.0% [##########] 2.6 KiB (10.6 KiB / s)
https://repo1.maven.org/maven2/com/twitter/chill_2.11/0.9.3/chill_2.…
100.0% [##########] 2.3 KiB (7.1 KiB / s)
https://repo1.maven.org/maven2/commons-net/commons-net/3.1/commons-n…
100.0% [##########] 18.0 KiB (26.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-launcher_2.11/…
100.0% [##########] 6.7 KiB (23.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/…
100.0% [##########] 3.3 KiB (10.8 KiB / s)
https://repo1.maven.org/maven2/net/razorvine/pyrolite/4.13/pyrolite-…
100.0% [##########] 3.5 KiB (6.1 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/containers/proje…
100.0% [##########] 3.7 KiB (11.1 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-parent/…
100.0% [##########] 11.8 KiB (33.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project-dist…
100.0% [##########] 18.1 KiB (51.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/xbean/xbean/4.8/xbean-4.8.…
100.0% [##########] 17.5 KiB (28.6 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/…
100.0% [##########] 7.6 KiB (22.3 KiB / s)
https://repo1.maven.org/maven2/net/java/jvnet-parent/3/jvnet-parent-…
100.0% [##########] 4.7 KiB (14.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-parent/1.8.2/avr…
100.0% [##########] 20.9 KiB (21.5 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/project/2.22.2/p…
100.0% [##########] 82.3 KiB (55.2 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.16/slf4j-p…
100.0% [##########] 13.2 KiB (20.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/7/apache-7.pom
100.0% [##########] 14.1 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/apache-curator/2.6…
100.0% [##########] 29.6 KiB (12.5 KiB / s)
https://repo1.maven.org/maven2/io/netty/netty-parent/4.1.17.Final/ne…
100.0% [##########] 50.2 KiB (18.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/41/…
100.0% [##########] 63.3 KiB (24.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/40/…
100.0% [##########] 62.6 KiB (16.3 KiB / s)
https://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-par…
100.0% [##########] 4.0 KiB (6.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/34/…
100.0% [##########] 54.6 KiB (11.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/23/…
100.0% [##########] 43.4 KiB (19.5 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/oss-parent/24/oss-paren…
100.0% [##########] 19.0 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/17/apache-17.pom
100.0% [##########] 15.7 KiB (15.3 KiB / s)
https://repo1.maven.org/maven2/net/java/jvnet-parent/4/jvnet-parent-…
100.0% [##########] 7.6 KiB (3.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis-j…
100.0% [##########] 5.4 KiB (2.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-toplevel/1.8.2/a…
100.0% [##########] 15.3 KiB (5.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.6.…
100.0% [##########] 42.8 KiB (10.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
100.0% [##########] 14.3 KiB (8.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom
100.0% [##########] 13.6 KiB (6.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/9/apache-9.pom
100.0% [##########] 14.8 KiB (7.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis-d…
100.0% [##########] 15.7 KiB (20.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/10/apache-10.pom
100.0% [##########] 14.5 KiB (13.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-main/2.6.5/h…
100.0% [##########] 17.7 KiB (8.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/geronimo/genesis/genesis/2…
100.0% [##########] 13.9 KiB (22.1 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-bom/2.4.0-b34/h…
100.0% [##########] 20.4 KiB (38.0 KiB / s)
https://repo1.maven.org/maven2/net/java/jvnet-parent/5/jvnet-parent-…
100.0% [##########] 8.7 KiB (26.2 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-core_2.11/3.5.3/jso…
100.0% [##########] 2.3 KiB (7.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 4.6 KiB (11.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 3.4 KiB (13.1 KiB / s)
https://repo1.maven.org/maven2/javax/validation/validation-api/1.1.0…
100.0% [##########] 7.7 KiB (9.3 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-api/2.4.0-b34/h…
100.0% [##########] 4.5 KiB (16.9 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/media/jersey-med…
100.0% [##########] 7.0 KiB (6.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h…
100.0% [##########] 28.7 KiB (25.9 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag…
100.0% [##########] 13.0 KiB (17.7 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava/16.0.1/guava-1…
100.0% [##########] 6.0 KiB (18.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-framework/…
100.0% [##########] 2.1 KiB (7.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6…
100.0% [##########] 4.5 KiB (17.9 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-co…
100.0% [##########] 4.8 KiB (12.4 KiB / s)
https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/…
100.0% [##########] 5.1 KiB (17.0 KiB / s)
https://repo1.maven.org/maven2/org/tukaani/xz/1.5/xz-1.5.pom
100.0% [##########] 1.9 KiB (5.5 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-an…
100.0% [##########] 1.2 KiB (4.2 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-locator/2.4.0-b…
100.0% [##########] 5.4 KiB (17.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/…
100.0% [##########] 2.3 KiB (3.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1…
100.0% [##########] 11.2 KiB (42.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.8.2/avro-i…
100.0% [##########] 5.0 KiB (2.1 KiB / s)
https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-…
100.0% [##########] 7.1 KiB (27.1 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-a…
100.0% [##########] 1.5 KiB (5.9 KiB / s)
https://repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.…
100.0% [##########] 3.4 KiB (12.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 5.5 KiB (8.6 KiB / s)
https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0.1/jav…
100.0% [##########] 20.9 KiB (11.6 KiB / s)
https://repo1.maven.org/maven2/commons-codec/commons-codec/1.9/commo…
100.0% [##########] 11.3 KiB (40.0 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/osgi-resource-locat…
100.0% [##########] 6.0 KiB (16.6 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-…
100.0% [##########] 3.9 KiB (3.0 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/external/javax.inje…
100.0% [##########] 6.3 KiB (10.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/…
100.0% [##########] 1.5 KiB (5.0 KiB / s)
https://repo1.maven.org/maven2/javax/annotation/javax.annotation-api…
100.0% [##########] 13.1 KiB (4.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.5…
100.0% [##########] 29.9 KiB (20.6 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl…
100.0% [##########] 1.2 KiB (525 B / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/media/project/2.…
100.0% [##########] 2.9 KiB (1.2 KiB / s)
https://repo1.maven.org/maven2/com/esotericsoftware/kryo-parent/4.0.…
100.0% [##########] 6.9 KiB (2.9 KiB / s)
https://repo1.maven.org/maven2/com/google/guava/guava-parent/16.0.1/…
100.0% [##########] 7.2 KiB (22.8 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/external/2.4.0-b34/…
100.0% [##########] 2.9 KiB (8.7 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/…
100.0% [##########] 7.6 KiB (2.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn/2.6.5/h…
100.0% [##########] 3.5 KiB (10.8 KiB / s)
https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-…
100.0% [##########] 9.4 KiB (27.3 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/pom/8/pom-8.pom
100.0% [##########] 10.3 KiB (3.3 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-…
100.0% [##########] 2.4 KiB (7.6 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag…
100.0% [##########] 2.7 KiB (8.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 6.6 KiB (1.8 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-parent/2.4.0-b3…
100.0% [##########] 46.7 KiB (49.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/33/…
100.0% [##########] 51.8 KiB (12.5 KiB / s)
https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer-…
100.0% [##########] 11.2 KiB (9.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/32/…
100.0% [##########] 51.5 KiB (26.6 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/project/…
100.0% [##########] 3.0 KiB (9.9 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/codehaus-parent/4/codeha…
100.0% [##########] 4.7 KiB (13.5 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-parent/…
100.0% [##########] 8.0 KiB (20.7 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/oss-parent/23/oss-paren…
100.0% [##########] 18.8 KiB (27.3 KiB / s)
https://repo1.maven.org/maven2/org/fusesource/fusesource-pom/1.9/fus…
100.0% [##########] 14.4 KiB (15.2 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/oss-parent/25/oss-paren…
100.0% [##########] 18.9 KiB (17.7 KiB / s)
https://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/min…
100.0% [##########] 3.2 KiB (11.1 KiB / s)
https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/j…
100.0% [##########] 3.8 KiB (13.1 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-utils/2.4.0-b34…
100.0% [##########] 5.0 KiB (15.2 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/external/aopallianc…
100.0% [##########] 6.6 KiB (23.5 KiB / s)
https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.…
100.0% [##########] 1.3 KiB (2.3 KiB / s)
https://repo1.maven.org/maven2/org/javassist/javassist/3.18.1-GA/jav…
100.0% [##########] 9.3 KiB (27.4 KiB / s)
https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons…
100.0% [##########] 17.1 KiB (25.5 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-ast_2.11/3.5.3/json…
100.0% [##########] 1.6 KiB (5.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.6.5/h…
100.0% [##########] 7.5 KiB (23.0 KiB / s)
https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient…
100.0% [##########] 7.6 KiB (6.8 KiB / s)
https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-…
100.0% [##########] 7.8 KiB (22.7 KiB / s)
https://repo1.maven.org/maven2/commons-configuration/commons-configu…
100.0% [##########] 13.0 KiB (22.3 KiB / s)
https://repo1.maven.org/maven2/org/objenesis/objenesis/2.5.1/objenes…
100.0% [##########] 2.7 KiB (2.7 KiB / s)
https://repo1.maven.org/maven2/commons-collections/commons-collectio…
100.0% [##########] 12.1 KiB (17.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-client/2.6…
100.0% [##########] 2.2 KiB (6.8 KiB / s)
https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.pom
100.0% [##########] 623 B (2.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 3.0 KiB (9.7 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-scalap_2.11/3.5.3/j…
100.0% [##########] 1.6 KiB (6.4 KiB / s)
https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/…
100.0% [##########] 5.2 KiB (6.1 KiB / s)
https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-…
100.0% [##########] 9.9 KiB (25.7 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11…
100.0% [##########] 2.5 KiB (7.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 3.2 KiB (11.9 KiB / s)
https://repo1.maven.org/maven2/org/htrace/htrace-core/3.0.4/htrace-c…
100.0% [##########] 2.8 KiB (8.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/…
100.0% [##########] 10.3 KiB (14.8 KiB / s)
https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-c…
100.0% [##########] 7.8 KiB (7.8 KiB / s)
https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5…
100.0% [##########] 8.2 KiB (7.6 KiB / s)
https://repo1.maven.org/maven2/com/google/google/1/google-1.pom
100.0% [##########] 1.5 KiB (4.8 KiB / s)
https://repo1.maven.org/maven2/org/mortbay/jetty/project/6.1.26/proj…
100.0% [##########] 9.0 KiB (26.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/39/…
100.0% [##########] 60.5 KiB (77.8 KiB / s)
https://repo1.maven.org/maven2/org/htrace/htrace/3.0.4/htrace-3.0.4.…
100.0% [##########] 6.4 KiB (8.2 KiB / s)
https://repo1.maven.org/maven2/org/objenesis/objenesis-parent/2.5.1/…
100.0% [##########] 16.1 KiB (13.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/4/apache-4.pom
100.0% [##########] 4.4 KiB (4.8 KiB / s)
https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer-…
100.0% [##########] 11.3 KiB (7.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/25/…
100.0% [##########] 47.2 KiB (13.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/17/…
100.0% [##########] 30.5 KiB (7.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-parent/11/…
100.0% [##########] 24.1 KiB (13.2 KiB / s)
https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-parent/10/jet…
100.0% [##########] 3.2 KiB (7.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/16/apache-16.pom
100.0% [##########] 15.0 KiB (12.6 KiB / s)
https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-parent/14/jet…
100.0% [##########] 16.1 KiB (8.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-…
100.0% [##########] 5.6 KiB (7.5 KiB / s)
https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8…
100.0% [##########] 6.8 KiB (8.8 KiB / s)
https://repo1.maven.org/maven2/xml-apis/xml-apis/1.3.04/xml-apis-1.3…
100.0% [##########] 1.8 KiB (4.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/…
100.0% [##########] 6.2 KiB (13.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-…
100.0% [##########] 3.8 KiB (6.2 KiB / s)
https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-c…
100.0% [##########] 1.6 KiB (796 B / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.…
100.0% [##########] 1.9 KiB (924 B / s)
https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inj…
100.0% [##########] 612 B (1.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-…
100.0% [##########] 11.8 KiB (12.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/…
100.0% [##########] 5.8 KiB (2.0 KiB / s)
https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-ap…
100.0% [##########] 5.4 KiB (3.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcompone…
100.0% [##########] 14.4 KiB (12.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/3/apache-3.pom
100.0% [##########] 3.3 KiB (2.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-…
100.0% [##########] 41.6 KiB (18.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server/…
100.0% [##########] 1.7 KiB (231 B / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/project/6/p…
100.0% [##########] 23.4 KiB (13.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/project/project/…
100.0% [##########] 26.1 KiB (14.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/apache/11/apache-11.pom
100.0% [##########] 14.5 KiB (21.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api…
100.0% [##########] 2.6 KiB (8.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0…
100.0% [##########] 2.3 KiB (6.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.…
100.0% [##########] 5.6 KiB (14.6 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/je…
100.0% [##########] 3.3 KiB (8.6 KiB / s)
https://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom
100.0% [##########] 6.2 KiB (16.4 KiB / s)
https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0…
100.0% [##########] 7.1 KiB (6.0 KiB / s)
https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-…
100.0% [##########] 962 B (779 B / s)
https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1…
100.0% [##########] 357 B (168 B / s)
https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-…
100.0% [##########] 2.3 KiB (857 B / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-par…
100.0% [##########] 1.5 KiB (5.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-parent/1…
100.0% [##########] 26.7 KiB (70.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcompone…
100.0% [##########] 11.4 KiB (26.5 KiB / s)
https://repo1.maven.org/maven2/com/google/inject/guice-parent/3.0/gu…
100.0% [##########] 12.7 KiB (19.5 KiB / s)
https://repo1.maven.org/maven2/com/google/google/5/google-5.pom
100.0% [##########] 2.4 KiB (7.2 KiB / s)
https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopallian…
100.0% [##########] 363 B (1.4 KiB / s)
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading org.scala-lang:scala-library:2.11.12
[error] Not found
[error] Not found
[error] not found: /home/wh/.ivy2/local/org.scala-lang/scala-library/2.11.12/ivys/ivy.xml
[error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.pom
[error] Error downloading org.codehaus.jackson:jackson-xc:1.9.13
[error] Not found
[error] Not found
[error] not found: /home/wh/.ivy2/local/org.codehaus.jackson/jackson-xc/1.9.13/ivys/ivy.xml
[error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.pom
[error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:246)
[error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$34(CoursierDependencyResolution.scala:215)
[error] at scala.util.Either$LeftProjection.map(Either.scala:573)
[error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:215)
[error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60)
[error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:52)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:102)
[error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:69)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$20(LibraryManagement.scala:115)
[error] at scala.util.control.Exception$Catch.apply(Exception.scala:228)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:115)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:96)
[error] at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:150)
[error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:129)
[error] at sbt.Classpaths$.$anonfun$updateTask0$5(Defaults.scala:2947)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:281)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
[error] at sbt.Execute.work(Execute.scala:290)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:281)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scala-lang:scala-library:2.11.12
[error] Not found
[error] Not found
[error] not found: /home/wh/.ivy2/local/org.scala-lang/scala-library/2.11.12/ivys/ivy.xml
[error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.pom
[error] Error downloading org.codehaus.jackson:jackson-xc:1.9.13
[error] Not found
[error] Not found
[error] not found: /home/wh/.ivy2/local/org.codehaus.jackson/jackson-xc/1.9.13/ivys/ivy.xml
[error] download error: Caught java.net.ConnectException: Connection refused (Connection refused) (Connection refused (Connection refused)) while downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.pom
[error] Total time: 125 s (02:05), completed Apr 29, 2022 12:12:47 PM
wh@master:~/sparkapp1$ /usr/local/sbt/sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[info] Loading project definition from /home/wh/sparkapp1/project
[info] Loading settings for project sparkapp1 from hello.sbt ...
[info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/)
[info] Updating
https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/…
100.0% [##########] 1.4 KiB (719 B / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.1…
100.0% [##########] 1.9 KiB (5.4 KiB / s)
[info] Resolved dependencies
[info] Updating
https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.12…
100.0% [##########] 2.3 KiB (9.3 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-c…
100.0% [##########] 2.1 KiB (3.3 KiB / s)
https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.pom
100.0% [##########] 19.4 KiB (18.6 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11…
100.0% [##########] 2.5 KiB (2.3 KiB / s)
[info] Resolved dependencies
[info] Fetching artifacts of
https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8…
100.0% [##########] 140.2 KiB (125.1 KiB / s)
https://repo1.maven.org/maven2/org/objenesis/objenesis/2.5.1/objenes…
100.0% [##########] 53.1 KiB (42.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-…
100.0% [##########] 43.9 KiB (32.0 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.…
100.0% [##########] 117.6 KiB (78.7 KiB / s)
https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-…
100.0% [##########] 22.8 KiB (10.7 KiB / s)
https://repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.…
100.0% [##########] 170.3 KiB (97.2 KiB / s)
https://repo1.maven.org/maven2/commons-net/commons-net/3.1/commons-n…
100.0% [##########] 267.0 KiB (61.4 KiB / s)
https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9…
100.0% [##########] 32.2 KiB (34.5 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl…
100.0% [##########] 226.8 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 59.9 KiB (28.3 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse…
100.0% [##########] 17.7 KiB (24.8 KiB / s)
https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-c…
100.0% [##########] 201.2 KiB (16.0 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-core_2.11/3.5.3/jso…
100.0% [##########] 673.8 KiB (17.2 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-crypto/1.0…
100.0% [##########] 131.4 KiB (10.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.6.5…
100.0% [##########] 25.4 KiB (10.5 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/external/aopallianc…
100.0% [##########] 14.4 KiB (9.4 KiB / s)
https://repo1.maven.org/maven2/com/ning/compress-lzf/1.0.3/compress-…
100.0% [##########] 78.0 KiB (11.0 KiB / s)

https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-…
100.0% [##########] 180.8 KiB (8.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 1.5 MiB (23.2 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.16/slf4j-api-…
100.0% [##########] 39.6 KiB (10.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/xbean/xbean-asm6-shaded/4.…
100.0% [##########] 276.3 KiB (18.5 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/…
100.0% [##########] 1.6 MiB (18.1 KiB / s)
https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5…
100.0% [##########] 521.0 KiB (21.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar
100.0% [##########] 1.2 MiB (15.7 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-jackson_2.11/3.5.3/…
100.0% [##########] 47.1 KiB (8.0 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11…
100.0% [##########] 655.4 KiB (13.7 KiB / s)
https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
100.0% [##########] 14.7 KiB (13.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-tags_2.11/2.4.…
100.0% [##########] 15.1 KiB (6.4 KiB / s)
https://repo1.maven.org/maven2/javax/servlet/javax.servlet-api/3.1.0…
100.0% [##########] 93.6 KiB (14.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1…
100.0% [##########] 357.0 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/com/twitter/chill_2.11/0.9.3/chill_2.…
100.0% [##########] 228.1 KiB (16.0 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.16/slf4j-…
100.0% [##########] 9.7 KiB (13.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/…
100.0% [##########] 148.3 KiB (16.0 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-scalap_2.11/3.5.3/j…
100.0% [##########] 600.4 KiB (12.3 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/external/javax.inje…
100.0% [##########] 5.8 KiB (3.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-kvstore_2.11/2…
100.0% [##########] 51.8 KiB (10.1 KiB / s)
https://repo1.maven.org/maven2/io/netty/netty-all/4.1.17.Final/netty…
100.0% [##########] 3.6 MiB (17.0 KiB / s)
https://repo1.maven.org/maven2/com/esotericsoftware/kryo-shaded/4.0.…
100.0% [##########] 401.2 KiB (18.3 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.1…
100.0% [##########] 26.4 KiB (15.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-framework/…
100.0% [##########] 180.9 KiB (14.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 66.2 KiB (21.9 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.…
100.0% [##########] 17.9 KiB (23.4 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-api/2.4.0-b34/h…
100.0% [##########] 174.8 KiB (19.8 KiB / s)
https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-…
100.0% [##########] 1021.2 KiB (15.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.5…
100.0% [##########] 3.2 MiB (14.3 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/osgi-resource-locat…
100.0% [##########] 19.8 KiB (6.3 KiB / s)
https://repo1.maven.org/maven2/org/json4s/json4s-ast_2.11/3.5.3/json…
100.0% [##########] 88.5 KiB (19.4 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-da…
100.0% [##########] 1.1 MiB (17.2 KiB / s)
https://repo1.maven.org/maven2/javax/activation/activation/1.1.1/act…
100.0% [##########] 67.8 KiB (29.7 KiB / s)
https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-…
100.0% [##########] 186.0 KiB (16.4 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-c…
100.0% [##########] 413.8 KiB (13.6 KiB / s)
https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopallian…
100.0% [##########] 4.4 KiB (3.3 KiB / s)
https://repo1.maven.org/maven2/javax/annotation/javax.annotation-api…
100.0% [##########] 25.7 KiB (28.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.…
100.0% [##########] 221.9 KiB (16.4 KiB / s)
https://repo1.maven.org/maven2/net/razorvine/pyrolite/4.13/pyrolite-…
100.0% [##########] 92.6 KiB (12.8 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-locator/2.4.0-b…
100.0% [##########] 177.0 KiB (10.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-…
100.0% [##########] 651.9 KiB (11.7 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/jcl-over-slf4j/1.7.16/jcl-o…
100.0% [##########] 16.0 KiB (15.3 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-co…
100.0% [##########] 252.9 KiB (20.8 KiB / s)
https://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/min…
100.0% [##########] 5.6 KiB (13.3 KiB / s)


https://repo1.maven.org/maven2/com/google/guava/guava/16.0.1/guava-1…
100.0% [##########] 2.1 MiB (4.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 516.2 KiB (18.0 KiB / s)
https://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.jar
100.0% [##########] 63.7 KiB (13.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h…
92.5% [######### ] 7.0 MiB (8.7 KiB / s)
https://repo1.maven.org/maven2/xml-apis/xml-apis/1.3.04/xml-apis-1.3…
100.0% [##########] 189.8 KiB (13.9 KiB / s)
https://repo1.maven.org/maven2/commons-configuration/commons-configu…
100.0% [##########] 291.8 KiB (17.4 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/…
75.1% [####### ] 3.3 MiB (5.5 KiB / s)
https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/j…
100.0% [##########] 173.0 KiB (16.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/…
100.0% [##########] 423.2 KiB (1.6 KiB / s)

https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.5/…
100.0% [##########] 468.6 KiB (12.4 KiB / s)
https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/…
100.0% [##########] 33.8 KiB (6.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/…
100.0% [##########] 774.4 KiB (2.5 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-…
100.0% [##########] 503.6 KiB (13.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-client/2.6…
100.0% [##########] 67.3 KiB (14.7 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/hk2/hk2-utils/2.4.0-b34…
100.0% [##########] 116.2 KiB (15.3 KiB / s)
https://repo1.maven.org/maven2/org/slf4j/jul-to-slf4j/1.7.16/jul-to-…
100.0% [##########] 4.5 KiB (4.5 KiB / s)
https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0.1/jav…
100.0% [##########] 112.8 KiB (302 B / s)
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-a…
100.0% [##########] 762.4 KiB (17.8 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2…
100.0% [##########] 1.5 MiB (11.8 KiB / s)
https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar
100.0% [##########] 478.4 KiB (11.1 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/media/jersey-med…
100.0% [##########] 71.0 KiB (6.4 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/containers/jerse…
100.0% [##########] 64.7 KiB (11.0 KiB / s)
https://repo1.maven.org/maven2/com/github/luben/zstd-jni/1.3.2-2/zst…
100.0% [##########] 2.2 MiB (15.6 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-cl…
100.0% [##########] 672.1 KiB (12.4 KiB / s)
https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-ap…
100.0% [##########] 102.7 KiB (20.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.8.2/avro-i…
100.0% [##########] 129.9 KiB (21.4 KiB / s)
https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.…
100.0% [##########] 1.2 MiB (15.4 KiB / s)
https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.8.2/avr…
100.0% [##########] 182.7 KiB (30.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.4.…
100.0% [##########] 1.9 MiB (27.4 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-graphit…
100.0% [##########] 20.7 KiB (24.6 KiB / s)
https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons…
100.0% [##########] 277.6 KiB (26.6 KiB / s)
https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0…
100.0% [##########] 693.8 KiB (22.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-…
100.0% [##########] 312.5 KiB (21.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.6.5/h…
100.0% [##########] 88.5 KiB (27.4 KiB / s)
https://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar
100.0% [##########] 85.3 KiB (34.4 KiB / s)
https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/je…
100.0% [##########] 66.2 KiB (31.8 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1…
100.0% [##########] 38.4 KiB (20.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0…
100.0% [##########] 78.0 KiB (14.2 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-serv…
100.0% [##########] 929.4 KiB (18.5 KiB / s)
https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inj…
100.0% [##########] 2.4 KiB (1004 B / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-launcher_2.11/…
100.0% [##########] 74.2 KiB (15.4 KiB / s)
https://repo1.maven.org/maven2/net/sf/py4j/py4j/0.10.7/py4j-0.10.7.j…
100.0% [##########] 119.9 KiB (13.5 KiB / s)
https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient…
100.0% [##########] 297.9 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.12/…
100.0% [##########] 5.5 MiB (33.2 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-an…
100.0% [##########] 45.9 KiB (35.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/2.4.…
100.0% [##########] 12.8 MiB (30.7 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-clie…
100.0% [##########] 163.5 KiB (58.7 KiB / s)
https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.5.1…
100.0% [##########] 197.2 KiB (32.0 KiB / s)
https://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/lz4-java-1.4.0…
100.0% [##########] 361.4 KiB (40.7 KiB / s)
https://repo1.maven.org/maven2/javax/validation/validation-api/1.1.0…
100.0% [##########] 62.3 KiB (36.9 KiB / s)
https://repo1.maven.org/maven2/commons-collections/commons-collectio…
100.0% [##########] 574.5 KiB (31.7 KiB / s)
https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.…
100.0% [##########] 15.5 KiB (10.6 KiB / s)
https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-c…
100.0% [##########] 40.2 KiB (20.5 KiB / s)
https://repo1.maven.org/maven2/io/netty/netty/3.9.9.Final/netty-3.9.…
100.0% [##########] 1.3 MiB (21.6 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/core/jersey-comm…
100.0% [##########] 682.0 KiB (29.4 KiB / s)
https://repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-…
100.0% [##########] 41.9 KiB (35.9 KiB / s)
https://repo1.maven.org/maven2/com/twitter/chill-java/0.9.3/chill-ja…
100.0% [##########] 57.3 KiB (24.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/2.…
100.0% [##########] 242.4 KiB (23.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/…
100.0% [##########] 39.5 KiB (37.8 KiB / s)
https://repo1.maven.org/maven2/org/glassfish/jersey/bundles/repackag…
100.0% [##########] 948.5 KiB (31.7 KiB / s)
https://repo1.maven.org/maven2/org/spark-project/spark/unused/1.0.0/…
100.0% [##########] 2.7 KiB (9.5 KiB / s)
https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1…
100.0% [##########] 184.2 KiB (39.6 KiB / s)
https://repo1.maven.org/maven2/org/javassist/javassist/3.18.1-GA/jav…
100.0% [##########] 697.5 KiB (24.9 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-unsafe_2.11/2.…
100.0% [##########] 48.0 KiB (36.5 KiB / s)
https://repo1.maven.org/maven2/org/tukaani/xz/1.5/xz-1.5.jar
100.0% [##########] 97.2 KiB (48.7 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-network-shuffl…
100.0% [##########] 69.2 KiB (59.5 KiB / s)
https://repo1.maven.org/maven2/org/htrace/htrace-core/3.0.4/htrace-c…
100.0% [##########] 30.5 KiB (38.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/spark/spark-network-common…
100.0% [##########] 2.3 MiB (29.0 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api…
100.0% [##########] 16.2 KiB (5.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6…
100.0% [##########] 1.8 MiB (24.1 KiB / s)
https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.7.1…
100.0% [##########] 1.9 MiB (13.7 KiB / s)
https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar
100.0% [##########] 262.2 KiB (16.0 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/…
100.0% [##########] 4.4 MiB (26.3 KiB / s)
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/h…
100.0% [##########] 7.6 MiB (27.3 KiB / s)






[warn] Canceling execution...
[error] Total time: 6131 s (01:42:11), completed Apr 29, 2022 1:56:34 PM
wh@master:~/sparkapp1$ /usr/local/sbt/sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[info] Loading project definition from /home/wh/sparkapp1/project
[info] Loading settings for project sparkapp1 from hello.sbt ...
[info] Set current project to Hello Project (in build file:/home/wh/sparkapp1/)
[info] Fetching artifacts of
https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11…
100.0% [##########] 654.7 KiB (152.1 KiB / s)
https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-…
100.0% [##########] 675.3 KiB (106.0 KiB / s)
https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.12…
100.0% [##########] 14.9 MiB (879.2 KiB / s)
[info] Fetched artifacts of
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to /home/wh/sparkapp1/target/scala-2.11/classes ...
https://repo1.maven.org/maven2/org/scala-sbt/compiler-bridge_2.11/1.…
100.0% [##########] 2.8 KiB (10.6 KiB / s)
https://repo1.maven.org/maven2/org/scala-sbt/compiler-bridge_2.11/1.…
100.0% [##########] 47.7 KiB (158.0 KiB / s)
[info] Non-compiled module 'compiler-bridge_2.11' for Scala 2.11.12. Compiling...
[info] Compilation completed in 16.01s.
[success] Total time: 38 s, completed Apr 29, 2022 1:57:29 PM
wh@master:~/sparkapp1$

cd ~/sparkapp1
/usr/local/sbt/sbt package
# 第一次需要下载文件,很费时间,后面再打包就很快了

9

运行

$SPARK_HOME/bin/spark-submit --class "hello" ~/sparkapp1/target/scala-2.11/hello-project_2.11-1.0.jar

10

11

输出了hello

为了更加清晰输出,替换hello.scala文件:

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
def main(args: Array[String]) {
val logFile = "file:///usr/local/spark/README.md" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}

删除~/sparkapp1下的project和target

重新编译打包:

12

参考链接

https://blog.csdn.net/m0_51195997/article/details/124094278

https://www.cnblogs.com/zjj999/p/12710702.html

https://yusure.cn/backend/308.html

https://www.cnblogs.com/midworld/p/15647016.html

http://dblab.xmu.edu.cn/blog/2137-2/

https://blog.csdn.net/J_oshua/article/details/109403952