Maven编译Flume-ng 1.5.0
确保可以上网,因为编译时要下载很多的jar 包1、安装Maven下载:apache-maven-3.0.5-bin.tar.gz解压:[sparkadmin@hadoop4 ~]$ tar -zxvfapache-maven-3.0.5-bin.tar.gz配置环境变量[sparkadmin@hadoop4 ~]$ vim .bashrcexpo
确保可以上网,因为编译时要下载很多的jar 包
1、安装Maven
下载:apache-maven-3.0.5-bin.tar.gz
解压:[sparkadmin@hadoop4 ~]$ tar -zxvfapache-maven-3.0.5-bin.tar.gz
配置环境变量
[sparkadmin@hadoop4 ~]$ vim .bashrc
export MAVEN_HOME=/home/sparkadmin/apache-maven-3.0.5
export PATH=.:$PATH:$MAVEN_HOME/bin
刷新配置:
[sparkadmin@hadoop4 ~]$ source .bashrc
检查是否安装成功
[sparkadmin@hadoop4 ~]$ mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.102s
[INFO] Finished at: Wed Sep 28 11:49:16 CST 2016
[INFO] Final Memory: 10M/481M
[INFO] ------------------------------------------------------------------------
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-site, site, post-site, site-deploy, pre-clean, clean, post-clean. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
出现上面信息表示maven安装成功。
2、编译Flume
下载:apache-flume-1.5.0-src.tar.gz
[sparkadmin@hadoop4 ~]$ tar -zxvf apache-flume-1.5.0-src.tar.gz
Flume-ng 1.5.0 默认的Hadoop版本是1.0.1,Hbase是0.92.1,在编译时指定Hadoop2,默认的hadoop2是hadoop 2.4.0
加参数:-Phadoop-2
<hadoop.version>1.0.1</hadoop.version>
<hbase.version>0.92.1</hbase.version>
编译:
[sparkadmin@hadoop4 apache-flume-1.5.0-src]$ mvn install -Phadoop-2 -DskipTests -Dtar
报错解决办法:
第一个错误
[ERROR] Failed to execute goal on project flume-hdfs-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-hdfs-sink:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]
apache-flume-1.5.0-src/flume-ng-sinks/flume-hdfs-sink下面的pom.xml中注释掉:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<version>test</version>
</dependency>
第二个错误:
[ERROR] Failed to execute goal on project flume-ng-hbase-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-ng-hbase-sink:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]
apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink下面的pom.xml中注释掉:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<scope>test</scope>
</dependency>
第三个错误:
[ERROR] Failed to execute goal on project flume-tools: Could not resolve dependencies for project org.apache.flume:flume-tools:jar:1.5.0: Failure to find org.apache.hadoop:hadoop-test:jar:2.4.0 in http://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of repo1.maven.org has elapsed or updates are forced -> [Help 1]
apache-flume-1.5.0-src/flume-tools下面的pom.xml中注释掉:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-test</artifactId>
<scope>test</scope>
</dependency>
第四个错误:
[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink: Could not resolve dependencies for project org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0: Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0 (compile), org.slf4j:slf4j-api:jar:1.6.1 (compile), org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?), org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided), org.apache.solr:solr-test-framework:jar:4.3.0 (test), org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test), junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com): Failed to transfer file: http://maven.twttr.com/ua_parser/ua-parser/1.3.0/ua-parser-1.3.0.pom. Return code is: 504 , ReasonPhrase:Gateway Timeout. -> [Help 1]
这是因为:ua-parser-1.3.0下载失败
在根目录apache-flume-1.5.0-src 下面的pom.xml 中只需要加一个新的repository即可:
<repository>
<id>p2.jfrog.org</id>
<url>http://p2.jfrog.org/libs-releases</url>
</repository>
[sparkadmin@hadoop4 apache-flume-1.5.0-src]$ mvn install -Phadoop-2 -DskipTests -Dtar
............................................
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Flume ...................................... SUCCESS [1.398s]
[INFO] Flume NG SDK ...................................... SUCCESS [2.502s]
[INFO] Flume NG Configuration ............................ SUCCESS [0.308s]
[INFO] Flume NG Core ..................................... SUCCESS [2.131s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.087s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.123s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.206s]
[INFO] Flume NG Channels ................................. SUCCESS [0.064s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.268s]
[INFO] Flume NG file-based channel ....................... SUCCESS [0.403s]
[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.204s]
[INFO] Flume NG Node ..................................... SUCCESS [0.272s]
[INFO] Flume NG Embedded Agent ........................... SUCCESS [0.578s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [0.672s]
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.368s]
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [1.567s]
[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [0.775s]
[INFO] Flume Sources ..................................... SUCCESS [0.038s]
[INFO] Flume Scribe Source ............................... SUCCESS [0.195s]
[INFO] Flume JMS Source .................................. SUCCESS [0.274s]
[INFO] Flume Twitter Source .............................. SUCCESS [0.194s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.036s]
[INFO] Flume legacy Avro source .......................... SUCCESS [0.702s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [0.183s]
[INFO] Flume NG Clients .................................. SUCCESS [0.029s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [2.799s]
[INFO] Flume NG Tools .................................... SUCCESS [1.268s]
[INFO] Flume NG distribution ............................. SUCCESS [21.332s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [19.271s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.916s
[INFO] Finished at: Wed Sep 28 17:43:25 CST 2016
[INFO] Final Memory: 89M/2968M
[INFO] ------------------------------------------------------------------------
出现上面信息表示编译成功。
最后将会在flume-ng-dist/target目录下产生以下的文件:
[sparkadmin@hadoop4 target]$ ll
total 29184
drwxrwxr-x 3 sparkadmin sparkadmin 4096 Sep 28 17:42 apache-flume-1.5.0-bin
-rw-rw-r-- 1 sparkadmin sparkadmin 27315535 Sep 28 17:43 apache-flume-1.5.0-bin.tar.gz
drwxrwxr-x 3 sparkadmin sparkadmin 4096 Sep 28 17:43 apache-flume-1.5.0-src
-rw-rw-r-- 1 sparkadmin sparkadmin 2545097 Sep 28 17:43 apache-flume-1.5.0-src.tar.gz
drwxrwxr-x 2 sparkadmin sparkadmin 4096 Sep 28 17:42 archive-tmp
drwxrwxr-x 3 sparkadmin sparkadmin 4096 Sep 28 17:42 maven-shared-archive-resources
-rw-rw-r-- 1 sparkadmin sparkadmin 1014 Sep 28 17:43 rat.txt
更多推荐
所有评论(0)