java—使用SpringXD将数据加载到Hadoop2.2

q35jwt9p  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(364)

我正在尝试从多个源创建数据流并将其加载到hdfs中。
为了证明概念,我在同一台机器上使用单节点hadoop installation和springxd。hadoop是与ambari一起部署的,报告显示它功能正常(hdp2.0)。spring xd安装在/var/spring xd中,环境变量$xd\u home设置为/var/spring xd/xd;spring的服务器和客户端都使用--hadoopdistro hdp20标志运行。但是,到目前为止,我只能在日志文件中获取这些消息:

18:00:06,346  INFO DeploymentsPathChildrenCache-0 Configuration.deprecation:840 - fs.default.name is deprecated. Instead, use fs.defaultFS
18:00:06,692  WARN DeploymentsPathChildrenCache-0 util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.ClasspathLoggingApplicationListener:55 - Application failed to start with classpath: [file:/var/spring-xd/xd/modules/processor/scripts/, file:/var/spring-xd/xd/config/, file:/var/spring-xd/xd/, file:/var/spring-xd/xd/lib/activation-1.1.jar, file:/var/spring-xd/xd/lib/amqp-client-3.2.4.jar, file:/var/spring-xd/xd/lib/aopalliance-1.0.jar, file:/var/spring-xd/xd/lib/args4j-2.0.16.jar, file:/var/spring-xd/xd/lib/asm-3.1.jar, file:/var/spring-xd/xd/lib/aspectjrt-1.6.6.jar, file:/var/spring-xd/xd/lib/aspectjweaver-1.6.6.jar, file:/var/spring-xd/xd/lib/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/cglib-2.2.1-v20090111.jar, file:/var/spring-xd/xd/lib/cloudfoundry-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/com.ibm.jbatch-tck-spi-1.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-1.7.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-core-1.8.0.jar, file:/var/spring-xd/xd/lib/commons-cli-1.2.jar, file:/var/spring-xd/xd/lib/commons-codec-1.7.jar, file:/var/spring-xd/xd/lib/commons-collections-3.2.1.jar, file:/var/spring-xd/xd/lib/commons-compress-1.4.1.jar, file:/var/spring-xd/xd/lib/commons-configuration-1.6.jar, file:/var/spring-xd/xd/lib/commons-daemon-1.0.13.jar, file:/var/spring-xd/xd/lib/commons-dbcp-1.2.2.jar, file:/var/spring-xd/xd/lib/commons-digester-1.8.jar, file:/var/spring-xd/xd/lib/commons-el-1.0.jar, file:/var/spring-xd/xd/lib/commons-fileupload-1.2.1.jar, file:/var/spring-xd/xd/lib/commons-httpclient-3.1.jar, file:/var/spring-xd/xd/lib/commons-io-2.4.jar, file:/var/spring-xd/xd/lib/commons-jexl-2.1.1.jar, file:/var/spring-xd/xd/lib/commons-lang-2.6.jar, file:/var/spring-xd/xd/lib/commons-math-2.1.jar, file:/var/spring-xd/xd/lib/commons-net-3.3.jar, file:/var/spring-xd/xd/lib/commons-pool-1.3.jar, file:/var/spring-xd/xd/lib/core-0.9.2.jar, file:/var/spring-xd/xd/lib/curator-client-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-framework-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-recipes-2.4.1.jar, file:/var/spring-xd/xd/lib/ehcache-core-2.3.0.jar, file:/var/spring-xd/xd/lib/freemarker-2.3.15.jar, file:/var/spring-xd/xd/lib/groovy-all-2.2.2.jar, file:/var/spring-xd/xd/lib/guava-14.0.1.jar, file:/var/spring-xd/xd/lib/guice-3.0.jar, file:/var/spring-xd/xd/lib/guice-servlet-3.0.jar, file:/var/spring-xd/xd/lib/hibernate-validator-4.3.1.Final.jar, file:/var/spring-xd/xd/lib/hsqldb-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-annotations-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-core-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-core-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/jackson-databind-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-mapper-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/javax.batch-api-1.0.jar, file:/var/spring-xd/xd/lib/javax.inject-1.jar, file:/var/spring-xd/xd/lib/javax.mail-1.5.0.jar, file:/var/spring-xd/xd/lib/jboss-logging-3.1.0.CR2.jar, file:/var/spring-xd/xd/lib/jcl-over-slf4j-1.7.6.jar, file:/var/spring-xd/xd/lib/jersey-guice-1.9.jar, file:/var/spring-xd/xd/lib/jersey-test-framework-grizzly2-1.9.jar, file:/var/spring-xd/xd/lib/jettison-1.2.jar, file:/var/spring-xd/xd/lib/joda-time-2.1.jar, file:/var/spring-xd/xd/lib/jolokia-core-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jmx-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jvm-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-spring-1.1.5.jar, file:/var/spring-xd/xd/lib/json-path-0.8.1.jar, file:/var/spring-xd/xd/lib/json-simple-1.1.jar, file:/var/spring-xd/xd/lib/json-smart-1.1.1.jar, file:/var/spring-xd/xd/lib/kite-data-core-0.12.1.jar, file:/var/spring-xd/xd/lib/kryo-2.22.jar, file:/var/spring-xd/xd/lib/lettuce-2.3.3.jar, file:/var/spring-xd/xd/lib/log4j-1.2.17.jar, file:/var/spring-xd/xd/lib/mysql-connector-java-5.1.23.jar, file:/var/spring-xd/xd/lib/netty-3.7.0.Final.jar, file:/var/spring-xd/xd/lib/objenesis-1.3.jar, file:/var/spring-xd/xd/lib/opencsv-2.3.jar, file:/var/spring-xd/xd/lib/paranamer-2.3.jar, file:/var/spring-xd/xd/lib/parquet-avro-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-column-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-common-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-encoding-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-format-1.0.0.jar, file:/var/spring-xd/xd/lib/parquet-generator-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-hadoop-1.2.5.jar, file:/var/spring-xd/xd/lib/postgresql-9.2-1002-jdbc4.jar, file:/var/spring-xd/xd/lib/slf4j-api-1.7.6.jar, file:/var/spring-xd/xd/lib/slf4j-log4j12-1.7.6.jar, file:/var/spring-xd/xd/lib/snakeyaml-1.12.jar, file:/var/spring-xd/xd/lib/snappy-java-1.0.5.jar, file:/var/spring-xd/xd/lib/spring-amqp-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-aop-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-manager-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-resources-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-core-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-infrastructure-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-integration-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-beans-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-actuator-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-autoconfigure-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-support-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-core-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-commons-1.6.2.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-redis-1.2.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-expression-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-hateoas-0.9.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-integration-amqp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-core-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-event-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-file-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-ftp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-groovy-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-http-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-jmx-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-redis-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-scripting-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-jdbc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-messaging-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-plugin-core-0.8.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-rabbit-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-retry-1.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-service-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/spring-tx-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-web-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-webmvc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-xd-analytics-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-batch-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-dirt-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-hadoop-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-spi-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-rest-domain-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-tuple-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/tomcat-el-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-core-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-logging-juli-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-jdbc-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-jsp-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-juli-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-servlet-api-7.0.35.jar, file:/var/spring-xd/xd/lib/uuid-3.2.jar, file:/var/spring-xd/xd/lib/validation-api-1.0.0.GA.jar, file:/var/spring-xd/xd/lib/xmlenc-0.52.jar, file:/var/spring-xd/xd/lib/xmlpull-1.1.3.1.jar, file:/var/spring-xd/xd/lib/xpp3_min-1.1.4c.jar, file:/var/spring-xd/xd/lib/xstream-1.4.4.jar, file:/var/spring-xd/xd/lib/xz-1.0.jar, file:/var/spring-xd/xd/lib/zookeeper-3.4.6.jar, file:/var/spring-xd/xd/lib/hdp20/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-annotations-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-auth-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-distcp-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-hdfs-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-core-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-jobclient-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-shuffle-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-streaming-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-api-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-client-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-nodemanager-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/jetty-util-6.1.26.jar, file:/var/spring-xd/xd/lib/hdp20/protobuf-java-2.5.0.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-batch-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-core-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-store-2.0.0.RC2-hdp20.jar]
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.AutoConfigurationReportLoggingInitializer:101 - 

Error starting ApplicationContext. To display the auto-configuration report enabled debug logging (start with --debug)

18:00:07,946 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache:550 - 
java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storeWriter': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: failed to initialize; nested exception is java.net.ConnectException: Call From CentOS/10.0.0.4 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.springframework.xd.dirt.server.ContainerRegistrar.deployStreamModule(ContainerRegistrar.java:448)
    at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:347)

我仍然能够成功地执行以下命令

hadoop fs ls /

从xd shell中成功获取结果。
我知道springxdapi,但现在想留在xdshell中。任何帮助都将不胜感激。

c9qzyr3d

c9qzyr3d1#

它无法连接到localhost:8020,即名称节点服务。很可能您的hadoop安装没有正确启动。

pdkcd3nj

pdkcd3nj2#

使用springxd单节点进行测试-它更简单,而且没有缩放器
请确保在hdfs中设置了访问权限,我以前见过这个问题。
如果hadoop在你桌面上的虚拟机上运行,而springxd在你的主机上,有时会出现ip/dns问题来阻止它。

pb3s4cty

pb3s4cty3#

检查防火墙规则并运行一些标准hadoop命令。
xd正在运行zookeeper,可能会被阻止

相关问题