无hdfs的hadoop代码的java执行

izj3ouym  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(227)

我在没有命令的情况下运行访问hdfs的代码时遇到问题:

  1. hadoop jar

下面是我尝试运行的代码:

  1. package com.infotel.mycompany.testhdfs;
  2. import java.io.BufferedReader;
  3. import java.io.IOException;
  4. import java.io.InputStreamReader;
  5. import org.apache.hadoop.conf.Configuration;
  6. import org.apache.hadoop.fs.FileSystem;
  7. import org.apache.hadoop.fs.Path;
  8. /**
  9. * Hello world!
  10. *
  11. */
  12. public class App
  13. {
  14. public static void main( String[] args ) throws IOException
  15. {
  16. Configuration config = new Configuration();
  17. config.addResource("/opt/hadoop-2.2.0/etc/hadoop/core-site.xml");
  18. config.set("fs.defaultFS", "hdfs://192.168.2.164/");
  19. FileSystem dfs = FileSystem.get(config);
  20. Path pt = new Path("/path/to/myfile");
  21. BufferedReader br = new BufferedReader(new InputStreamReader(dfs.open(pt)));
  22. String line;
  23. line = br.readLine();
  24. while(line != null) {
  25. System.out.println(line);
  26. line = br.readLine();
  27. }
  28. }
  29. }

我使用maven和以下pom.xml构建代码:

  1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  2. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  3. <modelVersion>4.0.0</modelVersion>
  4. <groupId>com.mycompany.bigdata</groupId>
  5. <artifactId>testhdfs</artifactId>
  6. <version>0.0.1-SNAPSHOT</version>
  7. <packaging>jar</packaging>
  8. <name>testhdfs</name>
  9. <url>http://maven.apache.org</url>
  10. <properties>
  11. <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  12. </properties>
  13. <dependencies>
  14. <dependency>
  15. <groupId>junit</groupId>
  16. <artifactId>junit</artifactId>
  17. <version>3.8.1</version>
  18. <scope>test</scope>
  19. </dependency>
  20. <dependency>
  21. <groupId>org.apache.hadoop</groupId>
  22. <artifactId>hadoop-core</artifactId>
  23. <version>0.20.2</version>
  24. </dependency>
  25. </dependencies>
  26. <build>
  27. <plugins>
  28. <plugin>
  29. <groupId>org.codehaus.mojo</groupId>
  30. <artifactId>exec-maven-plugin</artifactId>
  31. <configuration>
  32. <mainClass>com.infotel.bigdata.testhdfs.App</mainClass>
  33. </configuration>
  34. </plugin>
  35. </plugins>
  36. </build>
  37. </project>

使用此命令运行代码时:

  1. hadoop jar target/testhdfs-0.0.1-SNAPSHOT.jar com.infotel.mycompany.testhdfs.App

很好用。但是如果我用这个命令或从eclipse运行代码:

  1. mvn exec:java

我得到以下错误:

  1. java.lang.reflect.InvocationTargetException
  2. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  3. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  4. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  5. at java.lang.reflect.Method.invoke(Method.java:483)
  6. at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
  7. at java.lang.Thread.run(Thread.java:744)
  8. Caused by: java.io.FileNotFoundException: File /path/to/myfile does not exist.
  9. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
  10. at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
  11. at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
  12. at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
  13. at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
  14. at com.infotel.bigdata.testhdfs.App.main(App.java:30)

这是我的core-site.xml:

  1. <configuration>    
  2. <property>
  3. <name>fs.defaultFS</name>   
  4. <value>hdfs://chef</value>
  5. <description>NameNode URI </description>
  6. </property> 
  7. <property>       
  8. <name>hadoop.http.staticuser.user</name>  
  9. <value>hdfs</value>  
  10. </property>
  11. <property>
  12. <name>hadoop.tmp.dir</name>
  13. <value>/opt/hadoop-2.2.0/tmp</value>
  14. </property>
  15. </configuration>

我在尝试运行mapreduce时遇到了同样的问题,我必须使用hadoopjar命令,而不是从eclipse运行它。我正在使用hadoop2.2.0。看起来我完全错过了什么或不明白什么,谷歌没有帮助我。
如果有人有解决办法,我会非常感激。最终的目标是从servlet中的hdfs中检索文件,这就是为什么我不能使用 hadoop jar

mjqavswn

mjqavswn1#

我找到了解决办法。首先,我尝试使用hadoop安装中的所有库运行代码,命令如下:

  1. java -cp `hadoop classpath`:target/testhdfs-0.0.1-SNAPSHOT.jar com.infotel.bigdata.testhdfs.App

所以我发现这是jar版本的问题。通过盲目搜索,我将pom.xml改为:

  1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  2. xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  3. <modelVersion>4.0.0</modelVersion>
  4. <groupId>com.mycompany.bigdata</groupId>
  5. <artifactId>testhdfs</artifactId>
  6. <version>0.0.1-SNAPSHOT</version>
  7. <packaging>jar</packaging>
  8. <name>testhdfs</name>
  9. <url>http://maven.apache.org</url>
  10. <properties>
  11. <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  12. </properties>
  13. <dependencies>
  14. <dependency>
  15. <groupId>junit</groupId>
  16. <artifactId>junit</artifactId>
  17. <version>3.8.1</version>
  18. <scope>test</scope>
  19. </dependency>
  20. <dependency>
  21. <groupId>org.apache.hadoop</groupId>
  22. <artifactId>hadoop-common</artifactId>
  23. <version>2.2.0</version>
  24. </dependency>
  25. <dependency>
  26. <groupId>org.apache.hadoop</groupId>
  27. <artifactId>hadoop-hdfs</artifactId>
  28. <version>2.2.0</version>
  29. </dependency>
  30. </dependencies>
  31. <build>
  32. <plugins>
  33. <plugin>
  34. <groupId>org.codehaus.mojo</groupId>
  35. <artifactId>exec-maven-plugin</artifactId>
  36. <configuration>
  37. <mainClass>com.infotel.bigdata.testhdfs.App</mainClass>
  38. </configuration>
  39. </plugin>
  40. </plugins>
  41. </build>
  42. </project>

现在呢 mvn exec:java 好好工作。也许hadoop核心应该只用于旧的hadoop版本?有人知道更多关于hadoop核心和hadoop公共的知识吗?

展开查看全部
mftmpeh8

mftmpeh82#

在这个语句中,您可以添加name节点启动的端口。

  1. config.set("fs.defaultFS", "hdfs://192.168.2.164/");

apachehadoop中name node的默认端口是8020。

  1. config.set("fs.defaultFS", "hdfs://192.168.2.164:8020");

相关问题