如何执行mongohadoop连接器的传感器日志示例?

ppcbkaq5  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(235)

我想把mongodb和hadoop结合起来。我找到的是mongo hadoop连接器。但是我找不到关于这个例子的完整文档。
文件中有四个文件 mongo-hadoop/examples/sensors , build , run_job.sh , src , testdata_generator.js 分别是。我使用 testdata_generator.js ,dbs是 demo . 当我想跑的时候 run_job.sh ,有一个例外:

  1. MongoDB shell version: 2.6.1
  2. connecting to: demo
  3. false
  4. Exception in thread "main" java.lang.ClassNotFoundException: -D
  5. at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
  6. at java.security.AccessController.doPrivileged(Native Method)
  7. at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  8. at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  9. at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  10. at java.lang.Class.forName0(Native Method)
  11. at java.lang.Class.forName(Class.java:249)
  12. at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

运行\u job.sh

  1. # !/bin/sh
  2. mongo demo --eval "db.logs_aggregate.drop()"
  3. # Set your HADOOP_HOME directory here.
  4. # export HADOOP_HOME="/Users/mike/hadoop/hadoop-2.0.0-cdh4.3.0"
  5. export HADOOP_HOME="/home/hduser/hadoop"
  6. # FIRST PASS - map all the devices into an output collection
  7. declare -a job1_args
  8. job1_args=("jar" "`pwd`/build/libs/sensors-1.2.1-SNAPSHOT-hadoop_2.2.jar")
  9. # job1_args=(${job1_args[@]} "com.mongodb.hadoop.examples.sensors.Devices")
  10. job1_args=(${job1_args[@]} "-D" "mongo.job.input.format=com.mongodb.hadoop.MongoInputFormat")
  11. job1_args=(${job1_args[@]} "-D" "mongo.input.uri=mongodb://localhost:27017/demo.devices")
  12. job1_args=(${job1_args[@]} "-D" "mongo.job.mapper=com.mongodb.hadoop.examples.sensors.DeviceMapper")
  13. job1_args=(${job1_args[@]} "-D" "mongo.job.reducer=com.mongodb.hadoop.examples.sensors.DeviceReducer")
  14. job1_args=(${job1_args[@]} "-D" "mongo.job.output.key=org.apache.hadoop.io.Text")
  15. job1_args=(${job1_args[@]} "-D" "mongo.job.output.value=org.apache.hadoop.io.Text")
  16. job1_args=(${job1_args[@]} "-D" "mongo.output.uri=mongodb://localhost:27017/demo.logs_aggregate")
  17. job1_args=(${job1_args[@]} "-D" "mongo.job.output.format=com.mongodb.hadoop.MongoOutputFormat")
  18. $HADOOP_HOME/bin/hadoop "${job1_args[@]}" "$1"

我可以在我的计算机上运行基本的map/reduce示例,但是这个问题困扰了我很多天。。。
新编辑内容:
我可以通过以下步骤运行此示例:
编译 Devices.java , DeviceMapper.java , DeviceReducer.java ,和 SensorDataGenerator.java 到。类;命令是 javac -classpath [library files] -d [folders] Devices.java DeviceMapper.java DeviceReducer.java SensorDataGenerator.java 将.class文件编译成.jar;命令是 jar -cvf [jar file name] -C [path] 执行hadoop;命令是hadoop jar [jar file name] [class name] 但我不知道为什么我不能执行 run_job.sh 成功。
devices.java是本示例中的主java文件:

  1. public class Devices extends MongoTool {
  2. public Devices() throws UnknownHostException {
  3. Configuration conf = new Configuration();
  4. MongoConfig config = new MongoConfig(conf);
  5. setConf(conf);
  6. config.setInputFormat(MongoInputFormat.class);
  7. config.setInputURI("mongodb://localhost:27017/demo.devices");
  8. config.setOutputFormat(MongoOutputFormat.class);
  9. config.setOutputURI("mongodb://localhost:27017/demo.logs_aggregate");
  10. config.setMapper(DeviceMapper.class);
  11. config.setReducer(DeviceReducer.class);
  12. config.setMapperOutputKey(Text.class);
  13. config.setMapperOutputValue(Text.class);
  14. config.setOutputKey(IntWritable.class);
  15. config.setOutputValue(BSONWritable.class);
  16. new SensorDataGenerator().run();
  17. }
  18. public static void main(final String[] pArgs) throws Exception {
  19. System.exit(ToolRunner.run(new Devices(), pArgs));
  20. }
  21. }
krugob8w

krugob8w1#

使用gradle运行它。这些bash脚本有点过时,应该删除:
/梯度传感器数据

相关问题