nutch2.0依赖项设置

qc6wkl3g  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(220)

我正在部署模式下使用nutch2.0爬网URL:
我用蚂蚁做坚果
我在hadoop中使用了nutch-2.0.job。
我运行了以下命令: sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.InjectorJob /user/bluesky/nutch/urls sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.GeneratorJob sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.fetcher.FetcherJob (batch -id) 但当我运行最后一个命令时,它在日志中给了我错误:

2012-10-09 15:35:04,292 ERROR org.apache.nutch.fetcher.FetcherJob: Unexpected error for http://www.imdb.com/title/tt0327056/
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:668)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:540)
2012-10-09 15:35:04,293 ERROR org.apache.nutch.fetcher.FetcherJob: fetcher throwable caught
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题