x的jar-store关系问题局部问题，读取数据失败

bkhjykvo  于 2021-07-15  发布在  Pig
关注(0)|答案(0)|浏览(552)

第一种方法：使用pig-xmapreduce
通过hbase shell创建的hbase表

  1. Hbase table is created:
  2. hbase(main):003:0> list
  3. TABLE                                                                                                                                                                                                              
  4. clientes                                                                                                                                                                                                           
  5. 1 row(s)
  6. Took 0.0047 seconds                                                                                                                                                                                                
  7. => ["clientes"]

使用此代码将clientes.txt中的数据加载到dados（pig-x mapreduce）

  1. grunt> dados = LOAD 'file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt' USING PigStorage(',') AS (
  2.             id:chararray,
  3.             nome:chararray,
  4.             sobrenome:chararray,
  5.             idade:int,
  6.             funcao:chararray
  7. );

已检查带有转储护墙板的护墙板，但失败：

  1. 2021-03-07 19:00:32,390 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1615152557282_0002
  2. 2021-03-07 19:00:32,390 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases dados
  3. 2021-03-07 19:00:32,390 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: dados[1,8],dados[-1,-1] C:  R: 
  4. 2021-03-07 19:00:32,395 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  5. 2021-03-07 19:00:37,406 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
  6. 2021-03-07 19:00:37,406 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1615152557282_0002 has failed! Stop running all dependent jobs
  7. 2021-03-07 19:00:37,406 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  8. 2021-03-07 19:00:37,410 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032
  9. 2021-03-07 19:00:37,492 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Could not get Job info from RM for job job_1615152557282_0002. Redirecting to job history server.
  10. 2021-03-07 19:00:37,595 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 0: java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
  11. 2021-03-07 19:00:37,595 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
  12. 2021-03-07 19:00:37,597 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 
  14. HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
  15. 3.2.2   0.17.0  hadoop  2021-03-07 19:00:31 2021-03-07 19:00:37 UNKNOWN
  17. Failed!
  19. Failed Jobs:
  20. JobId   Alias   Feature Message Outputs
  21. job_1615152557282_0002  dados   MAP_ONLY    Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Wrong FS: hdfs://localhost:9000/user/hadoop, expected: file:///
  22.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:294)
  23.     at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:310)
  24.     at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:327)
  25.     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
  26.     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1565)
  27.     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1562)
  28.     at java.security.AccessController.doPrivileged(Native Method)
  29.     at javax.security.auth.Subject.doAs(Subject.java:422)
  30.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
  31.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1562)
  32.     at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
  33.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  34.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  35.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  36.     at java.lang.reflect.Method.invoke(Method.java:498)
  37.     at org.apache.pig.backend.hadoop.PigJobControl.submit(PigJobControl.java:128)
  38.     at org.apache.pig.backend.hadoop.PigJobControl.run(PigJobControl.java:205)
  39.     at java.lang.Thread.run(Thread.java:748)
  40.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:301)
  41. Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/user/hadoop, expected: file:///
  42.     at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
  43.     at org.apache.hadoop.fs.RawLocalFileSystem.setWorkingDirectory(RawLocalFileSystem.java:604)
  44.     at org.apache.hadoop.fs.FilterFileSystem.setWorkingDirectory(FilterFileSystem.java:307)
  45.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:250)
  46.     ... 18 more
  47.     hdfs://localhost:9000/tmp/temp-1169299097/tmp-2103156722,
  49. Input(s):
  50. Failed to read data from "file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt"
  52. Output(s):
  53. Failed to produce result in "hdfs://localhost:9000/tmp/temp-1169299097/tmp-2103156722"
  55. Counters:
  56. Total records written : 0
  57. Total bytes written : 0
  58. Spillable Memory Manager spill count : 0
  59. Total bags proactively spilled: 0
  60. Total records proactively spilled: 0
  62. Job DAG:
  63. job_1615152557282_0002
  65. 2021-03-07 19:00:37,597 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
  66. 2021-03-07 19:00:37,601 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias dados. Backend error : java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
  67. Details at logfile: /mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/pig_1615154395936.log

第二种方法：使用pig-x本地（倾倒护墙板工程）

  1. grunt> dados = LOAD 'file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt' USING PigStorage(',') AS (
  2. >> id:chararray,
  3. >> nome:chararray,
  4. >> sobrenome:chararray,
  5. >> idade:int,
  6. >> funcao:chararray
  7. >> );
  9. 2021-03-07 19:02:17,219 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
  10. 2021-03-07 19:02:17,222 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt:0+794
  11. 2021-03-07 19:02:17,226 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - File Output Committer Algorithm version is 2
  12. 2021-03-07 19:02:17,226 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
  13. 2021-03-07 19:02:17,241 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (PS Old Gen) of size 699400192 to monitor. collectionUsageThreshold = 489580128, usageThreshold = 489580128
  14. 2021-03-07 19:02:17,243 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
  15. 2021-03-07 19:02:17,253 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: dados[1,8],dados[-1,-1] C:  R: 
  16. 2021-03-07 19:02:17,266 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
  17. 2021-03-07 19:02:17,274 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local116575577_0001_m_000000_0 is done. And is in the process of committing
  18. 2021-03-07 19:02:17,280 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
  19. 2021-03-07 19:02:17,280 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.Task - Task attempt_local116575577_0001_m_000000_0 is allowed to commit now
  20. 2021-03-07 19:02:17,285 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local116575577_0001_m_000000_0' to file:/tmp/temp2133275539/tmp1539690224
  21. 2021-03-07 19:02:17,286 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.LocalJobRunner - map
  22. 2021-03-07 19:02:17,286 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local116575577_0001_m_000000_0' done.
  23. 2021-03-07 19:02:17,291 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.Task - Final Counters for attempt_local116575577_0001_m_000000_0: Counters: 16
  24.     File System Counters
  25.         FILE: Number of bytes read=1264
  26.         FILE: Number of bytes written=530456
  27.         FILE: Number of read operations=0
  28.         FILE: Number of large read operations=0
  29.         FILE: Number of write operations=0
  30.     Map-Reduce Framework
  31.         Map input records=20
  32.         Map output records=20
  33.         Input split bytes=414
  34.         Spilled Records=0
  35.         Failed Shuffles=0
  36.         Merged Map outputs=0
  37.         GC time elapsed (ms)=0
  38.         Total committed heap usage (bytes)=311427072
  39.     File Input Format Counters 
  40.         Bytes Read=0
  41.     File Output Format Counters 
  42.         Bytes Written=0
  43.     org.apache.pig.PigWarning
  44.         FIELD_DISCARDED_TYPE_CONVERSION_FAILED=1
  45. 2021-03-07 19:02:17,291 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.mapred.LocalJobRunner - Finishing task: attempt_local116575577_0001_m_000000_0
  46. 2021-03-07 19:02:17,291 [Thread-7] INFO  org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
  47. 2021-03-07 19:02:17,485 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  48. 2021-03-07 19:02:17,492 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  49. 2021-03-07 19:02:17,492 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  50. 2021-03-07 19:02:17,492 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
  51. 2021-03-07 19:02:17,493 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  52. 2021-03-07 19:02:17,536 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  53. 2021-03-07 19:02:17,540 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 
  55. HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
  56. 3.2.2   0.17.0  hadoop  2021-03-07 19:02:16 2021-03-07 19:02:17 UNKNOWN
  58. Success!
  60. Job Stats (time in seconds):
  61. JobId   Maps    Reduces MaxMapTime  MinMapTime  AvgMapTime  MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime    Alias   Feature Outputs
  62. job_local116575577_0001 1   0   n/a n/a n/a n/a 0   0   0   0   dados   MAP_ONLY    file:/tmp/temp2133275539/tmp1539690224,
  64. Input(s):
  65. Successfully read 20 records from: "file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt"
  67. Output(s):
  68. Successfully stored 20 records in: "file:/tmp/temp2133275539/tmp1539690224"
  70. Counters:
  71. Total records written : 20
  72. Total bytes written : 0
  73. Spillable Memory Manager spill count : 0
  74. Total bags proactively spilled: 0
  75. Total records proactively spilled: 0
  77. Job DAG:
  78. job_local116575577_0001
  80. 2021-03-07 19:02:17,542 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  81. 2021-03-07 19:02:17,544 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  82. 2021-03-07 19:02:17,551 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  83. 2021-03-07 19:02:17,558 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning FIELD_DISCARDED_TYPE_CONVERSION_FAILED 1 time(s).
  84. 2021-03-07 19:02:17,558 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
  85. 2021-03-07 19:02:17,563 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
  86. 2021-03-07 19:02:17,563 [main] WARN  org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
  87. 2021-03-07 19:02:17,570 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input files to process : 1
  88. 2021-03-07 19:02:17,570 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  89. (id,nome,sobrenome,,funcao)
  90. (c001,Josias,Silva,55,Analista de Mercado)
  91. (1100002,Pedro,Malan,74,Professor)
  92. (1100003,Maria,Maciel,34,Bombeiro)
  93. (1100004,Suzana,Bustamante,66,Analista de TI)
  94. (1100005,Karen,Moreira,74,Advogado)
  95. (1100006,Patricio,Teixeira,42,Veterinario)
  96. (1100007,Elisa,Haniero,43,Piloto)
  97. (1100008,Mauro,Bender,63,Marceneiro)
  98. (1100009,Mauricio,Wagner,39,Artista)
  99. (1100010,Douglas,Macedo,60,Escritor)
  100. (1100011,Francisco,McNamara,47,Cientista de Dados)
  101. (1100012,Sidney,Raynor,26,Escritor)
  102. (1100013,Maria,Moon,41,Gerente de Projetos)
  103. (1100014,Bete,Balanaira,65,Musico)
  104. (1100015,Julia,Peixoto,49,Especialista em TI)
  105. (1100016,Jeronimo,Wallace,52,Engenheiro de Dados)
  106. (1100017,Noeli,Laura,72,Cientista de Dados)
  107. (1100018,Jean,Junior,45,Desenvolvedor RPA)
  108. (1100019,Cristina,Garbim,63,Engenheiro Blockchain)

但把护墙板放进hbase://clientes'或将dado存储到'file:///home/hadoop/hadloop/pig\u output'失败：

  1. grunt> STORE dados INTO 'hbase://clientes' USING
  2. >> org.apache.pig.backend.hadoop.hbase.HBaseStorage(
  3. >> 'dados_clientes:nome
  4. >> dados_clientes:sobrenome
  5. >> dados_clientes:idade
  6. >> dados_clientes:funcao'
  7. >> );
  8. 2021-03-07 19:03:51,347 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local1289080477_0002
  9. 2021-03-07 19:03:51,347 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases dados
  10. 2021-03-07 19:03:51,347 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: dados[1,8],dados[-1,-1] C:  R: 
  11. 2021-03-07 19:03:51,349 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  12. 2021-03-07 19:03:51,349 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local1289080477_0002]
  13. 2021-03-07 19:03:51,835 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.hbase.mapreduce.TableOutputFormat - Created table instance for clientes
  14. 2021-03-07 19:03:51,839 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (PS Old Gen) of size 699400192 to monitor. collectionUsageThreshold = 489580128, usageThreshold = 489580128
  15. 2021-03-07 19:03:51,839 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
  16. 2021-03-07 19:03:51,843 [LocalJobRunner Map Task Executor #0] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: dados[1,8],dados[-1,-1] C:  R: 
  17. 2021-03-07 19:03:51,860 [LocalJobRunner Map Task Executor #0] INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x1780e985b4d000f
  18. 2021-03-07 19:03:51,866 [LocalJobRunner Map Task Executor #0] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x1780e985b4d000f closed
  19. 2021-03-07 19:03:51,866 [LocalJobRunner Map Task Executor #0-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1780e985b4d000f
  20. 2021-03-07 19:03:51,867 [Thread-10] INFO  org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
  21. 2021-03-07 19:03:51,870 [Thread-10] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local1289080477_0002
  22. java.lang.Exception: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
  23.     at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
  24.     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
  25. Caused by: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
  26.     at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
  27.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
  28.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
  29.     at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:670)
  30.     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
  31.     at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
  32.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
  33.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
  34.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
  35.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
  36.     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
  37.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
  38.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  39.     at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
  40.     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  41.     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  42.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  43.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  44.     at java.lang.Thread.run(Thread.java:748)
  45. Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
  46.     at java.util.ArrayList.rangeCheck(ArrayList.java:657)
  47.     at java.util.ArrayList.get(ArrayList.java:433)
  48.     at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:992)
  49.     at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
  50.     ... 18 more
  51. 2021-03-07 19:03:52,055 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
  52. 2021-03-07 19:03:52,055 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local1289080477_0002 has failed! Stop running all dependent jobs
  53. 2021-03-07 19:03:52,055 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  54. 2021-03-07 19:03:52,056 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  55. 2021-03-07 19:03:52,057 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  56. 2021-03-07 19:03:52,057 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
  57. 2021-03-07 19:03:52,058 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 
  59. HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
  60. 3.2.2   0.17.0  hadoop  2021-03-07 19:03:50 2021-03-07 19:03:52 UNKNOWN
  62. Failed!
  64. Failed Jobs:
  65. JobId   Alias   Feature Message Outputs
  66. job_local1289080477_0002    dados   MAP_ONLY    Message: Job failed!    hbase://clientes,
  68. Input(s):
  69. Failed to read data from "file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt"
  71. Output(s):
  72. Failed to produce result in "hbase://clientes"
  74. Counters:
  75. Total records written : 0
  76. Total bytes written : 0
  77. Spillable Memory Manager spill count : 0
  78. Total bags proactively spilled: 0
  79. Total records proactively spilled: 0
  81. Job DAG:
  82. job_local1289080477_0002
  84. 2021-03-07 19:03:52,058 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
  1. grunt> STORE dados INTO 'file:///home/hadoop/hadloop/pig_output' USING
  2. >> org.apache.pig.backend.hadoop.hbase.HBaseStorage(
  3. >> 'dados_clientes:nome
  4. >> dados_clientes:sobrenome
  5. >> dados_clientes:idade
  6. >> dados_clientes:funcao'
  7. >> );
  8. java.lang.Exception: java.lang.IllegalArgumentException: Illegal character code:47, </> at 0. User-space table qualifiers can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: ///home/hadoop/hadloop/pig_output
  9.     at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
  10.     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
  11. Caused by: java.lang.IllegalArgumentException: Illegal character code:47, </> at 0. User-space table qualifiers can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: ///home/hadoop/hadloop/pig_output
  12.     at org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:196)
  13.     at org.apache.hadoop.hbase.TableName.isLegalTableQualifierName(TableName.java:149)
  14.     at org.apache.hadoop.hbase.TableName.<init>(TableName.java:322)
  15.     at org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:358)
  16.     at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:449)
  17.     at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.<init>(TableOutputFormat.java:107)
  18.     at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.getRecordWriter(TableOutputFormat.java:153)
  19.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:83)
  20.     at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:659)
  21.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
  22.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
  23.     at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
  24.     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  25.     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  26.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  27.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  28.     at java.lang.Thread.run(Thread.java:748)
  29. 2021-03-07 19:05:10,476 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local1458581109_0003
  30. 2021-03-07 19:05:10,476 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases dados
  31. 2021-03-07 19:05:10,476 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: dados[1,8],dados[-1,-1] C:  R: 
  32. 2021-03-07 19:05:10,477 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  33. 2021-03-07 19:05:10,477 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
  34. 2021-03-07 19:05:10,477 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local1458581109_0003 has failed! Stop running all dependent jobs
  35. 2021-03-07 19:05:10,478 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  36. 2021-03-07 19:05:10,478 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  37. 2021-03-07 19:05:10,479 [main] WARN  org.apache.hadoop.metrics2.impl.MetricsSystemImpl - JobTracker metrics system already initialized!
  38. 2021-03-07 19:05:10,480 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
  39. 2021-03-07 19:05:10,480 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 
  41. HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
  42. 3.2.2   0.17.0  hadoop  2021-03-07 19:05:10 2021-03-07 19:05:10 UNKNOWN
  44. Failed!
  46. Failed Jobs:
  47. JobId   Alias   Feature Message Outputs
  48. job_local1458581109_0003    dados   MAP_ONLY    Message: Job failed!    file:///home/hadoop/hadloop/pig_output,
  50. Input(s):
  51. Failed to read data from "file:///mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/clientes.txt"
  53. Output(s):
  54. Failed to produce result in "file:///home/hadoop/hadloop/pig_output"
  56. Counters:
  57. Total records written : 0
  58. Total bytes written : 0
  59. Spillable Memory Manager spill count : 0
  60. Total bags proactively spilled: 0
  61. Total records proactively spilled: 0
  63. Job DAG:
  64. job_local1458581109_0003
  66. 2021-03-07 19:05:10,480 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!

正在运行的服务：

  1. (base) [hadoop@dataserver 1-HBase]$ jps
  2. 4160 SecondaryNameNode
  3. 11666 Main
  4. 5413 HQuorumPeer
  5. 5766 HRegionServer
  6. 6966 JobHistoryServer
  7. 4631 NodeManager
  8. 4457 ResourceManager
  9. 5578 HMaster
  10. 3835 DataNode
  11. 12382 Jps
  12. 3615 NameNode

hadoop版本：

  1. SUBCOMMAND may print help when invoked w/o parameters or with -h.
  2. (base) [hadoop@dataserver 1-HBase]$ hadoop version
  3. Hadoop 3.2.2
  4. Source code repository Unknown -r 7a3bc90b05f257c8ace2f76d74264906f0f7a932
  5. Compiled by hexiaoqiao on 2021-01-03T09:26Z
  6. Compiled with protoc 2.5.0
  7. From source with checksum 5a8f564f46624254b27f6a33126ff4
  8. This command was run using /opt/hadoop/share/hadoop/common/hadoop-common-3.2.2.jar

hbase版本：

  1. (base) [hadoop@dataserver 1-HBase]$ hbase version
  2. /opt/hadoop/libexec/hadoop-functions.sh: line 2366: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution
  3. /opt/hadoop/libexec/hadoop-functions.sh: line 2461: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution
  4. Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
  5. SLF4J: Class path contains multiple SLF4J bindings.
  6. SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  7. SLF4J: Found binding in [jar:file:/opt/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  8. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  9. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  10. HBase 2.2.0
  11. Source code repository file:///opt/hbase-rm/output/hbase-2.2.0-bin revision=Unknown
  12. Compiled by hbase-rm on Tue Jun 11 04:30:30 UTC 2019
  13. From source with checksum 63a465554927aeea3f1f0bcae63decff

清管器版本：

  1. (base) [hadoop@dataserver 1-HBase]$ pig version
  2. 2021-03-07 19:08:50,197 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
  3. 2021-03-07 19:08:50,199 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
  4. 2021-03-07 19:08:50,199 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
  5. 2021-03-07 19:08:50,263 [main] INFO  org.apache.pig.Main - Apache Pig version 0.17.0 (r1797386) compiled Jun 02 2017, 15:41:58
  6. 2021-03-07 19:08:50,263 [main] INFO  org.apache.pig.Main - Logging error messages to: /mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/pig_1615154930258.log
  7. 2021-03-07 19:08:50,536 [main] ERROR org.apache.pig.Main - ERROR 2997: Encountered IOException. File version does not exist
  8. Details at logfile: /mnt/win/GD/DS/1Formacao/3EngenhariaDeDadosHadoop/07/Arquivos/1-HBase/pig_1615154930258.log
  9. 2021-03-07 19:08:50,557 [main] INFO  org.apache.pig.Main - Pig script completed in 400 milliseconds (400 ms)
hbasemapreduceyarnJarapache-pig

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com