pig mapreduce作业失败

kq0g1dla  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(376)

嗨，我正在我的linusystem上运行一个简单的加载和转储命令。

grunt> a= load 'a.txt';

grunt> dump a;

我有以下错误

2016-04-07 08:27:07,029 [main] INFO 
 org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
 script: UNKNOWN 2016-04-07 08:27:07,171 [main] INFO 
 org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
 {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,
 GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter,
 MergeFilter, MergeForEach, PartitionFilterOptimizer,
 PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter,
 SplitFilter, StreamTypeCastInserter]} 2016-04-07 08:27:07,270 [main]
 INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
 - File concatenation threshold: 100 optimistic? false 2016-04-07 08:27:07,292 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
 - MR plan size before optimization: 1 2016-04-07 08:27:07,293 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
 - MR plan size after optimization: 1 2016-04-07 08:27:07,329 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM
 Metrics with processName=JobTracker, sessionId= 2016-04-07
 08:27:07,356 [main] INFO 
 org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script
 settings are added to the job 2016-04-07 08:27:07,365 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
 - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2016-04-07 08:27:07,406 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
 - Setting up single store job 2016-04-07 08:27:07,438 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 1 map-reduce job(s) waiting for submission. 2016-04-07 08:27:07,448 [JobControl] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot
 initialize JVM Metrics with processName=JobTracker, sessionId= -
 already initialized 2016-04-07 08:27:07,648 [JobControl] INFO 
 org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging
 area
 file:/tmp/hadoop-uc4/mapred/staging/uc41840016457/.staging/job_local1840016457_0001
 2016-04-07 08:27:07,648 [JobControl] INFO 
 org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob -
 PigLatin:DefaultJobName got an error while submitting ENOENT: No such
 file or directory
         at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
         at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
         at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:724)
         at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
         at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:600)
         at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:95)
         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:190)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:422)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
         at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:497)
         at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
         at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194)
         at java.lang.Thread.run(Thread.java:745)
         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276)
 2016-04-07 08:27:07,852 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - HadoopJobId: job_local1840016457_0001 2016-04-07 08:27:07,852 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Processing aliases a 2016-04-07 08:27:07,852 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - detailed locations: M: a[1,3] C:  R: 2016-04-07 08:27:07,856 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 0% complete 2016-04-07 08:27:07,863 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. 2016-04-07 08:27:07,863 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - job job_local1840016457_0001 has failed! Stop running all dependent jobs 2016-04-07 08:27:07,863 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - 100% complete 2016-04-07 08:27:07,867 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
 Metrics with processName=JobTracker, sessionId= - already initialized
 2016-04-07 08:27:07,868 [main] ERROR
 org.apache.pig.tools.pigstats.PigStats - ERROR 0:
 java.lang.IllegalStateException: Job in state DEFINE instead of
 RUNNING 2016-04-07 08:27:07,868 [main] ERROR
 org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce
 job(s) failed! 2016-04-07 08:27:07,871 [main] INFO 
 org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script
 Statistics:

 HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt    
 Features
 2.7.1.2.3.0.0-2557      0.15.0.2.3.0.0-2557     uc4     2016-04-07 08:27:07     2016-04-07 08:27:07     UNKNOWN

Failed!

 Failed Jobs: JobId   Alias   Feature Message Outputs
 job_local1840016457_0001        a       MAP_ONLY        Message:
 ENOENT: No such file or directory
         at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
         at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
         at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:724)
         at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
         at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:600)
         at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:95)
         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:190)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:422)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
         at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:497)
         at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
         at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194)
         at java.lang.Thread.run(Thread.java:745)
         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276)
         file:/tmp/temp44212715/tmp-142377885,

nput(s): Failed to read data from "file:///home/uc4/a.txt"
utput(s): 
 Failed to produce result in
 "file:/tmp/temp44212715/tmp-142377885"

 Counters: Total records written : 0 Total bytes written : 0 Spillable
 Memory Manager spill count : 0 Total bags proactively spilled: 0 Total
 records proactively spilled: 0

 Job DAG: job_local1840016457_0001

 2016-04-07 08:27:07,871 [main] INFO 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 - Failed! 2016-04-07 08:27:07,880 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator
 for alias a. Backend error : java.lang.IllegalStateException: Job in
 state DEFINE instead of RUNNING Details at logfile:
 /home/uc4/pig_1460010417251.log
apache-pigIllegalStateException

1条答案

按热度按时间
nhaq1z21

nhaq1z211#

消息：eNote:没有这样的文件或目录
提供完整有效路径，如：

a= load '/data/a.txt';
dump a;

我从我的文件所在的目录运行grunt
pig的java进程不是从执行的位置开始的 pig 但从Pig的家乡发出命令。
读取日志文件以获取正确的清管器错误消息： /home/uc4/pig_1460010417251.log

赞(0）分享  回复(0）举报  2021-06-21
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com