java—如何在控制台上抑制hadoop生成的日志消息

v1uwarro  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(628)

我使用的是hadoop2.5.2。我是hadoop和map reduce的新手。我正在尝试一个map reduce代码来执行情感分析。但是当我执行代码时，控制台上的hadoop生成了大量日志消息。这就是为什么要花费大量时间来给出每个句子的分析结果的原因。下面是日志消息。。。。

2016-05-12 23:03:05,396 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,397 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,400 WARN  mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-05-12 23:03:05,403 WARN  mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2016-05-12 23:03:05,412 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 3
2016-05-12 23:03:05,421 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:3
2016-05-12 23:03:05,430 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1427897879_0120
2016-05-12 23:03:05,444 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-05-12 23:03:05,445 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
2016-05-12 23:03:05,445 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-05-12 23:03:05,482 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-05-12 23:03:05,482 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
2016-05-12 23:03:05,483 WARN  conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-05-12 23:03:05,483 INFO  mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2016-05-12 23:03:05,483 INFO  mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1427897879_0120
2016-05-12 23:03:05,483 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2016-05-12 23:03:05,484 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-05-12 23:03:05,485 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2016-05-12 23:03:05,485 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,486 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,486 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,487 INFO  mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,487 INFO  mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,513 INFO  mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,513 INFO  mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,514 INFO  mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,514 INFO  mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,514 INFO  mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,516 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2016-05-12 23:03:05,516 INFO  mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2016-05-12 23:03:05,516 INFO  mapred.MapTask (MapTask.java:flush(1455)) - Spilling map output
2016-05-12 23:03:05,516 INFO  mapred.MapTask (MapTask.java:flush(1456)) - bufstart = 0; bufend = 17961; bufvoid = 104857600
2016-05-12 23:03:05,516 INFO  mapred.MapTask (MapTask.java:flush(1458)) - kvstart = 26214396(104857584); kvend = 26211024(104844096); length = 3373/6553600
2016-05-12 23:03:05,523 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1641)) - Finished spill 0
2016-05-12 23:03:05,524 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1427897879_0120_m_000000_0 is done. And is in the process of committing
2016-05-12 23:03:05,525 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,525 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1427897879_0120_m_000000_0' done.
2016-05-12 23:03:05,525 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,525 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000001_0
2016-05-12 23:03:05,525 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,526 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt~:0+17939
2016-05-12 23:03:05,526 INFO  mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,527 INFO  mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,550 INFO  mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,550 INFO  mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,550 INFO  mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,550 INFO  mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,550 INFO  mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,552 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -

我已经提到了在stackoverflow上已经问过的类似问题[如何在控制台上抑制hadoop日志消息，但在讨论其他明确的答案之前，它并没有帮助我。我试过使用一些论坛中建议的配置，但不起作用

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"

我还尝试通过设置以下值来编辑log4j.properties文件。。

hadoop.root.logger=WARN,DRFA 
hadoop.log.dir=. 
hadoop.log.file=hadoop.log

但我仍然无法摆脱hadoop mapreduce在运行时执行期间生成的日志消息。这也会延迟我在控制台上的输出。他们有什么解决方法或java代码，我可以嵌入到我的代码中并抑制消息。有什么建议，帮助…有人知道吗？？
谢谢！！

Javahadoopmapreducesentiment-analysis

1条答案

按热度按时间
uhry853o

uhry853o1#

我已经找到了解决方法。它所需要的只是更改mapreduce的配置文件。
1.mapreduce.map.log.level可以将值设置为off、fatal、error、warn、info、debug、trace和all。如果设置了“mapreduce.job.log4j properties file”，则可以覆盖该设置。
mapreduce.reduce.log.level还可以将值设置为off、fatal、error、warn、info、debug、trace和all。如果设置了“mapreduce.job.log4j properties file”，则可以覆盖该设置。因此最好确保未设置“mapreduce.job.log4j properties file”。
我们必须在mapred-site.xml中设置以下属性。

<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>

<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>

​现在我在控制台上看不到任何日志消息，但它也有缺点，因为在执行mapreduce代码时，没有日志消息可见，因此我们无法找出是否发生了任何错误。

赞(0）分享  回复(0）举报  2021-05-30
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com