如何使用mapreduce生成数据？

ao218c7q  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(374)

我想使用mapreduce和hadoop1.2.1生成数据。除了java和hadoop，我什么都不要我的意思是我不会使用hadoop生态系统等）
数据的格式应该是(csv格式）数字，x轴，y轴1,1,2\n 2,3,2\n 3,12,1\n 4531,2\n 5,234324\n。。。
数据大小至少大于1gb。
数据是在没有输入的情况下并行生成的。
我试过运行[mapreduce design patterns 182p{generating data}]的源代码。但在windows eclipse控制台和linux中都出现了类似的故障。
（参考=>http://www.amazon.com/mapreduce-design-patterns-effective-algorithms/dp/1449327176 )

//////////////////////////////copy from my window+eclipse-console//////////////////////////
4월 22, 2014 9:35:58 오전 org.apache.hadoop.util.NativeCodeLoader <clinit>
경고: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Cant help it, hence ignoring IOException setting persmission for path "file:/D:/debug/hadoop/tmp/mapred/staging/BISTel_RWoo1622699850/.staging": Failed to set permissions of path: D:\debug\hadoop\tmp\mapred\staging\BISTel_RWoo1622699850\.staging to 0700
Cant help it, hence ignoring IOException setting persmission for path "file:/D:/debug/hadoop/tmp/mapred/staging/BISTel_RWoo1622699850/.staging/job_local1622699850_0001": Failed to set permissions of path: D:\debug\hadoop\tmp\mapred\staging\BISTel_RWoo1622699850\.staging\job_local1622699850_0001 to 0700
4월 22, 2014 9:35:58 오전 org.apache.hadoop.mapred.JobClient$2 run
정보: Cleaning up the staging area file:/D:/debug/hadoop/tmp/mapred/staging/BISTel_RWoo1622699850/.staging/job_local1622699850_0001
Exception in thread "main" java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, ÁödµÈ ÆÄÀÏ; ã; ¼ö ¾ø
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:712)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:448)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:423)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(TrackerDistributedCacheManager.java:364)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.isPublic(TrackerDistributedCacheManager.java:328)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineCacheVisibilities(TrackerDistributedCacheManager.java:832)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestampsAndCacheVisibilities(TrackerDistributedCacheManager.java:756)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:843)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:734)
    at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:179)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at mrdp.ch7.RandomDataGenerationDriver.main(RandomDataGenerationDriver.java:237)
Caused by: java.io.IOException: CreateProcess error=2, ÁödµÈ ÆÄÀÏ; ã; ¼ö ¾ø
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
    at java.lang.ProcessImpl.start(ProcessImpl.java:136)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
    ... 24 more

    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:473)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:423)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(TrackerDistributedCacheManager.java:364)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.isPublic(TrackerDistributedCacheManager.java:328)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineCacheVisibilities(TrackerDistributedCacheManager.java:832)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestampsAndCacheVisibilities(TrackerDistributedCacheManager.java:756)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:843)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:734)
    at org.apache.hadoop.mapred.JobClient.access$400(JobClient.java:179)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
    at mrdp.ch7.RandomDataGenerationDriver.main(RandomDataGenerationDriver.java:237)
///////////////////////////////////////////////////////////////////////////////////////////
//////////copy from my local-linux-webpages: myserver:50030/jobtracker.jsp//////////
Hadoop map task list for job_201404151557_0017 on fox01

All Tasks

Task    Complete    Status  Start Time  Finish Time Errors  Counters
task_201404151557_0017_m_000000 0.00%

21-Apr-2014 16:17:00
21-Apr-2014 16:17:21 (20sec)
java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

0
task_201404151557_0017_m_000001 0.00%

21-Apr-2014 16:17:00
21-Apr-2014 16:17:21 (20sec)
java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

java.io.FileNotFoundException: /user/rwoo/test/wordList.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at java.io.FileReader.<init>(FileReader.java:41)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat$RandomStackoverflowRecordReader.initialize(RandomDataGenerationDriver.java:116)
    at mrdp.ch7.RandomDataGenerationDriver$RandomStackOverflowInputFormat.createRecordReader(RandomDataGenerationDriver.java:63)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:488)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

0
Javahadoopmapreduce

2条答案

按热度按时间
mgdq6dx1

mgdq6dx11#

使用cygwin可以在windows中安装hadoop。网上有很多关于它的文件。下面是一个很好的教程，可以在windows中为hadoop安装cygwin。

赞(0）分享  回复(0）举报  2021-06-03
ecr0jaav

ecr0jaav2#

我想你没有安装cygwin。或者cygwin/bin不在路径中？
对于windows+eclipse或windows上的hadoop，您需要cygwin。

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com