弹性Map减少超时java.io.ioexception:意外的流结束

cwxwcias  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(339)

我正在运行map reduce job（elastic map reduce emr）服务。对于小数据集，该作业可以正常工作，但对于大数据集（文件大小400mb），会出现以下异常
使用相同的大输入文件运行另一个作业很好，但为什么呢？

  1. Error: java.io.IOException: Unexpected end of stream pos=19520590 contentLength= 458603265
  2.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  3.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  4.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  5.     at java.io.DataInputStream.read(DataInputStream.java:100)
  6.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  7.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  8.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  9.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  10.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  11.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  12.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  13.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  14.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  15.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  16.     at java.security.AccessController.doPrivileged(Native Method)
  17.     at javax.security.auth.Subject.doAs(Subject.java:415)
  18.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  19.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  21. Error: java.io.IOException: Unexpected end of stream pos=222629174 contentLength= 458603265
  22.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  23.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  24.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  25.     at java.io.DataInputStream.read(DataInputStream.java:100)
  26.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  27.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  28.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  29.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  30.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  31.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  32.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  33.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  34.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  35.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  36.     at java.security.AccessController.doPrivileged(Native Method)
  37.     at javax.security.auth.Subject.doAs(Subject.java:415)
  38.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  39.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  41. Error: java.io.IOException: Unexpected end of stream pos=155898310 contentLength= 458603265
  42.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  43.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  44.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  45.     at java.io.DataInputStream.read(DataInputStream.java:100)
  46.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  47.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  48.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  49.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  50.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  51.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  52.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  53.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  54.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  55.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  56.     at java.security.AccessController.doPrivileged(Native Method)
  57.     at javax.security.auth.Subject.doAs(Subject.java:415)
  58.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  59.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  61. Error: java.io.IOException: Unexpected end of stream pos=88645447 contentLength= 458603265
  62.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  63.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  64.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  65.     at java.io.DataInputStream.read(DataInputStream.java:100)
  66.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  67.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  68.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  69.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  70.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  71.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  72.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  73.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  74.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  75.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  76.     at java.security.AccessController.doPrivileged(Native Method)
  77.     at javax.security.auth.Subject.doAs(Subject.java:415)
  78.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  79.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  81. Error: java.io.IOException: Unexpected end of stream pos=19520590 contentLength= 458603265
  82.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  83.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  84.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  85.     at java.io.DataInputStream.read(DataInputStream.java:100)
  86.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  87.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  88.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  89.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  90.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  91.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  92.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  93.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  94.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  95.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  96.     at java.security.AccessController.doPrivileged(Native Method)
  97.     at javax.security.auth.Subject.doAs(Subject.java:415)
  98.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  99.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  101. Error: java.io.IOException: Unexpected end of stream pos=291259038 contentLength= 458603265
  102.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  103.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  104.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  105.     at java.io.DataInputStream.read(DataInputStream.java:100)
  106.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  107.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  108.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  109.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  110.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  111.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  112.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  113.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  114.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  115.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  116.     at java.security.AccessController.doPrivileged(Native Method)
  117.     at javax.security.auth.Subject.doAs(Subject.java:415)
  118.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  119.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  121. Error: java.io.IOException: Unexpected end of stream pos=222629174 contentLength= 458603265
  122.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  123.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  124.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  125.     at java.io.DataInputStream.read(DataInputStream.java:100)
  126.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  127.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  128.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  129.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  130.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  131.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  132.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  133.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  134.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  135.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  136.     at java.security.AccessController.doPrivileged(Native Method)
  137.     at javax.security.auth.Subject.doAs(Subject.java:415)
  138.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  139.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  141. Error: java.io.IOException: Unexpected end of stream pos=155898310 contentLength= 458603265
  142.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  143.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  144.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  145.     at java.io.DataInputStream.read(DataInputStream.java:100)
  146.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  147.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  148.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  149.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  150.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  151.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  152.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  153.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  154.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  155.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  156.     at java.security.AccessController.doPrivileged(Native Method)
  157.     at javax.security.auth.Subject.doAs(Subject.java:415)
  158.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  159.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  161. Error: java.io.IOException: Unexpected end of stream pos=88645447 contentLength= 458603265
  162.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  163.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  164.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  165.     at java.io.DataInputStream.read(DataInputStream.java:100)
  166.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  167.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  168.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  169.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  170.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  171.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  172.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  173.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  174.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  175.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  176.     at java.security.AccessController.doPrivileged(Native Method)
  177.     at javax.security.auth.Subject.doAs(Subject.java:415)
  178.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  179.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  181. Error: java.io.IOException: Unexpected end of stream pos=19520590 contentLength= 458603265
  182.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  183.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  184.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  185.     at java.io.DataInputStream.read(DataInputStream.java:100)
  186.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  187.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  188.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  189.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  190.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  191.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  192.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  193.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  194.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  195.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  196.     at java.security.AccessController.doPrivileged(Native Method)
  197.     at javax.security.auth.Subject.doAs(Subject.java:415)
  198.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  199.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  201. Error: java.io.IOException: Unexpected end of stream pos=291259038 contentLength= 458603265
  202.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  203.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  204.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  205.     at java.io.DataInputStream.read(DataInputStream.java:100)
  206.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  207.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  208.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  209.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  210.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  211.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  212.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  213.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  214.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  215.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  216.     at java.security.AccessController.doPrivileged(Native Method)
  217.     at javax.security.auth.Subject.doAs(Subject.java:415)
  218.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  219.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  221. Error: java.io.IOException: Unexpected end of stream pos=222629174 contentLength= 458603265
  222.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  223.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  224.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  225.     at java.io.DataInputStream.read(DataInputStream.java:100)
  226.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  227.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  228.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  229.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  230.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  231.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  232.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  233.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  234.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  235.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  236.     at java.security.AccessController.doPrivileged(Native Method)
  237.     at javax.security.auth.Subject.doAs(Subject.java:415)
  238.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  239.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  241. Error: java.io.IOException: Unexpected end of stream pos=155898310 contentLength= 458603265
  242.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  243.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  244.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  245.     at java.io.DataInputStream.read(DataInputStream.java:100)
  246.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  247.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  248.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  249.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  250.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  251.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  252.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  253.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  254.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  255.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  256.     at java.security.AccessController.doPrivileged(Native Method)
  257.     at javax.security.auth.Subject.doAs(Subject.java:415)
  258.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  259.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
  261. Error: java.io.IOException: Unexpected end of stream pos=88645447 contentLength= 458603265
  262.     at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.read(NativeS3FileSystem.java:313)
  263.     at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
  264.     at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
  265.     at java.io.DataInputStream.read(DataInputStream.java:100)
  266.     at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  267.     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  268.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)
  269.     at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
  270.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
  271.     at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
  272.     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
  273.     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
  274.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
  275.     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  276.     at java.security.AccessController.doPrivileged(Native Method)
  277.     at javax.security.auth.Subject.doAs(Subject.java:415)
  278.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  279.     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

我用类似的配置运行了另一个作业，效果很好。有什么输入吗？

Javahadoopelastic-map-reduce

1条答案

按热度按时间
huwehgph

huwehgph1#

我在spark+hive中遇到了类似的问题。问题是我发出了一个配置单元查询来写入s3中的一个位置，由一个Dataframe支持，然后立即从中读取。比如：

  1. statusDF.createOrReplaceTempView("statusDF")
  2. spark.sql(s"INSERT OVERWRITE TABLE ${table} SELECT id, last_updated FROM statusDF")
  3. statusDF.show(5)

所以有一种可能是，您正在对同一个s3位置（但尚未完成）进行其他写入操作的过程中进行读取，这就产生了这个错误。

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com