一般microsoft错误消息

xj3cbfub  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(274)

'获取一般microsoft错误' Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid. '使用park shell将数据写入azure存储blob时。
我正在使用spark 2.4.4（2.7 hadoop发行版）发行版，并将以下2个特定于azure的文件复制到$spark\u install\u home/jars文件夹：
hadoop-azure-2.7.3.jar
azure存储7.0.0.jar
复制azure特定文件后连接到spark shell:>bin/spark shell--jars jars/hadoop-azure-2.7.3.jar，jars/azure-storage-7.0.0.jar
创建spark上下文以及适当的hadoop配置参数：

  1. scala> sc
  2. scala> sc.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
  3. scala> sc.hadoopConfiguration.set("fs.azure.account.key.ACCOUNT_NAME.blob.core.windows.net", "ACCOUNT_KEY")*

已成功从azure blob存储读取数据：

  1. scala> val df = spark.read.format("csv").option("inferSchema", "true").load("wasbs://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net/public_datasets/orders")*
  3. scala> df.show()
  4. +-----+----+---+--------+------+---------+                                      
  5. |  _c0| _c1|_c2|     _c3|   _c4|      _c5|
  6. +-----+----+---+--------+------+---------+
  7. |31614|1499| EF|54662084|3666.0|Household|
  8. |31615|1500| GH| 4727745|4119.0|Household|
  9. |31616|1501| OP|40921163|2292.0|Household|
  10. |31617|1502| IJ|11066508|7338.0|Household|
  11. |31618|1503| OP|11958569|3171.0|Household|
  12. |31619|1504| UV|29212069| 619.0|Household|
  13. |31620|1504| WX|69836336|4887.0|         |
  14. |31621|1506| UV|28518402|9498.0|     Loan|
  15. |31622|1507| ST|  156510|6619.0|Household|
  16. |31623|1508| OP|10600459|1657.7|  Leasing|
  17. |31624|1508| UV| 9055811|8967.0|Household|
  18. |31625|1509| EF|43322950|4943.0|Household|
  19. |31626|1509| QR|20525896|2320.0|         |
  20. |31627|1510| OP|82537191|9985.0|Household|
  21. |31628|1511| OP|43543106|1657.7|  Leasing|
  22. |31629|1512| YZ|42362171|8822.0|Household|
  23. |31630|1513| AB|52238965|1632.0|Household|
  24. |31631|1513| QR|85800695|3912.0|         |
  25. |31632|1513| OP|59476637|1316.0|Insurance|
  26. |31633|1513| GH|65290647| 689.0|         |
  27. +-----+----+---+--------+------+---------+
  28. only showing top 20 rows

将数据写回其他位置：

  1. df.write.mode("overwrite").csv("wasbs://CONTAINER_NAME@ACCOUNT_NAME.blob.core.windows.net/public_datasets/orders_modified/")

正在引发以下异常：

  1. Error committing the output of task: attempt_20200608142341_0006_m_000000_8
  2.    org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
  3.        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2482)
  4.        at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:424)
  5.        at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
  6.        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:531)
  7.        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:502)
  8.        at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
  9.        at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
  10.        at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
  11.        at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
  12.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
  13.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  14.        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  15.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  16.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  17.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  18.        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  19.        at org.apache.spark.scheduler.Task.run(Task.scala:123)
  20.        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  21.        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  22.        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  23.        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  24.        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  25.        at java.lang.Thread.run(Thread.java:748)
  26.    Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
  27.        at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
  28.        at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
  29.        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
  30.        at com.microsoft.azure.storage.blob.CloudBlob.startCopy(CloudBlob.java:883)
  31.        at com.microsoft.azure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:825)
  32.        at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:399)
  33.        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2449)
  34.        ... 22 more
  35.    20/06/08 14:23:45 ERROR Utils: Aborting task
  36.    org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
  37.        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2482)
  38.        at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:424)
  39.        at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1997)
  40.        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:531)
  41.        at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:502)
  42.        at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
  43.        at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
  44.        at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:225)
  45.        at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
  46.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
  47.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  48.        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  49.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  50.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  51.        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  52.        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  53.        at org.apache.spark.scheduler.Task.run(Task.scala:123)
  54.        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  55.        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  56.        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  57.        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  58.        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  59.        at java.lang.Thread.run(Thread.java:748)
  60.    Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
  61.        at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
  62.        at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
  63.        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
  64.        at com.microsoft.azure.storage.blob.CloudBlob.startCopy(CloudBlob.java:883)
  65.        at com.microsoft.azure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:825)
  66.        at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:399)
  67.        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2449)
  68.        ... 22 more

如果您有任何解决这个问题的建议，我们将不胜感激？

apache-sparkazure-storage

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com