s3n处的主机名uri无效://

yx2lnoni  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(378)

我在尝试访问s3 bucket中的对象时遇到这个错误。我可以访问s3 bucket中顶层的对象。但是当我尝试访问文件夹中的文件时,它不起作用。bucket/file.xml(工作),但bucket/folder/file.xml(不工作)输出无效的主机名uri异常。

  1. 17:28:20,652 DEBUG HttpConnection:691 - Open connection to output-4ssdgf7c9ae-838df-40se1-a105-7c8srfgsg6db166.s3.amazonaws.com:443
  2. 17:28:20,949 DEBUG header:69 - >> "GET /%2Foutput HTTP/1.1[\r][\n]"
  3. 17:28:20,949 DEBUG HttpMethodBase:1235 - Adding Host request header
  4. 17:28:20,956 DEBUG header:69 - >> "Content-Type: [\r][\n]"
  5. 17:28:20,957 DEBUG header:69 - >> "Date: Mon, 29 Aug 2016 11:58:20 GMT[\r][\n]"
  6. 17:28:20,957 DEBUG header:69 - >> "Authorization: AWS AsdggPZF42ULNsdsd:hJLLzNsxcvq44gNbbbndestXzL+XvP8cw=[\r][\n]"
  7. 17:28:20,957 DEBUG header:69 - >> "User-Agent: JetS3t/0.6.1 (Linux/3.16.0-77-generic; amd64; en; JVM 1.8.0_101)[\r][\n]"
  8. 17:28:20,957 DEBUG header:69 - >> "Host: output-4e5sfdg9ae-8zsdf-40s1-a105-7c8easfddb166.s3.amazonaws.com[\r][\n]"
  9. 17:28:20,958 DEBUG header:69 - >> "[\r][\n]"
  10. 17:28:21,798 DEBUG header:69 - << "HTTP/1.1 404 Not Found[\r][\n]"
  11. 17:28:21,800 DEBUG header:69 - << "x-amz-request-id: F280A20CBDGD76D0[\r][\n]"
  12. 17:28:21,800 DEBUG header:69 - << "x-amz-id-2: Z8Pq68OsdDEnhcEvEtrhlxkQ9b+bXYAbIxxssamS6+8vsbD9LSDFpzxb0KDFFvXA4[\r][\n]"
  13. 17:28:21,800 DEBUG header:69 - << "Content-Type: application/xml[\r][\n]"
  14. 17:28:21,801 DEBUG header:69 - << "Transfer-Encoding: chunked[\r][\n]"
  15. 17:28:21,801 DEBUG header:69 - << "Date: Mon, 29 Aug 2016 11:58:20 GMT[\r][\n]"
  16. 17:28:21,801 DEBUG header:69 - << "Server: AmazonS3[\r][\n]"
  17. 17:28:21,803 DEBUG HttpConnection:905 - Input data available
  18. 17:28:21,805 DEBUG RestS3Service:385 - Response for '/%2Foutput'. Content-Type: application/xml, Headers: [x-amz-request-id: F280A2SDF70D76D0
  19. , x-amz-id-2: Z8Pq68OgBSDFGhcEvEtrhlxkQ9b+bXYAbIxRAmdddamS6+8vsbD9Ltdmxs83KAkvXA4
  20. , Content-Type: application/xml
  21. , Transfer-Encoding: chunked
  22. , Date: Mon, 29 Aug 2016 11:58:20 GMT
  23. , Server: AmazonS3
  24. ]
  25. 17:28:21,805 WARN RestS3Service:393 - Response '/%2Foutput' - Unexpected response code 404, expected 200
  26. 17:28:21,806 WARN RestS3Service:402 - Response '/%2Foutput' - Received error response with XML message
  27. 17:28:21,808 DEBUG content:83 - << "1"
  28. 17:28:21,808 DEBUG content:83 - << "0"
  29. 17:28:21,808 DEBUG content:83 - << "a"
  30. 17:28:21,808 DEBUG content:83 - << "[\r]"
  31. 17:28:21,809 DEBUG content:69 - << "[\n]"
  32. 17:28:21,809 DEBUG content:69 - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
  33. 17:28:21,809 DEBUG content:83 - << "<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>/output</Key><RequestId>F280A20DGG076D0</RequestId><HostId>Z8Pq68OgSD23hcEvEtrhlxkQ9b+bXYAERTY54RamS6+8vsbD9LwDFE342AkvXA4</HostId></Error>"
  34. 17:28:21,810 DEBUG content:83 - << "[\r]"
  35. 17:28:21,810 DEBUG content:69 - << "[\n]"
  36. 17:28:21,810 DEBUG content:83 - << "0"
  37. 17:28:21,810 DEBUG content:83 - << "[\r]"
  38. 17:28:21,811 DEBUG content:69 - << "[\n]"
  39. 17:28:21,811 DEBUG content:83 - << "[\r]"
  40. 17:28:21,811 DEBUG content:69 - << "[\n]"
  41. 17:28:21,811 DEBUG HttpMethodBase:921 - Resorting to protocol version default close connection policy
  42. 17:28:21,812 DEBUG HttpMethodBase:925 - Should NOT close connection, using HTTP/1.1
  43. 17:28:21,812 DEBUG HttpConnection:1177 - Releasing connection back to connection manager.
  44. 17:28:21,812 DEBUG MultiThreadedHttpConnectionManager:943 - Freeing connection, hostConfig=HostConfiguration[host=https://output-4e57c9ae-838f-40e1-a105-7c8ed86db166.s3.amazonaws.com]
  45. 17:28:21,813 DEBUG IdleConnectionHandler:75 - Adding connection at: 1472471901813
  46. 17:28:21,813 DEBUG MultiThreadedHttpConnectionManager:926 - Notifying no-one, there are no waiting threads
  47. 17:28:21,813 DEBUG HttpMethodReleaseInputStream:142 - Released HttpMethod as its response data stream is fully consumed
  48. 17:28:21,816 DEBUG RestS3Service:516 - Releasing HttpClient connection after error: S3 GET failed for '/%2Foutput'
  49. 17:28:21,817 DEBUG JobClient:982 - Creating splits at file:/tmp/hadoop-java-dev/mapred/staging/java-dev930738962/.staging/job_local93034362_0001
  50. 17:28:21,825 WARN LoadSnappy:46 - Snappy native library not loaded
  51. 17:28:21,827 DEBUG FileSystem:1441 - Creating filesystem for s3://lukup_assets/live_channel_xmls/10%20tv.xml
  52. 17:28:21,827 INFO JobClient:1023 - Cleaning up the staging area file:/tmp/hadoop-java-dev/mapred/staging/java-dev930738962/.staging/job_local930743452_0001
  53. java.lang.IllegalArgumentException: Invalid hostname in URI s3://asset_fdp/live_records/400%20tv.xml
  54. ******end job

我得到上述例外:

  1. java.lang.IllegalArgumentException: Invalid hostname in URI s3://asset_fdp/live_records/400%20tv.xml

在上面的'asset\u fdp'是我的bucket名称,'live records'是bucket中的文件夹。

huus2vyu

huus2vyu1#

我猜这可能是您的密钥权限的问题?
也许您可以访问bucket\u name,但不能访问bucket\u name/*?
检查此项:
我需要一个amazons3用户可以完全访问一个bucket
但更重要的是,查看您的调试日志,我看到一些东西看起来有点像您的aws私钥。我希望不是这样?如果是,你应该把它从这个公共论坛上删除。。。

相关问题