我正在尝试在HDFS和我的GCP云存储之间复制数据。这不是一次性的数据拷贝。在第一次复制后，我只想复制新文件，更新文件。如果文件在本地删除，也应该从云存储中删除。

然而，我意识到的是，当目标是云时，基于快照差异的拷贝不起作用。

有可能进行这种同步吗？

-更新标志似乎不适用于云存储。它会复制所有文件，即使没有更改也是如此。

命令

hadoop distcp --conf hdfs.conf -update -delete hdfs:///tmp/test_distcp gs://onpremhadoopfiles-123/

带有快照差异的命令

hadoop distcp --conf test.conf -update -diff  test_distcp test_distcp_new  hdfs:///tmp/test_distcp gs://xxxx-123/

错误

Jul 29, 2022 9:56:31 AM com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase configure
WARNING: No working directory configured, using default: 'gs://onpremhadoopfiles-123/'
22/07/29 09:56:32 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=true, useRdiff=false, fromSnapshot=test_distcp, toSnapshot=test_distcp_new, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs:/tmp/test_distcp], targetPath=gs://xxx-123/, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false}, sourcePaths=[hdfs:/tmp/test_distcp], targetPathExists=true, preserveRawXattrsfalse
22/07/29 09:56:32 INFO client.RMProxy: Connecting to ResourceManager at xxx.xxx.com/xx.xx.xx.x:8032
22/07/29 09:56:33 ERROR tools.DistCp: Exception encountered
java.lang.IllegalArgumentException: The FileSystems needs to be DistributedFileSystem for using snapshot-diff-based distcp
        at org.apache.hadoop.tools.DistCpSync.preSyncCheck(DistCpSync.java:98)
        at org.apache.hadoop.tools.DistCpSync.sync(DistCpSync.java:149)
        at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:88)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:205)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:182)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)