例如，如果我的输入文件有500MB，我想这分裂250MB每个，如果我的输入文件是600MB块大小应该是300MB
y53ybaqx1#
If you are loading files into hdfs you can put with dfs.blocksize oprtion, you can calculate parameter in a shell depending on size.
hdfs dfs -D dfs.blocksize=268435456 -put myfile /some/hdfs/location
If you already have files in HDFS and want to change it's block size, you need to rewrite it.(1) move file to tmp location:
hdfs dfs -mv /some/hdfs/location/myfile /tmp
(2) Copy it back with -D dfs.blocksize=268435456
-D dfs.blocksize=268435456
hdfs dfs -D dfs.blocksize=268435456 -cp /tmp/myfile /some/hdfs/location
1条答案
按热度按时间y53ybaqx1#
If you are loading files into hdfs you can put with dfs.blocksize oprtion, you can calculate parameter in a shell depending on size.
If you already have files in HDFS and want to change it's block size, you need to rewrite it.
(1) move file to tmp location:
(2) Copy it back with
-D dfs.blocksize=268435456