我在一个文件夹中有大约200k个文件,我想根据出生时间(创建时间)来组织它们。我写了下面的脚本,但太慢了。我想改进一下。我该怎么做?
# !/usr/bin/env bash
echo Input directory is $1
input_directory=$1
## Find those files that are older than a month
inputfiles=$(hadoop fs -ls $input_directory | sed '1d;s/ */ /g' | cut -d\ -f8)
for filename in $inputfiles
do
echo processing $filename
hadoop fs -test -d $filename
lastcommand=$?
if [ "$lastcommand" == "1" ];then
year=$(date -d "`hadoop fs -stat $filename`" +%Y)
month=$(date -d "`hadoop fs -stat $filename`" +%m)
hadoop fs -test -d $input_directory/$year-$month
lastcommand2=$?
[[ "$lastcommand2" == "1" ]] && hadoop fs -mkdir -p $input_directory/$year-$month;
hadoop fs -mv $filename $input_directory/$year-$month/
else
echo not a file
fi
done
1条答案
按热度按时间im9ewurl1#
我能够使用hadoop文件系统重命名命令来移动苍蝇,而且效果很好。把时间从小时缩短到一分钟。谢谢您