我正在hdfs中存储一个500 mb或更大的视频文件。因为它大于块大小,所以它将被分发。我必须先收集或只处理第一块数据(这里是视频文件),因为它只包含序列头。我该怎么做,或者如何在hadoop中找到文件的第一个数据块?
qij5mzcb1#
如果你想读第一个街区,你可以得到一个 InputStream 从 FileSystem 并读取字节,直到它达到预定的数量(示例块大小64mb将是6410241024字节)。下面是一个例子(尽管64mb是大量的数据。如果您认为您需要的数据早于64mb,只需更改bytesleet)
InputStream
FileSystem
import java.io.EOFException;import java.io.InputStream;import java.io.OutputStream;import java.net.URI;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.zookeeper.common.IOUtils;public class TestReaderFirstBlock { private static final String uri = "hdfs://localhost:9000/path/to/file"; private static int bytesLeft = 64 * 1024 * 1024; private static final byte[] buffer = new byte[4096]; public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(URI.create(uri), conf); InputStream is = fs.open(new Path(uri)); OutputStream out = System.out; while (bytesLeft > 0) { int read = is.read(buffer, 0, Math.min(bytesLeft, buffer.length)); if (read == -1) { throw new EOFException("Unexpected end of data"); } out.write(buffer, 0, read); bytesLeft -= read; } IOUtils.closeStream(is); }}
import java.io.EOFException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.zookeeper.common.IOUtils;
public class TestReaderFirstBlock {
private static final String uri = "hdfs://localhost:9000/path/to/file";
private static int bytesLeft = 64 * 1024 * 1024;
private static final byte[] buffer = new byte[4096];
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
InputStream is = fs.open(new Path(uri));
OutputStream out = System.out;
while (bytesLeft > 0) {
int read = is.read(buffer, 0, Math.min(bytesLeft, buffer.length));
if (read == -1) {
throw new EOFException("Unexpected end of data");
}
out.write(buffer, 0, read);
bytesLeft -= read;
IOUtils.closeStream(is);
1条答案
按热度按时间qij5mzcb1#
如果你想读第一个街区,你可以得到一个
InputStream
从FileSystem
并读取字节,直到它达到预定的数量(示例块大小64mb将是6410241024字节)。下面是一个例子(尽管64mb是大量的数据。如果您认为您需要的数据早于64mb,只需更改bytesleet)