使用hiveudf解压缩列数据

piv4azn7 于 2021-06-26 发布在 Hive

关注(0)|答案(1)|浏览(500)

context:decompress the 使用hive udf evaluate（）方法的列数据
例外情况：
失败，出现异常java.io.ioexception:org.apache.hadoop.hive.ql.metadata.hiveexception：无法执行方法public static org.apache.hadoop.io.text test.udfdecompressor.evaluate（java.lang.string）在对象测试上抛出org.apache.hadoop.hive.ql.metadata.hiveexception。udfdecompressor@1008df1e 类test.udfdecompressor的参数{xï¿½ï¿½}千瓦â¸²ï¿½_一ï¿½ï¿½ï¿½ï¿½ï¿½ö¤ï¿½\ï¿½ï¿½a-b型ï¿½我ï¿½@`ï¿½ï¿½ï¿½ï¿½ï¿½"ï¿½第三次ï¿½我ï¿½ï¿½ï¿½ï¿½$_ï¿½eï¿½ï¿½ } 大小1
源代码：

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.zip.DataFormatException;
import java.util.zip.InflaterInputStream;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
public class Decompress extends UDF{
public static String evaluate(String data1) throws IOException, DataFormatException{
ByteArrayInputStream bao=new ByteArrayInputStream(data1.getBytes());
InflaterInputStream iis= new InflaterInputStream(bao);
String out="";
byte[] bt=new byte[1024];
int len=-1;
while ((len =iis.read(bt))!=-1){ 
out += new String(Arrays.copyOf(bt, len));
}
JavaStringObjectInspector stringInspector;
stringInspector = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
String ip = stringInspector.getPrimitiveJavaObject(out);
//return new String(ip.getBytes(Charset.forName("UTF-8")));
//return new String(ip.getBytes(Charset.forName("UTF-8")));
return ip;
}
}

我尝试了多种方法来使用gzib、zlibjavaapi解压，但是遇到了相同的错误。有谁能帮助我解决这个问题，并建议使用hiveudf解压列数据的正确方法吗
提前谢谢。

Java Hive udf compression

来源：https://stackoverflow.com/questions/44061561/decompress-column-data-using-hive-udf

1条答案

按热度按时间

frebpwbc1#

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.InflaterInputStream;
public class Decompress extends UDF {
    private final Text r = new Text();
    public Text evaluate(BytesWritable bw) throws IOException {
        ByteArrayInputStream zipped = new ByteArrayInputStream(bw.getBytes());
        InflaterInputStream inflater = new InflaterInputStream(zipped);
        ByteArrayOutputStream unzipped = new ByteArrayOutputStream();
        byte[] bt = new byte[1024];
        int len;
        while ((len = inflater.read(bt)) != -1) {
            unzipped.write(bt, 0, len);
        }
        r.clear();
        r.set(unzipped.toByteArray());
        return r;
    }
}

展开查看全部

赞(0）回复(0）举报 2021-06-26

我来回答

使用hiveudf解压缩列数据

1条答案

相关问题

热门标签

最新问答