mapreduce作业在map=12%时挂起,我怎么能通过它呢?

dgjrabp2  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(256)

我写了一个hiveql脚本,比如:

create temporary function getdomainips as 'com.is.mail.domainspf.IpFromDomainExtract';
select Domain,getdomainips(Domain) as ips from tmp_domain;

ipfromdomainextract类如下:

package com.is.mail.domainspf;

import org.apache.hadoop.hive.ql.exec.UDF;

import java.util.ArrayList;
import java.util.Arrays;

public class IpFromDomainExtract extends UDF {
    public ArrayList<String> evaluate(String domain) {
        try {
            if (domain == null || "".equals(domain)) {
                return new ArrayList<String>();
            }
            return new ArrayList<String>(Arrays.asList(UrlDomainDigHelper.getARecord(domain).split(",")));
        } catch (Exception e) {
            return new ArrayList<String>();
        }
    }
}

import java.net.UnknownHostException;

import org.xbill.DNS.ARecord;
import org.xbill.DNS.ExtendedResolver;
import org.xbill.DNS.Lookup;
import org.xbill.DNS.Record;
import org.xbill.DNS.TextParseException;
import org.xbill.DNS.Type;

/**

* use dnsjava jar package
* /

class UrlDomainDigHelper {
    private static ExtendedResolver resolver;

    public static String getARecord(String d) throws TextParseException, UnknownHostException {
        if (resolver == null) {
            synchronized (UrlDomainDigHelper.class) {
                if (resolver == null) {
                    ExtendedResolver tmpresolver = new ExtendedResolver();
                    tmpresolver.setTimeout(5);
                    resolver = tmpresolver;
                }
            }
        }
        Lookup lookup = new Lookup(d, Type.A);
        if (resolver != null)
            lookup.setResolver(resolver);
        Record[] records = lookup.run();
        StringBuilder sb = new StringBuilder();
        if (records != null) {
            for (int i = 0; i < records.length; i++) {
                ARecord mx = (ARecord) records[i];
                sb.append(mx.getAddress().getHostAddress()).append(",");
            }
        }
        if (sb.length() > 0)
            sb.setLength(sb.length() - 1);
        return sb.toString();
    }
}

当我运行hiveql脚本时,我陷入map=12%的困境,如下所示:
2016-01-06 16:14:06701第一阶段map=12%,reduce=0%,累计cpu 105.67秒2016-01-06 16:15:07172第一阶段map=12%,reduce=0%,累计cpu 106.7秒2016-01-06 16:16:07317第一阶段map=12%,reduce=0%,累计cpu 107.87秒2016-01-06 16:17:07501第一阶段map=12%,reduce=0%,累计cpu 108.84秒2016-01-06 16:18:07,680第1阶段map=12%,reduce=0%,累计cpu 109.71秒2016-01-06 16:19:07870第1阶段map=12%,reduce=0%,累计cpu 110.37秒2016-01-06 16:20:08014第1阶段map=12%,reduce=0%,累计cpu 111.5秒2016-01-06 16:21:08234第1阶段map=12%,reduce=0%,累计cpu 112.76秒2016-01-06 16:22:08494第1阶段map=12%,reduce=0%,累计cpu 113.78秒2016-01-06 16:23:08789第1阶段map=12%,reduce=0%,累计cpu 114.97秒2016-01-06 16:24:09191第1阶段map=12%,reduce=0%,累计cpu 115.84秒2016-01-06 16:25:09537第1阶段map=12%,reduce=0%,累计cpu 116.79秒2016-01-06 16:26:09779第1阶段map=12%,reduce=0%,累计cpu 117.69秒2016-01-06 16:27:10106第1阶段map=12%,减少=0%,累计cpu 118.91秒2016-01-06 16:28:10213第1阶段map=12%,减少=0%,累计cpu 119.94秒2016-01-06 16:29:10826第1阶段map=12%,减少=0%,累计cpu 120.76秒2016-01-06 16:30:11158第1阶段map=12%,减少=0%,累计cpu 122.2秒2016-01-06 16:31:11433一级map=12%,减少=0%,累计cpu 123.26秒2016-01-06 16:32:11564一级map=12%,减少=0%,累计cpu 124.28秒2016-01-06 16:33:12093一级map=12%,减少=0%,累计cpu 124.9秒2016-01-06 16:34:12319一级map=12%,减少=0%,累计cpu 125.82秒2016-01-06 16:35:12556第一阶段map=12%,reduce=0%,累计cpu 126.79秒2016-01-06 16:36:12978第一阶段map=12%,reduce=0%,累计cpu 127.57秒
当我使用jstack查看mapreduce作业流程信息时,得到以下结果:
“main”prio=10 tid=0x000000000122e000 nid=0x6908 in object.wait()[0x00007fc884fa7000]java.lang.thread.state:在java.lang.object.wait(本机方法)等待(在对象监视器上)-在java.lang.object.wait(对象)等待<0x00000000eab6b780>(org.xbill.dns.extendedresolver$resolution)。java:502)在org.xbill.dns.extendedresolver$resolution.start(extendedresolver。java:111)-在org.xbill.dns.extendedresolver.send(extendedresolver)处锁定<0x00000000eab6b780>(一个org.xbill.dns.extendedresolver$resolution)。java:358)在org.xbill.dns.lookup.lookup(lookup。java:477)在org.xbill.dns.lookup.resolve(lookup。java:529)在org.xbill.dns.lookup.run(查找。java:546)
我想我遇到了一个死锁,如果我已经使用了最新的dnsjavajar包(2.1.7版本),怎么解决这个问题呢?非常感谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题