使用stdout popen捕获sqoop日志

jvlzgdj9 于 2021-06-03 发布在 Sqoop

关注(0)|答案(1)|浏览(621)

下面是我正在运行的调用sqoop的python代码，但是除了下面几行之外，它没有捕获日志
警告：/usr/hdp/2.6.4.0-91/acumulo不存在！accumulo进口将失败。请将$accumulo\u home设置为accumulo安装的根目录。

import subprocess
job = "sqoop-import --direct --connect 'jdbc:sqlserver://host' --username myuser --password-file /user/ivr_sqoop --table data_app_det  --delete-target-dir   --verbose --split-by attribute_name_id --where \"db_process_time BETWEEN ('2018-07-15') and ('9999-12-31')\""
print job
with open('save.txt','w') as fp:
    proc = subprocess.Popen(job, stdout=fp, stderr=subprocess.PIPE, shell=True)
    stdout, stderr = proc.communicate()
    print "Here is the return code :: " + str(proc.returncode)
    print stdout`

如果我打电话的方式有问题，请告诉我。
注意：单个sqoop cmd运行良好并生成所有日志。
我也试过下面的方法，结果是一样的

import subprocess
job = "sqoop-import --direct --connect 'jdbc:sqlserver://host' --username myuser --password-file /user/ivr_sqoop --table data_app_det  --delete-target-dir   --verbose --split-by attribute_name_id --where \"db_process_time BETWEEN ('2018-07-15') and ('9999-12-31')\""
proc = subprocess.Popen(job, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
stdout, stderr = proc.communicate()

并在cmd的末尾使用'2>mylog.log'

import subprocess
job = "sqoop-import --direct --connect 'jdbc:sqlserver://host' --username myuser --password-file /user/ivr_sqoop --table data_app_det  --delete-target-dir   --verbose --split-by attribute_name_id --where \"db_process_time BETWEEN ('2018-07-15') and ('9999-12-31')\" > mylog.log "
proc = subprocess.Popen(job, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdout, stderr = proc.communicate()

我发现了下面类似的问题，但也没有答案。
子进程popen：忽略accumulo警告并继续执行sqoop

sqoop python subprocess popen

来源：https://stackoverflow.com/questions/51424854/capturing-sqoop-logs-using-stdout-popen

1条答案

按热度按时间

bxfogqkk1#

既然你添加了 shell=True ，它没有捕获sqoop日志。请删除 shell=True 从你的命令中添加 universal_newlines=True ，它将显示控制台日志。
工作代码：

import subprocess
import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
# Function to run Hadoop command
def run_unix_cmd(args_list):
    """
    run linux commands
    """
    print('Running system command: {0}'.format(' '.join(args_list)))
    proc = subprocess.Popen(args_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
    s_output, s_err = proc.communicate()
    s_return =  proc.returncode
    return s_return, s_output, s_err
# Create Sqoop Job
def sqoop_job():
    """
    Create Sqoop job
    """
    cmd = ['sqoop', 'import', '--connect', 'jdbc:oracle:thin:@//host:port/schema', '--username', 'user','--password', 'XX', '--query', '"your query"', '-m', '1', '--target-dir', 'tgt_dir']
    print(cmd)
    (ret, out, err) = run_unix_cmd(cmd)
    print(ret, out, err)
    if ret == 0:
        logging.info('Success.')
    else:
        logging.info('Error.')
if __name__ == '__main__':
    sqoop_job()

展开查看全部

赞(0）回复(0）举报 2021-06-03

我来回答

使用stdout popen捕获sqoop日志

1条答案

相关问题

热门标签

最新问答