我使用Spark将数据从Hive Table写入Kinetica,使用的是这个jar:kinetica-spark-7.0.6.1-jar-with-dependencies.jar
。但是,当我运行spark-submit
时,jar中的记录器将使用其凭据打印JDBC连接字符串,如下所示:
...
22/03/03 03:00:58 INFO spark.LoaderParams: Using JDBC connection string: jdbc:kinetica://10.xx.xx.xx:9191;UID=xxx;PWD=xxx
22/03/03 03:00:58 INFO spark.LoaderParams: Installing truststore to bypass certificate check.
22/03/03 03:00:58 INFO spark.LoaderParams: Using URL(s) http://10.xx.xx.xx:9191to create a GPUdb connection
22/03/03 03:00:59 INFO spark.LoaderParams: Connecting to http://10.xx.xx.xx:9191 as user <xxx>
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 47
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 42
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 53
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 52
...
它来自于这段代码:kinetica.spark.LoaderParams
我想专门为这个spark.LoaderParams
类将日志级别设置为更高的级别,这样连接字符串就不会显示在日志中的任何地方。
我能做到吗?
下面是我的log4j.properties
配置:
log4j.rootLogger=${root.logger}
root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=WARN
log4j.logger.org.eclipse.jetty=INFO
log4j.logger.org.spark-project.jetty=INFO
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}
log4j.logger.jobLogger=INFO, RollingAppenderU
log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.fileAppender.MaxFileSize=1MB
log4j.appender.fileAppender.MaxBackupIndex=1
1条答案
按热度按时间mpgws1up1#
在我的配置中,我使用以下命令记录
WARN
处的LoaderParams
语句以及来自INFO
处Kinetica Spark连接器的所有其他内容: