我正在尝试从 mapr fs origin
到 mapr fs destination
(这不是我的用例,只是为了测试而做这个简单的移动)。当你试图 validate
在这个管道中,我在暂存区看到的错误消息是:
HADOOPFS_11 - Cannot connect to the filesystem. Check if the Hadoop FS location: 'maprfs:///mapr/mycluster.cluster.local' is valid or not: 'java.io.IOException: Provided Subject must contain a KerberosPrincipal
轮胎的不同变化 hadoop fs URI
字段(例如mfs:///mapr/mycluster.cluster.local、maprfs:///mycluster.cluster.local)似乎没有帮助。在尝试验证后查看日志,我明白了
2018-01-04 10:28:56,686 mfs2mfs/mapr2sqlserver850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Created source of type: com.streamsets.pipeline.stage.origin.maprfs.ClusterMapRFSSource@16978460 DClusterSourceOffsetCommitter *admin preview-pool-1-thread-3
2018-01-04 10:28:56,697 mfs2mfs/mapr2sqlserver850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Error connecting to FileSystem: java.io.IOException: Provided Subject must contain a KerberosPrincipal ClusterHdfsSource *admin preview-pool-1-thread-3
java.io.IOException: Provided Subject must contain a KerberosPrincipal
....
2018-01-04 10:20:39,159 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Authentication Config: ClusterHdfsSource *admin preview-pool-1-thread-3
2018-01-04 10:20:39,159 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 ERROR Issues: Issue[instance='MapRFS_01' service='null' group='HADOOP_FS' config='null' message='HADOOPFS_11 - Cannot connect to the filesystem. Check if the Hadoop FS location: 'maprfs:///mapr/mycluster.cluster.local' is valid or not: 'java.io.IOException: Provided Subject must contain a KerberosPrincipal''] ClusterHdfsSource *admin preview-pool-1-thread-3
2018-01-04 10:20:39,169 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Validation Error: Failed to configure or connect to the 'maprfs:///mapr/mycluster.cluster.local' Hadoop file system: java.io.IOException: Provided Subject must contain a KerberosPrincipal HdfsTargetConfigBean *admin 0 preview-pool-1-thread-3
java.io.IOException: Provided Subject must contain a KerberosPrincipal
....
然而,据我所知,系统没有运行 Keberos
,所以这个错误消息对我来说有点混乱。取消注解 #export SDC_JAVA_OPTS="-Dmaprlogin.password.enabled=true ${SDC_JAVA_OPTS}"
在sdc环境中,用于本机mapr身份验证的变量文件似乎对问题没有帮助(即使在运行streamsets mapr安装脚本之前重新安装并注解这一行)。
有人知道发生了什么事以及如何解决吗?谢谢。
1条答案
按热度按时间v64noz0r1#
这个答案是在mapr社区论坛上提供的,对我有用(使用
mapr v6.0
). 请注意,此处的说明与streamsets文档当前提供的说明不同。在这些说明中,我是以用户身份登录的root
.在按照文档安装streamset(和mapr先决条件)之后。。。
更改流集的所有者
$SDC_DIST
或者$SDC_HOME
mapr用户(或您计划用于hadoop模拟的任何其他用户)的位置:$chown -R mapr:mapr $SDC_DIST
(对我来说,这是/opt/streamsets datacollector目录)。对…也一样$SDC_CONF
(/etc/sdc
对我来说)以及/var/lib/sdc
以及var/log/sdc
.在
$SDC_DIST/libexec/sdcd-env.sh
,将用户名和组名(靠近文件顶部)设置为mapr user“mapr”,并启用mapr密码登录。文件的最终外观应如下所示:编辑文件
/usr/lib/systemd/system/sdc.service
看起来像:Environment=SDC_JAVA_OPTS=-Dmaprlogin.passowrd.enabled=true
2018-01-16 14:26:10,883
ingest2sa_demodata_batch/ingest2sademodatabatchadca8442-cb00-4a0e-929b-df2babe4fd41
ERROR Error in Slave Runner: ClusterRunner *admin
runner-pool-2-thread-29
com.streamsets.datacollector.runner.PipelineRuntimeException:
CONTAINER_0800 - Pipeline
'ingest2sademodatabatchadca8442-cb00-4a0e-929b-df2babe4fd41'
validation error : HADOOPFS_11 - Cannot connect to the filesystem.
Check if the Hadoop FS location: 'maprfs:///' is valid or not:
'java.io.IOException: Provided Subject must contain a
KerberosPrincipal'