hadoop连接

dtcbnfnu  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(464)

我正在尝试使用SpringData-hadoop从本地机器的ide在远程集群上执行mr代码
hadoop 1.1.2、spring 3.2.4、spring data hadoop 1.0.0
我的bean配置文件。applicationcontext.xml如下所示：

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <beans xmlns="http://www.springframework.org/schema/beans"
  3.     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hdp="http://www.springframework.org/schema/hadoop"
  4.     xmlns:context="http://www.springframework.org/schema/context"
  5.     xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd 
  6.     http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
  7.     http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.2.xsd">
  9.     <context:property-placeholder location="resources/hadoop.properties" />
  11.     <hdp:configuration file-system-uri="${hd.fs}" job-tracker-uri="${hd.jobtracker.uri}">
  13.     </hdp:configuration>
  15.     <hdp:job id="wc-job" mapper="com.hadoop.basics.WordCounter.WCMapper"
  16.         reducer="com.hadoop.basics.WordCounter.WCReducer" input-path="${wordcount.input.path}"
  17.         output-path="${wordcount.output.path}" user="bigdata">
  18.     </hdp:job>
  20.     <hdp:job-runner id="myjobs-runner" job-ref="wc-job"
  21.         run-at-startup="true" />
  23.     <hdp:resource-loader id="resourceLoader" uri="${hd.fs}"
  24.         user="bigdata" />   
  25. </beans>

hadoop.properties属性

  1. hd.fs=hdfs://cloudx-843-770:9000
  2. hd.jobtracker.uri=cloudx-843-770:9001
  4. wordcount.input.path=/scratchpad/input/Childhood_days.txt
  5. wordcount.output.path=/scratchpad/output

我正在做的java类“运行方式…”

  1. package com.hadoop.basics;
  3. import java.io.IOException;
  4. import java.util.StringTokenizer;
  6. import org.apache.hadoop.io.IntWritable;
  7. import org.apache.hadoop.io.Text;
  8. import org.apache.hadoop.mapreduce.Mapper;
  9. import org.apache.hadoop.mapreduce.Reducer;
  10. import org.springframework.context.support.AbstractApplicationContext;
  11. import org.springframework.context.support.ClassPathXmlApplicationContext;
  13. public class WordCounter {
  15.     private static IntWritable one = new IntWritable(1);
  17.     public class WCMapper extends Mapper<Text, Text, Text, IntWritable> {
  19.         @Override
  20.         protected void map(
  21.                 Text key,
  22.                 Text value,
  23.                 org.apache.hadoop.mapreduce.Mapper<Text, Text, Text, IntWritable>.Context context)
  24.                 throws IOException, InterruptedException {
  25.             // TODO Auto-generated method stub
  26.             StringTokenizer strTokenizer = new StringTokenizer(value.toString());
  27.             Text token = new Text();
  29.             while (strTokenizer.hasMoreTokens()) {
  30.                 token.set(strTokenizer.nextToken());
  31.                 context.write(token, one);
  32.             }
  34.         }
  35.     }
  37.     public class WCReducer extends
  38.             Reducer<Text, IntWritable, Text, IntWritable> {
  39.         @Override
  40.         protected void reduce(
  41.                 Text key,
  42.                 Iterable<IntWritable> values,
  43.                 org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>.Context context)
  44.                 throws IOException, InterruptedException {
  45.             // TODO Auto-generated method stub
  47.             int sum = 0;
  49.             for (IntWritable value : values) {
  50.                 sum += value.get();
  51.             }
  53.             context.write(key, new IntWritable(sum));
  54.         }
  55.     }
  57.     public static void main(String[] args) {
  58.         AbstractApplicationContext context = new ClassPathXmlApplicationContext(
  59.                 "applicationContext.xml", WordCounter.class);
  60.         System.out.println("Word Count Application Running");
  61.         context.registerShutdownHook();
  62.     }
  63. }

输出为：

  1. Aug 22, 2013 9:59:02 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
  2. INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1815338: startup date [Thu Aug 22 09:59:02 IST 2013]; root of context hierarchy
  3. Aug 22, 2013 9:59:03 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
  4. INFO: Loading XML bean definitions from class path resource [com/hadoop/basics/applicationContext.xml]
  5. Aug 22, 2013 9:59:03 AM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties
  6. INFO: Loading properties file from class path resource [resources/hadoop.properties]
  7. Aug 22, 2013 9:59:03 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
  8. INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  9. Aug 22, 2013 9:59:03 AM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  10. INFO: Starting job [wc-job]
  11. Aug 22, 2013 9:59:03 AM org.apache.hadoop.security.UserGroupInformation doAs
  12. SEVERE: PriviledgedActionException as:bigdata via 298790 cause:org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  13. Aug 22, 2013 9:59:03 AM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  14. WARNING: Cannot start job [wc-job]
  15. org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  16.     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
  17.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  18.     at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source)
  19.     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
  20.     at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499)
  21.     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490)
  22.     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:473)
  23.     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
  24.     at java.security.AccessController.doPrivileged(Native Method)
  25.     at javax.security.auth.Subject.doAs(Unknown Source)
  26.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  27.     at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
  28.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
  29.     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
  30.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197)
  31.     at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
  32.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168)
  33.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160)
  34.     at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52)
  35.     at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44)
  36.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541)
  37.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
  38.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
  39.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  40.     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  41.     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  42.     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  43.     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  44.     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
  45.     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  46.     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  47.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:197)
  48.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:172)
  49.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:158)
  50.     at com.hadoop.basics.WordCounter.main(WordCounter.java:58)
  52. Aug 22, 2013 9:59:03 AM org.springframework.beans.factory.support.DefaultSingletonBeanRegistry destroySingletons
  53. INFO: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  54. Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'myjobs-runner': Invocation of init method failed; nested exception is java.lang.IllegalStateException: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  55.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1482)
  56.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
  57.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  58.     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  59.     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  60.     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  61.     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  62.     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
  63.     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  64.     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  65.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:197)
  66.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:172)
  67.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:158)
  68.     at com.hadoop.basics.WordCounter.main(WordCounter.java:58)
  69. Caused by: java.lang.IllegalStateException: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  70.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:209)
  71.     at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
  72.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168)
  73.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160)
  74.     at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52)
  75.     at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44)
  76.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541)
  77.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
  78.     ... 13 more
  79. Caused by: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  80.     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
  81.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  82.     at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source)
  83.     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
  84.     at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499)
  85.     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490)
  86.     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:473)
  87.     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
  88.     at java.security.AccessController.doPrivileged(Native Method)
  89.     at javax.security.auth.Subject.doAs(Unknown Source)
  90.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  91.     at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
  92.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
  93.     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
  94.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197)
  95.     ... 20 more

很明显，在集群上无法识别用户298790（我的本地windows机器用户）——这就是为什么在配置文件中。文件
我在文档中提到的作业配置中指定了user=“bigdata”。
医生。还提到：
shdp遵循hdfs权限，使用当前用户的身份（默认情况下）与文件系统进行交互。尤其是，hdfsresourceloader在进行模式匹配时，只考虑它认为可以看到的文件，而不执行任何特权操作。但是，可以指定不同的用户，这意味着resourceloader使用该用户的权限与hdfs进行交互-但是这遵守了api中的用户模拟规则，我决定使用hdfsresourceloader，但在文档中找不到任何示例甚至配置-有人能提供任何指针吗？
根据hadoop安全模拟，我认为我需要将我的windows用户298790添加到远程集群机（ubuntu）用户组中，还需要添加我的windows主机名，在大量用户和不断变化的windows客户机的情况下，我发现这是不可行的。如果我的假设是正确的，那么怎样才能避免添加和配置所有这些用户呢？
/将更改添加到core-site.xml/

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  4. <!-- Put site-specific property overrides in this file. -->
  6. <configuration>
  8. <property>
  9. <name>fs.default.name</name>
  10. <value>hdfs://cloudx-843-770:9000</value>
  11. </property>
  13. <property>
  14. <name>hadoop.proxyuser.298790.groups</name>
  15. <value>bigdata</value>
  16. <description>Allow the superuser bigdatato impersonate any members of the group bigdata</description>
  17. </property>
  19. <property>
  20. <name>hadoop.proxyuser.298790.hosts</name>
  21. <value>*</value>
  22. <description>The superuser can connect only from INFVA03351 to impersonate a user</description>
  23. </property>
  25. </configuration>

我重新启动了所有hadoop进程，但错误仍然存在。
然后，我决定创建一个新的用户viz。并将其添加到组bigdata中以进行模拟：

  1. root@cloudx-843-770:/home/bigdata# useradd -G bigdata 298790
  2.     root@cloudx-843-770:/home/bigdata#
  3.     root@cloudx-843-770:/home/bigdata#
  4.     root@cloudx-843-770:/home/bigdata# usermod -G bigdata 298790
  5.     root@cloudx-843-770:/home/bigdata#
  6.     root@cloudx-843-770:/home/bigdata# su 298790
  7.     $ groups
  8.     298790 bigdata
  9. root@cloudx-843-770:/home/bigdata#
  10. root@cloudx-843-770:/home/bigdata# cat /etc/passwd
  11. root:x:0:0:root:/root:/bin/bash
  12. daemon:x:1:1:daemon:/usr/sbin:/bin/sh
  13. bin:x:2:2:bin:/bin:/bin/sh
  14. sys:x:3:3:sys:/dev:/bin/sh
  15. sync:x:4:65534:sync:/bin:/bin/sync
  16. games:x:5:60:games:/usr/games:/bin/sh
  17. man:x:6:12:man:/var/cache/man:/bin/sh
  18. lp:x:7:7:lp:/var/spool/lpd:/bin/sh
  19. mail:x:8:8:mail:/var/mail:/bin/sh
  20. news:x:9:9:news:/var/spool/news:/bin/sh
  21. uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
  22. proxy:x:13:13:proxy:/bin:/bin/sh
  23. www-data:x:33:33:www-data:/var/www:/bin/sh
  24. backup:x:34:34:backup:/var/backups:/bin/sh
  25. list:x:38:38:Mailing List Manager:/var/list:/bin/sh
  26. irc:x:39:39:ircd:/var/run/ircd:/bin/sh
  27. gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
  28. nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
  29. libuuid:x:100:101::/var/lib/libuuid:/bin/sh
  30. syslog:x:101:103::/home/syslog:/bin/false
  31. mysql:x:102:105:MySQL Server,,,:/nonexistent:/bin/false
  32. messagebus:x:103:106::/var/run/dbus:/bin/false
  33. whoopsie:x:104:107::/nonexistent:/bin/false
  34. landscape:x:105:110::/var/lib/landscape:/bin/false
  35. sshd:x:106:65534::/var/run/sshd:/usr/sbin/nologin
  36. tomcat6:x:107:113::/usr/share/tomcat6:/bin/false
  37. coesystem:x:1000:1000:coesystem,,,:/home/coesystem:/bin/bash
  38. hpcc:x:999:1001:hpcc Runtime User:/home/hpcc:/bin/sh
  39. hduser:x:1001:1002:hduser,1,1,1,1:/home/hduser:/bin/bash
  40. bigdata:x:1002:1003:Big Data,1,1,1,1:/home/bigdata:/bin/bash
  41. 298790:x:1003:1004::/home/298790:/bin/sh

但是现在，当我尝试停止（然后启动）集群时，它会请求所有进程的密码：

  1. bigdata@cloudx-843-770:~/hadoop_ecosystem/apache_hadoop/hadoop-1.1.2/bin$ stop-all.sh
  2. Warning: $HADOOP_HOME is deprecated.
  4. stopping jobtracker
  5. bigdata@localhost's password:
  6. localhost: stopping tasktracker
  7. stopping namenode
  8. bigdata@localhost's password:
  9. localhost: stopping datanode
  10. bigdata@localhost's password:
  11. localhost: stopping secondarynamenode

现在，错误被稍微修改了一下-它首先无法连接，然后无法模拟：

  1. Aug 22, 2013 5:14:17 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
  2. INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@922804: startup date [Thu Aug 22 17:14:17 IST 2013]; root of context hierarchy
  3. Aug 22, 2013 5:14:17 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
  4. INFO: Loading XML bean definitions from class path resource [com/hadoop/basics/applicationContext.xml]
  5. Aug 22, 2013 5:14:17 PM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties
  6. INFO: Loading properties file from class path resource [resources/hadoop.properties]
  7. Aug 22, 2013 5:14:17 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
  8. INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  9. Aug 22, 2013 5:14:18 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  10. INFO: Starting job [wc-job]
  11. Aug 22, 2013 5:14:20 PM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
  12. INFO: Retrying connect to server: cloudx-843-770/172.25.37.135:9001. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  13. Aug 22, 2013 5:14:22 PM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
  14. INFO: Retrying connect to server: cloudx-843-770/172.25.37.135:9001. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  15. Aug 22, 2013 5:14:24 PM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
  16. INFO: Retrying connect to server: cloudx-843-770/172.25.37.135:9001. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  17. Aug 22, 2013 5:14:26 PM org.apache.hadoop.ipc.Client$Connection handleConnectionFailure
  18. INFO: Retrying connect to server: cloudx-843-770/172.25.37.135:9001. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
  19. Aug 22, 2013 5:14:36 PM org.apache.hadoop.security.UserGroupInformation doAs
  20. SEVERE: PriviledgedActionException as:bigdata via 298790 cause:org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  21. Aug 22, 2013 5:14:36 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  22. WARNING: Cannot start job [wc-job]
  23. org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  24.     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
  25.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  26.     at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source)
  27.     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
  28.     at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499)
  29.     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490)
  30.     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:473)
  31.     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
  32.     at java.security.AccessController.doPrivileged(Native Method)
  33.     at javax.security.auth.Subject.doAs(Unknown Source)
  34.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  35.     at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
  36.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
  37.     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
  38.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197)
  39.     at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
  40.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168)
  41.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160)
  42.     at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52)
  43.     at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44)
  44.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541)
  45.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
  46.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
  47.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  48.     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  49.     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  50.     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  51.     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  52.     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
  53.     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  54.     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  55.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:197)
  56.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:172)
  57.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:158)
  58.     at com.hadoop.basics.WordCounter.main(WordCounter.java:58)
  60. Aug 22, 2013 5:14:36 PM org.springframework.beans.factory.support.DefaultSingletonBeanRegistry destroySingletons
  61. INFO: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  62. Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'myjobs-runner': Invocation of init method failed; nested exception is java.lang.IllegalStateException: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  63.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1482)
  64.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
  65.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  66.     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  67.     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  68.     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  69.     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  70.     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
  71.     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  72.     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  73.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:197)
  74.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:172)
  75.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:158)
  76.     at com.hadoop.basics.WordCounter.main(WordCounter.java:58)
  77. Caused by: java.lang.IllegalStateException: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  78.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:209)
  79.     at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
  80.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168)
  81.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160)
  82.     at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52)
  83.     at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44)
  84.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541)
  85.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
  86.     ... 13 more
  87. Caused by: org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata
  88.     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
  89.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  90.     at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source)
  91.     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
  92.     at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499)
  93.     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490)
  94.     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:473)
  95.     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
  96.     at java.security.AccessController.doPrivileged(Native Method)
  97.     at javax.security.auth.Subject.doAs(Unknown Source)
  98.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  99.     at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
  100.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
  101.     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
  102.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197)
  103.     ... 20 more
hadoopspring-dataspring-data-hadoopspring-integration

1条答案

按热度按时间
rhfm7lfc

rhfm7lfc1#

如果您希望将一个用户模拟成另一个用户，apachehadoop要求您的服务端配置允许这样做。
这意味着，如果您以“foo”身份运行，并且希望实际以“bar”身份提交作业，那么您的namenode/jobtracker要求其加载的core-site.xml配置允许“foo”代理其他用户，并且通常在namenode/jobtracker的core-site.xml中必须存在如下内容：

  1. <property>
  2.   <name>hadoop.proxyuser.foo.groups</name>
  3.   <value>*</value>
  4. </property>
  6. <property>
  7.   <name>hadoop.proxyuser.foo.hosts</name>
  8.   <value>*</value>
  9. </property>

这将允许用户foo模拟任何其他用户（*对于组），并在从任何主机提交时这样做（*对于主机）。
虽然这不是必需的，但是明确建议定义的用户和组需要存在于namenode上，以便权限、组解析等以适当的方式工作。更多信息请参见：http://www.cloudera.com/blog/2012/03/authorization-and-authentication-in-hadoop/
文档位于http://static.springsource.org/spring-hadoop/docs/1.0.x/reference/html/security.html#security：kerberos可以更清楚地解决这个问题。

展开查看全部
赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com