我正试图用hadoop在macosx10.9.2上建立一个开发单节点集群。我尝试过各种在线教程,最近的一个就是这个。总结一下我所做的:
1) $ brew install hadoop
它在/usr/local/cellar/hadoop/2.2.0中安装了hadoop2.2.0
2) 已配置的环境变量。以下是我的.bash\u配置文件的相关部分:
### Java_HOME
export JAVA_HOME="$(/usr/libexec/java_home)"
### HADOOP Environment variables
export HADOOP_PREFIX="/usr/local/Cellar/hadoop/2.2.0"
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_CONF_DIR=$HADOOP_PREFIX/libexec/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export CLASSPATH=$CLASSPATH:.
export CLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/common/hadoop-common-2.2.0.jar
export CLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar
3) 配置的hdfs
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/Cellar/hadoop/2.2.0/hdfs/datanode</value>
<description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/Cellar/hadoop/2.2.0/hdfs/namenode</value>
<description>Path on the local filesystem where the NameNode stores the namespace and transaction logs persistently.</description>
</property>
</configuration>
3) 配置的core-site.xml
<!-- Let Hadoop modules know where the HDFS NameNode is at! -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost/</value>
<description>NameNode URI</description>
</property>
4) 已配置的站点.xml
<configuration>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
<description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
<description>Maximum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
<description>The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>2</value>
<description>The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value. </description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
<description>Physical memory, in MB, to be made available to running containers</description>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
<description>Number of CPU cores that can be allocated for containers.</description>
</property>
</configuration>
5) 然后我尝试使用以下格式格式化namenode:
$HADOOP_PREFIX/bin/hdfs namenode -format
这给了我一个错误:error:找不到或加载主类org.apache.hadoop.hdfs.server.namenode.namenode。
我查看了hdfs代码,运行它的行基本上相当于调用
$java org.apache.hadoop.hdfs.server.namenode.NameNode.
考虑到这是一个类路径问题,我尝试了一些方法
a) 将hadoop-common-2.2.0.jar和hadoop-hdfs-2.2.0.jar添加到类路径,如您在上面的my.bash\u profile脚本中看到的
b) 添加行
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
我的.bashèu配置文件(我后来删除了它,因为它似乎没有任何帮助)
c) 我还考虑编写一个shell脚本,将$hadoop\u home/libexec/share/hadoop中的每个jar添加到$hadoop\u类路径中,但这似乎是不必要的,而且很容易出现将来的问题。
你知道为什么我总是出现这样的错误吗:找不到或加载主类org.apache.hadoop.hdfs.server.namenode.namenode?提前谢谢。
3条答案
按热度按时间ctehm74n1#
由于brew包的布局方式,您需要将hadoop\前缀指向包中的libexec文件夹:
然后从conf目录的声明中删除libexec:
5fjcxozz2#
我也有同样的问题,那是因为权利的“根”。跑
hadoop
或者hdfs
命令sudo
如前所述:bksxznpy3#
尝试
$HADOOP_PREFIX/bin/hadoop namenode -format
而不是$hadoop\u prefix/bin/hdfs namenode-format