hadoop(伪分布式操作)_大数据知识库

我想在archlinux中使用hadoop，所以我在这里尝试了(https://wiki.archlinux.org/index.php/hadoop)，但我能做到。
我试过。。。
从aur安装hadoop（hadoop 3.0.0-1）
权限更改使用 $sudo chmod o+r /etc/hadoop && sudo chmod o+rw /tmp/hadoop/log ... 将此代码添加到
/etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/lib/jvm/java-8-openjdk 在/etc/conf.d/hadoop中更改名称 HADOOP_USERNAME="<my user name>" （我可以ssh到localhost而不需要密码短语，所以我没有为ssh做任何事情）
尝试 hadoop namenode -format 我收到一些温暖的信息

WARNING: HADOOP_SLAVES has been replaced by HADOOP_WORKERS. Using value of HADOOP_SLAVES.
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

所以，我改变了 /etc/profile.d/hadoop.sh 如下所示

export HADOOP_CONF_DIR=/etc/hadoop  
export HADOOP_LOG_DIR=/tmp/hadoop/log  
export HADOOP_WORKERS=/etc/hadoop/workers  
export HADOOP_PID_DIR=/tmp/hadoop/run

我知道 hadoop namenode -format 再次（我的shell中找不到hdfs）
关闭我的hadoop我收到了这些信息

[<my user name>@<pc name> etc]$ hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

2017-12-29 05:08:01,679 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = <my user name>/10.0.2.15
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.0.0
STARTUP_MSG:   classpath =  ...
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z
STARTUP_MSG:   java = 1.8.0_144

************************************************************/

2017-12-29 05:08:01,685 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2017-12-29 05:08:01,688 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-d2e0965e-76ff-456b-8552-9da5706b8a6b
2017-12-29 05:08:02,355 INFO namenode.FSEditLog: Edit logging is async:true
2017-12-29 05:08:02,389 INFO namenode.FSNamesystem: KeyProvider: null
2017-12-29 05:08:02,391 INFO namenode.FSNamesystem: fsLock is fair: true
2017-12-29 05:08:02,392 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: fsOwner             = <my user name> (auth:SIMPLE)
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: supergroup          = supergroup
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: isPermissionEnabled = true
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: HA Enabled: false
2017-12-29 05:08:02,452 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
 2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
 2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
 2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Dec 29 05:08:02
 2017-12-29 05:08:02,488 INFO util.GSet: Computing capacity for map BlocksMap
 2017-12-29 05:08:02,488 INFO util.GSet: VM type       = 64-bit
 2017-12-29 05:08:02,490 INFO util.GSet: 2.0% max memory 878.5 MB = 17.6 MB
 2017-12-29 05:08:02,494 INFO util.GSet: capacity      = 2^21 = 2097152 entries
 2017-12-29 05:08:02,546 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
 2017-12-29 05:08:02,555 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: defaultReplication         = 3
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplication             = 512
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: minReplication             = 1
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
 2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
 2017-12-29 05:08:02,625 INFO util.GSet: Computing capacity for map INodeMap
 2017-12-29 05:08:02,626 INFO util.GSet: VM type       = 64-bit
 2017-12-29 05:08:02,626 INFO util.GSet: 1.0% max memory 878.5 MB = 8.8 MB
 2017-12-29 05:08:02,626 INFO util.GSet: capacity      = 2^20 = 1048576 entries
 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: ACLs enabled? false
 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
 2017-12-29 05:08:02,627 INFO namenode.FSDirectory: XAttrs enabled? true
 2017-12-29 05:08:02,627 INFO namenode.NameNode: Caching file names occurring more than 10 times
 2017-12-29 05:08:02,631 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true
 2017-12-29 05:08:02,643 INFO util.GSet: Computing capacity for map cachedBlocks
 2017-12-29 05:08:02,643 INFO util.GSet: VM type       = 64-bit
 2017-12-29 05:08:02,643 INFO util.GSet: 0.25% max memory 878.5 MB = 2.2 MB
 2017-12-29 05:08:02,643 INFO util.GSet: capacity      = 2^18 = 262144 entries
 2017-12-29 05:08:02,649 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
 2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
 2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
 2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
 2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
  2017-12-29 05:08:02,685 INFO util.GSet: Computing capacity for map NameNodeRetryCache
  2017-12-29 05:08:02,685 INFO util.GSet: VM type       = 64-bit
 2017-12-29 05:08:02,685 INFO util.GSet: 0.029999999329447746% max memory 878.5 MB = 269.9 KB
 2017-12-29 05:08:02,686 INFO util.GSet: capacity      = 2^15 = 32768 entries
 Re-format filesystem in Storage Directory /tmp/hadoop-elect/dfs/name ? (Y or N) Y
 2017-12-29 05:08:04,846 INFO namenode.FSImage: Allocated new BlockPoolId: BP-142461510-10.0.2.15-1514491684838
 2017-12-29 05:08:04,846 INFO common.Storage: Will remove files: [/tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000, /tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000.md5, /tmp/hadoop-<my user name>/dfs/name/current/seen_txid, /tmp/hadoop-<my user name>/dfs/name/current/VERSION]
2017-12-29 05:08:04,863 INFO common.Storage: Storage directory /tmp/hadoop-<my user name>/dfs/name has been successfully formatted.
2017-12-29 05:08:04,870 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
  2017-12-29 05:08:04,981 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 387 bytes saved in 0 seconds.
  2017-12-29 05:08:04,986 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
  2017-12-29 05:08:04,989 INFO namenode.NameNode: SHUTDOWN_MSG: 
  /************************************************************
  SHUTDOWN_MSG: Shutting down NameNode at <my user name>/10.0.2.15

************************************************************/

启动服务使用 sudo systemctl start hadoop-datanode(jobtracker etc...) 检查服务使用 sudo systemctl status hadoop-... ```
● hadoop-datanode.service - Hadoop Datanode
Loaded: loaded (/usr/lib/systemd/system/hadoop-datanode.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:29:15 JST; 53s ago
Process: 30450 ExecStart=/usr/bin/hadoop datanode (code=exited, status=1/FAILURE)
Main PID: 30450 (code=exited, status=1/FAILURE)

Dec 29 05:29:14 hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2728)
Dec 29 05:29:14 hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2872)
Dec 29 05:29:14 hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2896)
Dec 29 05:29:14 hadoop[30450]: 2017-12-29 05:29:14,859 INFO util.ExitUtil: Exiting with status 1: java.io.IOException: Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Dec 29 05:29:14 hadoop[30450]: 2017-12-29 05:29:14,865 INFO datanode.DataNode: SHUTDOWN_MSG:
Dec 29 05:29:14 hadoop[30450]:
/************************************************************
Dec 29 05:29:14 hadoop[30450]: SHUTDOWN_MSG: Shutting down DataNode at elect/10.0.2.15
Dec 29 05:29:14 hadoop[30450]:

************************************************************/

Dec 29 05:29:15 elect systemd[1]: hadoop-datanode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:29:15 systemd[1]: hadoop-datanode.service: Failed with result 'exit-code'.

● hadoop-namenode.service - Hadoop Namenode
Loaded: loaded (/usr/lib/systemd/system/hadoop-namenode.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:29:13 JST; 10min ago
Process: 30449 ExecStart=/usr/bin/hadoop namenode (code=exited, status=1/FAILURE)
Main PID: 30449 (code=exited, status=1/FAILURE)

Dec 29 05:29:13 hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:929)
Dec 29 05:29:13 hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
Dec 29 05:29:13 hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
Dec 29 05:29:13 hadoop[30449]: 2017-12-29 05:29:13,366 INFO util.ExitUtil: Exiting with status 1: java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
Dec 29 05:29:13 hadoop[30449]: 2017-12-29 05:29:13,368 INFO namenode.NameNode: SHUTDOWN_MSG:
Dec 29 05:29:13 hadoop[30449]:
/************************************************************
Dec 29 05:29:13 hadoop[30449]: SHUTDOWN_MSG: Shutting down NameNode at /10.0.2.15
Dec 29 05:29:13 hadoop[30449]:

************************************************************/

Dec 29 05:29:13 systemd[1]: hadoop-namenode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:29:13 systemd[1]: hadoop-namenode.service: Failed with result 'exit-code'.

● hadoop-jobtracker.service - Hadoop Jobtracker
Loaded: loaded (/usr/lib/systemd/system/hadoop-jobtracker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:26:04 JST; 17min ago
Process: 29726 ExecStart=/usr/bin/hadoop jobtracker (code=exited, status=1/FAILURE)
Main PID: 29726 (code=exited, status=1/FAILURE)

Dec 29 05:26:04 hadoop[29726]: queue get information regarding JobQueues
Dec 29 05:26:04 hadoop[29726]: sampler sampler
Dec 29 05:26:04 hadoop[29726]: streaming launch a mapreduce streaming job
Dec 29 05:26:04 hadoop[29726]: version print the version
Dec 29 05:26:04 hadoop[29726]: Daemon Commands:
Dec 29 05:26:04 hadoop[29726]: tput: No value for $TERM and no -T specified
Dec 29 05:26:04 systemd[1]: hadoop-jobtracker.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:26:04 hadoop[29726]: historyserver run job history servers as a standalone daemon
Dec 29 05:26:04 hadoop[29726]: SUBCOMMAND may print help when invoked w/o parameters or with -h.
Dec 29 05:26:04 systemd[1]: hadoop-jobtracker.service: Failed with result 'exit-code'.

● hadoop-secondarynamenode.service - Hadoop Secondary Namenode
Loaded: loaded (/usr/lib/systemd/system/hadoop-secondarynamenode.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:25:59 JST; 20min ago
Process: 29433 ExecStart=/usr/bin/hadoop secondarynamenode (code=exited, status=1/FAILURE)
Main PID: 29433 (code=exited, status=1/FAILURE)

Dec 29 05:25:59 hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:231)
Dec 29 05:25:59 hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:194)
Dec 29 05:25:59 hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:690)
Dec 29 05:25:59 hadoop[29433]: 2017-12-29 05:25:59,387 INFO util.ExitUtil: Exiting with status 1: ExitException
Dec 29 05:25:59 hadoop[29433]: 2017-12-29 05:25:59,400 INFO namenode.SecondaryNameNode: SHUTDOWN_MSG:
Dec 29 05:25:59 hadoop[29433]:
/************************************************************
Dec 29 05:25:59 hadoop[29433]: SHUTDOWN_MSG: Shutting down
SecondaryNameNode at /10.0.2.15
Dec 29 05:25:59 hadoop[29433]:

************************************************************/

Dec 29 05:25:59 systemd[1]: hadoop-secondarynamenode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:25:59 systemd[1]: hadoop-secondarynamenode.service: Failed with result 'exit-code'.

● hadoop-tasktracker.service - Hadoop Tasktracker
Loaded: loaded (/usr/lib/systemd/system/hadoop-tasktracker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:26:04 JST; 23min ago
Process: 29727 ExecStart=/usr/bin/hadoop tasktracker (code=exited, status=1/FAILURE)
Main PID: 29727 (code=exited, status=1/FAILURE)

Dec 29 05:26:04 hadoop[29727]: queue get information regarding JobQueues
Dec 29 05:26:04 hadoop[29727]: sampler sampler
Dec 29 05:26:04 hadoop[29727]: streaming launch a mapreduce streaming job
Dec 29 05:26:04 hadoop[29727]: version print the version
Dec 29 05:26:04 hadoop[29727]: Daemon Commands:
Dec 29 05:26:04 hadoop[29727]: tput: No value for $TERM and no -T specified
Dec 29 05:26:04 hadoop[29727]: historyserver run job history servers as a standalone daemon
Dec 29 05:26:04 hadoop[29727]: SUBCOMMAND may print help when invoked w/o parameters or with -h.
Dec 29 05:26:04 systemd[1]: hadoop-tasktracker.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:26:04 systemd[1]: hadoop-tasktracker.service: Failed with result 'exit-code'.

我找不到这个解决办法
　　 请告诉我解决办法。

hadoop(伪分布式操作)

暂无答案！

相关问题

热门标签

最新问答