无法使用sqoop导入数据

ht4b089n  于 2021-06-02  发布在  Hadoop
关注(0)|答案(3)|浏览(548)

我想使用sqoop将数据从mysql导入远程配置单元。我已经在中间件机器上安装了sqoop。运行此命令时:

  1. sqoop import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://192.168.2.146:3306/fir --username root -P -m 1 --table beard_size_list --connect jdbc:hive2://192.168.2.141:10000/efir --username oracle -P -m 1 --hive-table lnd_beard_size_list --hive-import;

这个命令正确吗?我可以将数据从远程mysql导入远程配置单元吗?
当我运行此命令时,它会继续尝试连接到资源管理器:

  1. 17/11/01 10:54:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.1.0-129
  2. Enter password:
  3. 17/11/01 10:54:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
  4. for output. You can override
  5. 17/11/01 10:54:10 INFO tool.BaseSqoopTool: delimiters with --fields-
  6. terminated-by, etc.
  7. 17/11/01 10:54:10 WARN sqoop.ConnFactory: Parameter --driver is set to an
  8. explicit driver however appropriate connection manager is not being set (via
  9. --connection-manager). Sqoop is going to fall back to
  10. org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which
  11. connection manager should be used next time.
  12. 17/11/01 10:54:10 INFO manager.SqlManager: Using default fetchSize of 1000
  13. 17/11/01 10:54:10 INFO tool.CodeGenTool: Beginning code generation
  14. 17/11/01 10:54:11 INFO manager.SqlManager: Executing SQL statement: SELECT
  15. t.* FROM beard_size_list AS t WHERE 1=0
  16. 17/11/01 10:54:11 INFO manager.SqlManager: Executing SQL statement: SELECT
  17. t.* FROM beard_size_list AS t WHERE 1=0
  18. 17/11/01 10:54:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
  19. /usr/hdp/2.6.1.0-129/hadoop-mapreduce
  20. Note: /tmp/sqoop-
  21. oracle/compile/d93080265a09913fbfe9e06e92d314a3/beard_size_list.java uses or
  22. overrides a deprecated API.
  23. Note: Recompile with -Xlint:deprecation for details.
  24. 17/11/01 10:54:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-
  25. oracle/compile/d93080265a09913fbfe9e06e92d314a3/beard_size_list.jar
  26. 17/11/01 10:54:15 INFO mapreduce.ImportJobBase: Beginning import of
  27. beard_size_list
  28. 17/11/01 10:54:15 INFO Configuration.deprecation: mapred.jar is deprecated.
  29. Instead, use mapreduce.job.jar
  30. 17/11/01 10:54:15 INFO manager.SqlManager: Executing SQL statement: SELECT
  31. t.* FROM beard_size_list AS t WHERE 1=0
  32. 17/11/01 10:54:17 INFO Configuration.deprecation: mapred.map.tasks is
  33. deprecated. Instead, use mapreduce.job.maps
  34. 17/11/01 10:54:17 INFO client.RMProxy: Connecting to ResourceManager at
  35. hortonworksn2.com/192.168.2.191:8050
  36. 17/11/01 10:54:17 INFO client.AHSProxy: Connecting to Application History
  37. server at hortonworksn2.com/192.168.2.191:10200
  38. 17/11/01 10:54:19 INFO ipc.Client: Retrying connect to server:
  39. hortonworksn2.com/192.168.2.191:8050. Already tried 0 time(s); retry policy
  40. is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
  41. MILLISECONDS)
  42. 17/11/01 10:54:20 INFO ipc.Client: Retrying connect to server:
  43. hortonworksn2.com/192.168.2.191:8050. Already tried 1 time(s); retry policy
  44. is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
  45. MILLISECONDS)
  46. 17/11/01 10:54:21 INFO ipc.Client: Retrying connect to server:
  47. hortonworksn2.com/192.168.2.191:8050. Already tried 2 time(s); retry policy
  48. is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
  49. MILLISECONDS)
  50. 17/11/01 10:54:22 INFO ipc.Client: Retrying connect to server:
  51. hortonworksn2.com/192.168.2.191:8050. Already tried 3 time(s); retry policy
  52. is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
  53. MILLISECONDS)
  54. 17/11/01 10:54:23 INFO ipc.Client: Retrying connect to server:
  55. hortonworksn2.com/192.168.2.191:8050. Already tried 4 time(s); retry policy
  56. is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
  57. MILLISECONDS)

它试图连接的端口是8050,但实际的端口是8033。我怎样才能解决这个问题?

ibps3vxo

ibps3vxo1#

为什么在代码中添加了两次-connect语句?尝试以下代码: sqoop import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://192.168.2.146:3306/fir --username root -P -m 1 --split-by beard_size_list_table_primary_key --table beard_size_list --target-dir /user/data/raw/beard_size_list --fields-terminated-by "," --hive-import --create-hive-table --hive-table dbschema.beard_size_list ###注:
创建配置单元表–确定如果配置单元表已存在,则设置作业是否会失败。在这种情况下,如果您创建了hiveexternaltable并设置了targetdir路径,它将起作用

kgqe7b3p

kgqe7b3p2#

请尝试以下命令:
sqoop导入--驱动程序com.mysql.jdbc.driver--连接jdbc:mysql用法://192.168.2.146:3306/fir—用户名root-p-m1—表格大小\u列表;

x6h2sr28

x6h2sr283#

请检查以下属性是否设置为 yarn-site.xml 正确地

  1. <name>yarn.resourcemanager.address</name>
  2. <value>192.168.2.191:8033</value>

相关问题