java—在本地模式下运行一个简单的级联应用程序

pxy2qtax  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(339)

我对cascading/hadoop不太熟悉,正在尝试以本地模式(即在内存中)运行一个简单的示例。该示例仅复制一个文件:

  1. import java.util.Properties;
  2. import cascading.flow.Flow;
  3. import cascading.flow.FlowConnector;
  4. import cascading.flow.FlowDef;
  5. import cascading.flow.local.LocalFlowConnector;
  6. import cascading.pipe.Pipe;
  7. import cascading.property.AppProps;
  8. import cascading.scheme.hadoop.TextLine;
  9. import cascading.tap.Tap;
  10. import cascading.tap.hadoop.Hfs;
  11. public class CascadingTest {
  12. public static void main(String[] args) {
  13. Properties properties = new Properties();
  14. AppProps.setApplicationJarClass( properties, CascadingTest.class );
  15. FlowConnector flowConnector = new LocalFlowConnector();
  16. // create the source tap
  17. Tap inTap = new Hfs( new TextLine(), "D:\\git_workspace\\Impatient\\part1\\data\\rain.txt" );
  18. // create the sink tap
  19. Tap outTap = new Hfs( new TextLine(), "D:\\git_workspace\\Impatient\\part1\\data\\out.txt" );
  20. // specify a pipe to connect the taps
  21. Pipe copyPipe = new Pipe( "copy" );
  22. // connect the taps, pipes, etc., into a flow
  23. FlowDef flowDef = FlowDef.flowDef()
  24. .addSource( copyPipe, inTap )
  25. .addTailSink( copyPipe, outTap );
  26. // run the flow
  27. Flow flow = flowConnector.connect( flowDef );
  28. flow.complete();
  29. }
  30. }

下面是我得到的错误:

  1. 09-25-12 11:30:38,114 INFO - AppProps - using app.id: 9C82C76AC667FDAA2F6969A0DF3949C6
  2. Exception in thread "main" cascading.flow.planner.PlannerException: could not build flow from assembly: [java.util.Properties cannot be cast to org.apache.hadoop.mapred.JobConf]
  3. at cascading.flow.planner.FlowPlanner.handleExceptionDuringPlanning(FlowPlanner.java:515)
  4. at cascading.flow.local.planner.LocalPlanner.buildFlow(LocalPlanner.java:84)
  5. at cascading.flow.FlowConnector.connect(FlowConnector.java:454)
  6. at com.x.y.CascadingTest.main(CascadingTest.java:37)
  7. Caused by: java.lang.ClassCastException: java.util.Properties cannot be cast to org.apache.hadoop.mapred.JobConf
  8. at cascading.tap.hadoop.Hfs.sourceConfInit(Hfs.java:78)
  9. at cascading.flow.local.LocalFlowStep.initTaps(LocalFlowStep.java:77)
  10. at cascading.flow.local.LocalFlowStep.getInitializedConfig(LocalFlowStep.java:56)
  11. at cascading.flow.local.LocalFlowStep.createFlowStepJob(LocalFlowStep.java:135)
  12. at cascading.flow.local.LocalFlowStep.createFlowStepJob(LocalFlowStep.java:38)
  13. at cascading.flow.planner.BaseFlowStep.getFlowStepJob(BaseFlowStep.java:588)
  14. at cascading.flow.BaseFlow.initializeNewJobsMap(BaseFlow.java:1162)
  15. at cascading.flow.BaseFlow.initialize(BaseFlow.java:184)
  16. at cascading.flow.local.planner.LocalPlanner.buildFlow(LocalPlanner.java:78)
  17. ... 2 more
j2cgzkjk

j2cgzkjk1#

欢迎来到级联-
我刚刚回答了级联用户列表,但简而言之,问题是本地和hadoop模式类的混合。。此代码具有localflowconnector,但随后使用hfs抽头。
当我回到“不耐烦”教程中使用的类时,它会正确运行:https://gist.github.com/3784194

oaxa6hgo

oaxa6hgo2#

更详细一点:不能在级联中混合本地类和hadoop类,因为它们假设不同且不兼容的环境。在您的例子中,您正在尝试使用hadoop taps创建一个本地流,后者需要hadoop JobConf 而不是 Properties 用于配置本地抽头的对象。
如果您使用 cascading.tap.local.FileTap 而不是 cascading.tap.hadoop.Hfs .

eoxn13cs

eoxn13cs3#

是的,您需要使用lfs(本地文件系统)tap而不是hfs(hadoop文件系统)。
您还可以在本地模式下/从eclipse使用junit测试用例(带有级联unittestjar)测试代码。
http://www.cascading.org/2012/08/07/cascading-for-the-impatient-part-6/

相关问题