如何利用/导入flink的testharness类和bazel?

5gfr0r5j  于 2021-06-26  发布在  Flink
关注(0)|答案(1)|浏览(395)

我的直觉可能是正确的使用 Package 和/或分类器。为了找到构建文件的正确导入标记,我建议访问 .cache/bazel/bazel_user/.../external/maven/BUILD file 将文章编辑到工作设置中
我想使用bazel构建工具测试我自己的async flink stream操作符richayncFunction。
这与如何使用flink的testharness类基本相同?
主要的问题是我找不到进口 org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness 以及测试蒸汽操作员所需的所有其他代码。
我试图遵循提供的答案,但我与bazel语法斗争,问自己是否有可能在bazel中使用这些导入。
我正在使用最新的bazel版本和intellij2019.3.4以及bazel插件。
链接的答案假设maven依赖:

  1. <dependency>
  2. <groupId>org.apache.flink</groupId>
  3. <artifactId>flink-test-utils-junit</artifactId>
  4. <version>${flink.version}</version>
  5. </dependency>
  6. <dependency>
  7. <groupId>org.apache.flink</groupId>
  8. <artifactId>flink-streaming-java_2.11</artifactId>
  9. <version>${flink.version}</version>
  10. <scope>test</scope>
  11. <type>test-jar</type>
  12. </dependency>
  13. <dependency>
  14. <groupId>org.apache.flink</groupId>
  15. <artifactId>flink-runtime_2.11</artifactId>
  16. <version>${flink.version}</version>
  17. <scope>test</scope>
  18. <type>test-jar</type>
  19. </dependency>

我的工作区是这样的

  1. ...
  2. http_archive(
  3. name = "rules_jvm_external",
  4. sha256 = RULES_JVM_EXTERNAL_SHA,
  5. strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
  6. url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
  7. )
  8. load("@rules_jvm_external//:defs.bzl", "maven_install")
  9. load("@rules_jvm_external//:defs.bzl", "artifact")
  10. load("@rules_jvm_external//:specs.bzl", "maven")
  11. load("@io_grpc_grpc_java//:repositories.bzl", "IO_GRPC_GRPC_JAVA_ARTIFACTS")
  12. load("@io_grpc_grpc_java//:repositories.bzl", "IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS")
  13. maven_install(
  14. artifacts = [
  15. ...
  16. "org.apache.commons:commons-lang3:3.9",
  17. "org.javatuples:javatuples:1.2",
  18. "junit:junit:4.13",
  19. "org.testcontainers:testcontainers:1.14.1",
  20. "org.testcontainers:kafka:1.14.1",
  21. "org.testcontainers:postgresql:1.14.1",
  22. "commons-io:commons-io:2.6",
  23. "com.google.code.findbugs:jsr305:1.3.9",
  24. "com.google.errorprone:error_prone_annotations:2.0.18",
  25. "com.google.j2objc:j2objc-annotations:1.1",
  26. "com.google.protobuf:protobuf-java:3.11.1",
  27. "com.google.protobuf:protobuf-java-util:3.6.1",
  28. "info.picocli:picocli:4.1.0",
  29. "org.slf4j:slf4j-log4j12:1.7.5",
  30. "org.slf4j:slf4j-api:1.7.28",
  31. "com.github.jasync-sql:jasync-postgresql:1.0.11",
  32. "com.github.jasync-sql:jasync-common:1.0.11",
  33. "org.postgresql:postgresql:42.2.5",
  34. "org.mongodb:mongodb-driver-reactivestreams:4.0.2",
  35. "org.mongodb:mongodb-driver-core:4.0.2",
  36. "org.mongodb:bson:4.0.2",
  37. "org.reactivestreams:reactive-streams:1.0.3",
  38. "joda-time:joda-time:2.9.7",
  39. "org.apache.kafka:kafka-clients:2.4.0",
  40. # "io.grpc:grpc-netty-shaded:%s" % GRPC_JAVA_VERSION,
  41. # "io.grpc:grpc-protobuf:%s" % GRPC_JAVA_VERSION,
  42. # "io.grpc:grpc-stub:%s" % GRPC_JAVA_VERSION,
  43. "org.apache.flink:flink-core:%s" % FLINK_VERSION,
  44. "org.apache.flink:flink-java:%s" % FLINK_VERSION,
  45. "org.apache.flink:flink-streaming-java_%s:%s" % (SCALA_VERSION, FLINK_VERSION),
  46. "org.apache.flink:flink-connector-kafka-0.11_%s:%s" % (SCALA_VERSION, FLINK_VERSION),
  47. "org.apache.flink:flink-cep_2.11:%s" % FLINK_VERSION,
  48. ] + IO_GRPC_GRPC_JAVA_ARTIFACTS,
  49. generate_compat_repositories = True,
  50. override_targets = IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS,
  51. repositories = [
  52. "https://jcenter.bintray.com/",
  53. "https://maven.google.com",
  54. "https://repo1.maven.org/maven2",
  55. ],
  56. )
  57. maven_install(
  58. name = "testing",
  59. artifacts = [
  60. maven.artifact(
  61. group = "org.apache.flink",
  62. artifact = "flink-runtime_2.11",
  63. version = FLINK_VERSION,
  64. classifier = "tests",
  65. packaging = "test-jar",
  66. ),
  67. maven.artifact(
  68. group = "org.apache.flink",
  69. artifact = "flink-streaming-java_2.11",
  70. version = FLINK_VERSION,
  71. classifier = "tests",
  72. packaging = "test-jar",
  73. ),
  74. maven.artifact(
  75. group = "org.apache.flink",
  76. artifact = "flink-test-utils-junit",
  77. version = FLINK_VERSION,
  78. ),
  79. "org.apache.flink:flink-tests:%s" % FLINK_VERSION,
  80. ],
  81. repositories = [
  82. "https://jcenter.bintray.com/",
  83. "https://maven.google.com",
  84. "https://repo1.maven.org/maven2",
  85. ],
  86. )
  87. load("@maven//:compat.bzl", "compat_repositories")
  88. compat_repositories()
  89. ...

测试maven安装“group”是我试图弄清楚如何导入所需的依赖项,因为据我所知,bazelmaven语法中没有显式的“type”和“scope”。
测试生成文件如下所示

  1. load("@rules_java//java:defs.bzl", "java_test")
  2. load("@rules_jvm_external//:specs.bzl", "maven")
  3. load("@rules_jvm_external//:defs.bzl", "artifact", "maven_install")
  4. java_test(
  5. name = "some_module",
  6. size = "medium",
  7. srcs = ["DataOperatorTests.java"],
  8. tags = [
  9. "docker",
  10. "integration",
  11. ],
  12. test_class = "org.some.project.DataOperatorTests",
  13. deps = [
  14. "@maven//:com_google_protobuf_protobuf_java",
  15. "@maven//:junit_junit_4_13",
  16. "@maven//:org_apache_flink_flink_connector_kafka_0_11_2_11",
  17. "@maven//:org_apache_flink_flink_core",
  18. "@maven//:org_apache_flink_flink_java",
  19. "@maven//:org_javatuples_javatuples",
  20. "@maven//:org_mongodb_bson",
  21. "@maven//:org_mongodb_mongodb_driver_core",
  22. "@maven//:org_mongodb_mongodb_driver_reactivestreams",
  23. "@maven//:org_testcontainers_kafka",
  24. "@maven//:org_testcontainers_testcontainers",
  25. "@maven//:org_apache_flink_flink_streaming_java_2_11",
  26. "@testing//:org_apache_flink_flink_runtime_2_11_tests",
  27. "@testing//:org_apache_flink_flink_streaming_java_2_11_tests",
  28. "@testing//:org_apache_flink_flink_test_utils_junit",
  29. "@testing//:org_apache_flink_flink_tests",
  30. ],
  31. )
  32. test_suite(name = "smoke",tags = ["-docker","-internal",],)
  33. test_suite(name = "unit",tags = ["-internal","unit",],)
  34. test_suite(name = "integration",tags = ["-internal","integration",],)
  35. test_suite(name = "internal")

通过上述设置,以下错误不再发生-留作参考
这些包似乎存在,但在实际的测试构建文件中,我在同步过程中遇到了这个错误。

  1. Error:(5, 1) no such target '@testing//:org_apache_flink_flink_streaming_java_2_11': target 'org_apache_flink_flink_streaming_java_2_11' not declared in package '' defined by /home/user/.cache/bazel/_bazel_user/7b62e4e31c70ee640c6d33972433da28/external/testing/BUILD and referenced by '//core/src/test/java/org/some/project/some_module:some_module'
zysjyyx4

zysjyyx41#

我自己解决了这个问题。将上述代码调整为合适的解决方案。
总而言之。
当maven存储库是这样给出的:

  1. <dependency>
  2. <groupId>org.apache.flink</groupId>
  3. <artifactId>flink-test-utils-junit</artifactId>
  4. <version>${flink.version}</version>
  5. </dependency>
  6. <dependency>
  7. <groupId>org.apache.flink</groupId>
  8. <artifactId>flink-streaming-java_2.11</artifactId>
  9. <version>${flink.version}</version>
  10. <scope>test</scope>
  11. <type>test-jar</type>
  12. </dependency>
  13. <dependency>
  14. <groupId>org.apache.flink</groupId>
  15. <artifactId>flink-runtime_2.11</artifactId>
  16. <version>${flink.version}</version>
  17. <scope>test</scope>
  18. <type>test-jar</type>
  19. </dependency>

相应的bazel工作区文件必须包含如下条目:

  1. load("@rules_jvm_external//:defs.bzl", "maven_install")
  2. load("@rules_jvm_external//:defs.bzl", "artifact")
  3. load("@rules_jvm_external//:specs.bzl", "maven")
  4. maven_install(
  5. ...
  6. artifacts = [
  7. maven.artifact(
  8. group = "org.apache.flink",
  9. artifact = "flink-runtime_2.11",
  10. version = FLINK_VERSION,
  11. classifier = "tests",
  12. packaging = "test-jar",
  13. ),
  14. maven.artifact(
  15. group = "org.apache.flink",
  16. artifact = "flink-streaming-java_2.11",
  17. version = FLINK_VERSION,
  18. classifier = "tests",
  19. packaging = "test-jar",
  20. ),
  21. maven.artifact(
  22. group = "org.apache.flink",
  23. artifact = "flink-test-utils-junit",
  24. version = FLINK_VERSION,
  25. ),
  26. "org.apache.flink:flink-tests:%s" % FLINK_VERSION,
  27. ],
  28. repositories = [
  29. "https://jcenter.bintray.com/",
  30. "https://maven.google.com",
  31. "https://repo1.maven.org/maven2",
  32. ],
  33. )

有关依赖关系的详细说明,请参见:https://github.com/bazelbuild/rules_jvm_external#detailed-依赖信息规范和此处:https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/api.md#mavenartifact
要在构建文件中找到要导入的结果目标,请使用@jin提到的方法

  1. bazel query @maven//:all | grep flink_flink_streaming

在大多数情况下,目标可能只有后缀 _tests 以下是生成文件的必要导入:

  1. load("@rules_jvm_external//:specs.bzl", "maven")
  2. load("@rules_jvm_external//:defs.bzl", "artifact", "maven_install")
  3. java_test(
  4. name = "some_module",
  5. size = "medium",
  6. srcs = ["DataOperatorTests.java"],
  7. tags = [
  8. "docker",
  9. "integration",
  10. ],
  11. test_class = "org.some.project.DataOperatorTests",
  12. deps = [
  13. ...
  14. "@maven//:org_apache_flink_flink_runtime_2_11_tests",
  15. "@maven//:org_apache_flink_flink_streaming_java_2_11_tests",
  16. "@maven//:org_apache_flink_flink_test_utils_junit",
  17. "@maven//:org_apache_flink_flink_tests",
  18. ],
  19. )
展开查看全部

相关问题