cmake Confluent-Kafka多阶段构建问题

brqmpdu1  于 2024-01-09  发布在  Kafka
关注(0)|答案(1)|浏览(330)

我有一个多阶段的Docker构建，如下所示，

  1. # Use an Amazon Linux 2 base image
  2. FROM amazonlinux:2.0.20230418.0 AS builder
  4. # Install build tools and dependencies
  5. RUN yum install -y gcc-c++ make openssl-devel zlib-devel librdkafka-devel && yum groupinstall -y  "Development Tools"
  7. # Download librdkafka 2.2.0 source
  8. RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0 \
  9.     && ./configure --prefix /usr && make -j 3 && make install && ldconfig
  11. # Build librdkafka
  12. WORKDIR /usr/src/librdkafka-2.2.0
  13. RUN ./configure --prefix=/usr \
  14.     && make \
  15.     && make install
  17. # Final image
  18. FROM amazonlinux:2.0.20230418.0
  20. # Copy only librdkafka binaries from builder stage
  21. COPY --from=builder /usr/lib/librdkafka* /usr/lib/
  22. COPY --from=builder /usr/include/librdkafka /usr/include/librdkafka
  24. # Set LD_LIBRARY_PATH to include librdkafka library
  25. ENV LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH
  27. # Install Python and required packages
  28. RUN yum install -y python2.7 python-pip
  30. # Install confluent-kafka Python library
  31. RUN pip install confluent-kafka=2.2.0
  33. # Copy your application code
  34. COPY . /app
  35. WORKDIR /app
  37. # Command to run your application
  38. CMD ["python", "main.py"]

字符串
在我试图安装confluent-kafka后，从源代码构建librdkafka的一些原因，我总是结束了以下错误

  1. Agent pid 2036891
  2. Identity added: /home/john.doe/.ssh/id_rsa (john.doe@abc-corp)
  3. #1 [internal] load build definition from Dockerfile
  4. #1 sha256:d2adfb92a6221bec17bf4c12942bbff41d7b7280b2cf1b54bcb01e340ea55d1d
  5. #1 transferring dockerfile: 3.27kB done
  6. #1 DONE 0.0s
  7. # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  8. #REMVOED THESE LOGS AS IT WAS A SUCCESS
  9. # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  10. # 
  11. #
  12. #11 84.51 Complete!
  13. #11 DONE 85.3s
  15. #12 [builder 3/5] RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig
  17. #12 37.40 Turn off this advice by setting config variable advice.detachedHead to false
  18. #12 37.40 
  19. #12 37.40 HEAD is now at e75de5be Generates a random salt only when (#4350)
  20. #12 38.27 checking for OS or distribution... ok (amzn)
  21. #12 38.27 checking for C compiler from CC env... failed
  22. #12 38.28 checking for gcc (by command)... ok
  23. #12 38.29 checking for C++ compiler from CXX env... failed
  24. #12 38.30 checking for C++ compiler (g++)... ok
  25. #12 38.32 checking executable ld... ok
  26. #12 38.33 checking executable nm... ok
  27. #12 38.34 checking executable objdump... ok
  28. #12 38.35 checking executable strip... ok
  29. #12 38.36 checking executable libtool... ok
  30. #12 38.36 checking executable ranlib... ok
  31. #12 38.37 checking for pkgconfig (by command)... ok
  32. #12 38.38 checking for install (by command)...mklove/modules/configure.base: line 1181: which: command not found
  33. #12 38.38  failed
  34. #
  35. # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  36. # REMOVED A BUNCH OF LOGS HERE
  37. # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  38. #
  39. -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MetadataImpl.cpp -o MetadataImpl.o
  40. #12 63.97 Generating pkg-config file rdkafka++.pc
  41. #12 63.97 Generating pkg-config file rdkafka++-static.pc
  42. #12 64.36 Creating shared library librdkafka++.so.1
  43. #12 64.36 Creating static library librdkafka++.a
  44. #12 64.36 g++  -shared -Wl,-soname,librdkafka++.so.1 RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.so.1 -L../src -lrdkafka
  45. #12 64.36 ar rcs librdkafka++.a RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o
  46. #12 64.38 cp librdkafka++.a librdkafka++-dbg.a
  47. #12 64.46 cp librdkafka++.so.1 librdkafka++-dbg.so.1
  48. #12 64.47 Creating librdkafka++.so symlink
  49. #12 64.47 rm -f "librdkafka++.so" && ln -s "librdkafka++.so.1" "librdkafka++.so"
  50. #12 64.47 Checking librdkafka++ integrity
  51. #12 64.47 librdkafka++.so.1              OK
  52. #12 64.47 librdkafka++.a                 OK
  53. #12 64.47 make[1]: Leaving directory `/opt/librdkafka/src-cpp'
  54. #12 64.47 make -C examples
  55. #12 64.48 make[1]: Entering directory `/opt/librdkafka/examples'
  56. #12 64.48 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_example.c -o rdkafka_example  \
  57. #12 64.48       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  58. #12 64.48 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_performance.c -o rdkafka_performance  \
  59. #12 64.48       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  60. #12 64.48 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_example.cpp -o rdkafka_example_cpp  \
  61. #12 64.48       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  62. #12 64.88 # rdkafka_example is ready
  63. #12 64.89 #
  64. #12 64.89 # Run producer (write messages on stdin)
  65. #12 64.89 ./rdkafka_example -P -t <topic> -p <partition>
  66. #12 64.89 
  67. #12 64.89 # or consumer
  68. #12 64.89 ./rdkafka_example -C -t <topic> -p <partition>
  69. #12 64.89 
  70. #12 64.89 #
  71. #12 64.90 # More usage options:
  72. #12 64.90 ./rdkafka_example -h
  73. #12 64.90 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_complex_consumer_example.c -o rdkafka_complex_consumer_example  \
  74. #12 64.90       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  75. #12 65.08 # rdkafka_performance is ready
  76. #12 65.08 #
  77. #12 65.08 # Run producer
  78. #12 65.08 ./rdkafka_performance -P -t <topic> -p <partition> -s <msgsize>
  79. #12 65.08 
  80. #12 65.08 # or consumer
  81. #12 65.08 ./rdkafka_performance -C -t <topic> -p <partition>
  82. #12 65.08 
  83. #12 65.09 #
  84. #12 65.09 # More usage options:
  85. #12 65.09 ./rdkafka_performance -h
  86. #12 65.09 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_complex_consumer_example.cpp -o rdkafka_complex_consumer_example_cpp  \
  87. #12 65.09       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  88. #12 65.30 # rdkafka_complex_consumer_example is ready
  89. #12 65.31 #
  90. #12 65.31 ./rdkafka_complex_consumer_example <topic[:part]> <topic2[:part]> ..
  91. #12 65.31 
  92. #12 65.31 #
  93. #12 65.31 # More usage options:
  94. #12 65.31 ./rdkafka_complex_consumer_example -h
  95. #12 65.31 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp kafkatest_verifiable_client.cpp -o kafkatest_verifiable_client  \
  96. #12 65.31       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  97. #12 65.65 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src producer.c -o producer  \
  98. #12 65.65       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  99. #12 65.95 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src consumer.c -o consumer  \
  100. #12 65.95       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  101. #12 66.14 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src idempotent_producer.c -o idempotent_producer  \
  102. #12 66.14       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  103. #12 66.23 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src transactions.c -o transactions  \
  104. #12 66.23       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  105. #12 66.41 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src delete_records.c -o delete_records  \
  106. #12 66.41       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  107. #12 66.57 g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp openssl_engine_example.cpp -o openssl_engine_example_cpp  \
  108. #12 66.57       ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  109. #12 66.71 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src list_consumer_groups.c -o list_consumer_groups  \
  110. #12 66.71       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  111. #12 67.15 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src describe_consumer_groups.c -o describe_consumer_groups  \
  112. #12 67.15       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  113. #12 67.41 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src list_consumer_group_offsets.c -o list_consumer_group_offsets  \
  114. #12 67.41       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  115. #12 67.51 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src alter_consumer_group_offsets.c -o alter_consumer_group_offsets  \
  116. #12 67.51       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  117. #12 67.78 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src incremental_alter_configs.c -o incremental_alter_configs  \
  118. #12 67.78       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  119. #12 67.80 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src user_scram.c -o user_scram  \
  120. #12 67.80       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  121. #12 67.91 gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src misc.c -o misc  \
  122. #12 67.91       ../src/librdkafka.a -lm -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  123. #12 68.19 make[1]: Leaving directory `/opt/librdkafka/examples'
  124. #12 68.20 Updating CONFIGURATION.md
  125. #12 68.25 Checking  integrity
  126. #12 68.25 CONFIGURATION.md               OK
  127. #12 68.25 examples/rdkafka_example       OK
  128. #12 68.25 examples/rdkafka_performance   OK
  129. #12 68.25 examples/rdkafka_example_cpp   OK
  130. #12 68.46 make[1]: Entering directory `/opt/librdkafka/src'
  131. #12 68.46 Checking librdkafka integrity
  132. #12 68.47 librdkafka.so.1                OK
  133. #12 68.47 librdkafka.a                   OK
  134. #12 68.47 Symbol visibility              OK
  135. #12 68.47 make[1]: Leaving directory `/opt/librdkafka/src'
  136. #12 68.49 make[1]: Entering directory `/opt/librdkafka/src-cpp'
  137. #12 68.50 Generating pkg-config file rdkafka++-static.pc
  138. #12 68.50 Checking librdkafka++ integrity
  139. #12 68.50 librdkafka++.so.1              OK
  140. #12 68.50 librdkafka++.a                 OK
  141. #12 68.50 make[1]: Leaving directory `/opt/librdkafka/src-cpp'
  142. #12 68.54 make[1]: Entering directory `/opt/librdkafka/src'
  143. #12 68.54 Install librdkafka to /usr
  144. #12 68.54 d $DESTDIR/usr/include/librdkafka
  145. #12 68.54 /bin/sh: d: command not found
  146. #12 68.54 make[1]: [lib-install] Error 127 (ignored)
  147. #12 68.54 d $DESTDIR/usr/lib
  148. #12 68.54 /bin/sh: d: command not found
  149. #12 68.54 make[1]: [lib-install] Error 127 (ignored)
  150. #12 68.54 rdkafka.h rdkafka_mock.h $DESTDIR/usr/include/librdkafka
  151. #12 68.54 /bin/sh: rdkafka.h: command not found
  152. #12 68.54 make[1]: *** [lib-install] Error 127
  153. #12 68.54 make[1]: Leaving directory `/opt/librdkafka/src'
  154. #12 68.54 make: *** [install-subdirs] Error 2
  155. #12 ERROR: executor failed running [/bin/sh -c cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig]: exit code: 2
  156. ------
  157.  > [builder 3/5] RUN cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig:
  158. ------
  159. executor failed running [/bin/sh -c cd /opt/ && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && git checkout v2.2.0     && ./configure --prefix /usr && make -j 3 && make install && ldconfig]: exit code: 2
  161. kafka_test                                           Error on build
  163. job_done

型
任何帮助将不胜感激，谢谢

cmake

1条答案

按热度按时间
qq24tv8q

qq24tv8q1#

你的错误在这里

  1. #12 68.54 /bin/sh: d: command not found
  2. #12 68.54 make[1]: [lib-install] Error 127 (ignored)
  3. #12 68.54 d $DESTDIR/usr/lib
  4. #12 68.54 /bin/sh: d: command not found
  5. #12 68.54 make[1]: [lib-install] Error 127 (ignored)

字符串
这应该是cd命令，而不是d ...
你应该从YUM仓库安装，而不是从源代码克隆和构建。
否则，confluent-kafka-python会自动将其包含在二元轮中

赞(0）分享  回复(0）举报  2024-01-09
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com