flink运行python文件失败,错误为“pyflink不支持3.5之前的python版本”

h7appiyu  于 2021-06-26  发布在  Flink
关注(0)|答案(1)|浏览(545)

5月1日增加:
我在apache flink的jira上看到了一个关于这个错误的问题,也许有帮助?
我的系统是centos7,python版本3.6.8,pyflink版本1.10.0;
我遵循本教程并尝试运行pyflink文件;但我不断得到下面的错误;
我试过用-pyarch和-pyexec venv.zip/venv/bin/python3选项来运行它,但是没用。
我还将t_env.get_config().set_python_executable(“python3”)添加到py文件中,但仍然存在相同的错误:

[root@localhost pyflink]# flink run -m  localhost:8081  -pyarch venv.zip -pyexec venv.zip/venv/bin/python3  -py test_split_label.py 
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/wyl/test_file/pyflink/test_split_label.py", line 58, in <module>
    from pyflink.datastream import StreamExecutionEnvironment
  File "/tmp/pyflink/89713583-7229-4c30-93c9-4d543c360437/bd1f0746-4704-44e2-ba9e-3ef2bea03354pyflink.zip/pyflink/__init__.py", line 23, in <module>
RuntimeError: Python versions prior to 3.5 are not supported for PyFlink [sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)].
org.apache.flink.client.program.OptimizerPlanEnvironment$ProgramAbortException
    at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
    at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)

我的默认python版本已设置为python3.6:

[root@localhost pyflink]# python
Python 3.6.8 (default, Aug  7 2019, 17:28:10) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)

似乎我应该设置一些环境变量?但我不知道我应该设置什么名字和路径。现在我有一个$flink\u的家。
非常感谢您的帮助。

smdnsysy

smdnsysy1#

这是因为您使用的客户端的python环境是2.7。什么api set_python_executable do正在集群端设置python可执行文件。因此,如果您的pyflink版本是1.10,那么您需要在3.5+python环境中使用virtualenv,您可以参考文档https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/python/common_questions.html#execute-pyflink作业与python虚拟环境。在pyflink1.11中,您可以直接配置 PYFLINK_CLIENT_EXECUTABLE 在客户端分离python可执行文件。

相关问题