我尝试在Dataproc Serverless Spark pyspark作业中使用ElasticSearch包。我只在Dataproc Serverless中遇到这个包的问题。

import os
print("Current dir:", os.getcwd())
print("Current dir list:", os.listdir('.'))
 
import pandas
import statsmodels
import platform
import numpy
 
print("Python version:", platform.python_version())
print("Pandas version:", pandas.__version__)
print("statsmodel version:", statsmodels.__version__ )
print("Numpy version:", numpy.__version__)

#import elasticsearch as es
from elasticsearch import Elasticsearch as es
print("elasticsearch version:", es.__version__ )

下面是这段代码的输出。

Current dir: /tmp/srvls-batch-7554fe27-4044-4341-ae79-ffe9488ea385
Current dir list: ['pyspark_venv.tar.gz', '.test_sls.py.crc', 'test_sls.py']
Python version: 3.9.15
Pandas version: 1.4.4
statsmodel version: 0.13.5
Numpy version: 1.21.6
Traceback (most recent call last):
  File "/tmp/srvls-batch-7554fe27-4044-4341-ae79-ffe9488ea385/test_sls.py", line 16, in <module>
    from elasticsearch import Elasticsearch as es
ModuleNotFoundError: No module named 'elasticsearch'

我按照下面的步骤来设置venv的这项工作，
https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-virtualenv
并使用--archives选项。有没有人可以纠正我，如果我错过了什么。先谢谢你了