试试看

fs = pa.hdfs.connect(...)
fs.read_parquet('/path/to/hdfs-file', **other_options)

或

import pyarrow.parquet as pq
with fs.open(path) as f:
    pq.read_table(f, **read_options)

我打开了https://issues.apache.org/jira/browse/ARROW-1848，想添加一些更明确的文档

我通过Pydoop库和engine = pyarrow尝试了同样的方法，它对我来说非常有效。

!pip install pydoop pyarrow
import pydoop.hdfs as hd

#read files via Pydoop and return df

def readParquetFilesPydoop(path):
    with hd.open(path) as f:
        df = pd.read_parquet(f ,engine='pyarrow')
        logger.info ('file: ' +  path  +  ' : ' + str(df.shape))
        return df

你可以阅读和写作与pyarrow所描绘的接受的答案。然而，那里提供的API早就被弃用了，并且不适用于最新版本的Hadoop。用途：

from pyarrow import fs
import pyarrow.parquet as pq

# connect to hadoop
hdfs = fs.HadoopFileSystem('hostname', 8020) 

# will read single file from hdfs
with hdfs.open_input_file(path) as pqt:
     df = pq.read_table(pqt).to_pandas()

# will read directory full of partitioned parquets (ie. from spark)
df = pq.ParquetDataset(path, hdfs).read().to_pandas()