问题验证

• 我已经在文档和discord上寻找答案。

问题

当我尝试使用SentenceWindowNodeParser运行一个摄取管道时，我得到了以下错误：
Traceback (most recent call last): File "/home/ubuntu/projects/wimi/src/llama_index/ingest_sentence_window.py", line 38, in <module> pipeline.run(documents=documents, num_workers=4) File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/site-packages/llama_index/core/ingestion/pipeline.py", line 738, in run nodes_parallel = p.starmap( File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/multiprocessing/pool.py", line 375, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks put(task) File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/miniconda3/envs/wimi/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) AttributeError: Can't pickle local object 'split_by_sentence_tokenizer.<locals>.split'
代码(我实现了一个自定义读取器，返回list[Document]):

from qdrant_client import QdrantClient
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.ingestion import IngestionPipeline
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from wimi_commons.llama_index.documents import JSONReader, DummyTextCleaner
from pathlib import Path

DATA_DIR = str(Path(__file__).parents[2] / "data")

if __name__=="__main__":
    client = QdrantClient(host="localhost", grpc_port=6334, prefer_grpc=True)
    vector_store = QdrantVectorStore(client=client, collection_name="intfloat-multilingual-e5-large-collection-llama-index-sentence-window")
    embed_model = HuggingFaceEmbedding(model_name="intfloat/multilingual-e5-large")
    documents =JSONReader(
        file_path=f"{DATA_DIR}/solr_with_content_llama_index.json",
        content_key="text"
    ).load_data()
    node_parser = SentenceWindowNodeParser.from_defaults(
        window_size=3,
        window_metadata_key="window",
        original_text_metadata_key="original_text",
    )
    pipeline = IngestionPipeline(
        transformations=[
            SentenceSplitter(chunk_size=2000, chunk_overlap=0),
            node_parser,
            embed_model,
        ],
        vector_store=vector_store,
    )
    pipeline.run(documents=documents, num_workers=4)
    pipeline.persist(f"{DATA_DIR}/pipeline_storage")

我不知道我在这里做错了什么...有什么建议吗？