llama_index [Bug]: QdrantVectorStore解析器在响应中总是期望一个名为"text"的键,

vwkv1x7d  于 8个月前  发布在  其他
关注(0)|答案(2)|浏览(74)

Bug描述

parse_to_query_result 方法在 QdrantVectorStore 中总是期望元数据中有一个名为 "text" 的键。如果元数据中没有名为 "text" 的键，这将引发错误。
llama-index/vector_stores/qdrant/base.py 文件的第740行

  1. def parse_to_query_result(self, response: List[Any]) -> VectorStoreQueryResult:
  2.         """
  3. Convert vector store response to VectorStoreQueryResult.
  5. Args:
  6. response: List[Any]: List of results returned from the vector store.
  7. """
  8.         nodes = []
  9.         similarities = []
  10.         ids = []
  12.         for point in response:
  13.             payload = cast(Payload, point.payload)
  14.             try:
  15.                 node = metadata_dict_to_node(payload)
  16.             except Exception:
  17.                 metadata, node_info, relationships = legacy_metadata_dict_to_node(
  18.                     payload
  19.                 )
  21.                 node = TextNode(
  22.                     id_=str(point.id),
  23.                     text=payload.get("text"),  # <----- this should not be hardcoded
  24.                     metadata=metadata,
  25.                     start_char_idx=node_info.get("start", None),
  26.                     end_char_idx=node_info.get("end", None),
  27.                     relationships=relationships,
  28.                 )
  29.             nodes.append(node)
  30.             similarities.append(point.score)
  31.             ids.append(str(point.id))
  33.         return VectorStoreQueryResult(nodes=nodes, similarities=similarities, ids=ids)

这种情况不应该发生，应该在 vectorDB 元数据中使用 "text"。用户应该能够通过参数作为文本内容的键名传递，或者应该得到警告而不是解析错误。

版本

0.10.39

重现步骤

  1. 在 Qdrant 中创建一个元数据中没有 "text" 键的集合。
  2. 通过任何查询尝试从集合中检索任何节点。
  1. vec_db_client = qdrant_client.QdrantClient(
  2.     host=QDRANT_HOST,
  3.     port=443,
  4.     https=True,
  5. )
  7. vec_index = VectorStoreIndex.from_vector_store(
  8.     vector_store=QdrantVectorStore(
  9.         client=vec_db_client, collection_name=collection
  10.     )
  11. )
  13. retriever = VectorIndexRetriever(index=vec_index, similarity_top_k=10)
  15. query_engine = RetrieverQueryEngine(
  16.     retriever=retriever,
  17.     node_postprocessors=postprocessors,
  18. )
  20. query_engine.retrieve("query")  # <---- Raises error

相关日志/回溯

  1. {
  2. 	"name": "ValidationError",
  3. 	"message": "1 validation error for TextNode
  4. text
  5. none is not an allowed value (type=type_error.none.not_allowed)",
  6. 	"stack": "---------------------------------------------------------------------------
  7. ValueError                                Traceback (most recent call last)
  8. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/vector_stores/qdrant/base.py:757, in QdrantVectorStore.parse_to_query_result(self, response)
  9. 756 try:
  10. --> 757     node = metadata_dict_to_node(payload)
  11. 758 except Exception:
  13. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/vector_stores/utils.py:70, in metadata_dict_to_node(metadata, text)
  14. 69 if node_json is None:
  15. ---> 70     raise ValueError(\"Node content not found in metadata dict.\")
  16. 72 node: BaseNode
  18. ValueError: Node content not found in metadata dict.
  20. During handling of the above exception, another exception occurred:
  22. ValidationError                           Traceback (most recent call last)
  23. Cell In[6], line 1
  24. ----> 1 result = search.query(query=\"query?\", collection=\"test\")
  26. Cell In[1], line 210, in Search.query(self, query, collection)
  27. 204 # Query engine is used only for retrieval
  28. 205 query_engine = RetrieverQueryEngine(
  29. 206     retriever=retriever,
  30. 207     node_postprocessors=postprocessors,
  31. 208 )
  32. --> 210 retrieved_nodes = query_engine.retrieve(query)
  33. 212 logger.info(f\"Retrieved nodes: {retrieved_nodes}\")
  34. 214 if len(retrieved_nodes) == 0:
  36. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/query_engine/retriever_query_engine.py:144, in RetrieverQueryEngine.retrieve(self, query_bundle)
  37. 143 def retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
  38. --> 144     nodes = self._retriever.retrieve(query_bundle)
  39. 145     return self._apply_node_postprocessors(nodes, query_bundle=query_bundle)
  41. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:274, in Dispatcher.span.<locals>.wrapper(func, instance, args, kwargs)
  42. 270 self.span_enter(
  43. 271     id_=id_, bound_args=bound_args, instance=instance, parent_id=parent_id
  44. 272 )
  45. 273 try:
  46. --> 274     result = func(*args, **kwargs)
  47. 275 except BaseException as e:
  48. 276     self.event(SpanDropEvent(span_id=id_, err_str=str(e)))
  50. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/base/base_retriever.py:244, in BaseRetriever.retrieve(self, str_or_query_bundle)
  51. 239 with self.callback_manager.as_trace(\"query\"):
  52. 240     with self.callback_manager.event(
  53. 241         CBEventType.RETRIEVE,
  54. 242         payload={EventPayload.QUERY_STR: query_bundle.query_str},
  55. 243     ) as retrieve_event:
  56. --> 244         nodes = self._retrieve(query_bundle)
  57. 245         nodes = self._handle_recursive_retrieval(query_bundle, nodes)
  58. 246         retrieve_event.on_end(
  59. 247             payload={EventPayload.NODES: nodes},
  60. 248         )
  62. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py:274, in Dispatcher.span.<locals>.wrapper(func, instance, args, kwargs)
  63. 270 self.span_enter(
  64. 271     id_=id_, bound_args=bound_args, instance=instance, parent_id=parent_id
  65. 272 )
  66. 273 try:
  67. --> 274     result = func(*args, **kwargs)
  68. 275 except BaseException as e:
  69. 276     self.event(SpanDropEvent(span_id=id_, err_str=str(e)))
  71. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/indices/vector_store/retrievers/retriever.py:101, in VectorIndexRetriever._retrieve(self, query_bundle)
  72. 95     if query_bundle.embedding is None and len(query_bundle.embedding_strs) > 0:
  73. 96         query_bundle.embedding = (
  74. 97             self._embed_model.get_agg_embedding_from_queries(
  75. 98                 query_bundle.embedding_strs
  76. 99             )
  77. 100         )
  78. --> 101 return self._get_nodes_with_embeddings(query_bundle)
  80. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/core/indices/vector_store/retrievers/retriever.py:177, in VectorIndexRetriever._get_nodes_with_embeddings(self, query_bundle_with_embeddings)
  81. 173 def _get_nodes_with_embeddings(
  82. 174     self, query_bundle_with_embeddings: QueryBundle
  83. 175 ) -> List[NodeWithScore]:
  84. 176     query = self._build_vector_store_query(query_bundle_with_embeddings)
  85. --> 177     query_result = self._vector_store.query(query, **self._kwargs)
  86. 178     return self._build_node_list_from_query_result(query_result)
  88. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/vector_stores/qdrant/base.py:605, in QdrantVectorStore.query(self, query, **kwargs)
  89. 598 else:
  90. 599     response = self._client.search(
  91. 600         collection_name=self.collection_name,
  92. 601         query_vector=query_embedding,
  93. 602         limit=query.similarity_top_k,
  94. 603         query_filter=query_filter,
  95. 604     )
  96. --> 605     return self.parse_to_query_result(response)
  98. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/llama_index/vector_stores/qdrant/base.py:763, in QdrantVectorStore.parse_to_query_result(self, response)
  99. 758 except Exception:
  100. 759     metadata, node_info, relationships = legacy_metadata_dict_to_node(
  101. 760         payload
  102. 761     )
  103. --> 763     node = TextNode(
  104. 764         id_=str(point.id),
  105. 765         text=payload.get(\"text\"),
  106. 766         metadata=metadata,
  107. 767         start_char_idx=node_info.get(\"start\", None),
  108. 768         end_char_idx=node_info.get(\"end\", None),
  109. 769         relationships=relationships,
  110. 770     )
  111. 771 nodes.append(node)
  112. 772 similarities.append(point.score)
  114. File ~/miniconda3/envs/gptllama/lib/python3.10/site-packages/pydantic/v1/main.py:341, in BaseModel.__init__(__pydantic_self__, **data)
  115. 339 values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
  116. 340 if validation_error:
  117. --> 341     raise validation_error
  118. 342 try:
  119. 343     object_setattr(__pydantic_self__, '__dict__', values)
  121. ValidationError: 1 validation error for TextNode
  122. text
  123. none is not an allowed value (type=type_error.none.not_allowed)"
  124. }
llama_index

2条答案

按热度按时间
r7xajy2e

r7xajy2e1#

你好！我在这里帮助你解决遇到的任何问题。如果你有任何问题或需要修复bug的帮助，请随时提问！

要解决QdrantVectorStore类中parse_to_query_result方法在元数据中期望有一个"text"键的问题，你可以修改该方法，在尝试访问它之前检查"text"键是否存在。如果键不存在，你可以优雅地处理它，通过设置默认值或跳过操作。

以下是一个示例修改：

  1. def parse_to_query_result(self, response: List[Any]) -> VectorStoreQueryResult:
  2.     """
  3. Convert vector store response to VectorStoreQueryResult.
  5. Args:
  6. response: List[Any]: List of results returned from the vector store.
  7. """
  8.     nodes = []
  9.     similarities = []
  10.     ids = []
  12.     for point in response:
  13.         payload = cast(Payload, point.payload)
  14.         try:
  15.             node = metadata_dict_to_node(payload)
  16.         except Exception:
  17.             metadata, node_info, relationships = legacy_metadata_dict_to_node(
  18.                 payload
  19.             )
  21.             text = payload.get("text", "Default text")  # Check if "text" key exists
  22.             node = TextNode(
  23.                 id_=str(point.id),
  24.                 text=text,
  25.                 metadata=metadata,
  26.                 start_char_idx=node_info.get("start", None),
  27.                 end_char_idx=node_info.get("end", None),
  28.                 relationships=relationships,
  29.             )
  30.         nodes.append(node)
  31.         ids.append(str(point.id))
  32.         try:
  33.             similarities.append(point.score)
  34.         except AttributeError:
  35.             # certain requests do not return a score
  36.             similarities.append(1.0)
  38.     return VectorStoreQueryResult(nodes=nodes, similarities=similarities, ids=ids)

在这个修改中，payload.get("text", "Default text")行检查负载中是否存在"text"键。如果不存在，它将text设置为"Default text"。这种方法确保方法通过提供默认值1(https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/indices/vector_store/retrievers/retriever.py)2(https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/vector_stores/utils.py)3(https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/vector_stores/llama-index-vector-stores-qdrant/llama_index/vector_stores/qdrant/base.py)来优雅地处理缺失的"text"键。

  • 要继续与Dosu交流，请提及@dosu。*
展开查看全部
赞(0）分享  回复(0）举报  8个月前
xzlaal3s

xzlaal3s2#

我遇到了相同的错误：

  1. ValidationError: 1 validation error for TextNode
  2. text
  3.   none is not an allowed value (type=type_error.none.not_allowed)

我在几周前创建了一个VectoreStoreIndex,并从之前创建的集合中获取数据。经过几个小时的调试和尝试一些操作后，我意识到在qdrant中使用openai嵌入创建了一个新的集合后，解决了这个错误。我没有深入研究到底是什么导致了这个bug,但我猜测qdrant向量存储的创建方式发生了变化，这与旧版本创建嵌入的方式不匹配。
我不确定这是否有意义，但它解决了错误。如果有人遇到相同的问题，请分享一下。

赞(0）分享  回复(0）举报  8个月前
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com