检查其他资源

• 为这个问题添加了一个非常描述性的标题。
• 使用集成搜索在LangChain文档中进行了搜索。
• 使用GitHub搜索找到了一个类似的问题，但没有找到。
• 我确信这是LangChain中的一个错误，而不是我的代码。
• 通过更新到LangChain的最新稳定版本(或特定集成包)无法解决此错误。

示例代码

import bs4
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
blog_docs = loader.load()

from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=300, 
    chunk_overlap=50)

splits = text_splitter.split_documents(blog_docs)

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(documents=splits, 
                                    embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever()

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(documents=splits, 
                                    embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
docs = retriever.get_relevant_documents("What is Task Decomposition?")

print(f"number of documents - {len(docs)}")
for doc in docs:
  print(f"document content - `{doc.__dict__}")

打印出的值是
文档内容 - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.
1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', 'metadata': {'source': ' https://lilianweng.github.io/posts/2023-06-23-agent/'} , 'type': 'Document'}
文档内容 - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.
1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', 'metadata': {'source': ' https://lilianweng.github.io/posts/2023-06-23-agent/'} , 'type': 'Document'}
文档内容 - {'page_content': 'Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps。 CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process。
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure。 The search process can be BFS (breadth-first search)或DFS(深度优先搜索)与每个状态由分类器(通过提示)或多数投票进行评估。
任务分解可以通过(1)LLM简单提示如“XYZ的步骤。
1。”，“实现XYZ的子目标是什么？”，(2)使用任务特定指令；例如“编写小说大纲”。用于写小说，或者(3)与人类输入完成。', 'metadata': {'source': ' https://lilianweng.github.io/posts/2023-06-23-agent/'} , 'type': 'Document'}
文档内容 - {'page_content': '资源：

1. 用于搜索和信息收集的互联网访问。
2. 长期记忆管理。
3. GPT-3.5驱动的代理，用于分配简单任务。
4. 文件输出。

性能评估：

1. 不断回顾和分析您的行动，确保您正在尽最大努力执行。
2. 建设性地自我批评您的整体行为。
3. 反思过去的决策和策略，以完善您的方法。
4. 每个命令都有成本，因此要明智高效。尽量在最少的步骤中完成任务。', 'metadata': {'source': ' https://lilianweng.github.io/posts/2023-06-23-agent/'} , 'type': 'Document'}
   如您所见，有3个文档是相同的。我检查了分割并包含52个文档，但

res = vectorstore.get()
res.keys()

len(res['documents'])

的值为156,所以我认为每个文档存储了3次而不是1次。

错误信息和堆栈跟踪(如果适用)

• 无响应*

描述

我尝试在玩具示例中将chroma用作检索器，除了在应用“get_relevant_documents”时获得不同的文档外，我还得到了相同的文档3次。

系统信息

langchain==0.2.1
langchain-community==0.2.1
langchain-core==0.2.3
langchain-openai==0.1.8
langchain-text-splitters==0.2.0
langchainhub==0.1.17
Linux
Python 3.10.12
我在Colab上运行