refactor: 统一导入方式，移除 sys.path 操作
Some checks failed
构建并部署 AI Agent 服务 / deploy (push) Failing after 6m22s
Some checks failed
构建并部署 AI Agent 服务 / deploy (push) Failing after 6m22s
- 重构所有模块导入，移除 sys.path.insert - 统一使用 from backend.xxx 的绝对导入方式 - rag_core 包内使用相对导入（from .xxx） - 移动 visualize_graph.py 到 tools/ 目录 - 添加必要的 __init__.py 文件 - 清理废弃文档和脚本
This commit is contained in:
0
backend/__init__.py
Normal file
0
backend/__init__.py
Normal file
@@ -6,10 +6,7 @@ from typing import Optional, Dict, Any
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加项目路径
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
|
||||
|
||||
from app.model_services.chat_services import get_chat_service
|
||||
from backend.app.model_services.chat_services import get_chat_service
|
||||
|
||||
|
||||
class IntentType(Enum):
|
||||
|
||||
@@ -1,179 +0,0 @@
|
||||
# MCP 集成系统
|
||||
|
||||
## 概述
|
||||
|
||||
这是一个统一的外部接口管理层，集成了 MCP (Model Context Protocol)，同时支持数据库缓存和降级到模拟数据。
|
||||
|
||||
## 架构设计
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 子图 (Subgraphs) │
|
||||
│ contact_api │ dictionary_api │ news_api │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ MCP Manager (统一入口) │
|
||||
│ ┌─────────────────────────────────────────────────┐ │
|
||||
│ │ Adapters (适配器层) │ │
|
||||
│ │ ContactAdapter │ DictionaryAdapter │ NewsAdapter│ │
|
||||
│ └─────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ MCP Client │ │ Database │ │ Mock Data │
|
||||
│ (真实服务) │ │ (缓存层) │ │ (降级层) │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
backend/app/mcp/
|
||||
├── __init__.py # 模块初始化
|
||||
├── mcp_manager.py # MCP管理器（统一入口）
|
||||
├── mcp_client.py # MCP客户端
|
||||
├── base_adapter.py # 适配器基类
|
||||
├── mcp_config.example.yaml # 配置示例
|
||||
├── mcp_example.py # 使用示例
|
||||
└── adapters/
|
||||
├── __init__.py
|
||||
├── contact_adapter.py # 通讯录适配器
|
||||
├── dictionary_adapter.py# 词典适配器
|
||||
└── news_adapter.py # 新闻适配器
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 基本使用（自动降级）
|
||||
|
||||
现有的子图API已经无缝迁移，无需修改代码：
|
||||
|
||||
```python
|
||||
# 通讯录 - 和之前一样使用
|
||||
from backend.app.subgraphs.contact.api_client import contact_api
|
||||
|
||||
contacts = await contact_api.list_contacts(user_id="default")
|
||||
|
||||
# 词典 - 和之前一样使用
|
||||
from backend.app.subgraphs.dictionary.api_client import dictionary_api
|
||||
|
||||
word_data = await dictionary_api.query_word(word="ephemeral")
|
||||
|
||||
# 新闻 - 和之前一样使用
|
||||
from backend.app.subgraphs.news_analysis.api_client import news_api
|
||||
|
||||
news_list = await news_api.query_news(query="AI")
|
||||
```
|
||||
|
||||
### 2. 直接使用MCP管理器
|
||||
|
||||
```python
|
||||
from backend.app.mcp import mcp_manager, ContactAdapter, DictionaryAdapter, NewsAdapter
|
||||
|
||||
# 注册适配器
|
||||
mcp_manager.register_adapter(ContactAdapter())
|
||||
mcp_manager.register_adapter(DictionaryAdapter())
|
||||
mcp_manager.register_adapter(NewsAdapter())
|
||||
|
||||
# 初始化
|
||||
await mcp_manager.initialize()
|
||||
|
||||
# 统一调用接口
|
||||
result = await mcp_manager.execute(
|
||||
"dictionary",
|
||||
"query_word",
|
||||
word="serendipity",
|
||||
user_id="default"
|
||||
)
|
||||
|
||||
print(f"来源: {result.source}") # mcp_dictionary / database / mock
|
||||
print(f"数据: {result.data}")
|
||||
```
|
||||
|
||||
### 3. 配置MCP服务器
|
||||
|
||||
复制配置示例：
|
||||
|
||||
```bash
|
||||
cp backend/app/mcp/mcp_config.example.yaml backend/app/mcp/mcp_config.yaml
|
||||
```
|
||||
|
||||
编辑 `mcp_config.yaml`，启用需要的MCP服务器：
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
# Gmail 邮件服务
|
||||
gmail:
|
||||
type: stdio
|
||||
command: npx
|
||||
args:
|
||||
- "-y"
|
||||
- "@modelcontextprotocol/server-gmail"
|
||||
enabled: true
|
||||
|
||||
# GitHub
|
||||
github:
|
||||
type: stdio
|
||||
command: npx
|
||||
args:
|
||||
- "-y"
|
||||
- "@modelcontextprotocol/server-github"
|
||||
env:
|
||||
GITHUB_PERSONAL_ACCESS_TOKEN: "ghp_your_token_here"
|
||||
enabled: true
|
||||
```
|
||||
|
||||
## 特性
|
||||
|
||||
### 1. 三层降级策略
|
||||
|
||||
- **MCP层**: 优先使用真实的MCP服务
|
||||
- **数据库层**: 其次使用数据库缓存
|
||||
- **模拟层**: 最后降级到模拟数据，确保系统始终可用
|
||||
|
||||
### 2. 统一接口
|
||||
|
||||
所有外部服务都通过 `mcp_manager.execute()` 统一调用，返回标准化的 `AdapterResult`。
|
||||
|
||||
### 3. 向后兼容
|
||||
|
||||
保留了原有的 `api_client` 接口，现有代码无需修改即可使用新系统。
|
||||
|
||||
### 4. 可扩展
|
||||
|
||||
通过继承 `BaseAdapter` 可以轻松添加新的适配器。
|
||||
|
||||
## 创建自定义适配器
|
||||
|
||||
```python
|
||||
from backend.app.mcp import BaseAdapter, AdapterResult
|
||||
|
||||
class MyAdapter(BaseAdapter):
|
||||
name = "my_service"
|
||||
description = "我的自定义服务"
|
||||
|
||||
async def execute(self, action: str, **kwargs) -> AdapterResult:
|
||||
# 1. 尝试MCP
|
||||
# 2. 尝试数据库
|
||||
# 3. 降级到模拟
|
||||
pass
|
||||
|
||||
# 注册
|
||||
mcp_manager.register_adapter(MyAdapter())
|
||||
```
|
||||
|
||||
## 可用的MCP服务器
|
||||
|
||||
- **@modelcontextprotocol/server-filesystem** - 文件系统访问
|
||||
- **@modelcontextprotocol/server-github** - GitHub 集成
|
||||
- **@modelcontextprotocol/server-gmail** - Gmail 邮件
|
||||
- **@modelcontextprotocol/server-brave-search** - 网页搜索
|
||||
- 更多社区服务器...
|
||||
|
||||
## 完整示例
|
||||
|
||||
参见 `backend/app/mcp/mcp_example.py` 获取完整的使用示例。
|
||||
@@ -1,363 +0,0 @@
|
||||
# RAG 召回率与相关性评估指南
|
||||
|
||||
本指南介绍如何评估 RAG 系统的召回率（Recall）和相关性（Relevance）。
|
||||
|
||||
---
|
||||
|
||||
## 📊 核心概念
|
||||
|
||||
### 1. 召回率 (Recall)
|
||||
|
||||
召回率衡量的是：**在所有相关文档中，有多少被检索出来了？**
|
||||
|
||||
```
|
||||
Recall@k = (前 k 个结果中的相关文档数量) / (总相关文档数量)
|
||||
```
|
||||
|
||||
例如：
|
||||
- 总共有 5 篇相关文档
|
||||
- 检索返回 10 篇，其中 3 篇是相关的
|
||||
- Recall@10 = 3/5 = 60%
|
||||
|
||||
### 2. 精确率 (Precision)
|
||||
|
||||
精确率衡量的是：**在检索出来的文档中，有多少是相关的？**
|
||||
|
||||
```
|
||||
Precision@k = (前 k 个结果中的相关文档数量) / k
|
||||
```
|
||||
|
||||
例如：
|
||||
- 检索返回 10 篇，其中 3 篇是相关的
|
||||
- Precision@10 = 3/10 = 30%
|
||||
|
||||
### 3. F1 分数 (F1 Score)
|
||||
|
||||
F1 分数是召回率和精确率的调和平均数：
|
||||
|
||||
```
|
||||
F1@k = 2 * Recall@k * Precision@k / (Recall@k + Precision@k)
|
||||
```
|
||||
|
||||
### 4. 平均倒数排名 (MRR)
|
||||
|
||||
MRR 衡量第一个相关文档的排名：
|
||||
|
||||
```
|
||||
MRR = 1/m * sum(1/rank_i for i=1..m)
|
||||
```
|
||||
|
||||
其中 rank_i 是第 i 个相关文档第一次出现的排名。
|
||||
|
||||
例如：
|
||||
- 测试用例 1：第一个相关文档在第 2 位 → 1/2 = 0.5
|
||||
- 测试用例 2：第一个相关文档在第 1 位 → 1/1 = 1.0
|
||||
- 测试用例 3：第一个相关文档在第 3 位 → 1/3 ≈ 0.333
|
||||
- MRR = (0.5 + 1.0 + 0.333) / 3 ≈ 0.611
|
||||
|
||||
### 5. 相关性评分
|
||||
|
||||
相关性评分评估检索到的文档与查询的相关程度，通常使用：
|
||||
- 人工标注（Human Evaluation）
|
||||
- LLM 评估（LLM-as-a-Judge）
|
||||
- 相关性模型（Cross-Encoder）
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ 如何评估
|
||||
|
||||
### 方法一：使用内置评估模块
|
||||
|
||||
我们的项目已经内置了评估模块 `app.rag.evaluate`。
|
||||
|
||||
#### 1. 准备测试用例
|
||||
|
||||
首先，需要准备带有标注的测试用例：
|
||||
|
||||
```python
|
||||
from app.rag.evaluate import RetrievalTestCase
|
||||
|
||||
test_cases = [
|
||||
RetrievalTestCase(
|
||||
query="什么是 RAG 系统？",
|
||||
relevant_doc_ids=["doc_rag_1", "doc_rag_2", "doc_rag_3"],
|
||||
expected_answer="RAG 是 Retrieval-Augmented Generation 的缩写..."
|
||||
),
|
||||
RetrievalTestCase(
|
||||
query="如何使用 LangChain？",
|
||||
relevant_doc_ids=["doc_langchain_1", "doc_langchain_2"],
|
||||
expected_answer="LangChain 的使用步骤包括..."
|
||||
),
|
||||
# 更多测试用例...
|
||||
]
|
||||
```
|
||||
|
||||
**重要提示：**
|
||||
- 每个查询需要知道哪些文档是相关的
|
||||
- 相关文档需要有唯一的 ID
|
||||
- expected_answer 是可选的，用于评估答案质量
|
||||
|
||||
#### 2. 运行评估
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from app.rag.evaluate import RAGEvaluator, generate_test_report
|
||||
|
||||
# 初始化评估器
|
||||
evaluator = RAGEvaluator(rag_pipeline, test_cases)
|
||||
|
||||
# 运行评估
|
||||
metrics = asyncio.run(evaluator.evaluate_retrieval(k_list=[1, 3, 5, 10]))
|
||||
|
||||
# 生成报告
|
||||
report = generate_test_report(metrics)
|
||||
print(report)
|
||||
```
|
||||
|
||||
#### 3. 运行示例脚本
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python scripts/evaluate_rag.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 方法二：手动计算召回率
|
||||
|
||||
如果你想手动计算，步骤如下：
|
||||
|
||||
#### 步骤 1：准备测试数据
|
||||
|
||||
准备一个测试查询列表，每个查询对应相关文档的 ID：
|
||||
|
||||
```python
|
||||
test_queries = [
|
||||
{
|
||||
"query": "什么是 RAG？",
|
||||
"relevant_ids": ["doc1", "doc3", "doc5"]
|
||||
},
|
||||
{
|
||||
"query": "如何优化 RAG？",
|
||||
"relevant_ids": ["doc2", "doc4"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### 步骤 2：运行检索
|
||||
|
||||
对于每个查询，运行 RAG 检索，记录返回的文档 ID：
|
||||
|
||||
```python
|
||||
def run_retrieval(query):
|
||||
"""运行检索，返回文档 ID 列表"""
|
||||
docs = rag_pipeline.retrieve(query)
|
||||
return [doc.metadata["id"] for doc in docs]
|
||||
```
|
||||
|
||||
#### 步骤 3：计算召回率
|
||||
|
||||
```python
|
||||
def calculate_recall(retrieved_ids, relevant_ids, k):
|
||||
"""计算 Recall@k"""
|
||||
top_k = retrieved_ids[:k]
|
||||
relevant_in_top_k = set(top_k) & set(relevant_ids)
|
||||
recall = len(relevant_in_top_k) / len(relevant_ids)
|
||||
return recall
|
||||
|
||||
# 示例
|
||||
retrieved = ["doc1", "doc2", "doc3", "doc4", "doc5"]
|
||||
relevant = ["doc1", "doc3", "doc5"]
|
||||
print(f"Recall@3: {calculate_recall(retrieved, relevant, k=3):.2%}") # 2/3 = 66.67%
|
||||
print(f"Recall@5: {calculate_recall(retrieved, relevant, k=5):.2%}") # 3/3 = 100%
|
||||
```
|
||||
|
||||
#### 步骤 4：聚合结果
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
all_recalls_at_1 = []
|
||||
all_recalls_at_3 = []
|
||||
all_recalls_at_5 = []
|
||||
|
||||
for test_case in test_queries:
|
||||
retrieved = run_retrieval(test_case["query"])
|
||||
recall_1 = calculate_recall(retrieved, test_case["relevant_ids"], k=1)
|
||||
recall_3 = calculate_recall(retrieved, test_case["relevant_ids"], k=3)
|
||||
recall_5 = calculate_recall(retrieved, test_case["relevant_ids"], k=5)
|
||||
|
||||
all_recalls_at_1.append(recall_1)
|
||||
all_recalls_at_3.append(recall_3)
|
||||
all_recalls_at_5.append(recall_5)
|
||||
|
||||
print(f"Average Recall@1: {np.mean(all_recalls_at_1):.2%}")
|
||||
print(f"Average Recall@3: {np.mean(all_recalls_at_3):.2%}")
|
||||
print(f"Average Recall@5: {np.mean(all_recalls_at_5):.2%}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 方法三：评估相关性
|
||||
|
||||
评估相关性有几种方法：
|
||||
|
||||
#### 方案 A：使用 LLM 评估（LLM-as-a-Judge）
|
||||
|
||||
```python
|
||||
from app.rag.evaluate import RelevanceEvaluator
|
||||
|
||||
# 初始化评估器
|
||||
evaluator = RelevanceEvaluator(llm)
|
||||
|
||||
# 评估相关性
|
||||
score, reason = asyncio.run(evaluator.evaluate_relevance(query, document))
|
||||
|
||||
print(f"相关性评分: {score}/5")
|
||||
print(f"理由: {reason}")
|
||||
```
|
||||
|
||||
#### 方案 B：使用重排模型评分
|
||||
|
||||
重排模型本身可以给出相关性分数：
|
||||
|
||||
```python
|
||||
from app.model_services import get_rerank_service
|
||||
|
||||
rerank_service = get_rerank_service()
|
||||
|
||||
# 获取相关性分数
|
||||
scores = rerank_service.compute_scores(
|
||||
query="什么是 RAG？",
|
||||
documents=["doc1", "doc2", "doc3"]
|
||||
)
|
||||
```
|
||||
|
||||
#### 方案 C：人工标注
|
||||
|
||||
最准确但也最耗时的方法是让人工标注相关性：
|
||||
|
||||
```python
|
||||
# 相关性评分标准
|
||||
relevance_levels = {
|
||||
5: "完全相关，直接回答了问题",
|
||||
4: "高度相关，包含关键信息",
|
||||
3: "部分相关，有一些相关信息",
|
||||
2: "弱相关，提及但不太相关",
|
||||
1: "不相关，基本无关",
|
||||
0: "完全无关"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 如何解释结果
|
||||
|
||||
### 召回率低怎么办？
|
||||
|
||||
如果 Recall@k 低，可能的原因：
|
||||
|
||||
1. **检索器召回能力不足**
|
||||
- 嵌入模型不合适
|
||||
- 检索算法太简单
|
||||
- 解决方案：改用更好的嵌入模型、使用混合检索
|
||||
|
||||
2. **查询理解不够**
|
||||
- 查询改写效果不好
|
||||
- 解决方案：增加查询改写的多样性
|
||||
|
||||
3. **文档分块策略不好**
|
||||
- 分块太小/太大
|
||||
- 解决方案：调整 chunk_size，使用父子分块
|
||||
|
||||
### 精确率低怎么办？
|
||||
|
||||
如果 Precision@k 低，可能的原因：
|
||||
|
||||
1. **检索结果噪声多**
|
||||
- 解决方案：加强重排序
|
||||
|
||||
2. **文档切分有问题**
|
||||
- 不相关的片段也被检索到
|
||||
- 解决方案：改进切分策略
|
||||
|
||||
---
|
||||
|
||||
## 🎯 评估最佳实践
|
||||
|
||||
### 1. 测试用例构建
|
||||
|
||||
- ✅ **覆盖多样的查询类型**：事实型、概念型、操作型
|
||||
- ✅ **每个查询有多个相关文档**：避免单点依赖
|
||||
- ✅ **包含难例**：测试边界情况
|
||||
- ✅ **定期更新**：随着知识库变化更新测试用例
|
||||
|
||||
### 2. 评估指标选择
|
||||
|
||||
- **快速迭代**：关注 Recall@3, Recall@5
|
||||
- **正式发布**：完整评估所有指标
|
||||
- **用户体验**：同时评估答案质量
|
||||
|
||||
### 3. A/B 测试
|
||||
|
||||
当你改进 RAG 系统时，使用 A/B 测试：
|
||||
|
||||
```python
|
||||
# A 版本（旧版本）
|
||||
metrics_a = evaluator.evaluate_retrieval()
|
||||
|
||||
# B 版本（新版本）
|
||||
metrics_b = evaluator_new.evaluate_retrieval()
|
||||
|
||||
# 对比
|
||||
print(f"Recall@5 改进: {metrics_b.recall_at_k[5] - metrics_a.recall_at_k[5]:.2%}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 完整评估报告示例
|
||||
|
||||
运行评估后，会生成这样的报告：
|
||||
|
||||
```
|
||||
================================================================================
|
||||
RAG 系统评估报告
|
||||
================================================================================
|
||||
|
||||
【召回率 Recall@k】
|
||||
Recall@1: 60.00%
|
||||
Recall@3: 85.00%
|
||||
Recall@5: 95.00%
|
||||
Recall@10: 100.00%
|
||||
|
||||
【精确率 Precision@k】
|
||||
Precision@1: 100.00%
|
||||
Precision@3: 90.00%
|
||||
Precision@5: 80.00%
|
||||
Precision@10: 55.00%
|
||||
|
||||
【F1 分数 F1@k】
|
||||
F1@1: 0.7500
|
||||
F1@3: 0.8718
|
||||
F1@5: 0.8636
|
||||
F1@10: 0.7097
|
||||
|
||||
【平均倒数排名 MRR】: 0.8500
|
||||
|
||||
================================================================================
|
||||
指标说明：
|
||||
- Recall@k: 前 k 个结果中包含多少比例的相关文档
|
||||
- Precision@k: 前 k 个结果中有多少比例是相关文档
|
||||
- F1@k: 召回率和精确率的调和平均数
|
||||
- MRR: 第一个相关文档的排名的倒数的平均值
|
||||
================================================================================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 相关文件
|
||||
|
||||
- `backend/app/rag/evaluate.py` - 评估模块
|
||||
- `backend/scripts/evaluate_rag.py` - 评估示例脚本
|
||||
- `backend/app/rag/pipeline.py` - RAG 流水线
|
||||
- `backend/app/model_services/` - 模型服务
|
||||
@@ -8,11 +8,6 @@ import logging
|
||||
from typing import List
|
||||
from pathlib import Path
|
||||
|
||||
# 添加父目录到路径，支持从 app.model_services 导入
|
||||
backend_root = Path(__file__).parent.parent
|
||||
if str(backend_root) not in sys.path:
|
||||
sys.path.insert(0, str(backend_root))
|
||||
|
||||
from .config import LLAMACPP_EMBEDDING_URL, LLAMACPP_API_KEY
|
||||
from langchain_core.embeddings import Embeddings
|
||||
|
||||
@@ -36,7 +31,7 @@ class LlamaCppEmbedder:
|
||||
|
||||
# 直接获取统一嵌入服务
|
||||
try:
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.app.model_services import get_embedding_service
|
||||
self._fallback_embeddings = get_embedding_service()
|
||||
logger.info("✅ 统一嵌入服务加载成功")
|
||||
except Exception as e:
|
||||
|
||||
@@ -13,7 +13,9 @@ from langchain_classic.retrievers import ParentDocumentRetriever
|
||||
from langchain_text_splitters import RecursiveCharacterTextSplitter, TextSplitter
|
||||
from langchain_core.stores import BaseStore
|
||||
|
||||
from rag_core import LlamaCppEmbedder, QdrantVectorStore, create_docstore
|
||||
from .embedders import LlamaCppEmbedder
|
||||
from .vector_store import QdrantVectorStore
|
||||
from .store import create_docstore
|
||||
|
||||
|
||||
def create_parent_retriever(
|
||||
|
||||
@@ -1,143 +0,0 @@
|
||||
"""
|
||||
RAG 评估示例脚本
|
||||
演示如何使用 RAGEvaluator 评估召回率和相关性
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加项目路径
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
|
||||
|
||||
from app.rag.evaluate import (
|
||||
RAGEvaluator,
|
||||
RelevanceEvaluator,
|
||||
RetrievalTestCase,
|
||||
generate_test_report
|
||||
)
|
||||
from app.rag.pipeline import RAGPipeline
|
||||
from app.model_services import get_chat_service, get_embedding_service
|
||||
|
||||
|
||||
async def main():
|
||||
print("=" * 80)
|
||||
print("RAG 系统评估示例")
|
||||
print("=" * 80)
|
||||
print()
|
||||
|
||||
# 1. 准备测试用例
|
||||
print("【1/4】准备测试用例...")
|
||||
test_cases = [
|
||||
RetrievalTestCase(
|
||||
query="什么是 RAG 系统？",
|
||||
relevant_doc_ids=["doc_rag_1", "doc_rag_2", "doc_rag_3"],
|
||||
expected_answer="RAG 是 Retrieval-Augmented Generation 的缩写，是一种结合检索和生成的技术..."
|
||||
),
|
||||
RetrievalTestCase(
|
||||
query="如何使用 LangChain 构建 RAG？",
|
||||
relevant_doc_ids=["doc_langchain_1", "doc_langchain_2"],
|
||||
expected_answer="使用 LangChain 构建 RAG 的步骤包括：1) 准备文档 2) 向量化 3) 构建检索器 4) 组合生成..."
|
||||
),
|
||||
RetrievalTestCase(
|
||||
query="什么是向量数据库？",
|
||||
relevant_doc_ids=["doc_vector_db_1", "doc_qdrant_1"],
|
||||
expected_answer="向量数据库是专门用于存储和检索向量嵌入的数据库，如 Qdrant、Pinecone 等..."
|
||||
),
|
||||
RetrievalTestCase(
|
||||
query="如何优化 RAG 的检索质量？",
|
||||
relevant_doc_ids=["doc_optimize_1", "doc_rerank_1", "doc_fusion_1"],
|
||||
expected_answer="优化 RAG 检索质量的方法包括：重排序、查询改写、结果融合、混合检索等..."
|
||||
),
|
||||
RetrievalTestCase(
|
||||
query="LangGraph 是什么？",
|
||||
relevant_doc_ids=["doc_langgraph_1"],
|
||||
expected_answer="LangGraph 是 LangChain 的扩展，用于构建状态感知的多步工作流..."
|
||||
),
|
||||
]
|
||||
print(f" 已加载 {len(test_cases)} 个测试用例")
|
||||
print()
|
||||
|
||||
# 2. 初始化 RAG 系统（这里使用模拟）
|
||||
print("【2/4】初始化 RAG 系统...")
|
||||
|
||||
# 注意：实际使用时，这里应该初始化真实的 RAGPipeline
|
||||
# 这里为了演示，我们创建一个模拟的 RAG 类
|
||||
class MockRAGPipeline:
|
||||
def __init__(self):
|
||||
# 模拟的文档库
|
||||
self.mock_docs = {
|
||||
"doc_rag_1": "RAG 是 Retrieval-Augmented Generation 的缩写...",
|
||||
"doc_rag_2": "RAG 系统由检索器和生成器两部分组成...",
|
||||
"doc_rag_3": "RAG 的工作流程是：查询 -> 检索 -> 生成...",
|
||||
"doc_langchain_1": "LangChain 是用于构建 LLM 应用的框架...",
|
||||
"doc_langchain_2": "LangChain 提供了多种工具和集成...",
|
||||
"doc_vector_db_1": "向量数据库用于存储向量嵌入...",
|
||||
"doc_qdrant_1": "Qdrant 是一个开源的向量数据库...",
|
||||
"doc_optimize_1": "RAG 优化方法包括重排序和查询改写...",
|
||||
"doc_rerank_1": "重排序使用 Cross-Encoder 重新排序检索结果...",
|
||||
"doc_fusion_1": "结果融合使用 RRF 算法合并多个检索结果...",
|
||||
"doc_langgraph_1": "LangGraph 用于构建状态机工作流...",
|
||||
}
|
||||
|
||||
async def aretrieve(self, query: str):
|
||||
"""模拟检索，返回相关文档"""
|
||||
from langchain_core.documents import Document
|
||||
|
||||
# 简单的关键词匹配模拟
|
||||
results = []
|
||||
for doc_id, content in self.mock_docs.items():
|
||||
if any(keyword in query.lower() for keyword in ["rag", "检索"]):
|
||||
if "rag" in doc_id.lower():
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
elif any(keyword in query.lower() for keyword in ["langchain", "构建"]):
|
||||
if "langchain" in doc_id.lower():
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
elif any(keyword in query.lower() for keyword in ["向量", "数据库", "qdrant"]):
|
||||
if "vector" in doc_id.lower() or "qdrant" in doc_id.lower():
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
elif any(keyword in query.lower() for keyword in ["优化", "重排", "融合"]):
|
||||
if "optimize" in doc_id.lower() or "rerank" in doc_id.lower() or "fusion" in doc_id.lower():
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
elif any(keyword in query.lower() for keyword in ["langgraph"]):
|
||||
if "langgraph" in doc_id.lower():
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
|
||||
# 如果没有匹配到，返回一些通用结果
|
||||
if not results:
|
||||
for doc_id, content in list(self.mock_docs.items())[:3]:
|
||||
results.append(Document(page_content=content, metadata={"id": doc_id}))
|
||||
|
||||
return results
|
||||
|
||||
rag_pipeline = MockRAGPipeline()
|
||||
print(" RAG 系统已初始化（模拟）")
|
||||
print()
|
||||
|
||||
# 3. 评估检索质量
|
||||
print("【3/4】评估检索质量...")
|
||||
evaluator = RAGEvaluator(rag_pipeline, test_cases)
|
||||
metrics = await evaluator.evaluate_retrieval(k_list=[1, 3, 5, 10])
|
||||
print(" 评估完成")
|
||||
print()
|
||||
|
||||
# 4. 生成报告
|
||||
print("【4/4】生成评估报告...")
|
||||
report = generate_test_report(metrics)
|
||||
print(report)
|
||||
print()
|
||||
|
||||
# 5. 保存报告
|
||||
report_file = os.path.join(os.path.dirname(__file__), 'rag_evaluation_report.txt')
|
||||
with open(report_file, 'w', encoding='utf-8') as f:
|
||||
f.write(report)
|
||||
print(f" 报告已保存到：{report_file}")
|
||||
print()
|
||||
|
||||
print("=" * 80)
|
||||
print("评估完成！")
|
||||
print("=" * 80)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
0
frontend/__init__.py
Normal file
0
frontend/__init__.py
Normal file
@@ -1,9 +0,0 @@
|
||||
"""
|
||||
AI Agent 前端模块
|
||||
采用分层架构设计，包含配置、状态、API客户端和UI组件
|
||||
"""
|
||||
|
||||
from logger import debug, info, warning, error
|
||||
|
||||
__version__ = "2.0.0"
|
||||
__all__ = ["debug", "info", "warning", "error"]
|
||||
@@ -38,10 +38,6 @@ from .config import (
|
||||
)
|
||||
|
||||
# 从 rag_core 重新导出常用组件
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "backend"))
|
||||
|
||||
from backend.rag_core import (
|
||||
LlamaCppEmbedder,
|
||||
QdrantVectorStore,
|
||||
|
||||
@@ -11,19 +11,8 @@ from dotenv import load_dotenv
|
||||
# 加载 .env 文件
|
||||
load_dotenv()
|
||||
|
||||
# 添加项目根目录和 backend 目录到 Python 路径
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "backend"))
|
||||
|
||||
# 导入方式：条件导入，支持作为脚本运行和作为包导入
|
||||
if __name__ == "__main__":
|
||||
# 作为脚本直接运行时使用绝对导入
|
||||
from rag_indexer.index_builder import IndexBuilder, IndexBuilderConfig
|
||||
from rag_indexer.splitters import SplitterType
|
||||
else:
|
||||
# 作为包导入时使用相对导入
|
||||
from .index_builder import IndexBuilder, IndexBuilderConfig
|
||||
from .splitters import SplitterType
|
||||
from rag_indexer.index_builder import IndexBuilder, IndexBuilderConfig
|
||||
from rag_indexer.splitters import SplitterType
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
|
||||
@@ -12,9 +12,6 @@ from pathlib import Path
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Union, Optional, Any, Dict
|
||||
|
||||
# 添加 backend 目录到路径以导入 rag_core
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "backend"))
|
||||
|
||||
from httpx import RemoteProtocolError
|
||||
from langchain_core.documents import Document
|
||||
from langchain_core.embeddings import Embeddings
|
||||
@@ -27,16 +24,11 @@ from qdrant_client.http.models import SparseVectorParams
|
||||
from .loaders import DocumentLoader
|
||||
from .splitters import SplitterType, get_splitter
|
||||
|
||||
# 从 rag_core 导入
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "backend"))
|
||||
|
||||
from rag_core import LlamaCppEmbedder, QdrantVectorStore, create_docstore, create_parent_retriever
|
||||
from backend.rag_core import LlamaCppEmbedder, QdrantVectorStore, create_docstore, create_parent_retriever
|
||||
|
||||
# 尝试导入新的 model_services（如果可用）
|
||||
try:
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.app.model_services import get_embedding_service
|
||||
HAS_MODEL_SERVICES = True
|
||||
except ImportError:
|
||||
HAS_MODEL_SERVICES = False
|
||||
|
||||
@@ -7,13 +7,8 @@ import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
# 添加项目根目录到 Python 路径
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from rag_core import QdrantVectorStore
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.rag_core import QdrantVectorStore
|
||||
from backend.app.model_services import get_embedding_service
|
||||
|
||||
|
||||
def check_qdrant_data():
|
||||
@@ -55,7 +50,7 @@ def check_qdrant_data():
|
||||
|
||||
def check_sparse_embedder():
|
||||
"""检查稀疏嵌入器"""
|
||||
from rag_core import get_sparse_embedder
|
||||
from backend.rag_core import get_sparse_embedder
|
||||
|
||||
print("\n" + "="*70)
|
||||
print("检查稀疏嵌入器...")
|
||||
|
||||
@@ -7,12 +7,9 @@ import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
|
||||
from qdrant_client import models
|
||||
from rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from backend.app.model_services import get_embedding_service
|
||||
|
||||
|
||||
def test_dense_retrieval():
|
||||
|
||||
@@ -7,13 +7,8 @@ import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
# 添加项目根目录到 Python 路径
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from rag_core import QdrantVectorStore
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.rag_core import QdrantVectorStore
|
||||
from backend.app.model_services import get_embedding_service
|
||||
|
||||
|
||||
async def delete_and_recreate():
|
||||
|
||||
@@ -6,10 +6,7 @@
|
||||
import sys
|
||||
import os
|
||||
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
|
||||
from rag_core.client import create_qdrant_client
|
||||
from backend.rag_core.client import create_qdrant_client
|
||||
|
||||
|
||||
def delete_collection():
|
||||
|
||||
@@ -7,12 +7,9 @@ import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
|
||||
from qdrant_client import models
|
||||
from rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from backend.app.model_services import get_embedding_service
|
||||
|
||||
|
||||
def check_qdrant_content():
|
||||
|
||||
@@ -10,12 +10,9 @@ import sys
|
||||
import uuid
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 添加项目根目录和 backend 目录到 Python 路径
|
||||
# 加载环境变量
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..")
|
||||
backend_dir = os.path.join(project_root, "backend")
|
||||
sys.path.insert(0, project_root)
|
||||
sys.path.insert(0, backend_dir)
|
||||
load_dotenv()
|
||||
load_dotenv(os.path.join(project_root, ".env"))
|
||||
|
||||
from backend.app.config import DB_URI
|
||||
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
|
||||
|
||||
@@ -6,14 +6,11 @@ import numpy as np
|
||||
from dotenv import load_dotenv
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
# 添加项目根目录和 backend 目录到 Python 路径
|
||||
# 加载环境变量
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..")
|
||||
backend_dir = os.path.join(project_root, "backend")
|
||||
sys.path.insert(0, project_root)
|
||||
sys.path.insert(0, backend_dir)
|
||||
load_dotenv()
|
||||
load_dotenv(os.path.join(project_root, ".env"))
|
||||
|
||||
from rag_core import LlamaCppEmbedder
|
||||
from backend.rag_core import LlamaCppEmbedder
|
||||
|
||||
QDRANT_URL = os.getenv("QDRANT_URL", "http://127.0.0.1:6333")
|
||||
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")
|
||||
|
||||
@@ -7,15 +7,6 @@
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加必要的路径
|
||||
project_root = os.path.dirname(os.path.abspath(__file__))
|
||||
frontend_src = os.path.join(project_root, "frontend", "src")
|
||||
backend_dir = os.path.join(project_root, "backend")
|
||||
|
||||
sys.path.insert(0, project_root)
|
||||
sys.path.insert(0, frontend_src)
|
||||
sys.path.insert(0, backend_dir)
|
||||
|
||||
print("=" * 60)
|
||||
print("前端导入测试")
|
||||
print("=" * 60)
|
||||
@@ -32,7 +23,7 @@ except Exception as e:
|
||||
# 测试 2: 导入配置
|
||||
print("\n[测试 2] 导入配置...")
|
||||
try:
|
||||
from config import config
|
||||
from frontend.src.config import config
|
||||
print(f"✅ config 导入成功: page_title={config.page_title}")
|
||||
except Exception as e:
|
||||
print(f"❌ 导入失败: {e}")
|
||||
@@ -40,7 +31,7 @@ except Exception as e:
|
||||
# 测试 3: 导入状态管理
|
||||
print("\n[测试 3] 导入状态管理...")
|
||||
try:
|
||||
from state import AppState
|
||||
from frontend.src.state import AppState
|
||||
print("✅ AppState 导入成功")
|
||||
except Exception as e:
|
||||
print(f"❌ 导入失败: {e}")
|
||||
@@ -48,7 +39,7 @@ except Exception as e:
|
||||
# 测试 4: 导入 API 客户端
|
||||
print("\n[测试 4] 导入 API 客户端...")
|
||||
try:
|
||||
from api_client import api_client
|
||||
from frontend.src.api_client import api_client
|
||||
print("✅ api_client 导入成功")
|
||||
except Exception as e:
|
||||
print(f"❌ 导入失败: {e}")
|
||||
@@ -56,9 +47,9 @@ except Exception as e:
|
||||
# 测试 5: 导入组件
|
||||
print("\n[测试 5] 导入组件...")
|
||||
try:
|
||||
from components.sidebar import render_sidebar
|
||||
from components.chat_area import render_chat_area
|
||||
from components.info_panel import render_info_panel
|
||||
from frontend.src.components.sidebar import render_sidebar
|
||||
from frontend.src.components.chat_area import render_chat_area
|
||||
from frontend.src.components.info_panel import render_info_panel
|
||||
print("✅ 所有组件导入成功")
|
||||
except Exception as e:
|
||||
print(f"❌ 导入失败: {e}")
|
||||
|
||||
@@ -15,10 +15,9 @@ import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 加载环境变量（Qdrant URL、PostgreSQL 连接等）
|
||||
load_dotenv()
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..")
|
||||
load_dotenv(os.path.join(project_root, ".env"))
|
||||
|
||||
# 添加项目根目录到路径
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
from pydantic import SecretStr
|
||||
from langchain_openai import ChatOpenAI
|
||||
from rag_indexer.index_builder import IndexBuilderConfig
|
||||
|
||||
@@ -6,18 +6,12 @@
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
# 添加项目根目录到 Python 路径
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from rag_indexer.index_builder import IndexBuilder
|
||||
from rag_indexer.splitters import SplitterType
|
||||
|
||||
from rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from app.model_services import get_embedding_service
|
||||
from backend.rag_core import QdrantVectorStore, get_sparse_embedder
|
||||
from backend.app.model_services import get_embedding_service
|
||||
from qdrant_client import models
|
||||
|
||||
|
||||
@@ -36,6 +30,7 @@ async def test_index_builder():
|
||||
)
|
||||
|
||||
# 测试文档路径
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
test_file = os.path.join(project_root, "data", "user_docs", "doublestory.txt")
|
||||
|
||||
if os.path.exists(test_file):
|
||||
|
||||
@@ -7,10 +7,7 @@ import asyncio
|
||||
import os
|
||||
import sys
|
||||
|
||||
project_root = os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
sys.path.insert(0, os.path.join(project_root, "backend"))
|
||||
|
||||
from app.rag.retriever import create_hybrid_retriever, create_parent_hybrid_retriever
|
||||
from backend.app.rag.retriever import create_hybrid_retriever, create_parent_hybrid_retriever
|
||||
|
||||
|
||||
def test_hybrid_retriever():
|
||||
|
||||
@@ -2,16 +2,16 @@
|
||||
"""
|
||||
LangGraph 图结构可视化脚本
|
||||
快速查看节点和边的连接关系
|
||||
运行方式：python backend/app/graph/visualize_graph.py
|
||||
运行方式：python tools/visualize_graph.py
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 确定项目根目录（Agent1 目录）
|
||||
# 当前文件位置：backend/app/graph/visualize_graph.py
|
||||
# 向上 4 级到 Agent1
|
||||
PROJECT_ROOT = Path(__file__).parent.parent.parent.parent
|
||||
# 当前文件位置：tools/visualize_graph.py
|
||||
# 向上 1 级到 Agent1
|
||||
PROJECT_ROOT = Path(__file__).parent.parent
|
||||
BACKEND_DIR = PROJECT_ROOT / "backend"
|
||||
|
||||
# 关键：把 backend 目录加入 sys.path，这样才能找到 rag_core
|
||||
@@ -23,9 +23,9 @@ if str(PROJECT_ROOT) not in sys.path:
|
||||
|
||||
load_dotenv(PROJECT_ROOT / ".env")
|
||||
|
||||
from app.agent.agent_service import AIAgentService
|
||||
from app.config import DB_URI
|
||||
from app.main_graph.checkpoint.postgres.aio import AsyncPostgresSaver
|
||||
from backend.app.agent.agent_service import AIAgentService
|
||||
from backend.app.config import DB_URI
|
||||
from backend.app.main_graph.checkpoint.postgres.aio import AsyncPostgresSaver
|
||||
import asyncio
|
||||
|
||||
|
||||
Reference in New Issue
Block a user