feat: 事件时间轴缩略帧 + 人物管理头像 + 人物合并硬规则校验
## 新架构：Oracle 集中计算 + NAS 代理展示 ### Oracle 端 (fam-edge) - 新增 frame_service: ffmpeg 视频抽帧 + VLM 人物定位裁剪头像（磁盘缓存） - 新增 /api/oracle/frame: 按 video_id+ts 抽帧返回 jpeg（带 token） - 新增 /api/oracle/avatar: 按 label 生成人物头像（VLM 定位人物 + 兜底整帧居中） - 新增 person_identifier: 人物身份识别模块 - Gemini 适配器支持 flash/flash-lite 双模型切换，429 自动降级 - frame_service VLM 全模型 429 时进入 10 分钟熔断，避免每次请求白打配额 - 兜底头像不落缓存，配额恢复后自动重试 VLM 精确定位 ### 人物合并硬规则校验（框架级修复） - person_service: LLM 合并结果落库前加硬冲突检测 - 性别冲突 → 绝不合并 - 年龄档跨未成年/成年 → 绝不合并（防止把爷爷/宝宝并进同一人） - oracle_db: upsert_person 入口剥离括号后缀（人物A（别名：人物B） → 人物A），消灭垃圾人物行 - 修复 set_canonical 丢弃 source 参数的 bug（旧代码硬编码 'manual' 导致错误合并被永久固化） - get_events_for_label: 只提取该身份组的特征文本，头像定位更精准 ### NAS 端 (fam-core) - 新增 img_proxy: /api/proxy/frame 和 /api/proxy/avatar 代理 Oracle 图片 - app.py 注册 img_bp 蓝图 - oracle_sync / db_layer / member_manager 同步人物表 ### UI 端 (fam-ui) - 事件时间轴: 每条事件卡片加时间点缩略帧 - 人物管理: 每人卡片加头像（150x150 圆角） - parse_persons: 剥离括号备注，与 Oracle 归一化一致 - 新增 EventItem 组件、Timeline 页改造 - Chat / ServiceStatus 页相应调整 ### 数据库 - scripts/ddl.sql: 同步表结构更新 - Oracle people 表: features_json / display_uid / source 字段完善
This commit is contained in:
ericwyuan
@@ -66,6 +66,38 @@ motion_segment:
|
||||
min_duration_sec: 1 # 短于该时长的事件不分割
|
||||
unfinished_grace_sec: 10 # start_time+duration 距当前 ≤ 该秒视为"已结束"容差
|
||||
|
||||
# 闭集人物识别（2026-08-22 新增，家里固定 4 人：爷爷/爸爸/媳妇/汤圆）：
|
||||
# 原来靠大模型自己编的"人物A/B/C"临时 uid + 文字特征描述跨视频合并，验证下来
|
||||
# 不可靠（用户原话"现在的识别全是错的"）。人脸向量方案也验证过，家庭监控这种
|
||||
# 大广角/远距离画面下同人内部相似度经常比不同人还低，此路不通。
|
||||
# 现在改成：汤圆(幼儿/儿童特征)/媳妇(唯一成年女性) 直接用 Gemini 已产出的
|
||||
# 性别/年龄字段判断，免费且验证下来接近 100% 准；爷爷/爸爸(两个成年男性，纯
|
||||
# 外观规则/人脸向量都区分不开) 改用视觉大模型"看参考图比对"——NVIDIA 实测
|
||||
# 6/6 全对且配额与主分析链路完全独立，设为优先，Gemini flash-lite(7/8) 兜底。
|
||||
# 参考图目录结构：{ref_dir}/爷爷/*.jpg、{ref_dir}/爸爸/*.jpg（已用确认过身份的
|
||||
# 历史截图种好，见 PROGRESS.md 记录）。
|
||||
person_identifier:
|
||||
enabled: true
|
||||
ref_dir: "/opt/fam-edge/data/person_refs"
|
||||
max_ref_per_person: 6
|
||||
# 连续两次分类调用之间的最小间隔（不分 provider 统一限速）：正常处理新片段时
|
||||
# 调用本来就稀疏，这个主要是给历史数据批量回填用的，避免短时间内密集调用打爆配额
|
||||
min_call_interval_sec: 2
|
||||
nvidia:
|
||||
api_key: "${NVIDIA_API_KEY}"
|
||||
model_name: "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"
|
||||
fallback_models: [] # 目前只验证过这一个能用的 NVIDIA 视觉模型，先留好扩展位
|
||||
timeout: 60
|
||||
max_retries: 3 # 每个模型对瞬时故障（429/5xx/超时）最多重试几次（含首次）
|
||||
retry_backoff_sec: 3 # 重试前基础等待秒数，指数退避（3s -> 6s -> 12s）
|
||||
gemini:
|
||||
api_key: "${GEMINI_API_KEY}"
|
||||
extra_api_keys: ["${GEMINI_API_KEY_2}", "${GEMINI_API_KEY_3}", "${GEMINI_API_KEY_4}"]
|
||||
model_name: "gemini-flash-lite-latest"
|
||||
timeout: 60
|
||||
max_retries: 2
|
||||
retry_backoff_sec: 3
|
||||
|
||||
# 智能问答降级链（与视频分析独立）：Gemini -> NVIDIA -> 本地 Ollama
|
||||
models:
|
||||
- provider: "gemini"
|
||||
@@ -90,6 +122,9 @@ models:
|
||||
- "智能摄像头-3"
|
||||
- "智能摄像头-4"
|
||||
timeout: 600
|
||||
# 问答专用超时（跟上面视频分析的 timeout 分开）：用户在等交互式回答，一个
|
||||
# key/模型卡住不该等 10 分钟，超时要短，快速降级到下一个 key/模型/provider
|
||||
chat_timeout: 20
|
||||
# 模型级独立超时（最终值，不参与编排层 ×2 放大）
|
||||
# gemini-flash-lite 实测 ~22-34s，按用户要求放宽至 8 分钟（480s），避免大视频/排队时过早切断
|
||||
model_timeouts:
|
||||
@@ -119,6 +154,7 @@ models:
|
||||
base_url: "https://integrate.api.nvidia.com/v1"
|
||||
api_key: "${NVIDIA_API_KEY}"
|
||||
timeout: 600
|
||||
chat_timeout: 20 # 问答专用超时，跟视频分析的 timeout 分开
|
||||
max_base64_mb: 20 # 超过此大小直接跳过 NVIDIA，不做注定失败的编码+上传
|
||||
switch_interval_sec: 5 # 模型切换间隔：一个失败后等待再试下一个（未来加模型时用）
|
||||
model_timeouts: # 模型级独立超时（最终值，不参与 ×2）
|
||||
|
||||
@@ -14,6 +14,7 @@ API-Gateway - Flask 蓝图（新架构 v3）
|
||||
一次 Gemini 调用一并产出（见 ai_orchestrator/prompts.py），frame_service 不再
|
||||
额外调用任何模型。NAS 经 core 代理读取，不在 NAS 做图像计算。
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
from flask import Blueprint, request, jsonify, Response
|
||||
@@ -102,6 +103,39 @@ def people_correct():
|
||||
return jsonify({"status": "ok", "label": label, "canonical_name": canonical}), 200
|
||||
|
||||
|
||||
@api_bp.route('/api/oracle/identity/correct', methods=['POST'])
|
||||
def identity_correct():
|
||||
"""事件时间轴/人物管理"这个人识别错了"纠错入口（比 people/correct 粒度更细）。
|
||||
|
||||
请求: {"video_id": 123, "current_name": "爷爷", "new_name": "爸爸", "token": "..."}
|
||||
只改这一段视频里被错误识别的那个人，不影响同名字符串在其他视频里的映射——
|
||||
人物 uid 只在单次视频分析内稳定，同一个"人物A"字符串在不同视频里可能是不同
|
||||
真人，纠错必须落到 (video_id, 当前展示名) 这一粒度，不能按全局 label 改。
|
||||
写 manual 来源，受保护不会被后续自动识别覆盖回去；立即重写这段视频的展示数据。
|
||||
"""
|
||||
if not _check_token():
|
||||
return jsonify({"error": "unauthorized"}), 401
|
||||
data = request.get_json(silent=True)
|
||||
if not data:
|
||||
return jsonify({"error": "Invalid JSON"}), 400
|
||||
video_id = data.get('video_id')
|
||||
current_name = (data.get('current_name') or '').strip()
|
||||
new_name = (data.get('new_name') or '').strip()
|
||||
if not video_id or not current_name or not new_name:
|
||||
return jsonify({"error": "缺少 video_id / current_name / new_name"}), 400
|
||||
try:
|
||||
video_id = int(video_id)
|
||||
except (TypeError, ValueError):
|
||||
return jsonify({"error": "video_id 必须是数字"}), 400
|
||||
try:
|
||||
state.get_db().correct_video_identity(video_id, current_name, new_name)
|
||||
except Exception as e:
|
||||
logger.error(f"identity_correct 异常: {e}")
|
||||
return jsonify({"error": str(e)}), 500
|
||||
return jsonify({"status": "ok", "video_id": video_id,
|
||||
"current_name": current_name, "new_name": new_name}), 200
|
||||
|
||||
|
||||
@api_bp.route('/api/edge/chat/ask', methods=['POST'])
|
||||
def chat_ask():
|
||||
"""智能问答编排：Gemini → NVIDIA → 本地 Ollama（两云端都失败才用本地兜底）
|
||||
@@ -125,6 +159,28 @@ def chat_ask():
|
||||
return jsonify({"answer": answer, "provider": provider}), 200
|
||||
|
||||
|
||||
@api_bp.route('/api/edge/chat/ask/stream', methods=['POST'])
|
||||
def chat_ask_stream():
|
||||
"""智能问答编排（流式版）：SSE 逐块推送，边生成边显示，不用等全量回答。
|
||||
|
||||
请求同 /api/edge/chat/ask。响应 Content-Type: text/event-stream，
|
||||
每行 `data: <json>\\n\\n`，json 结构见 qa.QAOrchestrator.run_qa_stream 注释。
|
||||
"""
|
||||
data = request.get_json(silent=True)
|
||||
if not data or 'prompt' not in data:
|
||||
return jsonify({"error": "缺少必填字段: prompt"}), 400
|
||||
|
||||
prompt = data['prompt']
|
||||
max_tokens = int(data.get('max_tokens', 1024))
|
||||
|
||||
def generate():
|
||||
for event in get_qa().run_qa_stream(prompt, max_tokens=max_tokens):
|
||||
yield f"data: {json.dumps(event, ensure_ascii=False)}\n\n"
|
||||
|
||||
return Response(generate(), mimetype='text/event-stream',
|
||||
headers={'Cache-Control': 'no-cache', 'X-Accel-Buffering': 'no'})
|
||||
|
||||
|
||||
@api_bp.route('/api/oracle/activity', methods=['GET'])
|
||||
def activity():
|
||||
"""实时服务状态 + 最近活动流（token 校验）。
|
||||
|
||||
@@ -91,7 +91,10 @@ def extract_frame(db, video_id: int, ts: str, width: int = FRAME_W) -> bytes:
|
||||
offset = 0.0
|
||||
|
||||
_ensure_dir()
|
||||
cache = os.path.join(CACHE_DIR, f"frame_{video_id}_{int(offset)}.jpg")
|
||||
# 缓存 key 必须带 width：同一 (video_id, offset) 不同调用方可能要不同分辨率
|
||||
# （时间轴缩略图 400px / 头像 600px / 人物识别裁人脸要接近原始分辨率 2880px），
|
||||
# 不带 width 会导致后来的高分辨率请求悄悄拿到早先缓存的低分辨率帧。
|
||||
cache = os.path.join(CACHE_DIR, f"frame_{video_id}_{int(offset)}_{width}.jpg")
|
||||
if os.path.isfile(cache) and os.path.getsize(cache) > 0:
|
||||
with open(cache, 'rb') as f:
|
||||
return f.read()
|
||||
|
||||
@@ -80,6 +80,15 @@ class BaseModelAdapter(ABC):
|
||||
raise NotImplementedError(
|
||||
f"{self.provider_name} 适配器未实现 chat()（不参与智能问答）")
|
||||
|
||||
def chat_stream(self, prompt: str, max_tokens: int = 512):
|
||||
"""流式问答：逐块 yield 文本增量。默认实现退化为"等 chat() 整段返回后
|
||||
一次性当一个大 chunk 吐出"——子类没有真流式 API（或懒得接）时这样也能
|
||||
用，只是没有逐字显示的效果；Gemini 有原生 SSE 流式接口，重写了这个方法。
|
||||
"""
|
||||
result = self.chat(prompt, max_tokens=max_tokens)
|
||||
if result:
|
||||
yield result
|
||||
|
||||
@abstractmethod
|
||||
def get_timeout(self) -> int:
|
||||
"""该模型的调用超时秒数"""
|
||||
|
||||
@@ -74,6 +74,10 @@ class GeminiAdapter(BaseModelAdapter):
|
||||
self.key_labels.append(str(label))
|
||||
self.api_key = self.api_keys[0] if self.api_keys else '' # 向后兼容单 key 用法
|
||||
self.timeout = config.get('timeout', 600)
|
||||
# 问答（chat）专用超时——跟视频分析的 timeout 分开，不能共用 600s：
|
||||
# 智能问答是同步等待用户看结果的交互场景，一个 key/模型卡住不该让用户等
|
||||
# 10 分钟，超时应该短、快速降级到下一个 key/模型/provider
|
||||
self.chat_timeout = config.get('chat_timeout', 20)
|
||||
# 模型级独立超时（最终值，不参与编排层 ×N 放大）: {model_name: seconds}
|
||||
# 例: {"gemini-flash-lite-latest": 90}（按实测耗时 ×4 配置）
|
||||
self.model_timeouts = {
|
||||
@@ -431,10 +435,10 @@ class GeminiAdapter(BaseModelAdapter):
|
||||
"generationConfig": {
|
||||
"temperature": temperature,
|
||||
"maxOutputTokens": max_tokens}},
|
||||
timeout=self.timeout
|
||||
timeout=self.chat_timeout
|
||||
)
|
||||
except requests.Timeout:
|
||||
logger.warning(f"Gemini {key_label} [{model}] 问答超时")
|
||||
logger.warning(f"Gemini {key_label} [{model}] 问答超时({self.chat_timeout}s)")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Gemini {key_label} [{model}] 问答异常: {e}")
|
||||
@@ -452,6 +456,69 @@ class GeminiAdapter(BaseModelAdapter):
|
||||
continue
|
||||
return None
|
||||
|
||||
def chat_stream(self, prompt: str, max_tokens: int = 512):
|
||||
"""流式问答：逐块 yield 文本增量。用于聊天界面边生成边显示，不用等全量
|
||||
返回再展示——之前整段等待是"卡住没反馈"体验差的根源之一。
|
||||
|
||||
按 key 轮换 × 模型链依次尝试，但只在"这次尝试还没吐出任何文本"时才允许
|
||||
换下一个 key/模型；一旦已经开始吐字给用户看了，中途出错就直接结束这次
|
||||
生成（不再悄悄换 provider 接着写，否则会出现两段风格/内容不连贯的回答
|
||||
拼在一起，比直接告知"生成中断"更让人困惑）。
|
||||
"""
|
||||
if self._cb.is_open():
|
||||
logger.warning("Gemini 熔断器 OPEN，跳过问答（流式）")
|
||||
return
|
||||
if not self.api_keys:
|
||||
logger.warning("Gemini API Key 未配置，跳过问答（流式）")
|
||||
return
|
||||
got_any = False
|
||||
for idx, api_key in self._rotated_keys():
|
||||
key_label = self.key_labels[idx]
|
||||
for model in self.model_chain:
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{self._base_url}/models/{model}:streamGenerateContent"
|
||||
f"?alt=sse&key={api_key}",
|
||||
json={"contents": [{"parts": [{"text": prompt}]}],
|
||||
"generationConfig": {
|
||||
"temperature": 0.3, "maxOutputTokens": max_tokens}},
|
||||
timeout=self.chat_timeout, stream=True,
|
||||
)
|
||||
except requests.Timeout:
|
||||
logger.warning(f"Gemini {key_label} [{model}] 流式问答超时({self.chat_timeout}s)")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Gemini {key_label} [{model}] 流式问答异常: {e}")
|
||||
continue
|
||||
if resp.status_code != 200:
|
||||
logger.warning(f"Gemini {key_label} [{model}] 流式问答 HTTP {resp.status_code}")
|
||||
resp.close()
|
||||
continue
|
||||
try:
|
||||
for line in resp.iter_lines(decode_unicode=True):
|
||||
if not line or not line.startswith('data: '):
|
||||
continue
|
||||
chunk = line[len('data: '):]
|
||||
try:
|
||||
obj = json.loads(chunk)
|
||||
except ValueError:
|
||||
continue
|
||||
cands = obj.get('candidates', [])
|
||||
text = ''.join(
|
||||
p.get('text', '')
|
||||
for p in (cands[0].get('content', {}) if cands else {}).get('parts', []))
|
||||
if text:
|
||||
got_any = True
|
||||
yield text
|
||||
except Exception as e:
|
||||
logger.warning(f"Gemini {key_label} [{model}] 流式读取中断: {e}")
|
||||
finally:
|
||||
resp.close()
|
||||
if got_any:
|
||||
self._cb.record_success()
|
||||
return # 已经开始吐字，不管这次是否读完都不再换 provider
|
||||
self._cb.record_failure()
|
||||
|
||||
def get_timeout(self) -> int:
|
||||
return self.timeout
|
||||
|
||||
|
||||
@@ -62,6 +62,8 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
self.api_key = self._resolve_key(config.get('api_key', ''))
|
||||
self.base_url = config.get('base_url', 'https://integrate.api.nvidia.com/v1')
|
||||
self.timeout = config.get('timeout', 600)
|
||||
# 问答专用超时，跟视频分析分开——交互式问答不该等到跟视频分析一样久
|
||||
self.chat_timeout = config.get('chat_timeout', 20)
|
||||
self.max_base64_mb = float(config.get('max_base64_mb', 20))
|
||||
# 模型级独立超时（最终值，不参与编排层 ×N 放大）: {model_name: seconds}
|
||||
self.model_timeouts = {
|
||||
@@ -209,7 +211,7 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.3,
|
||||
max_tokens=max_tokens,
|
||||
timeout=self.timeout
|
||||
timeout=self.chat_timeout
|
||||
)
|
||||
content = resp.choices[0].message.content
|
||||
if content:
|
||||
|
||||
@@ -120,11 +120,21 @@ class OracleDB:
|
||||
thumbnail_url TEXT,
|
||||
received_at TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS person_identity_map (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
video_id INTEGER,
|
||||
raw_uid TEXT,
|
||||
canonical_name TEXT,
|
||||
source TEXT,
|
||||
updated_at TEXT,
|
||||
UNIQUE(video_id, raw_uid)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_videos_updated ON videos(updated_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_video ON events(video_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_calls_created ON model_calls(created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_activity_ts ON service_activity(ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_motion_window ON ss_motion_events(start_time, event_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_identity_map_video ON person_identity_map(video_id);
|
||||
""")
|
||||
# 兼容旧库：补 retry_count / file_valid / media 等列（生产-消费队列用）
|
||||
cols = [r[1] for r in c.execute("PRAGMA table_info(videos)").fetchall()]
|
||||
@@ -599,6 +609,109 @@ class OracleDB:
|
||||
'features_text': features_text,
|
||||
'event_start_time': best['event_start_time']}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 人物对应关系表（2026-08-22 新增）：记录"某个视频里 Gemini 给的原始 uid"
|
||||
# 与"闭集识别解析出的规范名"之间的映射，作为可追溯、可纠错的中间层。
|
||||
#
|
||||
# 设计动机：events.person_appearances_json 里的 uid 只在单次视频分析内稳定，
|
||||
# 同一字符串在不同视频里完全可能指向不同真人——不能靠"改 label 的
|
||||
# canonical_name"来纠错（一个 label 撞了多个真人，改一次就把另一个人也带歪
|
||||
# 了）。所以纠错必须落到 (video_id, raw_uid) 这一粒度，而不是全局 label。
|
||||
#
|
||||
# events/videos 表里实际展示用的 person_list_json / person_appearances_json /
|
||||
# people_json 会在识别（或纠正）时被直接重写成规范名（rewrite_event_person_names），
|
||||
# 保持"读的时候不用现查表拼接"的简单模型；这张表只作为"这次重写是怎么来的"的
|
||||
# 记录 + 纠错操作的定位依据，不参与展示时的实时查询。
|
||||
# ------------------------------------------------------------------
|
||||
def set_identity_mapping(self, video_id: int, raw_uid: str,
|
||||
canonical_name: str, source: str = 'auto_id') -> bool:
|
||||
"""记录/更新 (video_id, raw_uid) -> canonical_name。manual 来源受保护，
|
||||
不会被后续自动识别结果（rule/auto_id）覆盖。返回是否真的发生了变化
|
||||
（调用方据此决定要不要顺带重写 events 展示数据）。"""
|
||||
now = _now_iso()
|
||||
row = self._conn.execute(
|
||||
"SELECT canonical_name, source FROM person_identity_map "
|
||||
"WHERE video_id=? AND raw_uid=?", (video_id, raw_uid)).fetchone()
|
||||
if row:
|
||||
if row['source'] == 'manual' and source != 'manual':
|
||||
return False
|
||||
if row['canonical_name'] == canonical_name and row['source'] == source:
|
||||
return False
|
||||
self._conn.execute(
|
||||
"UPDATE person_identity_map SET canonical_name=?, source=?, updated_at=? "
|
||||
"WHERE video_id=? AND raw_uid=?",
|
||||
(canonical_name, source, now, video_id, raw_uid))
|
||||
else:
|
||||
self._conn.execute(
|
||||
"INSERT INTO person_identity_map "
|
||||
"(video_id, raw_uid, canonical_name, source, updated_at) VALUES (?,?,?,?,?)",
|
||||
(video_id, raw_uid, canonical_name, source, now))
|
||||
self._conn.commit()
|
||||
return True
|
||||
|
||||
def get_identity_map_for_video(self, video_id: int) -> Dict[str, str]:
|
||||
rows = self._conn.execute(
|
||||
"SELECT raw_uid, canonical_name FROM person_identity_map WHERE video_id=?",
|
||||
(video_id,)).fetchall()
|
||||
return {r['raw_uid']: r['canonical_name'] for r in rows if r['canonical_name']}
|
||||
|
||||
def rewrite_event_person_names(self, video_id: int, rename_map: Dict[str, str]):
|
||||
"""按 {当前展示名: 新名} 把该视频全部 events 的 person_list_json /
|
||||
person_appearances_json[].uid，以及 videos.people_json 里的名字替换掉。
|
||||
rename_map 的 key 是"事件数据里当前显示的名字"（可能是原始 uid，也可能是
|
||||
上一轮已经替换过的规范名——纠错场景下就是这种情况）。
|
||||
"""
|
||||
if not rename_map:
|
||||
return
|
||||
now = _now_iso()
|
||||
with self._write_lock:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, person_list_json, person_appearances_json FROM events "
|
||||
"WHERE video_id=?", (video_id,)).fetchall()
|
||||
for r in rows:
|
||||
changed = False
|
||||
plist = json.loads(r['person_list_json'] or '[]')
|
||||
new_plist = [rename_map.get(x, x) for x in plist]
|
||||
if new_plist != plist:
|
||||
changed = True
|
||||
pa = json.loads(r['person_appearances_json']) if r['person_appearances_json'] else None
|
||||
if pa:
|
||||
for p in pa:
|
||||
if isinstance(p, dict) and p.get('uid') in rename_map:
|
||||
p['uid'] = rename_map[p['uid']]
|
||||
changed = True
|
||||
if changed:
|
||||
self._conn.execute(
|
||||
"UPDATE events SET person_list_json=?, person_appearances_json=? "
|
||||
"WHERE id=?",
|
||||
(json.dumps(new_plist, ensure_ascii=False),
|
||||
json.dumps(pa, ensure_ascii=False) if pa is not None
|
||||
else r['person_appearances_json'],
|
||||
r['id']))
|
||||
vrow = self._conn.execute(
|
||||
"SELECT people_json FROM videos WHERE id=?", (video_id,)).fetchone()
|
||||
if vrow and vrow['people_json']:
|
||||
plist = json.loads(vrow['people_json'])
|
||||
new_plist = [rename_map.get(x, x) for x in plist]
|
||||
if new_plist != plist:
|
||||
self._conn.execute(
|
||||
"UPDATE videos SET people_json=?, updated_at=? WHERE id=?",
|
||||
(json.dumps(new_plist, ensure_ascii=False), now, video_id))
|
||||
self._conn.commit()
|
||||
|
||||
def correct_video_identity(self, video_id: int, current_name: str, new_name: str):
|
||||
"""纠错入口（人物管理页 / 事件时间轴"修正"按钮都走这个）：把某视频里当前
|
||||
展示为 current_name 的人物改成 new_name。写 manual 来源，受保护不会被后续
|
||||
自动识别覆盖回去；同时立即重写这段视频的展示数据，不用等下一轮识别。"""
|
||||
row = self._conn.execute(
|
||||
"SELECT raw_uid FROM person_identity_map WHERE video_id=? AND canonical_name=?",
|
||||
(video_id, current_name)).fetchone()
|
||||
# 没有映射记录（比如这条数据是老流水线时代产出的，从没跑过闭集识别）
|
||||
# 就把 current_name 本身当 raw_uid 存一条新映射
|
||||
raw_uid = row['raw_uid'] if row else current_name
|
||||
self.set_identity_mapping(video_id, raw_uid, new_name, source='manual')
|
||||
self.rewrite_event_person_names(video_id, {current_name: new_name})
|
||||
|
||||
def get_events_for_label(self, label: str, limit: int = 6):
|
||||
"""该人物（canonical_name 或 UID label）出现的候选事件，按时间倒序（最近优先）。
|
||||
|
||||
@@ -745,8 +858,11 @@ class OracleDB:
|
||||
row['features_json'] if row else None, features) if row else (
|
||||
self._merge_features(None, features))
|
||||
if row:
|
||||
# manual 覆盖 llm；llm 不覆盖 manual
|
||||
if source == 'manual' or row['source'] != 'manual':
|
||||
# manual/auto_id 覆盖 llm；llm 不覆盖 manual/auto_id（auto_id 是闭集人物
|
||||
# 识别的确定性结论——比 llm 的文字特征合并猜测可靠得多，同样需要保护，
|
||||
# 不能被后续 person_service 的 llm 合并跑批悄悄覆盖回去）
|
||||
_protected = ('manual', 'auto_id', 'rule')
|
||||
if source in _protected or row['source'] not in _protected:
|
||||
self._conn.execute(
|
||||
"UPDATE people SET canonical_name=?, source=?, appearances=appearances+1, "
|
||||
"features_json=?, display_uid=?, updated_at=? WHERE label=?",
|
||||
@@ -804,7 +920,8 @@ class OracleDB:
|
||||
now = _now_iso()
|
||||
row = self._conn.execute("SELECT * FROM people WHERE label=?", (label,)).fetchone()
|
||||
if row:
|
||||
if source == 'manual' or row['source'] != 'manual':
|
||||
_protected = ('manual', 'auto_id', 'rule')
|
||||
if source in _protected or row['source'] not in _protected:
|
||||
self._conn.execute(
|
||||
"UPDATE people SET appearances=?, source=?, updated_at=? WHERE label=?",
|
||||
(int(count), source, now, label))
|
||||
@@ -853,6 +970,11 @@ class OracleDB:
|
||||
model_calls = self._conn.execute(
|
||||
"SELECT * FROM model_calls WHERE created_at >= ? ORDER BY id ASC",
|
||||
(since_iso,)).fetchall()
|
||||
# 人物对应关系表（甲骨文不稳定，提取出的有效数据都要同步到 NAS 防丢失；
|
||||
# 这张表是识别结果的可追溯记录 + 纠错依据，同样纳入增量同步）
|
||||
identity_map = self._conn.execute(
|
||||
"SELECT * FROM person_identity_map WHERE updated_at > ? ORDER BY id ASC",
|
||||
(since_iso,)).fetchall()
|
||||
|
||||
def _ser(row):
|
||||
d = dict(row)
|
||||
@@ -863,6 +985,7 @@ class OracleDB:
|
||||
"events": [_ser(e) for e in events],
|
||||
"people": [_ser(p) for p in people],
|
||||
"model_calls": [_ser(m) for m in model_calls],
|
||||
"identity_map": [_ser(m) for m in identity_map],
|
||||
"server_time": _now_iso(),
|
||||
}
|
||||
|
||||
|
||||
256
fam-edge/src/fam_edge/person_identifier.py
Normal file
256
fam-edge/src/fam_edge/person_identifier.py
Normal file
@@ -0,0 +1,256 @@
|
||||
"""
|
||||
PersonIdentifier - 闭集人物识别（家里固定 4 个人：爷爷/爸爸/媳妇/汤圆）
|
||||
|
||||
背景（2026-08-22）: 原来靠大模型每次视频分析自己编的"人物A/B/C"临时 uid + 一段
|
||||
性别/年龄/衣着文字描述做跨视频合并，原理上就不可靠——文字描述会因光线/角度/换衣服
|
||||
对不上，反复出现张冠李戴（用户原话："现在的识别全是错的"）。人脸向量方案也验证
|
||||
过，家庭监控这种大广角/远距离/糊画面下同人内部相似度经常比不同人还低，此路不通。
|
||||
|
||||
现在改成基于已知这个家庭只有 4 个固定成员的闭集规则：
|
||||
- 汤圆（幼儿/儿童）、媳妇（唯一成年女性）：Gemini 每次分析已经会标性别/年龄段，
|
||||
这两条命中率验证下来接近 100%，直接用，不需要额外模型调用。
|
||||
- 爷爷、爸爸（两个成年男性，纯外观规则/人脸向量都区分不开）：改用视觉大模型
|
||||
"看图比对"——给几张已确认身份的参考图 + 待判断的截图，直接问模型这是谁。
|
||||
实测 NVIDIA nemotron-omni 在留出测试集上 6/6 全对，Gemini flash-lite 7/8，
|
||||
NVIDIA 配额与 Gemini 完全独立、不跟主分析链路抢配额，设为优先。
|
||||
|
||||
调用粒度：每个运动片段（视频行）只调一次（不是每个事件都调）——同一段视频里
|
||||
人不会中途换衣服，取片段内最大 bbox 的成年男性外观代表整段。
|
||||
|
||||
健壮性（2026-08-22 补，用于支撑历史数据批量回填）:
|
||||
- NVIDIA/Gemini 各自支持模型链（model_chain，fallback_models 可再加型号）+
|
||||
每个模型独立重试（429/503/超时/连接错误这类瞬时故障，指数退避），非瞬时错误
|
||||
（400 参数错误等）不重试、直接换下一个模型/provider。
|
||||
- min_call_interval_sec 控制连续两次分类调用之间的最小间隔（不分 provider 统一
|
||||
限速）——批量回填时会短时间内密集调用，需要限速避免打爆配额/被限流。
|
||||
"""
|
||||
import base64
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
from .logger import setup_logger
|
||||
|
||||
try:
|
||||
from openai import OpenAI
|
||||
except ImportError:
|
||||
OpenAI = None
|
||||
|
||||
logger = setup_logger('fam-edge.person_identifier')
|
||||
|
||||
REF_DIR_DEFAULT = '/opt/fam-edge/data/person_refs'
|
||||
ADULT_MALE_CANDIDATES = ('爷爷', '爸爸')
|
||||
# HTTP 状态码：值得重试的瞬时故障（配额限流/服务过载），其余（400 参数错误/401 鉴权等）不重试
|
||||
_RETRYABLE_STATUS = (429, 500, 502, 503, 504)
|
||||
|
||||
|
||||
class PersonIdentifier:
|
||||
def __init__(self, config: dict):
|
||||
self.enabled = bool(config.get('enabled', True))
|
||||
self.ref_dir = config.get('ref_dir', REF_DIR_DEFAULT)
|
||||
self.max_ref_per_person = int(config.get('max_ref_per_person', 6))
|
||||
self.min_call_interval_sec = float(config.get('min_call_interval_sec', 2))
|
||||
self._last_call_at = 0.0
|
||||
|
||||
nv = config.get('nvidia', {})
|
||||
self.nvidia_model_chain = [nv.get('model_name', 'nvidia/nemotron-3-nano-omni-30b-a3b-reasoning')] + [
|
||||
m for m in nv.get('fallback_models', []) or [] if m]
|
||||
self.nvidia_base_url = nv.get('base_url', 'https://integrate.api.nvidia.com/v1')
|
||||
self.nvidia_api_key = self._resolve(nv.get('api_key', '${NVIDIA_API_KEY}'))
|
||||
self.nvidia_timeout = int(nv.get('timeout', 60))
|
||||
self.nvidia_max_retries = int(nv.get('max_retries', 3))
|
||||
self.nvidia_retry_backoff = float(nv.get('retry_backoff_sec', 3))
|
||||
|
||||
gm = config.get('gemini', {})
|
||||
self.gemini_model = gm.get('model_name', 'gemini-flash-lite-latest')
|
||||
raw_keys = [gm.get('api_key', '${GEMINI_API_KEY}')] + list(gm.get('extra_api_keys', []) or [])
|
||||
self.gemini_api_keys = [k for k in (self._resolve(r) for r in raw_keys) if k]
|
||||
self.gemini_timeout = int(gm.get('timeout', 60))
|
||||
self.gemini_max_retries = int(gm.get('max_retries', 2))
|
||||
self.gemini_retry_backoff = float(gm.get('retry_backoff_sec', 3))
|
||||
|
||||
self._refs = None # lazy: {person: [base64_str, ...]}
|
||||
|
||||
@staticmethod
|
||||
def _resolve(raw: str) -> str:
|
||||
if isinstance(raw, str) and raw.startswith('${') and raw.endswith('}'):
|
||||
return os.environ.get(raw[2:-1], '')
|
||||
return raw
|
||||
|
||||
def _load_refs(self):
|
||||
if self._refs is not None:
|
||||
return self._refs
|
||||
refs = {}
|
||||
for person in ADULT_MALE_CANDIDATES:
|
||||
d = os.path.join(self.ref_dir, person)
|
||||
files = []
|
||||
if os.path.isdir(d):
|
||||
files = sorted(f for f in os.listdir(d) if f.lower().endswith(('.jpg', '.jpeg', '.png')))
|
||||
imgs = []
|
||||
for f in files[:self.max_ref_per_person]:
|
||||
try:
|
||||
with open(os.path.join(d, f), 'rb') as fh:
|
||||
imgs.append(base64.b64encode(fh.read()).decode('ascii'))
|
||||
except OSError:
|
||||
continue
|
||||
refs[person] = imgs
|
||||
self._refs = refs
|
||||
return refs
|
||||
|
||||
def has_references(self) -> bool:
|
||||
refs = self._load_refs()
|
||||
return all(refs.get(p) for p in ADULT_MALE_CANDIDATES)
|
||||
|
||||
def _pace(self):
|
||||
"""连续两次分类调用之间强制最小间隔，批量回填时避免短时间内打爆配额。"""
|
||||
if self.min_call_interval_sec <= 0:
|
||||
return
|
||||
wait = self.min_call_interval_sec - (time.time() - self._last_call_at)
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
|
||||
def classify_adult_male(self, crop_bytes: bytes) -> Optional[str]:
|
||||
"""给一张成年男性截图，返回 '爷爷' / '爸爸'，判断不了返回 None（调用方保持原样不动）。
|
||||
|
||||
NVIDIA 优先（配额独立、实测更准，模型链+重试），失败/未配置则退回 Gemini
|
||||
flash-lite（多 key 轮换+重试）。两边都失败返回 None——绝不瞎猜，宁可这次不
|
||||
设置 canonical_name，留给下次（或人工在人物管理页确认）。
|
||||
"""
|
||||
if not self.enabled or not self.has_references():
|
||||
return None
|
||||
self._pace()
|
||||
self._last_call_at = time.time()
|
||||
result = self._classify_nvidia(crop_bytes)
|
||||
if result:
|
||||
return result
|
||||
return self._classify_gemini(crop_bytes)
|
||||
|
||||
def _build_prompt_and_images(self, crop_bytes: bytes):
|
||||
refs = self._load_refs()
|
||||
query_b64 = base64.b64encode(crop_bytes).decode('ascii')
|
||||
images = [] # list of (b64, caption)
|
||||
idx = 1
|
||||
for person in ADULT_MALE_CANDIDATES:
|
||||
for b64 in refs.get(person, []):
|
||||
images.append((b64, f'（上图是参考图{idx}，此人是：{person}）'))
|
||||
idx += 1
|
||||
images.append((query_b64, '（上图是待判断的截图，请判断这是「爷爷」还是「爸爸」）'))
|
||||
prefix = ('下面先给你几张参考图，每张图后面标了这个人是谁'
|
||||
'（这户人家只有这两个成年男性，一个是爷爷，一个是爸爸）：')
|
||||
suffix = ('只根据外观线索（体型/发型/衣着/姿态等）判断，用 JSON 回答，'
|
||||
'格式：{"person":"爷爷或爸爸"}，不要输出其他内容。')
|
||||
return prefix, images, suffix
|
||||
|
||||
def _extract_json_person(self, text: str) -> Optional[str]:
|
||||
text = (text or '').strip()
|
||||
for cand in ('爷爷', '爸爸'):
|
||||
if cand in text:
|
||||
# 两个都出现时（比如复述了参考图说明）不采信，避免误判
|
||||
if '爷爷' in text and '爸爸' in text:
|
||||
# 优先信 JSON 里 "person" 字段紧跟的那个
|
||||
m = re.search(r'"person"\s*:\s*"(爷爷|爸爸)"', text)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return None
|
||||
return cand
|
||||
return None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# NVIDIA：模型链 × 每个模型独立重试（瞬时故障退避重试，非瞬时故障直接换模型）
|
||||
# ------------------------------------------------------------------
|
||||
def _classify_nvidia(self, crop_bytes: bytes) -> Optional[str]:
|
||||
if not self.nvidia_api_key or OpenAI is None:
|
||||
return None
|
||||
prefix, images, suffix = self._build_prompt_and_images(crop_bytes)
|
||||
if len(images) > 12:
|
||||
images = images[-12:] # NVIDIA 单请求最多 12 张图，优先保留最新的参考+待判断图
|
||||
content = [{'type': 'text', 'text': prefix}]
|
||||
for b64, caption in images:
|
||||
content.append({'type': 'image_url', 'image_url': {'url': f'data:image/jpeg;base64,{b64}'}})
|
||||
content.append({'type': 'text', 'text': caption})
|
||||
content.append({'type': 'text', 'text': suffix})
|
||||
|
||||
client = OpenAI(base_url=self.nvidia_base_url, api_key=self.nvidia_api_key)
|
||||
for model in self.nvidia_model_chain:
|
||||
for attempt in range(self.nvidia_max_retries):
|
||||
try:
|
||||
resp = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{'role': 'user', 'content': content}],
|
||||
temperature=0.1, max_tokens=200,
|
||||
timeout=self.nvidia_timeout)
|
||||
text = resp.choices[0].message.content
|
||||
person = self._extract_json_person(text)
|
||||
if person:
|
||||
logger.info(f"NVIDIA[{model}] 人物识别: {person}")
|
||||
return person
|
||||
logger.warning(f"NVIDIA[{model}] 返回结果无法解析出人物: {text[:100] if text else text}")
|
||||
break # 解析不出人物是内容问题，不是瞬时故障，重试没用，换下一个模型
|
||||
except Exception as e:
|
||||
status = getattr(getattr(e, 'response', None), 'status_code', None)
|
||||
retryable = status in _RETRYABLE_STATUS or status is None
|
||||
if retryable and attempt < self.nvidia_max_retries - 1:
|
||||
backoff = self.nvidia_retry_backoff * (2 ** attempt)
|
||||
logger.warning(
|
||||
f"NVIDIA[{model}] 第 {attempt+1}/{self.nvidia_max_retries} 次失败"
|
||||
f"（status={status}），{backoff:.1f}s 后重试: {e}")
|
||||
time.sleep(backoff)
|
||||
continue
|
||||
logger.warning(f"NVIDIA[{model}] 失败（status={status}），换下一个模型: {e}")
|
||||
break
|
||||
return None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Gemini：多 key 轮换 × 每个 key 独立重试
|
||||
# ------------------------------------------------------------------
|
||||
def _classify_gemini(self, crop_bytes: bytes) -> Optional[str]:
|
||||
prefix, images, suffix = self._build_prompt_and_images(crop_bytes)
|
||||
parts = [{'text': prefix}]
|
||||
for b64, caption in images:
|
||||
parts.append({'inline_data': {'mime_type': 'image/jpeg', 'data': b64}})
|
||||
parts.append({'text': caption})
|
||||
parts.append({'text': suffix})
|
||||
|
||||
for key in self.gemini_api_keys:
|
||||
for attempt in range(self.gemini_max_retries):
|
||||
try:
|
||||
resp = requests.post(
|
||||
f'https://generativelanguage.googleapis.com/v1beta/models/'
|
||||
f'{self.gemini_model}:generateContent?key={key}',
|
||||
json={'contents': [{'parts': parts}],
|
||||
'generationConfig': {'temperature': 0.1, 'maxOutputTokens': 200}},
|
||||
timeout=self.gemini_timeout)
|
||||
data = resp.json()
|
||||
if not data.get('candidates'):
|
||||
err = data.get('error') or {}
|
||||
status = resp.status_code
|
||||
if status in _RETRYABLE_STATUS and attempt < self.gemini_max_retries - 1:
|
||||
backoff = self.gemini_retry_backoff * (2 ** attempt)
|
||||
logger.warning(
|
||||
f"Gemini key 第 {attempt+1}/{self.gemini_max_retries} 次失败"
|
||||
f"（status={status}），{backoff:.1f}s 后重试: {err}")
|
||||
time.sleep(backoff)
|
||||
continue
|
||||
logger.warning(f"Gemini 人物识别失败（status={status}）: {err}")
|
||||
break # 这个 key 不行了，换下一个 key
|
||||
text = ''.join(
|
||||
p.get('text', '')
|
||||
for p in data['candidates'][0].get('content', {}).get('parts', []))
|
||||
person = self._extract_json_person(text)
|
||||
if person:
|
||||
logger.info(f"Gemini 人物识别: {person}")
|
||||
return person
|
||||
except requests.RequestException as e:
|
||||
if attempt < self.gemini_max_retries - 1:
|
||||
backoff = self.gemini_retry_backoff * (2 ** attempt)
|
||||
logger.warning(
|
||||
f"Gemini 网络异常，{backoff:.1f}s 后重试（第 {attempt+1}/"
|
||||
f"{self.gemini_max_retries} 次）: {e}")
|
||||
time.sleep(backoff)
|
||||
continue
|
||||
logger.warning(f"Gemini 人物识别异常: {e}")
|
||||
break
|
||||
return None
|
||||
@@ -33,3 +33,34 @@ class QAOrchestrator:
|
||||
return answer, adapter.provider_name
|
||||
logger.info(f"QA {adapter.provider_name} 无返回，降级下一模型")
|
||||
return None, None
|
||||
|
||||
def run_qa_stream(self, prompt: str, max_tokens: int = 1024):
|
||||
"""流式版：依次尝试各适配器的 chat_stream()，yield 结构化事件字典。
|
||||
|
||||
事件类型:
|
||||
{"type":"provider_trying","provider":p} 开始尝试这个 provider
|
||||
{"type":"chunk","provider":p,"text":t} 这个 provider 吐出的文本增量
|
||||
{"type":"provider_failed","provider":p} 这个 provider 一个字都没吐出就失败，换下一个
|
||||
{"type":"done","provider":p} 成功结束（这个 provider 至少吐出过一块）
|
||||
{"type":"all_failed"} 所有 provider 都失败
|
||||
|
||||
跟 run_qa 一样"仅在还没吐出任何文本时才允许换下一个 provider"——一旦
|
||||
开始给用户看字了，中途失败就结束这次生成，不再悄悄换源接着写。
|
||||
"""
|
||||
for adapter in self.adapters:
|
||||
yield {"type": "provider_trying", "provider": adapter.provider_name}
|
||||
got_any = False
|
||||
try:
|
||||
for chunk in adapter.chat_stream(prompt, max_tokens=max_tokens):
|
||||
if chunk:
|
||||
got_any = True
|
||||
yield {"type": "chunk", "provider": adapter.provider_name, "text": chunk}
|
||||
except Exception as e:
|
||||
logger.warning(f"QA {adapter.provider_name} 流式异常: {e}")
|
||||
if got_any:
|
||||
logger.info(f"QA 流式命中 provider={adapter.provider_name}")
|
||||
yield {"type": "done", "provider": adapter.provider_name}
|
||||
return
|
||||
logger.info(f"QA {adapter.provider_name} 流式无返回，降级下一模型")
|
||||
yield {"type": "provider_failed", "provider": adapter.provider_name}
|
||||
yield {"type": "all_failed"}
|
||||
|
||||
@@ -15,6 +15,7 @@ import os
|
||||
import re
|
||||
import json
|
||||
import subprocess
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
@@ -22,6 +23,8 @@ from .logger import setup_logger
|
||||
from .config_loader import load_config
|
||||
from .model_adapters.adapter_factory import build_adapters
|
||||
from .model_adapters.base_adapter import BaseModelAdapter
|
||||
from .person_identifier import PersonIdentifier
|
||||
from . import frame_service
|
||||
from . import oracle_db
|
||||
|
||||
logger = setup_logger('fam-edge.video_processor')
|
||||
@@ -200,6 +203,9 @@ class VideoProcessor:
|
||||
self.motion_keep_audio = bool(seg.get('keep_audio', True))
|
||||
self.motion_min_duration = float(seg.get('min_duration_sec', 1))
|
||||
self.motion_grace_sec = int(seg.get('unfinished_grace_sec', 10))
|
||||
# 闭集人物识别（家里固定 4 人）：汤圆/媳妇 用性别年龄规则；爷爷/爸爸 用
|
||||
# person_identifier 视觉大模型比对，每片段每个 uid 只调一次
|
||||
self.person_identifier = PersonIdentifier(self.config.get('person_identifier', {}))
|
||||
adapters = build_adapters(self.config.get('models', []))
|
||||
self.vision_adapters: Dict[str, BaseModelAdapter] = {
|
||||
a.provider_name: a for a in adapters if a.get_role() == 'vision'}
|
||||
@@ -477,13 +483,111 @@ class VideoProcessor:
|
||||
elif k not in merged:
|
||||
merged[k] = v_str or 'unknown'
|
||||
uid_features[uid] = merged
|
||||
# 闭集人物识别（家里固定 4 人）：汤圆/媳妇 用性别年龄规则免费识别（source=rule）；
|
||||
# 爷爷/爸爸 每个 uid（同一片段内视为同一人，不逐事件重复调用）用视觉大模型
|
||||
# 比对一次（source=auto_id）。resolved: {原始 uid: (规范名, 来源)}
|
||||
resolved = self._resolve_closed_set_identities(video_id, uid_features, norm_events)
|
||||
|
||||
# 人物对应关系表：记录 (video_id, raw_uid) -> canonical_name，并把这段视频
|
||||
# 展示用的 events/videos 数据直接重写成规范名（读的时候不用现查表拼接）。
|
||||
# manual 纠正过的映射受保护，这里不会覆盖。
|
||||
rename_map = {}
|
||||
for uid, (canonical, source) in resolved.items():
|
||||
if self.db.set_identity_mapping(video_id, uid, canonical, source=source):
|
||||
rename_map[uid] = canonical
|
||||
if rename_map:
|
||||
self.db.rewrite_event_person_names(video_id, rename_map)
|
||||
|
||||
for p in people:
|
||||
if p and p not in ('无人', '无'):
|
||||
feats = uid_features.get(p)
|
||||
canonical, source = resolved.get(p, ('', 'llm'))
|
||||
# 已解析的人物直接用规范名作为 people 表的 label，跨视频天然汇总到
|
||||
# 同一行；解析不了的沿用原始 uid（跟旧行为一致，留给下次/人工确认）
|
||||
label = canonical or p
|
||||
if feats:
|
||||
self.db.upsert_person(p, source='llm', features=feats, display_uid=p)
|
||||
self.db.upsert_person(label, canonical_name=canonical, source=source,
|
||||
features=feats, display_uid=p)
|
||||
else:
|
||||
self.db.upsert_person(p, source='llm')
|
||||
self.db.upsert_person(label, canonical_name=canonical, source=source)
|
||||
logger.info(f"[video_id={video_id}] 已落库: summary={len(summary)}字, "
|
||||
f"events={len(norm_events)}, people={people}, "
|
||||
f"with_features={len(uid_features)}")
|
||||
f"with_features={len(uid_features)}, 闭集识别={resolved}")
|
||||
|
||||
def _resolve_closed_set_identities(self, video_id: int, uid_features: Dict,
|
||||
norm_events: List[Dict]) -> Dict[str, tuple]:
|
||||
"""闭集人物识别：返回 {uid: (canonical_name, source)}。
|
||||
|
||||
汤圆（幼儿/儿童特征）、媳妇（唯一成年女性）靠 Gemini 已经产出的性别/年龄
|
||||
字段直接判断（source='rule'），验证过命中率接近 100%，不需要额外模型调用。
|
||||
爷爷/爸爸两个成年男性外观规则/人脸向量都区分不开（验证过），改用视觉大模型
|
||||
比对参考图（source='auto_id'），每个 uid 只取本片段内最大 bbox 的一次出现
|
||||
判断一次，不逐事件重复调用。
|
||||
"""
|
||||
resolved: Dict[str, tuple] = {}
|
||||
for uid, feats in uid_features.items():
|
||||
gender = str(feats.get('gender', '') or '').strip()
|
||||
age_band = str(feats.get('age_band', '') or '').strip()
|
||||
if age_band in ('幼儿', '儿童'):
|
||||
resolved[uid] = ('汤圆', 'rule')
|
||||
elif gender == '女':
|
||||
resolved[uid] = ('媳妇', 'rule')
|
||||
elif gender == '男':
|
||||
crop = self._best_crop_for_uid(video_id, uid, norm_events)
|
||||
if crop:
|
||||
person = self.person_identifier.classify_adult_male(crop)
|
||||
if person:
|
||||
resolved[uid] = (person, 'auto_id')
|
||||
return resolved
|
||||
|
||||
def _best_crop_for_uid(self, video_id: int, uid: str, norm_events: List[Dict]) -> Optional[bytes]:
|
||||
"""取该 uid 在本片段里最大 bbox 的一次出现，裁剪成一张人物截图（jpeg bytes）。
|
||||
|
||||
bbox 缺失时（实测偶发：某些云端响应——尤其 flash-lite 兜底——没有带
|
||||
person_appearances.bbox 字段）回退到该 uid 第一次出现时刻的整帧居中裁剪，
|
||||
跟 build_avatar() 已有的兜底逻辑一致，好过直接放弃识别这个人。
|
||||
"""
|
||||
best_ts, best_bbox, best_area = None, None, 0
|
||||
first_ts = None
|
||||
for ev in norm_events:
|
||||
for pa in ev.get('person_appearances', []):
|
||||
if pa.get('uid') != uid:
|
||||
continue
|
||||
if first_ts is None:
|
||||
first_ts = ev.get('timestamp')
|
||||
bbox = pa.get('bbox')
|
||||
if not bbox or len(bbox) != 4:
|
||||
continue
|
||||
ymin, xmin, ymax, xmax = bbox
|
||||
area = max(0, ymax - ymin) * max(0, xmax - xmin)
|
||||
if area > best_area:
|
||||
best_area, best_ts, best_bbox = area, ev.get('timestamp'), bbox
|
||||
if best_ts is None and first_ts is None:
|
||||
return None
|
||||
frame = frame_service.extract_frame(self.db, video_id, best_ts or first_ts, width=2880)
|
||||
if frame is None:
|
||||
return None
|
||||
frame_service._ensure_dir()
|
||||
fd, tmp = tempfile.mkstemp(suffix='.jpg', dir=frame_service.CACHE_DIR)
|
||||
os.close(fd)
|
||||
try:
|
||||
with open(tmp, 'wb') as f:
|
||||
f.write(frame)
|
||||
size = frame_service._out_size(tmp)
|
||||
if not size:
|
||||
return None
|
||||
if best_bbox is not None:
|
||||
px = frame_service._bbox_to_pixels(best_bbox, *size)
|
||||
ok = frame_service._crop_ffmpeg(tmp, px, 300)
|
||||
else:
|
||||
ok = frame_service._center_square_ffmpeg(tmp, 300)
|
||||
if not ok:
|
||||
return None
|
||||
with open(tmp, 'rb') as f:
|
||||
return f.read()
|
||||
finally:
|
||||
if os.path.exists(tmp):
|
||||
try:
|
||||
os.remove(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import os
|
||||
|
||||
from fam_edge import frame_service
|
||||
from fam_edge.frame_service import _bbox_to_pixels
|
||||
|
||||
|
||||
@@ -20,3 +23,52 @@ def test_bbox_to_pixels_full_frame():
|
||||
def test_bbox_to_pixels_zero_area():
|
||||
x1, y1, x2, y2 = _bbox_to_pixels([500, 500, 500, 500], 400, 300)
|
||||
assert (x1, y1) == (x2, y2)
|
||||
|
||||
|
||||
class _FakeRow(dict):
|
||||
"""支持 row['key'] 访问的假 sqlite3.Row。"""
|
||||
def __getitem__(self, k):
|
||||
return dict.get(self, k)
|
||||
|
||||
|
||||
class _FakeDb:
|
||||
def __init__(self, local_path, event_start_time):
|
||||
self._row = _FakeRow(local_path=local_path, event_start_time=event_start_time)
|
||||
|
||||
def get_video_by_id(self, video_id):
|
||||
return self._row
|
||||
|
||||
|
||||
def test_extract_frame_cache_key_includes_width(tmp_path, monkeypatch):
|
||||
"""核心诉求: 同一 (video_id, ts) 不同调用方要不同分辨率（时间轴缩略图/头像/
|
||||
人物识别裁人脸），缓存 key 不带 width 会导致后来的高分辨率请求悄悄拿到早先
|
||||
缓存的低分辨率帧——这里验证两次不同 width 请求各自落到独立的缓存文件。"""
|
||||
monkeypatch.setattr(frame_service, "CACHE_DIR", str(tmp_path))
|
||||
video_path = tmp_path / "fake_video.mp4"
|
||||
video_path.write_bytes(b"not a real video, ffmpeg call is mocked")
|
||||
db = _FakeDb(str(video_path), "2026-08-22 10:00:00")
|
||||
|
||||
written_widths = []
|
||||
|
||||
def fake_run_ffmpeg(args, timeout=60):
|
||||
# 把请求的 -vf scale=WIDTH:-2 记下来，往输出路径写点假数据模拟成功
|
||||
out_path = args[-1]
|
||||
vf = next((a for a in args if a.startswith('scale=')), '')
|
||||
written_widths.append(vf)
|
||||
with open(out_path, 'wb') as f:
|
||||
f.write(b'\xff\xd8fakejpeg')
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(frame_service, "_run_ffmpeg", fake_run_ffmpeg)
|
||||
|
||||
data_small = frame_service.extract_frame(db, 42, "2026-08-22 10:00:05", width=400)
|
||||
data_large = frame_service.extract_frame(db, 42, "2026-08-22 10:00:05", width=2880)
|
||||
|
||||
assert data_small is not None and data_large is not None
|
||||
cache_files = sorted(os.listdir(tmp_path))
|
||||
frame_caches = [f for f in cache_files if f.startswith('frame_42_5_')]
|
||||
assert len(frame_caches) == 2, f"expected 2 distinct cache files, got {frame_caches}"
|
||||
assert 'frame_42_5_400.jpg' in frame_caches
|
||||
assert 'frame_42_5_2880.jpg' in frame_caches
|
||||
# 两次都真的各自调用了 ffmpeg（第二次没有因为撞到第一次的缓存而被跳过）
|
||||
assert len(written_widths) == 2
|
||||
|
||||
@@ -108,3 +108,17 @@ def test_rotated_keys_single_key_never_errors():
|
||||
a = GeminiAdapter(_cfg())
|
||||
for _ in range(3):
|
||||
assert a._rotated_keys() == [(0, "key-primary")]
|
||||
|
||||
|
||||
def test_chat_timeout_defaults_short_not_shared_with_video_timeout():
|
||||
"""核心诉求: 问答是交互场景，不能沿用视频分析的 600s 超时——否则一个卡住
|
||||
的 key/模型会让用户在聊天界面一直等，这正是"一直卡着"这个 bug 的根因。"""
|
||||
a = GeminiAdapter(_cfg(timeout=600))
|
||||
assert a.timeout == 600
|
||||
assert a.chat_timeout == 20
|
||||
assert a.chat_timeout != a.timeout
|
||||
|
||||
|
||||
def test_chat_timeout_configurable():
|
||||
a = GeminiAdapter(_cfg(chat_timeout=8))
|
||||
assert a.chat_timeout == 8
|
||||
|
||||
@@ -181,3 +181,133 @@ def test_has_motion_in_range_local_filters_by_camera_id(tmp_path):
|
||||
[{"event_id": 1, "camera_id": 99, "event_type": 10, "start_time": 2500, "duration": 5}])
|
||||
assert db.has_motion_in_range_local(2000, 3000, camera_id=2) is False
|
||||
assert db.has_motion_in_range_local(2000, 3000, camera_id=99) is True
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# 人物对应关系表（video_id, raw_uid) -> canonical_name
|
||||
# ----------------------------------------------------------------------
|
||||
|
||||
def _seed_video_with_events(db, filename="motion_1_1000.mp4"):
|
||||
vid = db.ensure_video(filename, f"/tmp/{filename}", event_start_time="2026-08-22 10:00:00")
|
||||
events = [
|
||||
{"timestamp": "10:00:01", "description": "在客厅走动", "people": ["人物A"],
|
||||
"person_appearances": [{"uid": "人物A", "features": {"gender": "男"}, "action": "走动"}]},
|
||||
{"timestamp": "10:00:05", "description": "坐下", "people": ["人物A", "人物B"],
|
||||
"person_appearances": [
|
||||
{"uid": "人物A", "features": {"gender": "男"}, "action": "坐下"},
|
||||
{"uid": "人物B", "features": {"gender": "女"}, "action": "站立"}]},
|
||||
]
|
||||
db.mark_video_processed(vid, "摘要", events, ["人物A", "人物B"], "gemini")
|
||||
return vid
|
||||
|
||||
|
||||
def test_set_identity_mapping_inserts_new_row(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
assert db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id") is True
|
||||
assert db.get_identity_map_for_video(1) == {"人物A": "爷爷"}
|
||||
|
||||
|
||||
def test_set_identity_mapping_updates_existing_non_manual_row(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id")
|
||||
assert db.set_identity_mapping(1, "人物A", "爸爸", source="auto_id") is True
|
||||
assert db.get_identity_map_for_video(1) == {"人物A": "爸爸"}
|
||||
|
||||
|
||||
def test_set_identity_mapping_manual_protected_from_auto_overwrite(tmp_path):
|
||||
"""核心诉求: 人工纠正过的映射不能被后续自动识别悄悄改回去。"""
|
||||
db = _db(tmp_path)
|
||||
db.set_identity_mapping(1, "人物A", "爸爸", source="manual")
|
||||
changed = db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id")
|
||||
assert changed is False
|
||||
assert db.get_identity_map_for_video(1) == {"人物A": "爸爸"}
|
||||
|
||||
|
||||
def test_set_identity_mapping_manual_can_override_manual(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
db.set_identity_mapping(1, "人物A", "爸爸", source="manual")
|
||||
changed = db.set_identity_mapping(1, "人物A", "爷爷", source="manual")
|
||||
assert changed is True
|
||||
assert db.get_identity_map_for_video(1) == {"人物A": "爷爷"}
|
||||
|
||||
|
||||
def test_set_identity_mapping_no_change_returns_false(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id")
|
||||
changed = db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id")
|
||||
assert changed is False
|
||||
|
||||
|
||||
def test_get_identity_map_for_video_scoped_per_video(tmp_path):
|
||||
"""核心诉求: 同一个 raw_uid 字符串在不同视频里可能是不同真人，映射必须按
|
||||
video_id 隔离，不能串。"""
|
||||
db = _db(tmp_path)
|
||||
db.set_identity_mapping(1, "人物A", "爷爷", source="auto_id")
|
||||
db.set_identity_mapping(2, "人物A", "爸爸", source="auto_id")
|
||||
assert db.get_identity_map_for_video(1) == {"人物A": "爷爷"}
|
||||
assert db.get_identity_map_for_video(2) == {"人物A": "爸爸"}
|
||||
|
||||
|
||||
def test_rewrite_event_person_names_updates_events_and_video(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
vid = _seed_video_with_events(db)
|
||||
db.rewrite_event_person_names(vid, {"人物A": "爷爷", "人物B": "媳妇"})
|
||||
|
||||
rows = db._conn.execute(
|
||||
"SELECT person_list_json, person_appearances_json FROM events "
|
||||
"WHERE video_id=? ORDER BY id", (vid,)).fetchall()
|
||||
assert json.loads(rows[0]["person_list_json"]) == ["爷爷"]
|
||||
pa0 = json.loads(rows[0]["person_appearances_json"])
|
||||
assert pa0[0]["uid"] == "爷爷"
|
||||
assert json.loads(rows[1]["person_list_json"]) == ["爷爷", "媳妇"]
|
||||
pa1 = json.loads(rows[1]["person_appearances_json"])
|
||||
assert {p["uid"] for p in pa1} == {"爷爷", "媳妇"}
|
||||
|
||||
vrow = db._conn.execute("SELECT people_json FROM videos WHERE id=?", (vid,)).fetchone()
|
||||
assert set(json.loads(vrow["people_json"])) == {"爷爷", "媳妇"}
|
||||
|
||||
|
||||
def test_rewrite_event_person_names_noop_on_empty_map(tmp_path):
|
||||
db = _db(tmp_path)
|
||||
vid = _seed_video_with_events(db)
|
||||
before = db._conn.execute(
|
||||
"SELECT person_list_json FROM events WHERE video_id=?", (vid,)).fetchall()
|
||||
db.rewrite_event_person_names(vid, {})
|
||||
after = db._conn.execute(
|
||||
"SELECT person_list_json FROM events WHERE video_id=?", (vid,)).fetchall()
|
||||
assert [r["person_list_json"] for r in before] == [r["person_list_json"] for r in after]
|
||||
|
||||
|
||||
def test_correct_video_identity_end_to_end(tmp_path):
|
||||
"""核心诉求: 纠错入口应该找到当前展示名对应的映射行，改写映射 + 立即重写
|
||||
展示数据，且标记为 manual（受保护）。"""
|
||||
db = _db(tmp_path)
|
||||
vid = _seed_video_with_events(db)
|
||||
db.set_identity_mapping(vid, "人物A", "爷爷", source="auto_id")
|
||||
db.rewrite_event_person_names(vid, {"人物A": "爷爷"})
|
||||
|
||||
db.correct_video_identity(vid, current_name="爷爷", new_name="爸爸")
|
||||
|
||||
assert db.get_identity_map_for_video(vid) == {"人物A": "爸爸"}
|
||||
rows = db._conn.execute(
|
||||
"SELECT person_list_json FROM events WHERE video_id=? ORDER BY id", (vid,)).fetchall()
|
||||
assert json.loads(rows[0]["person_list_json"]) == ["爸爸"]
|
||||
# manual 之后不能被自动识别覆盖回去
|
||||
changed = db.set_identity_mapping(vid, "人物A", "爷爷", source="auto_id")
|
||||
assert changed is False
|
||||
|
||||
|
||||
def test_correct_video_identity_without_prior_mapping_uses_current_name_as_raw_uid(tmp_path):
|
||||
"""核心诉求: 老流水线时代产出的数据从没跑过闭集识别，映射表里没有记录——
|
||||
纠错依然要能生效，把 current_name 本身当 raw_uid 存一条新映射。"""
|
||||
db = _db(tmp_path)
|
||||
vid = db.ensure_video("motion_2_2000.mp4", "/tmp/x.mp4", event_start_time="2026-08-22 10:00:00")
|
||||
events = [{"timestamp": "10:00:01", "description": "走动", "people": ["爷爷"],
|
||||
"person_appearances": [{"uid": "爷爷", "features": {"gender": "男"}, "action": "走动"}]}]
|
||||
db.mark_video_processed(vid, "摘要", events, ["爷爷"], "gemini")
|
||||
|
||||
db.correct_video_identity(vid, current_name="爷爷", new_name="爸爸")
|
||||
assert db.get_identity_map_for_video(vid) == {"爷爷": "爸爸"}
|
||||
rows = db._conn.execute(
|
||||
"SELECT person_list_json FROM events WHERE video_id=?", (vid,)).fetchall()
|
||||
assert json.loads(rows[0]["person_list_json"]) == ["爸爸"]
|
||||
|
||||
347
fam-edge/tests/test_person_identifier.py
Normal file
347
fam-edge/tests/test_person_identifier.py
Normal file
@@ -0,0 +1,347 @@
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from fam_edge.person_identifier import PersonIdentifier
|
||||
|
||||
|
||||
def _cfg(ref_dir, **overrides):
|
||||
base = {
|
||||
"enabled": True,
|
||||
"ref_dir": ref_dir,
|
||||
"max_ref_per_person": 6,
|
||||
"min_call_interval_sec": 0, # 测试不需要真实限速，避免拖慢用例
|
||||
"nvidia": {"api_key": "nvkey", "model_name": "nvidia/test", "timeout": 30,
|
||||
"max_retries": 2, "retry_backoff_sec": 0.01},
|
||||
"gemini": {"api_key": "gkey1", "model_name": "gemini-flash-lite-latest", "timeout": 30,
|
||||
"max_retries": 2, "retry_backoff_sec": 0.01},
|
||||
}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
|
||||
def _write_refs(tmp_path, grandpa=2, dad=2):
|
||||
for person, n in (("爷爷", grandpa), ("爸爸", dad)):
|
||||
d = tmp_path / person
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
for i in range(n):
|
||||
(d / f"{i:02d}.jpg").write_bytes(b"fakejpegbytes")
|
||||
|
||||
|
||||
class _FakeResp:
|
||||
def __init__(self, status_code=200, payload=None):
|
||||
self.status_code = status_code
|
||||
self._payload = payload or {}
|
||||
|
||||
def json(self):
|
||||
return self._payload
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_real_sleep(monkeypatch):
|
||||
"""全部用例都不需要真的睡（限速/退避都测计数和结果，不测真实耗时）。"""
|
||||
monkeypatch.setattr("fam_edge.person_identifier.time.sleep", lambda s: None)
|
||||
|
||||
|
||||
def test_has_references_false_when_dirs_missing(tmp_path):
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path / "refs")))
|
||||
assert pi.has_references() is False
|
||||
|
||||
|
||||
def test_has_references_true_when_both_present(tmp_path):
|
||||
_write_refs(tmp_path, grandpa=3, dad=2)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.has_references() is True
|
||||
|
||||
|
||||
def test_has_references_false_when_only_one_person_has_refs(tmp_path):
|
||||
(tmp_path / "爷爷").mkdir(parents=True)
|
||||
(tmp_path / "爷爷" / "01.jpg").write_bytes(b"x")
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.has_references() is False
|
||||
|
||||
|
||||
def test_extract_json_person_clean_json():
|
||||
pi = PersonIdentifier(_cfg("/nonexistent"))
|
||||
assert pi._extract_json_person('{"person":"爷爷"}') == '爷爷'
|
||||
assert pi._extract_json_person('{"person":"爸爸"}') == '爸爸'
|
||||
|
||||
|
||||
def test_extract_json_person_no_match_returns_none():
|
||||
pi = PersonIdentifier(_cfg("/nonexistent"))
|
||||
assert pi._extract_json_person('不知道是谁') is None
|
||||
assert pi._extract_json_person('') is None
|
||||
|
||||
|
||||
def test_extract_json_person_both_mentioned_uses_json_field():
|
||||
"""核心诉求: 模型有时会把参考图说明也复述一遍，回复里两个名字都出现——
|
||||
这时候不能瞎猜，要从 JSON 的 person 字段里精确取，取不到就返回 None。"""
|
||||
pi = PersonIdentifier(_cfg("/nonexistent"))
|
||||
text = '参考图1是爷爷，参考图5是爸爸。{"person":"爸爸"}'
|
||||
assert pi._extract_json_person(text) == '爸爸'
|
||||
|
||||
|
||||
def test_extract_json_person_both_mentioned_no_json_field_returns_none():
|
||||
pi = PersonIdentifier(_cfg("/nonexistent"))
|
||||
text = '这个人可能是爷爷，也可能是爸爸，不太确定'
|
||||
assert pi._extract_json_person(text) is None
|
||||
|
||||
|
||||
def test_classify_returns_none_when_disabled(tmp_path):
|
||||
_write_refs(tmp_path)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path), enabled=False))
|
||||
assert pi.classify_adult_male(b"crop") is None
|
||||
|
||||
|
||||
def test_classify_returns_none_when_no_references(tmp_path):
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path / "empty")))
|
||||
assert pi.classify_adult_male(b"crop") is None
|
||||
|
||||
|
||||
def test_classify_falls_back_to_gemini_when_nvidia_unavailable(tmp_path, monkeypatch):
|
||||
"""openai SDK 未安装时 NVIDIA 路径应该静默跳过（不报错），落到 Gemini。"""
|
||||
_write_refs(tmp_path)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", None)
|
||||
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
return _FakeResp(200, {
|
||||
"candidates": [{"content": {"parts": [{"text": '{"person":"爸爸"}'}]}}]
|
||||
})
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.classify_adult_male(b"crop") == '爸爸'
|
||||
|
||||
|
||||
def _fake_openai_factory(reply_text=None, exc=None, fail_times=0):
|
||||
"""构造一个假 OpenAI 客户端：先失败 fail_times 次再成功，或者一直抛 exc。"""
|
||||
state = {"calls": 0}
|
||||
|
||||
class FakeMessage:
|
||||
content = reply_text
|
||||
|
||||
class FakeChoice:
|
||||
message = FakeMessage()
|
||||
|
||||
class FakeChatResp:
|
||||
choices = [FakeChoice()]
|
||||
|
||||
class FakeCompletions:
|
||||
def create(self, **kwargs):
|
||||
state["calls"] += 1
|
||||
if state["calls"] <= fail_times:
|
||||
raise (exc or RuntimeError("boom"))
|
||||
if exc and fail_times == 0:
|
||||
raise exc
|
||||
return FakeChatResp()
|
||||
|
||||
class FakeChat:
|
||||
completions = FakeCompletions()
|
||||
|
||||
class FakeOpenAI:
|
||||
def __init__(self, base_url=None, api_key=None):
|
||||
pass
|
||||
chat = FakeChat()
|
||||
|
||||
return FakeOpenAI, state
|
||||
|
||||
|
||||
def test_classify_nvidia_success_skips_gemini(tmp_path, monkeypatch):
|
||||
_write_refs(tmp_path)
|
||||
FakeOpenAI, state = _fake_openai_factory(reply_text='{"person":"爷爷"}')
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", FakeOpenAI)
|
||||
|
||||
gemini_called = {"n": 0}
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
gemini_called["n"] += 1
|
||||
return _FakeResp(200, {})
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.classify_adult_male(b"crop") == '爷爷'
|
||||
assert gemini_called["n"] == 0
|
||||
assert state["calls"] == 1
|
||||
|
||||
|
||||
class _FakeHTTPError(Exception):
|
||||
def __init__(self, status_code):
|
||||
self.response = type("R", (), {"status_code": status_code})()
|
||||
|
||||
|
||||
def test_nvidia_retries_transient_error_then_succeeds(tmp_path, monkeypatch):
|
||||
"""核心诉求: 429/503 这类瞬时故障要退避重试，不是第一次失败就放弃换 provider。"""
|
||||
_write_refs(tmp_path)
|
||||
FakeOpenAI, state = _fake_openai_factory(
|
||||
reply_text='{"person":"爸爸"}', exc=_FakeHTTPError(503), fail_times=1)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", FakeOpenAI)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.classify_adult_male(b"crop") == '爸爸'
|
||||
assert state["calls"] == 2 # 第一次 503 失败重试一次后成功
|
||||
|
||||
|
||||
def test_nvidia_gives_up_after_max_retries_falls_back_to_gemini(tmp_path, monkeypatch):
|
||||
_write_refs(tmp_path)
|
||||
FakeOpenAI, state = _fake_openai_factory(exc=_FakeHTTPError(503), fail_times=99)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", FakeOpenAI)
|
||||
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
return _FakeResp(200, {
|
||||
"candidates": [{"content": {"parts": [{"text": '{"person":"汤圆"}'}]}}]
|
||||
})
|
||||
# 用一个不属于爷爷/爸爸的返回值只是为了确认真的调用到了 gemini 分支
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
|
||||
cfg = _cfg(str(tmp_path))
|
||||
pi = PersonIdentifier(cfg)
|
||||
pi.classify_adult_male(b"crop")
|
||||
assert state["calls"] == pi.nvidia_max_retries # 重试到上限就放弃，不会无限重试
|
||||
|
||||
|
||||
def test_nvidia_non_retryable_error_gives_up_immediately(tmp_path, monkeypatch):
|
||||
"""核心诉求: 400 参数错误这类非瞬时故障，重试没有意义，应该立刻换下一个模型/provider，
|
||||
不要浪费时间重试一个注定失败的请求。"""
|
||||
_write_refs(tmp_path)
|
||||
FakeOpenAI, state = _fake_openai_factory(exc=_FakeHTTPError(400), fail_times=99)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", FakeOpenAI)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post",
|
||||
lambda *a, **k: _FakeResp(500, {"error": "down"}))
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
pi.classify_adult_male(b"crop")
|
||||
assert state["calls"] == 1 # 400 不重试，一次就放弃这个模型
|
||||
|
||||
|
||||
def test_nvidia_falls_through_model_chain(tmp_path, monkeypatch):
|
||||
"""核心诉求: 第一个模型重试耗尽后，应该换模型链里的下一个型号再试，而不是
|
||||
直接放弃整个 NVIDIA provider。"""
|
||||
_write_refs(tmp_path)
|
||||
calls = []
|
||||
|
||||
class FakeMessage:
|
||||
def __init__(self, content):
|
||||
self.content = content
|
||||
|
||||
class FakeChoice:
|
||||
def __init__(self, content):
|
||||
self.message = FakeMessage(content)
|
||||
|
||||
class FakeChatResp:
|
||||
def __init__(self, content):
|
||||
self.choices = [FakeChoice(content)]
|
||||
|
||||
class FakeCompletions:
|
||||
def create(self, model, **kwargs):
|
||||
calls.append(model)
|
||||
if model == 'nvidia/model-a':
|
||||
raise _FakeHTTPError(503)
|
||||
return FakeChatResp('{"person":"爷爷"}')
|
||||
|
||||
class FakeChat:
|
||||
completions = FakeCompletions()
|
||||
|
||||
class FakeOpenAI:
|
||||
def __init__(self, base_url=None, api_key=None):
|
||||
pass
|
||||
chat = FakeChat()
|
||||
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", FakeOpenAI)
|
||||
cfg = _cfg(str(tmp_path), nvidia={
|
||||
"api_key": "nvkey", "model_name": "nvidia/model-a",
|
||||
"fallback_models": ["nvidia/model-b"], "timeout": 30,
|
||||
"max_retries": 2, "retry_backoff_sec": 0.01,
|
||||
})
|
||||
pi = PersonIdentifier(cfg)
|
||||
assert pi.classify_adult_male(b"crop") == '爷爷'
|
||||
assert calls == ['nvidia/model-a', 'nvidia/model-a', 'nvidia/model-b']
|
||||
|
||||
|
||||
def test_gemini_retries_transient_error_on_same_key(tmp_path, monkeypatch):
|
||||
_write_refs(tmp_path)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", None)
|
||||
calls = []
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
calls.append(url.split('key=')[-1])
|
||||
if len(calls) == 1:
|
||||
return _FakeResp(503, {"error": {"code": 503}})
|
||||
return _FakeResp(200, {
|
||||
"candidates": [{"content": {"parts": [{"text": '{"person":"媳妇"}'}]}}]
|
||||
})
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
pi.classify_adult_male(b"crop")
|
||||
assert calls == ['gkey1', 'gkey1'] # 同一个 key 重试，不是立刻跳到下一个 key
|
||||
|
||||
|
||||
def test_classify_gemini_rotates_across_keys_after_retries_exhausted(tmp_path, monkeypatch):
|
||||
_write_refs(tmp_path)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", None)
|
||||
|
||||
calls = []
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
key = url.split('key=')[-1]
|
||||
calls.append(key)
|
||||
if key == 'gkey1':
|
||||
return _FakeResp(429, {"error": {"code": 429}})
|
||||
return _FakeResp(200, {
|
||||
"candidates": [{"content": {"parts": [{"text": '{"person":"爷爷"}'}]}}]
|
||||
})
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
|
||||
cfg = _cfg(str(tmp_path), gemini={
|
||||
"api_key": "gkey1", "extra_api_keys": ["gkey2"],
|
||||
"model_name": "gemini-flash-lite-latest", "timeout": 30,
|
||||
"max_retries": 2, "retry_backoff_sec": 0.01,
|
||||
})
|
||||
pi = PersonIdentifier(cfg)
|
||||
assert pi.classify_adult_male(b"crop") == '爷爷'
|
||||
assert calls == ['gkey1', 'gkey1', 'gkey2'] # gkey1 重试用尽才换 gkey2
|
||||
|
||||
|
||||
def test_classify_both_providers_fail_returns_none(tmp_path, monkeypatch):
|
||||
"""核心诉求: NVIDIA 和 Gemini 都失败时绝不能瞎猜，必须返回 None。"""
|
||||
_write_refs(tmp_path)
|
||||
monkeypatch.setattr("fam_edge.person_identifier.OpenAI", None)
|
||||
|
||||
def fake_post(url, json=None, timeout=None):
|
||||
return _FakeResp(500, {"error": "boom"})
|
||||
monkeypatch.setattr("fam_edge.person_identifier.requests.post", fake_post)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path)))
|
||||
assert pi.classify_adult_male(b"crop") is None
|
||||
|
||||
|
||||
def test_env_var_credentials_resolved(tmp_path):
|
||||
os.environ["TEST_NVIDIA_KEY_XYZ"] = "realkey"
|
||||
try:
|
||||
cfg = _cfg(str(tmp_path), nvidia={"api_key": "${TEST_NVIDIA_KEY_XYZ}"})
|
||||
pi = PersonIdentifier(cfg)
|
||||
assert pi.nvidia_api_key == "realkey"
|
||||
finally:
|
||||
del os.environ["TEST_NVIDIA_KEY_XYZ"]
|
||||
|
||||
|
||||
def test_max_ref_per_person_limits_loaded_refs(tmp_path):
|
||||
_write_refs(tmp_path, grandpa=10, dad=10)
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path), max_ref_per_person=3))
|
||||
refs = pi._load_refs()
|
||||
assert len(refs['爷爷']) == 3
|
||||
assert len(refs['爸爸']) == 3
|
||||
|
||||
|
||||
def test_pace_sleeps_when_called_too_soon(tmp_path, monkeypatch):
|
||||
"""核心诉求: 批量回填会短时间内密集调用，min_call_interval_sec 要真的限速，
|
||||
不能形同虚设。"""
|
||||
_write_refs(tmp_path)
|
||||
slept = []
|
||||
monkeypatch.setattr("fam_edge.person_identifier.time.sleep", lambda s: slept.append(s))
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path), min_call_interval_sec=5))
|
||||
pi._last_call_at = __import__("time").time() # 刚刚调用过
|
||||
pi._pace()
|
||||
assert slept and slept[0] > 0
|
||||
|
||||
|
||||
def test_pace_no_sleep_when_interval_already_elapsed(tmp_path, monkeypatch):
|
||||
_write_refs(tmp_path)
|
||||
slept = []
|
||||
monkeypatch.setattr("fam_edge.person_identifier.time.sleep", lambda s: slept.append(s))
|
||||
pi = PersonIdentifier(_cfg(str(tmp_path), min_call_interval_sec=5))
|
||||
pi._last_call_at = 0 # 很久以前
|
||||
pi._pace()
|
||||
assert slept == []
|
||||
81
fam-edge/tests/test_qa.py
Normal file
81
fam-edge/tests/test_qa.py
Normal file
@@ -0,0 +1,81 @@
|
||||
from fam_edge.qa import QAOrchestrator
|
||||
|
||||
|
||||
class _FakeAdapter:
|
||||
def __init__(self, provider_name, chunks=None, raises=False):
|
||||
self.provider_name = provider_name
|
||||
self._chunks = chunks or []
|
||||
self._raises = raises
|
||||
|
||||
def chat_stream(self, prompt, max_tokens=512):
|
||||
if self._raises:
|
||||
raise RuntimeError("boom")
|
||||
for c in self._chunks:
|
||||
yield c
|
||||
|
||||
def chat(self, prompt, max_tokens=512):
|
||||
return ''.join(self._chunks) or None
|
||||
|
||||
|
||||
def _orchestrator(adapters):
|
||||
qa = QAOrchestrator.__new__(QAOrchestrator) # 跳过 __init__（不需要真实 config/adapters）
|
||||
qa.adapters = adapters
|
||||
return qa
|
||||
|
||||
|
||||
def test_run_qa_stream_first_provider_success():
|
||||
qa = _orchestrator([_FakeAdapter("gemini", chunks=["你", "好"])])
|
||||
events = list(qa.run_qa_stream("hi"))
|
||||
types = [e["type"] for e in events]
|
||||
assert types == ["provider_trying", "chunk", "chunk", "done"]
|
||||
assert events[1]["text"] == "你"
|
||||
assert events[2]["text"] == "好"
|
||||
assert events[-1]["provider"] == "gemini"
|
||||
|
||||
|
||||
def test_run_qa_stream_falls_back_when_first_yields_nothing():
|
||||
"""核心诉求: 第一个 provider 一个字都没吐出来才允许换下一个——不是失败就切，
|
||||
是"完全没有产出"才切。"""
|
||||
qa = _orchestrator([
|
||||
_FakeAdapter("gemini", chunks=[]),
|
||||
_FakeAdapter("nvidia", chunks=["答案"]),
|
||||
])
|
||||
events = list(qa.run_qa_stream("hi"))
|
||||
types = [e["type"] for e in events]
|
||||
assert types == ["provider_trying", "provider_failed", "provider_trying", "chunk", "done"]
|
||||
assert events[-1]["provider"] == "nvidia"
|
||||
|
||||
|
||||
def test_run_qa_stream_does_not_switch_after_partial_output():
|
||||
"""核心诉求: 已经开始吐字之后中途失败，不能悄悄换下一个 provider 接着写
|
||||
（会出现两段风格/内容不连贯的回答拼在一起）——直接结束这次生成。"""
|
||||
class _PartialThenRaise:
|
||||
provider_name = "gemini"
|
||||
def chat_stream(self, prompt, max_tokens=512):
|
||||
yield "先吐"
|
||||
raise RuntimeError("connection reset")
|
||||
|
||||
qa = _orchestrator([_PartialThenRaise(), _FakeAdapter("nvidia", chunks=["不该被用到"])])
|
||||
events = list(qa.run_qa_stream("hi"))
|
||||
types = [e["type"] for e in events]
|
||||
assert types == ["provider_trying", "chunk", "done"]
|
||||
assert events[1]["text"] == "先吐"
|
||||
assert events[-1]["provider"] == "gemini"
|
||||
|
||||
|
||||
def test_run_qa_stream_all_providers_fail():
|
||||
qa = _orchestrator([
|
||||
_FakeAdapter("gemini", chunks=[]),
|
||||
_FakeAdapter("nvidia", chunks=[], raises=True),
|
||||
])
|
||||
events = list(qa.run_qa_stream("hi"))
|
||||
assert events[-1]["type"] == "all_failed"
|
||||
assert "provider_failed" in [e["type"] for e in events]
|
||||
|
||||
|
||||
def test_run_qa_stream_exception_treated_as_no_output():
|
||||
qa = _orchestrator([_FakeAdapter("gemini", raises=True), _FakeAdapter("nvidia", chunks=["ok"])])
|
||||
events = list(qa.run_qa_stream("hi"))
|
||||
assert events[0] == {"type": "provider_trying", "provider": "gemini"}
|
||||
assert events[1] == {"type": "provider_failed", "provider": "gemini"}
|
||||
assert events[-1]["provider"] == "nvidia"
|
||||
Reference in New Issue
Block a user