refactor(fam-edge): 重构第一阶段 - 人物图片零额外调用 + 运行时稳定性 + 工程质量
人物图片功能重做: bbox 随核心视频分析那一次 Gemini 调用一并产出(prompts.py 加
person_appearances.bbox 字段, [ymin,xmin,ymax,xmax] 0-1000 归一化), frame_service
直接用存好的 bbox 裁剪头像/事件缩略图, 删除原来"展示时额外调用 Gemini 定位人物"的
整套逻辑(locate_person_bbox/VLM 校验/熔断), 从架构上消除与核心视频分析共抢配额的
问题; 用真实数据验证裁剪结果正确框住人物本体。
NVIDIA 模型修复: 实测原配置的 3 个模型均不可用(asset_id 引用 500/400, 不支持视频),
改用 nemotron-3-nano-omni 的 base64 内嵌视频方式(唯一实测打通), 加 max_base64_mb
防止对大文件做注定失败的编码。
Gemini 多 Key 轮换: 支持 extra_api_keys 配置多个独立项目的 key, 配额用尽时依次
换 key 重试(每换 key 需重新上传, Files API 按项目隔离)。
稳定性加固: CircuitBreaker HALF_OPEN 清空旧失败计数(修复探测一失败就重新 OPEN 的
bug); chat() 统一接入熔断器(原来只有视频分析路径检查); NVIDIA 适配器改用共享
json_parser(原来自己重复实现且不做 schema 校验); Gemini Files API 上传超时也尝试
清理远程孤儿文件; video_processor/video_queue 里直接操作 OracleDB._conn 的裸 SQL
改走新增的 set_event_start_time/mark_video_invalid/reset_video_to_pending 方法;
/health 加入队列线程存活状态; 密钥改用 ${ENV_VAR} 引用(.env 已支持自动加载),
不再明文写入 config.yaml。
工程质量: 新增 fam-edge/tests(32 个单元测试, 覆盖熔断器状态机/JSON 解析容错/
时间戳解析/bbox 坐标换算/多 key 解析), 新增 scripts/smoke_test.py(发版前接口
稳定性检查); 清理死代码(OllamaAdapter.analyze_frames、get_sync_delta 死分支、
未使用的 vision_timeout/max_concurrent_tasks 配置项); 修正 get_events_for_label
排序(改最近优先 + 过滤畸形历史时间戳)。
已部署 Oracle 并跑通 smoke test 全部 6 项检查。
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ericwyuan
@@ -2,17 +2,29 @@
|
||||
NvidiaVisionAdapter - NVIDIA NIM 云端 VLM 适配器
|
||||
|
||||
provider_name = "nvidia"
|
||||
模型: nvidia/nemotron-nano-12b-v2-vl（NIM 官方支持整视频 video_url 输入，内部自行采样帧）
|
||||
模型: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning（唯一实测确认可用的视频理解模型）
|
||||
角色: vision (整视频直出结构化 JSON) + 智能问答
|
||||
SDK: openai (NIM 兼容 OpenAI API 规范)
|
||||
整视频分析: 先经 NVIDIA Assets API 上传完整视频拿 asset_id，再以 video_url 引用单次调用
|
||||
—— 本地不切片、不抽帧（request payload 有 25MB 上限，base64 直塞不可行，必须走 Assets API）
|
||||
|
||||
整视频分析实测结论（2026-08-21 用真实短视频逐个探测）:
|
||||
- nemotron-3-nano-omni-30b-a3b-reasoning: video_url 只认 base64 data URI
|
||||
（`data:video/mp4;base64,<...>`），Assets API 的 asset_id 引用方式对它直接 500
|
||||
（报错 "Only base64 data URLs are supported for now"）——所以本适配器不再走
|
||||
Assets API 上传，直接 base64 内嵌整段视频。
|
||||
- nemotron-nano-12b-v2-vl: 需要走 NVCF 函数调用协议本身的 NVCF-ASSET-DIR/
|
||||
NVCF-FUNCTION-ASSET-IDS 请求头，而这两个头的值是 NVCF 服务端按内部路径生成、
|
||||
不是客户端能自己拼对的（实测传什么都 400 "Invalid NVCF-ASSET-DIR"），标准
|
||||
OpenAI 兼容 chat.completions 调用打不通，已从模型链移除。
|
||||
- meta/llama-3.2-11b-vision-instruct: 明确不支持视频输入（"At most 0 video(s)
|
||||
may be provided"），只能单图，已移除。
|
||||
base64 方案的代价是请求体大小受限（原实现注释称约 25MB 上限），所以本适配器会在
|
||||
上传前检查文件大小，超过 `max_base64_mb`（默认 20MB）直接放弃，不做注定失败的
|
||||
慢速编码+上传。真实监控视频压缩后通常在 20MB 上下，属于"够不到就正常降级到失败
|
||||
重试"，不是本地故意限制过窄。
|
||||
"""
|
||||
import base64
|
||||
import os
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
import requests
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
@@ -20,6 +32,7 @@ from .base_adapter import BaseModelAdapter
|
||||
from .circuit_breaker import CircuitBreaker
|
||||
from ..logger import setup_logger
|
||||
from ..ai_orchestrator.prompts import build_video_prompt
|
||||
from ..ai_orchestrator.json_parser import parse_vlm_json, VLMOutputInvalidError
|
||||
from ..config_loader import load_config
|
||||
|
||||
logger = setup_logger('fam-edge.nvidia_adapter')
|
||||
@@ -43,12 +56,13 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
def __init__(self, config: dict):
|
||||
super().__init__("nvidia", config)
|
||||
self.model_name = config.get(
|
||||
'model_name', 'nvidia/nemotron-nano-12b-v2-vl')
|
||||
'model_name', 'nvidia/nemotron-3-nano-omni-30b-a3b-reasoning')
|
||||
self.model_chain = [self.model_name] + [
|
||||
m for m in config.get('fallback_models', []) if m and m != self.model_name]
|
||||
self.api_key = self._resolve_key(config.get('api_key', ''))
|
||||
self.base_url = config.get('base_url', 'https://integrate.api.nvidia.com/v1')
|
||||
self.timeout = config.get('timeout', 600)
|
||||
self.max_base64_mb = float(config.get('max_base64_mb', 20))
|
||||
# 模型级独立超时（最终值，不参与编排层 ×N 放大）: {model_name: seconds}
|
||||
self.model_timeouts = {
|
||||
str(k): int(v) for k, v in (config.get('model_timeouts') or {}).items()}
|
||||
@@ -86,63 +100,8 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 整视频分析：Assets API 上传 -> video_url(asset_id) 单次调用
|
||||
# 整视频分析：base64 内嵌 video_url 单次调用（omni 只认 base64，不认 asset_id 引用）
|
||||
# ------------------------------------------------------------------
|
||||
ASSET_API = "https://api.nvcf.nvidia.com/v2/nvcf/assets"
|
||||
|
||||
def _upload_asset(self, video_path: str) -> Optional[str]:
|
||||
"""用 NVIDIA Assets API 上传大视频文件，返回 asset_id 供 video_url 引用。"""
|
||||
content_type = "video/mp4"
|
||||
upload_timeout = max(self.timeout, 900)
|
||||
try:
|
||||
r = requests.post(
|
||||
self.ASSET_API,
|
||||
headers={
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json={"contentType": content_type, "description": "fam-edge video asset"},
|
||||
timeout=60,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"NVIDIA 创建 asset 异常: {e}")
|
||||
return None
|
||||
if r.status_code not in (200, 201):
|
||||
logger.warning(f"NVIDIA 创建 asset 失败 HTTP {r.status_code}: {r.text[:200]}")
|
||||
return None
|
||||
try:
|
||||
j = r.json()
|
||||
asset_id = j.get("assetId")
|
||||
upload_url = j.get("uploadUrl")
|
||||
except ValueError:
|
||||
logger.warning("NVIDIA asset 响应解析失败")
|
||||
return None
|
||||
if not asset_id or not upload_url:
|
||||
logger.warning("NVIDIA asset 响应缺少 assetId/uploadUrl")
|
||||
return None
|
||||
try:
|
||||
with open(video_path, 'rb') as f:
|
||||
up = requests.put(
|
||||
upload_url,
|
||||
data=f,
|
||||
# 必须全小写 header 名且 content-type 值与 POST 的 contentType 一致：
|
||||
# 预签名 S3 URL 签名覆盖这两个值，不一致会 SignatureDoesNotMatch
|
||||
headers={"content-type": content_type,
|
||||
"x-amz-meta-nvcf-asset-description": "fam-edge video asset"},
|
||||
timeout=upload_timeout,
|
||||
)
|
||||
except requests.Timeout:
|
||||
logger.warning(f"NVIDIA 上传 asset 超时 ({upload_timeout}s)")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"NVIDIA 上传 asset 异常: {e}")
|
||||
return None
|
||||
if up.status_code not in (200, 201):
|
||||
logger.warning(f"NVIDIA 上传 asset 失败 HTTP {up.status_code}: {up.text[:200]}")
|
||||
return None
|
||||
logger.info(f"NVIDIA asset 上传成功: {asset_id}")
|
||||
return asset_id
|
||||
|
||||
def analyze_video(self, video_path: str,
|
||||
known_members_context: str,
|
||||
event_start_time: str = '') -> Optional[Dict]:
|
||||
@@ -156,10 +115,18 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
logger.warning(f"NVIDIA 视频文件不存在: {video_path}")
|
||||
return None
|
||||
|
||||
# asset 只上传一次，模型链内复用同一 assetId
|
||||
asset_id = self._upload_asset(video_path)
|
||||
if not asset_id:
|
||||
self._cb.record_failure()
|
||||
size_mb = os.path.getsize(video_path) / (1024 * 1024)
|
||||
if size_mb > self.max_base64_mb:
|
||||
logger.warning(
|
||||
f"NVIDIA 视频 {size_mb:.1f}MB 超过 base64 上限 {self.max_base64_mb}MB，"
|
||||
"跳过（不做注定失败的慢速编码）")
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(video_path, 'rb') as f:
|
||||
video_b64 = base64.b64encode(f.read()).decode()
|
||||
except Exception as e:
|
||||
logger.warning(f"NVIDIA 读取/编码视频失败: {e}")
|
||||
return None
|
||||
|
||||
prompt = self._build_video_prompt(known_members_context, event_start_time)
|
||||
@@ -176,7 +143,7 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "text", "text": prompt},
|
||||
{"type": "video_url", "video_url": {
|
||||
"url": f"data:video/mp4;asset_id={asset_id}"}}
|
||||
"url": f"data:video/mp4;base64,{video_b64}"}}
|
||||
]}],
|
||||
temperature=0.2,
|
||||
max_tokens=16384,
|
||||
@@ -192,22 +159,19 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
last_err = f"{model}_empty"
|
||||
self._sleep_switch(idx)
|
||||
continue
|
||||
data = self._parse_json(content)
|
||||
if not data or 'events' not in data:
|
||||
try:
|
||||
data = parse_vlm_json(content)
|
||||
except VLMOutputInvalidError as e:
|
||||
self._emit_model_call(model, started, duration, False, "json_parse_failed")
|
||||
logger.warning(f"NVIDIA [{model}] JSON 解析失败，切换下一模型: {content[:120]}")
|
||||
logger.warning(f"NVIDIA [{model}] JSON 解析失败，切换下一模型: {e}")
|
||||
last_err = f"{model}_json"
|
||||
self._sleep_switch(idx)
|
||||
continue
|
||||
self._emit_model_call(model, started, duration, True)
|
||||
self._cb.record_success()
|
||||
logger.info(f"NVIDIA [{model}] 整视频分析完成，events={len(data.get('events', []))}")
|
||||
return {
|
||||
"global_summary": str(data.get('global_summary', '')),
|
||||
"events": data.get('events', []),
|
||||
"people_mentioned": data.get('people_mentioned', []),
|
||||
"compute_provider": f"nvidia:{model}",
|
||||
}
|
||||
data['compute_provider'] = f"nvidia:{model}"
|
||||
return data
|
||||
except Exception as e:
|
||||
duration = time.time() - t0
|
||||
self._emit_model_call(model, started, duration, False, str(e))
|
||||
@@ -224,27 +188,6 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
logger.info(f"NVIDIA 等待 {self.switch_interval_sec}s 后切换下一模型")
|
||||
time.sleep(self.switch_interval_sec)
|
||||
|
||||
@staticmethod
|
||||
def _parse_json(content: str) -> Optional[dict]:
|
||||
content = content.strip()
|
||||
try:
|
||||
return json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
fence = re.search(r'```(?:json)?\s*(\{.*?\})\s*```', content, re.DOTALL)
|
||||
if fence:
|
||||
try:
|
||||
return json.loads(fence.group(1))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
brace = re.search(r'\{.*\}', content, re.DOTALL)
|
||||
if brace:
|
||||
try:
|
||||
return json.loads(brace.group(0))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return None
|
||||
|
||||
def _build_video_prompt(self, known_members: str, event_start_time: str) -> str:
|
||||
camera = load_config().get('gdrive_sync', {}).get('camera_name', '')
|
||||
return build_video_prompt(known_members, event_start_time, camera)
|
||||
@@ -253,22 +196,29 @@ class NvidiaVisionAdapter(BaseModelAdapter):
|
||||
# 智能问答：纯文本
|
||||
# ------------------------------------------------------------------
|
||||
def chat(self, prompt: str, max_tokens: int = 2048) -> Optional[str]:
|
||||
if self._cb.is_open():
|
||||
logger.warning("NVIDIA 熔断器 OPEN，跳过问答")
|
||||
return None
|
||||
if self._client is None:
|
||||
logger.warning("NVIDIA 客户端未初始化，跳过问答")
|
||||
return None
|
||||
try:
|
||||
resp = self._client.chat.completions.create(
|
||||
model=self.model_name,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.3,
|
||||
max_tokens=max_tokens,
|
||||
timeout=self.timeout
|
||||
)
|
||||
content = resp.choices[0].message.content
|
||||
return content.strip() if content else None
|
||||
except Exception as e:
|
||||
logger.warning(f"NVIDIA 问答异常: {e}")
|
||||
return None
|
||||
for model in self.model_chain:
|
||||
try:
|
||||
resp = self._client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
temperature=0.3,
|
||||
max_tokens=max_tokens,
|
||||
timeout=self.timeout
|
||||
)
|
||||
content = resp.choices[0].message.content
|
||||
if content:
|
||||
self._cb.record_success()
|
||||
return content.strip()
|
||||
except Exception as e:
|
||||
logger.warning(f"NVIDIA [{model}] 问答异常: {e}")
|
||||
self._cb.record_failure()
|
||||
return None
|
||||
|
||||
def get_timeout(self) -> int:
|
||||
return self.timeout
|
||||
|
||||
Reference in New Issue
Block a user