fix(ai): 队列会把共用的网关打到 502,加并发上限和间隔

排查生产上一直生成不出来,发现网关在返 502。上去看:机器好好的、systemd
说 active、5100 端口在监听——但它是 `gunicorn -w 1 --threads 4`,全部并发
就四个,而且 fam-edge 和摄像头项目也在用同一个。

我们这边一个请求占一个线程 2~5 分钟,NVIDIA 链重试起来最坏十七分钟(它自己
README 已知问题 #3)。而我写的 worker 是跑完一个立刻拉下一个,同步后还有八
个 scope 排队——等于拿满线程不撒手。这个 502 大概率是我打出来的,而且顺带
把另外两个项目也打下线了。

- 并发按整个部署计算,不是每个 gunicorn worker 一个:claim 前先数全局
  running(两个 worker 各跑「一个」就是两个并发)
- 每跑完一个任务停 20 秒,不只是空闲时才停
- 两个都可用环境变量调,注释里写清楚调大的代价是什么

补齐的历史数据晚二十分钟到没有任何人受影响;网关不响应是三个项目一起受影响。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
ericwyuan
2026-09-01 15:46:42 +08:00
parent 2488956f36
commit a2d6a5c57f
3 changed files with 76 additions and 5 deletions

View File

@@ -89,3 +89,16 @@ AI_MAX_TOKENS=1024
# before the answer starts, and at 1024 the reply was all thinking with the
# JSON truncated away.
AI_COACH_MAX_TOKENS=4000
# --- AI coach job queue ---
# The gateway runs `gunicorn -w 1 --threads 4` and is shared with fam-edge and
# the camera project: four concurrent requests for everyone, while one of ours
# holds a thread for 2-5 minutes. So this consumer runs one job at a time
# across the whole deployment (not one per Gunicorn worker) and waits between
# jobs. Raising either of these makes the backfill finish sooner at the cost of
# the shared box — a 502 there is a 502 for the other two projects as well.
AI_JOB_CONCURRENCY=1
AI_JOB_GAP_SECONDS=20
# Set AI_JOBS=false to stop consuming entirely (screens then show the computed
# figures with no model reading).
AI_JOBS=true