fix(ai): 队列会把共用的网关打到 502,加并发上限和间隔
排查生产上一直生成不出来,发现网关在返 502。上去看:机器好好的、systemd 说 active、5100 端口在监听——但它是 `gunicorn -w 1 --threads 4`,全部并发 就四个,而且 fam-edge 和摄像头项目也在用同一个。 我们这边一个请求占一个线程 2~5 分钟,NVIDIA 链重试起来最坏十七分钟(它自己 README 已知问题 #3)。而我写的 worker 是跑完一个立刻拉下一个,同步后还有八 个 scope 排队——等于拿满线程不撒手。这个 502 大概率是我打出来的,而且顺带 把另外两个项目也打下线了。 - 并发按整个部署计算,不是每个 gunicorn worker 一个:claim 前先数全局 running(两个 worker 各跑「一个」就是两个并发) - 每跑完一个任务停 20 秒,不只是空闲时才停 - 两个都可用环境变量调,注释里写清楚调大的代价是什么 补齐的历史数据晚二十分钟到没有任何人受影响;网关不响应是三个项目一起受影响。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -89,3 +89,16 @@ AI_MAX_TOKENS=1024
|
||||
# before the answer starts, and at 1024 the reply was all thinking with the
|
||||
# JSON truncated away.
|
||||
AI_COACH_MAX_TOKENS=4000
|
||||
|
||||
# --- AI coach job queue ---
|
||||
# The gateway runs `gunicorn -w 1 --threads 4` and is shared with fam-edge and
|
||||
# the camera project: four concurrent requests for everyone, while one of ours
|
||||
# holds a thread for 2-5 minutes. So this consumer runs one job at a time
|
||||
# across the whole deployment (not one per Gunicorn worker) and waits between
|
||||
# jobs. Raising either of these makes the backfill finish sooner at the cost of
|
||||
# the shared box — a 502 there is a 502 for the other two projects as well.
|
||||
AI_JOB_CONCURRENCY=1
|
||||
AI_JOB_GAP_SECONDS=20
|
||||
# Set AI_JOBS=false to stop consuming entirely (screens then show the computed
|
||||
# figures with no model reading).
|
||||
AI_JOBS=true
|
||||
|
||||
Reference in New Issue
Block a user