Commit d0d8a4b
committed
feat(people): 身份画像后台构建改为有界并发协调器
新增 identityProfileCoordinator,将原先每轮串行构建单个人物的 buildOneDirtyPerson
升级为“有界并发构建 + 串行写入提交 + 批末 ANN 合并调度”,提升 dirty 画像吞吐:
- coordinator(identity_profile_coordinator.go):worker 并发执行只读+纯计算
(ListProfileFaces + builder.Build),写入仍经 WriteQueue 串行提交,单 person
失败只 MarkFailed 不阻断同批;批末统一决定 ANN activate 还是 full rebuild。
- 前台让路:service.go 注入 SetForegroundBusyFn,复用 clusteringCoordinator
的 foregroundWaiterCount,前台有 merge/split/move 写操作时跳过新批次,已开始
的小批次允许完成,避免与前台争抢资源。
- 高/低优先级 dirty 分批:新增 repo.ListDirtyByReasons,前台操作产生的
dirty_reason 优先构建,避免被低优先级按 person_id 挤出。
- 时间预算:slice budget(默认 5s)软上限,超时不再派发新 worker。
- 新增配置项 build_workers(1-4)/dirty_batch_size/slice_budget_ms/
ann_rebuild_delta_threshold,含默认值与 Validate 校验。
- NewBackgroundDB 连接池 1→8,WAL 模式支持并发只读 worker 不互锁。
- 运行状态接口新增 IdentityCoordinatorStats(脱敏),定位瓶颈在
build/write/ANN rebuild 哪一阶段。
- 测试桩补齐 ListDirtyByReasons,SliceBuildsAtMostOnePerson 改为
SliceBuildsBoundedDirtyBatch。1 parent 21aaaa6 commit d0d8a4b
12 files changed
Lines changed: 1247 additions & 82 deletions
File tree
- backend
- internal
- model
- repository
- service
- pkg
- config
- database
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
96 | 100 | | |
97 | 101 | | |
98 | 102 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
718 | 718 | | |
719 | 719 | | |
720 | 720 | | |
721 | | - | |
722 | | - | |
723 | | - | |
724 | | - | |
725 | | - | |
726 | | - | |
727 | | - | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
728 | 729 | | |
729 | 730 | | |
730 | 731 | | |
| |||
799 | 800 | | |
800 | 801 | | |
801 | 802 | | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
802 | 828 | | |
803 | 829 | | |
804 | 830 | | |
| |||
Lines changed: 23 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
74 | 78 | | |
75 | 79 | | |
76 | 80 | | |
| |||
189 | 193 | | |
190 | 194 | | |
191 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
192 | 215 | | |
193 | 216 | | |
194 | 217 | | |
| |||
0 commit comments