Commit Graph

4 Commits

Author SHA1 Message Date
Misaka_Company
2ae5621090 feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit
- Add --append flag: diff source-table 总排号 against existing
  Common.Attachment SN and classify/write only the missing ones. --limit
  caps the per-run append count, --order sets the direction. Backed by
  new fetch_existing_attachment_sns() in db.py.
- Fix batch elapsed-time accounting: summarize_results() summed each
  task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor
  concurrency (cumulative work time, not real wall-clock — 100 tasks on
  8 workers reported ~5x the actual runtime). Callers now time
  classify_batch() via perf_counter and pass wall_clock_ms; both the
  write_attachments [汇总] line and main.py --summary report wall-clock
  separately from the cumulative sum.
- Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the
  human-readable [汇总] line; structured JSON --summary fields stay in ms.
- Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100
  bind-parameter hard limit (previously --append --limit 5000 failed at
  the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids
  (sn / id paths) and to the DELETE inside upsert_attachments.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 11:17:17 +08:00
Misaka_Company
ee566e3fbd refactor: --id 改为真实 ID 列,新增 --sn 对应总排号
此前接口 --id 实际按总排号列查询,与数据库真实 ID 字段语义混淆。
现明确区分两种键类型:
- --id  -> 数据库真实 ID 列(config id_field)
- --sn  -> 总排号列(config id_column,即原先 --id 的语义)
- --ids-file 视为总排号(键类型 sn)

db.py: fetch_params_by_ids 改为接受 (标识符, 键类型) 列表,返回
  {标识符: {param, sn}};id 键回取对应总排号;新增 id_field 配置读取
classifier.py: classify_batch/classify_single 透传键类型,输出 zong_pai_hao
  一律为回查到的总排号,not_found 时回退输入标识符便于追溯
main.py / write_attachments.py: 新增 --sn,--id 改指真实 ID,二者可混用;
  全表扫描回退仍按总排号(sn)查询
README.md: 对齐 --id/--sn 语义并补充 id_field 配置说明

注:config.yaml 含密钥被 .gitignore 忽略,id_field 仅存于本地配置;
db.py 在未配置 id_field 时优雅降级为按总排号查询并告警。

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 16:00:21 +08:00
Misaka_Company
622f348cf1 feat: 附件分类结果写入层 + 运行统计 + --id 指定总排号
- db.py: 新增 upsert_attachments/fetch_all_ids,实现 Common.Attachment 幂等写入
  (先删受影响 NS 旧行再批量插入,依赖唯一索引,可安全重跑)
- write_attachments.py: 写入入口,支持 --limit/--ids-file/--id(单个或逗号分隔多个)
  /--mode/--dry-run/--enable-other,运行结束打印 token 与缓存命中率汇总
- llm_client.py: LLMCallResult 捕获 usage/elapsed_ms/model/attempt
- classifier.py: classify_batch 结果透传 meta(耗时分两种、上下文 token、缓存命中率),
  新增 summarize_results 聚合批统计
- main.py: 新增 --summary 把批汇总打到 stderr,stdout 保持干净 JSON Lines
- order_logger.py: 每次 LLM 尝试补充 [调用统计] 段,便于排查耗时与缓存效果
- README.md: 对齐上述接口与统计说明

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 15:38:06 +08:00
Misaka_Company
2110e6f38c feat: add order-attachment LLM classifier
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 13:35:28 +08:00