feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit

- Add --append flag: diff source-table 总排号 against existing
  Common.Attachment SN and classify/write only the missing ones. --limit
  caps the per-run append count, --order sets the direction. Backed by
  new fetch_existing_attachment_sns() in db.py.
- Fix batch elapsed-time accounting: summarize_results() summed each
  task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor
  concurrency (cumulative work time, not real wall-clock — 100 tasks on
  8 workers reported ~5x the actual runtime). Callers now time
  classify_batch() via perf_counter and pass wall_clock_ms; both the
  write_attachments [汇总] line and main.py --summary report wall-clock
  separately from the cumulative sum.
- Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the
  human-readable [汇总] line; structured JSON --summary fields stay in ms.
- Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100
  bind-parameter hard limit (previously --append --limit 5000 failed at
  the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids
  (sn / id paths) and to the DELETE inside upsert_attachments.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
2026-07-28 11:17:17 +08:00
parent a014a945af
commit 2ae5621090
5 changed files with 166 additions and 42 deletions

View File

@@ -265,12 +265,21 @@ def classify_single(
)[0]
def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
def summarize_results(
results: list[dict[str, Any]],
wall_clock_ms: float | None = None,
) -> dict[str, Any]:
"""汇总一批 classify_batch 结果的运行统计。
从每条结果的 meta 里累加:总耗时、实际 LLM 调用次数、各类 token 总量
以及整体缓存命中率(按 prompt_cache_hit_tokens / prompt_tokens 加权)。
供 CLI 打印批统计用,不改变 classify_batch 的返回结构。
从每条结果的 meta 里累加:各结果 elapsed_ms 的总和、实际 LLM 调用次数
各类 token 总量,以及整体缓存命中率(按 prompt_cache_hit_tokens /
prompt_tokens 加权)。供 CLI 打印批统计用,不改变 classify_batch 的返回结构。
注意耗时两个字段的区别classify_batch 是并发执行的):
- total_elapsed_ms各结果自身耗时(elapsed_ms)的**累加**,即"累计工作量"
(相当于把这些任务串行跑的总时长),并发下会明显大于真实墙钟;
- wall_clock_ms调用方传入的整批分类**真实墙钟**耗时(在 classify_batch
外层用 perf_counter 计时),二者之比 ≈ 并发增益(≈ max_workers
"""
total_elapsed = 0.0
llm_calls = 0
@@ -291,7 +300,7 @@ def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
sum_hit += u.get("prompt_cache_hit_tokens", 0) or 0
sum_miss += u.get("prompt_cache_miss_tokens", 0) or 0
avg_cache_hit_rate = round(sum_hit / sum_prompt, 4) if sum_prompt else 0.0
return {
out = {
"count": len(results),
"total_elapsed_ms": round(total_elapsed, 1),
"llm_calls": llm_calls,
@@ -302,3 +311,6 @@ def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
"total_cache_miss_tokens": sum_miss,
"avg_cache_hit_rate": avg_cache_hit_rate,
}
if wall_clock_ms is not None:
out["wall_clock_ms"] = round(float(wall_clock_ms), 1)
return out