feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit
- Add --append flag: diff source-table 总排号 against existing Common.Attachment SN and classify/write only the missing ones. --limit caps the per-run append count, --order sets the direction. Backed by new fetch_existing_attachment_sns() in db.py. - Fix batch elapsed-time accounting: summarize_results() summed each task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor concurrency (cumulative work time, not real wall-clock — 100 tasks on 8 workers reported ~5x the actual runtime). Callers now time classify_batch() via perf_counter and pass wall_clock_ms; both the write_attachments [汇总] line and main.py --summary report wall-clock separately from the cumulative sum. - Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the human-readable [汇总] line; structured JSON --summary fields stay in ms. - Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100 bind-parameter hard limit (previously --append --limit 5000 failed at the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids (sn / id paths) and to the DELETE inside upsert_attachments. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -265,12 +265,21 @@ def classify_single(
|
||||
)[0]
|
||||
|
||||
|
||||
def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
def summarize_results(
|
||||
results: list[dict[str, Any]],
|
||||
wall_clock_ms: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""汇总一批 classify_batch 结果的运行统计。
|
||||
|
||||
从每条结果的 meta 里累加:总耗时、实际 LLM 调用次数、各类 token 总量、
|
||||
以及整体缓存命中率(按 prompt_cache_hit_tokens / prompt_tokens 加权)。
|
||||
供 CLI 打印批统计用,不改变 classify_batch 的返回结构。
|
||||
从每条结果的 meta 里累加:各结果 elapsed_ms 的总和、实际 LLM 调用次数、
|
||||
各类 token 总量,以及整体缓存命中率(按 prompt_cache_hit_tokens /
|
||||
prompt_tokens 加权)。供 CLI 打印批统计用,不改变 classify_batch 的返回结构。
|
||||
|
||||
注意耗时两个字段的区别(classify_batch 是并发执行的):
|
||||
- total_elapsed_ms:各结果自身耗时(elapsed_ms)的**累加**,即"累计工作量"
|
||||
(相当于把这些任务串行跑的总时长),并发下会明显大于真实墙钟;
|
||||
- wall_clock_ms:调用方传入的整批分类**真实墙钟**耗时(在 classify_batch
|
||||
外层用 perf_counter 计时),二者之比 ≈ 并发增益(≈ max_workers)。
|
||||
"""
|
||||
total_elapsed = 0.0
|
||||
llm_calls = 0
|
||||
@@ -291,7 +300,7 @@ def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
sum_hit += u.get("prompt_cache_hit_tokens", 0) or 0
|
||||
sum_miss += u.get("prompt_cache_miss_tokens", 0) or 0
|
||||
avg_cache_hit_rate = round(sum_hit / sum_prompt, 4) if sum_prompt else 0.0
|
||||
return {
|
||||
out = {
|
||||
"count": len(results),
|
||||
"total_elapsed_ms": round(total_elapsed, 1),
|
||||
"llm_calls": llm_calls,
|
||||
@@ -302,3 +311,6 @@ def summarize_results(results: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"total_cache_miss_tokens": sum_miss,
|
||||
"avg_cache_hit_rate": avg_cache_hit_rate,
|
||||
}
|
||||
if wall_clock_ms is not None:
|
||||
out["wall_clock_ms"] = round(float(wall_clock_ms), 1)
|
||||
return out
|
||||
|
||||
Reference in New Issue
Block a user