10 Commits

Author SHA1 Message Date
Misaka_Company
c83c47a631 feat: add four fine-type classifications and refine rules
- Add four new fine-type classifications: 阻尼器, 弯管, 检验报告, 抱箍
- Split 卡箍抱箍 into separate 卡箍 and 抱箍 entries
- Rename 检验记录过程性 to 检验记录
- Clarify rule: distinguish 阻尼帽/阻尼钉 (internal process) from 阻尼器 (external accessory)
- Add rule: 法兰隔膜 not counted as accessory (ignored in classification)
- Update README.md: DOC 12→13, PART 24→27, total 37→41 fine types
2026-07-28 14:53:03 +08:00
Misaka_Company
8c8d62ff06 feat(prompts): rename 检验记录过程性 -> 检验记录; drop 法兰隔膜 as attachment
Classification-boundary adjustments in prompts.py (the single source the
parser validates against, so parser.py needs no change):

- Rename fine type 检验记录过程性 -> 检验记录 in the 资料类 enumeration.
- 法兰隔膜 no longer counts as an attachment: removed from the 配件类
  enumeration (25 -> 24 fine types) and the prose example list, added an
  explicit rule bullet and 注意 note that it is ignored even when the
  param text mentions it, and fixed the stale comment that referenced it.
- Updated the few-shot example: input still mentions 法兰隔膜 but the
  expected output now lists only 紧固件, actively teaching the model to
  ignore 法兰隔膜.
- README category lists/counts synced (检验记录 rename; 配件类 25 -> 24).

Verified via parser: 检验记录 and 紧固件 validate; 检验记录过程性 and
法兰隔膜 are now rejected as non-enum fine types.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 12:27:03 +08:00
Misaka_Company
2ae5621090 feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit
- Add --append flag: diff source-table 总排号 against existing
  Common.Attachment SN and classify/write only the missing ones. --limit
  caps the per-run append count, --order sets the direction. Backed by
  new fetch_existing_attachment_sns() in db.py.
- Fix batch elapsed-time accounting: summarize_results() summed each
  task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor
  concurrency (cumulative work time, not real wall-clock — 100 tasks on
  8 workers reported ~5x the actual runtime). Callers now time
  classify_batch() via perf_counter and pass wall_clock_ms; both the
  write_attachments [汇总] line and main.py --summary report wall-clock
  separately from the cumulative sum.
- Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the
  human-readable [汇总] line; structured JSON --summary fields stay in ms.
- Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100
  bind-parameter hard limit (previously --append --limit 5000 failed at
  the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids
  (sn / id paths) and to the DELETE inside upsert_attachments.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 11:17:17 +08:00
pengq
a014a945af refactor: 引入 SQLAlchemy 多数据库抽象(PostgreSQL + SQL Server)
- 新增 orm.py:按 db_type 构建引擎(postgresql+psycopg2 / mssql+pyodbc),
  声明式 Attachment 模型,init_schema 幂等建表
- 重写 db.py 为 SQLAlchemy Core 实现(动态 Table + quote 跨库正确引用、
  id/sn 双键回查、跨库分页、先删后插幂等),对外签名不变
- 配置:config.yaml 默认 PostgreSQL,新增 config.mssql.yaml 保留 SQL Server,
  config_loader 支持可选 driver 与 db_type
- write_attachments.py 新增 --init-db
- 依赖 requirements.txt 增 sqlalchemy / psycopg2-binary(保留 pyodbc)
- README 对齐:修正目标表字段描述并新增数据库抽象层小节
2026-07-26 11:59:55 +08:00
Misaka_Company
70b6fb3767 feat: --limit 全表扫描支持按真实 ID 排序/升降序/范围过滤
此前 --limit 仅按总排号排序取前 N 个。现扩展为:
- 排序按数据库真实 ID 列(config.id_field),新增 --order {asc,desc}(默认 asc)
- 新增 --range START,END 按真实 ID 闭区间过滤(与 --order/--limit 可组合)
- 上述三参数仅全表扫描生效,与 --id/--sn/--ids-file 互斥

db.py: fetch_all_ids 新增 order/id_min/id_max 参数,用派生表
  (内层 DISTINCT 取 (总排号,ID) 配对,外层按 ID 排序)规避
  SQL Server "SELECT DISTINCT 时 ORDER BY 列须在选择列表" 的限制;
  id_field 未配置时降级按总排号排序并告警。
write_attachments.py: 新增 --order/--range CLI 参数并接线,含 --range
  格式校验(须为 START,END 两个整数);模块 docstring 补示例。
README.md: 写入用法块与项目结构注释补充 --order/--range 说明。

自测(小数据量,只读+部分 dry-run):--limit 3 升/降序、--range 800,805
升序、--range 800,805 --order desc --limit 3 均验证返回总排号按真实 ID
正确排序且在范围内;dry-run 不落库;--range 格式错误正确报错退出。

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 16:27:16 +08:00
Misaka_Company
fd047e2143 refactor: Attachment 表关联字段 NS 更名为 SN
数据库侧已将 Common.Attachment 的 NS 列重命名为 SN,同步更新代码:
- db.py: upsert_attachments 的 DELETE/INSERT 改用 [SN],distinct_sn 变量与
  相关注释、日志文案统一更新
- write_attachments.py: 模块 docstring 与 convert_results_to_rows 的行列说明
  由 (NS, ...) 改为 (SN, ...);写入目标描述 Common.Attachment.SN
- README.md: 落库约定与字段表中 NS 全部改为 SN

已用 --sn 26B10 实测写入并用 SELECT [SN] 验证数据正确落库。

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 16:14:05 +08:00
Misaka_Company
ee566e3fbd refactor: --id 改为真实 ID 列,新增 --sn 对应总排号
此前接口 --id 实际按总排号列查询,与数据库真实 ID 字段语义混淆。
现明确区分两种键类型:
- --id  -> 数据库真实 ID 列(config id_field)
- --sn  -> 总排号列(config id_column,即原先 --id 的语义)
- --ids-file 视为总排号(键类型 sn)

db.py: fetch_params_by_ids 改为接受 (标识符, 键类型) 列表,返回
  {标识符: {param, sn}};id 键回取对应总排号;新增 id_field 配置读取
classifier.py: classify_batch/classify_single 透传键类型,输出 zong_pai_hao
  一律为回查到的总排号,not_found 时回退输入标识符便于追溯
main.py / write_attachments.py: 新增 --sn,--id 改指真实 ID,二者可混用;
  全表扫描回退仍按总排号(sn)查询
README.md: 对齐 --id/--sn 语义并补充 id_field 配置说明

注:config.yaml 含密钥被 .gitignore 忽略,id_field 仅存于本地配置;
db.py 在未配置 id_field 时优雅降级为按总排号查询并告警。

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 16:00:21 +08:00
Misaka_Company
622f348cf1 feat: 附件分类结果写入层 + 运行统计 + --id 指定总排号
- db.py: 新增 upsert_attachments/fetch_all_ids,实现 Common.Attachment 幂等写入
  (先删受影响 NS 旧行再批量插入,依赖唯一索引,可安全重跑)
- write_attachments.py: 写入入口,支持 --limit/--ids-file/--id(单个或逗号分隔多个)
  /--mode/--dry-run/--enable-other,运行结束打印 token 与缓存命中率汇总
- llm_client.py: LLMCallResult 捕获 usage/elapsed_ms/model/attempt
- classifier.py: classify_batch 结果透传 meta(耗时分两种、上下文 token、缓存命中率),
  新增 summarize_results 聚合批统计
- main.py: 新增 --summary 把批汇总打到 stderr,stdout 保持干净 JSON Lines
- order_logger.py: 每次 LLM 尝试补充 [调用统计] 段,便于排查耗时与缓存效果
- README.md: 对齐上述接口与统计说明

Co-Authored-By: WorkBuddy <workbuddy@tencent.com>
2026-07-24 15:38:06 +08:00
Misaka_Company
4ca048bd26 docs: 修正文档与代码对齐问题
- 附件分类说明.md 第4节:澄清程序输出为完整记录(含
  zong_pai_hao/status/has_attachment),types 仅为分类结果字段,
  示例只截取 types 展示格式差异,消除"只输出 types"误导。
- README.md:项目结构由错误的 src/ 子目录布局改为实际根目录平铺
  (并补入已弃用的 attachment_classifier.py 旧 Excel 版);
  修正 src/prompts.py 为 prompts.py 的引用。
2026-07-24 14:06:05 +08:00
Misaka_Company
2110e6f38c feat: add order-attachment LLM classifier
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 13:35:28 +08:00