100 Commits

Author SHA1 Message Date
Misaka
bccf7cd396 fix(runtime): strip host-injected env vars when launching 安能 2026-08-02 14:14:34 +08:00
Misaka
c1bd53d832 feat(tasks): record trigger mode, target date and force in task history 2026-08-02 14:14:30 +08:00
Misaka
3c32720985 feat(sites): auto-screenshot on final download failure for debugging
所有站点 with_retry 在最后一次重试失败、reset 之前自动截图,
保存到 logs/screenshots/。网页站点走 Playwright page.screenshot(),
安能走 CDP Page.captureScreenshot。截图失败绝不阻塞任务流程。

- paths.py: 新增 SCREENSHOT_DIR (BASE_DIR/logs/screenshots/)
- runtime.py: 新增 capture_error_screenshot() 工具函数
- .gitignore: 新增 logs/ 忽略规则

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-02 11:18:21 +08:00
Misaka
53c71aeeac refactor(status): derive site ready/business_date from PG instead of ingest_state
_ready_flags 改从 PostgreSQL 直接查询(expected_record /
actual_record / baishi_daily_stats),target_date = today − offset。
消除因 ingested_at 日期比对导致的每日零点全站 ready 集体重置。

store.py: 新增 has_data(site, kind, target_date) 查 PG 数据存在性
runtime.py: _ready_flags 返回 (flags, dates) 同源元组,_apply_ready
  同步写入 business_date,修正 ready 与 business_date 不同源导致的
  前端日期标签漂移(如实到就绪却显示'前天')

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-02 09:22:54 +08:00
Misaka
837264f7b0 fix(zto): resolve real-today date picker failure on month-boundary days
The ZTO jQuery Date Range Picker always displays a dual-month view.
When today falls on the 1st (or early days) of a month, the same date
appears in both panels: a hidden ghost cell (month1, display:none) and
a visible cell (month2). Both carry the real-today CSS class, so .first
picks the hidden one, causing wait_for(visible) to timeout.

Replace DOM-based real-today time extraction with Python datetime
computation. Add _zto_find_visible_day to locate the actually visible
date cell (skipping hidden ghost cells, trying both midnight and
23:59:59 time variants). Fix _zto_flip_to_target_month to use the
same visibility-aware lookup.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:50:30 +08:00
Misaka
eaf56c1c9e refactor(status): derive readiness from ingest_state; drop Excel probe & /data
状态盘就绪态 {kind}_ready 改为从持久的 ingest_state 派生(DB 真相、重启不丢),
不再由心跳读 downloads/ Excel、也不在启动时重置:
- 心跳 _fresh_ingest/_ready_flags/_apply_ready:expected/actual_ready=该类今天入库成功;
  undelivered_ready=百世原生(今天入库) / 4 站派生(expected ∧ actual)。
- _persist_to_db 入库后调 _refresh_ready 立即派生(省 30s 心跳等待,与心跳同源);
  4 站 undelivered 连入 expected+actual,故 ingest_state 补记 expected/actual/undelivered 三行。
- _record_business_date 只写 business_date,不再碰 ready。
- 删 run_heartbeat 的 Excel 探测循环、DATA_FILENAMES、probe_data_file、遗物清理。
- state_store 增 set_business_date / set_ready(仅写单字段,不碰彼此)。
- 删 GET /data/{filename}(前端不再下载原始中转 Excel);/report 保留。

原则:Excel 只作「站点下载→入库」中转,状态/比对一律走 DB。
真机+单元验证:重启后心跳派生(百世今日入库→立即绿、重启不掉灰);韵达 expected-only
时 undelivered 不亮、补 actual 后亮(派生);百世原生;_ready_flags 7 例边界全过。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 23:47:23 +08:00
Misaka
dc7653c256 fix(db_compare): anchor comparison on actual_offset so yunda reads fresh data
比对以实到扫描日(actual.scan_time::date)为锚,锚点日期必须等于实到下载日 = actual_offset。
原 _site_undelivered_handler(runtime.py)与 _target_date_for(db_compare.py)误用
expected_offset 算锚点:韵达 exp=1 / act=0,锚点落到 today-1(昨天),读到历史数据。
两处改用 actual_offset 后韵达锚点 = today,反推出 expected 的昨天批次正确参与比对。
其余 3 站 exp == act == 0,锚点不变。

5 站真机验证:韵达 07-31 比对现读新鲜数据 107/104/3(修复前读昨天 116/115/1);
中通/安能/顺心 不变;全站汇总报表重新生成。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 22:58:12 +08:00
Misaka
3c7e9f2522 feat(db_compare): DB-based full summary report + wire 跑比对 to it
新增 build_full_report:4 站走 compare_site_date、百世走 baishi_daily_stats(基数) + undelivered_record(按 ingested_at 日期过滤),复用 compare.build_summary 渲染 KPI/柱状图/口径说明,产 output/应到未到数据.xlsx。跑比对入口(__compare__)从 compare.main() Excel 路径切换到 build_full_report。百世未到件用基数差(undelivered_pieces)与应到/已到自洽。

集成验证:前端跑比对 -> build_full_report -> /report 下载,各站未到件 顺心8/中通10/韵达2/安能0/百世8。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 21:56:39 +08:00
Misaka
8521c200ab feat(store): persist 百世 daily basis to PG (baishi_daily_stats)
百世应到/实到基数(应扫/已扫)原仅在 state_store(单值、无历史)。新增百世专用聚合表 baishi_daily_stats(site+business_date UPSERT),baishi 下载时抓到基数直接落库(store.upsert_baishi_daily_stats,一步,不绕 state_store→store)。state_store 双写保留以兼容旧 Excel 汇总(process_baishi),后续统一清理。

真机端到端验证通过:PG (2026-07-31, 194, 186, 8),state_store 一致。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 21:23:08 +08:00
Misaka
c6ad6a0ca2 fix(runtime): ingest before db_compare so first-run/force reads fresh data
DB 比对发生在入库之前,导致首次/force 时 PG 无当天数据,比对返回 None、不产出 Excel。在 _site_undelivered_handler 下载成功后、比对前,前置 _record_business_date + ingest_task,使比对能读到本次下载的数据。dispatch_task 后置 _persist_to_db 保持不变(对 undelivered 幂等重复一次,安全)。

四站真机验证通过(中通/韵达/安能 + 顺心):前置入库日志均出现在 db_compare 之前,force 首跑即产出 Excel。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 20:25:52 +08:00
Misaka_Company
541836fd1b feat(db_compare): add PostgreSQL-based comparison engine with SF handling
Replace Excel-based undelivered comparison with DB queries for all four
sites. The engine anchors on actual scan_time, reverse-lookups handover
batches, and compares expected vs actual waybill-by-waybill.

Shunxin SF waybills: use COUNT(*) instead of COUNT(DISTINCT piece_no)
since SF piece numbers are random and not derivable from the waybill.

Changes:
- db_compare.py: new module with compare_site_date(), compare_site_batch(),
  write_result_excel(), and POST /compare API endpoint
- runtime.py: switch _site_undelivered_handler from compare.write_site_file
  (Excel) to db_compare (DB); downloads succeed independently of comparison
- server.py: add POST /compare endpoint with date validation
- docs: implementation plan for Shunxin DB comparison

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 16:31:54 +08:00
Misaka_Company
95597fbb0c docs: add four-site comparison logic review report
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 14:48:24 +08:00
Misaka_Company
f66e6dd39e fix(yunda): invert handover number filter to keep empty rows
The previous filter kept rows with non-empty handover numbers
(派件/签收 scans), which were duplicate rows. The correct logic
is to keep rows with empty handover numbers (到/接件 scans).

- store.py: change != "" to == "" in ingest filter
- compare.py: add same filter before comparison (previously missing)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 14:44:50 +08:00
Misaka
09e05f8dfc feat(runtime): disable microphone by default to suppress permission popup
韵达等站点打开时会请求麦克风权限,触发浏览器系统级授权弹窗。在共享 context 上加
add_init_script,于每个页面/iframe 加载前覆盖 navigator.mediaDevices.getUserMedia
(及 webkit/moz 旧版) 为直接 reject(NotAllowedError: Permission disabled):站点调用时
立即被拒、不再弹出系统授权窗,且麦克风被真正挡住(非授权给它)。物流工作台无需音视频
采集,故对所有网页站点统一生效。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 20:58:46 +08:00
Misaka
047036f46b feat(yunda): cross-month calendar navigation for date-specific download
韵达两个流程的日期组件不同,分别适配跨月翻月:
- 应到(.layui-laydate 新版):读 .laydate-set-ym 当前年月,点 .laydate-prev-m/.laydate-next-m
  翻月,格子 td[lay-ymd='YYYY-M-D']。
- 实到(#laydate_box 旧版):读 #laydate_y/#laydate_m 输入框值,点 #laydate_MM 内
  .laydate_chprev/.laydate_chnext 翻月,格子 td[y][m][d]。
两处设日期改走 _yunda_pick_laydate_new / _yunda_pick_laydate_old,同月(offset 当月)
行为不变,仅跨月时翻月。年*12+月 比较天然支持跨年。

实到 #startDate/#endDate 保持普通 click(force=True 会在导航后 laydate 绑定完成前
抢先点击导致面板打不开);新增 #laydate_box:visible 就绪等待。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 20:26:45 +08:00
Misaka
21662d3944 feat(shunxin): cross-month calendar navigation for date-specific download
顺心 Ant Design 日期面板跨月导航:目标日期不在当前月视窗时,读面板头部
.ant-picker-year-btn/.ant-picker-month-btn 得当前年月,按差值点
.ant-picker-header-prev-btn/next-btn 翻到目标月再选格子。应到(车辆点到)与
实到(卸车扫描记录)两处设日期统一改走 _shunxin_pick_date。

同月(offset 当月)行为不变,仅在跨月时多走翻月;与中通 _zto_flip_to_target_month
思路对称,适配 Ant Design 面板。年*12+月 比较天然支持跨年。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 19:58:10 +08:00
Misaka_Company
4f0ef69739 fix(server): lower date backtrack limit from 90 to 31 days
Aneng only supports querying data up to 31 days back. The previous
90-day cap allowed dates that would silently fail at download time,
so restrict the POST /tasks `date` validation to a 31-day window.
2026-07-29 16:55:14 +08:00
Misaka_Company
5e2ef72afd feat(server): add date field to POST /tasks with legality validation
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:11:12 +08:00
Misaka_Company
fcb75643d0 feat(runtime): propagate date through dispatch chain and business-date snapshot
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:10:23 +08:00
Misaka_Company
cda764a305 feat(anneng): support date arg (date takes precedence over offset)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:09:30 +08:00
Misaka_Company
e51aab2e0b feat(shunxin): support date arg, propagate to both accounts
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:08:51 +08:00
Misaka_Company
bacccc43ab feat(yunda): support date arg (date takes precedence over offset)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:08:06 +08:00
Misaka_Company
e240d92ad9 feat(zto): support date arg via effective-offset (reuses cross-month nav)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:07:27 +08:00
Misaka_Company
30e9203fca feat(baishi): accept date kwarg (ignored) for unified dispatch signature
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:06:37 +08:00
Misaka_Company
93e48118bf docs: add implementation plan for date-specific download API
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:03:49 +08:00
Misaka_Company
a68fd46c51 docs: add design spec for date-specific download API (developer interface)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 12:58:08 +08:00
Misaka_Company
e2ae3acd68 feat(zto): cross-month calendar navigation instead of fallback to today
When the offset target date falls outside the current two-month view, flip .prev months (1 month/step) until the target day cell enters month1 view, instead of silently falling back to today. JS-dispatched clicks avoid the .date-range-length-tip hover interception.

Also switch the expected (#beginDate) day-cell click to _dom_click (matching actual #daterange): the range-length tooltip intercepts the second click on non-today cells during single-day range selection, causing timeouts.

Verified end-to-end: offset=45 (-> 2026-06-14, cross-month) expected task succeeded and downloaded 6-14 data without fallback.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 12:49:44 +08:00
Misaka_Company
f710622a3d feat: dedup expected data by handover_no before export submit
- store.get_existing_handover_nos: query PG expected_record.handover_no

- inject dedup skip before submitting export in zto/yunda/anneng/shunxin

- shunxin reads RTS handover_no from waybill-list view (method 1)

- force-redownload switch threaded via task_spec -> dispatch -> impl

- schema: add idx_expected_handover index

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 10:57:07 +08:00
Misaka
d74a35844f feat(schedule): 周期抓取调度取代每日定点下载
应到/实到数据各自独立周期抓取(启用+激活时段+频率),抓取与处理解耦,抓完自动落库(复用现有 _persist_to_db 钩子)。

- state_store: 新增 fetch_schedule 表(site,kind);所有连接加 timeout=3.0 防并发锁;get/set/get_all_fetch_schedules、create_task_if_idle(周期去重)、allowed_kinds;get_all_config 改返回 fetch_schedules;旧 set_schedule 标废弃。修复 get_all_fetch_schedules 列错位(_fetch_spec(r[2:]))。
- server: IntervalTrigger + _in_active_window(含跨午夜) + _enqueue_fetch(就绪门/激活窗口/去重) + _reschedule_fetch + lifespan 按(site,kind)注册 + FetchScheduleSpec + PUT/GET /config 新结构。
- runtime/store/BFF: 零改动。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 22:58:47 +08:00
Misaka
82ab6d8be3 feat(runtime): 服务模式任务执行不再激活浏览器窗口
网页站点(顺心/百世/中通/韵达)执行下载流程时不再把浏览器窗口置顶,避免打扰用户当前工作;流程在后台静默跑。启动登录阶段与交互模式不受影响。

- RuntimeContext 加 foreground 标志:True=任务执行时置顶(交互调试),False=后台静默(服务模式)
- launch_and_prepare(foreground=True) 透传;server worker 以 foreground=False 启动
- _web_handler 按 ctx.foreground 决定单 page 置顶;顺心(list) 透传给 shunxin_download
- shunxin 两个 download 加 foreground 参数,逐账号 bring_to_front 加条件
- 启动登录/初始弹窗清理的 bring_to_front 不变(启动时窗口需可见)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-27 23:17:10 +08:00
Misaka
29edcbfdbb fix(yunda): 导出面板适配 Element UI 改版
韵达导出面板从 jQuery(.allRight/#submitbutton) 改版为 Element UI,应到/实到导出均卡在全选字段步骤。

- actual/expected 改 Element UI 交互:全选 → 向右转移(el-icon-d-arrow-right) → 导出(el-icon-download) → 等 el-loading-mask → el-message-box 成功提示 → 确定
- 新增 _resolve_export_frame() 探测导出 iframe(实到=myFrame、应到=target1),都未命中时 dump 面板内 iframe 名便于排查
- 外层 layui-layer 弹层、查询、切导出服务下载逻辑不变

验证:韵达应到 49 条 / 实到 141 条下载成功。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-27 22:50:47 +08:00
Misaka_Company
5e4889e845 docs: sync docs with current package layout and auto-ingest hook
- config.example.yaml: replace stale site_yunda/main_router with new
  module paths (sites.yunda, runtime)
- CLAUDE.md: add ingest-one to DB CLI list; new subsection documenting
  the download->PostgreSQL auto-ingest hook (_persist_to_db, ingest_task,
  ingest_state, /status.ingest, auto_ingest config)
- README.md: store tree comment lists all 5 CLI commands (add ingest-one)
- docs: /status row notes the ingest field; anneng CDP guide snippet
  gets timeout=15
- cli/server.py: docstring run command -> python -m inbound_verify.cli.server

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 12:35:54 +08:00
Misaka_Company
0111734e9c fix(anneng): bound CDP websocket recv with 15s socket timeout
websocket.create_connection had no timeout, so a hung Runtime.evaluate
(Electron business tab not replying) blocked ws.recv() indefinitely — the
export-poll's 300s deadline and with_retry could never fire (observed as a
~22min hang on an 安能 expected download). A 15s socket timeout lets recv
raise WebSocketTimeoutException so wait_until/with_retry can fail and retry.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 12:22:22 +08:00
Misaka_Company
615661c275 feat(server): expose ingest state in /status; doc auto-ingest in README 2026-07-24 11:16:19 +08:00
Misaka_Company
327f77a727 fix(runtime): wrap ingest_enabled gate so _persist_to_db never raises
Move the store.ingest_enabled() gate inside the main try/except. Previously
it sat outside as a standalone block: ingest_enabled() -> _load_pg_config()
raises FileNotFoundError when config.yaml is absent/malformed, which escaped
_persist_to_db, was caught by dispatch_task's outer except, and flipped a
successful download to FAILED -- violating the hook's never-raise invariant.
Now config errors print a [warn], record ok=False, and return silently.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 11:10:45 +08:00
Misaka_Company
62a66467a9 feat(runtime): auto-ingest hook after successful download
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 11:04:04 +08:00
Misaka_Company
7211846375 feat(state_store): add ingest_state table + set/get helpers 2026-07-24 10:55:43 +08:00
Misaka_Company
1093256de6 fix(store): guard ingest_task against 百世 non-undelivered kinds 2026-07-24 10:46:51 +08:00
Misaka_Company
0c51a41cfc feat(store): add kind-level ingest_task + ingest-one CLI subcommand
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 10:40:51 +08:00
Misaka_Company
70f518a1c3 feat(store): add auto_ingest config + pg connect/statement timeouts
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 10:34:27 +08:00
Misaka_Company
81dea38310 docs: add ingest-hook design spec and implementation plan
Spec for mounting PostgreSQL ingest as a best-effort, kind-level,
synchronous hook on runtime.dispatch_task after a successful download,
plus the 5-task implementation plan.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 10:27:00 +08:00
Misaka_Company
f61f52bcbe docs: align docs with post-restructure package layout
Update README tree (compare.py + domain.py), CLAUDE.md module refs, and the three docs/ deep-dives to the current inbound_verify package (compare/domain/sites/cli). Correct the statistics review report's methodology to the current 口径 (应到=交接件数, 实到=直接数单号去重, 未到=应到−实到) per compare.py docstring; remove ghost-script references in the CDP guide.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 10:12:40 +08:00
Misaka_Company
79c84a5a0c fix: retry launch page.goto to absorb transient DNS/timeout
Third launch failure this session (after two 顺心 load-timeouts and a 中通 ERR_NAME_NOT_RESOLVED) — each a transient network blip that killed the whole worker because launch_and_prepare opened sites with no retry. Wrap each site open+goto in _open_page: up to 3 attempts (2s apart) on domcontentloaded, so transient DNS/timeout is absorbed; only persistent failure (all 3) aborts. All 5 sites + 安能 now launch clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 09:06:02 +08:00
Misaka_Company
62c44b262a fix: launch page.goto uses domcontentloaded to avoid slow-site timeout
launch_and_prepare opened each site with Playwright's default wait_until='load', which waits for every resource (ads/trackers/images). sxne.sxjdfreight.com (顺心) reliably takes >30s to fire load, so its goto timed out and — launch not being retried — killed the whole worker, leaving only one page open. Switch the two launch gotos to wait_until='domcontentloaded' (return as soon as the DOM is ready; the readiness polling still gates login). Behavior for already-fast sites unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 08:59:56 +08:00
Misaka_Company
23042c272b refactor: rename expected_undelivered to compare
git mv expected_undelivered.py -> compare.py; update the 3 importers (store/runtime/router) to import compare. All public names (main, write_site_file, _read_business_dates) unchanged. Also add the Tier 2 plan doc.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 08:40:04 +08:00
Misaka_Company
91ef8970b8 refactor: extract domain.py (shared site/file/colmap config)
Move ALL_REPORT_SITES / SITE_UNDELIVERED_FILE / BAISHI_FILE / BAISHI_COLUMNS / arrived_pieces_* / STATIONS / _site_cfg out of expected_undelivered into a new leaf module inbound_verify/domain.py. store.py and runtime.py now read site/file config from domain directly instead of through the compare engine (store keeps eu only for _read_business_dates). Behavior identical.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 08:38:18 +08:00
Misaka_Company
bd0df8909f refactor: expected_undelivered uses paths.DOWNLOAD_DIR/OUTPUT_DIR (Tier 2)
Remove the duplicate BASE/DOWNLOADS/OUTPUT self-anchor (the one that caused the Tier 1 hotfix bug when the file moved into the package). DOWNLOADS/OUTPUT now come from paths.py; OUTFILE derives from OUTPUT_DIR. Behavior identical.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-24 08:34:57 +08:00
Misaka_Company
17293bee79 fix: correct expected_undelivered.BASE anchor after package move
Tier 1 regression: expected_undelivered.py carries its own BASE=dirname(__file__) anchor (a duplicate of paths.py). The package move dropped the file one level deeper, so BASE resolved to inbound_verify/ and DOWNLOADS/OUTPUT pointed at non-existent inbound_verify/downloads|output — while site downloads write to the project-root dirs via paths.py. process()/write_site_file() thus skipped the compare with '[跳过] downloads 下缺少 ...', returned None/False, and every undelivered task for the 4 web/app sites reported failed (重试耗尽) despite the files downloading fine. Fix: anchor BASE two levels up (mirrors paths.py). Verified: write_site_file returns True; all 4 undelivered tasks now success with 未到 files generated.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 13:55:48 +08:00
Misaka_Company
5ac618c041 docs: update stale module-name references in comments and docstrings
Replace pre-rename references (main_router, db_store, server.py) in code comments and docstrings with their new locations (cli/router, store, cli/server, runtime). Comment-only; no behavior change.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:26:34 +08:00
Misaka_Company
a07b9b435b docs: update README/CLAUDE.md for package layout; add Tier 1 spec and plan
Rewrite run commands to python -m inbound_verify.* (and console_script aliases); add pip install -e . to env prep; refresh the directory tree. Also commit the design spec and Tier 1 implementation plan under docs/superpowers/.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:21:45 +08:00
Misaka_Company
7065b88269 refactor: move flat modules into inbound_verify package (Tier 1, behavior-identical)
Relocate 12 root .py modules into inbound_verify/ (sites/, cli/ subpackages). Rewrite all internal imports to package-qualified; drop the site_ prefix on the 5 site modules and their 28 call sites. Fix paths.py BASE_DIR to anchor at the project root. Add main() entry wrappers (cli/router, cli/server, store). No behavior change.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:21:31 +08:00
Misaka_Company
0174e64a04 chore: scaffold inbound_verify package and pyproject
Empty package markers (inbound_verify/, sites/, cli/) plus pyproject.toml with console_scripts entry points and dependency floors.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-23 12:20:54 +08:00
Misaka
b9f726c30a 引入 PostgreSQL 持久化:到货核销数据入库管道
- 新增 db_store.py:解析 downloads/ 下各站应到/实到/未到 Excel,幂等 UPSERT 入库(统一核心列 + raw JSONB 兜底,单号按文本存)
- 新增 schema.sql:三表(expected/actual/undelivered_record),隔离到专用 schema inbound_verify(CREATE SCHEMA + search_path)
- 韵达实到业务清洗:抛弃「交接单号」为空行 + 按子单号去重
- 同步 config.example.yaml(postgres 配置)与 requirements.txt(psycopg[binary])

本次仅手动入库管道;自动挂载到下载流程留待后续。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-22 23:04:21 +08:00
Misaka
bbe9e3e619 百世「应到/实到」抓取并写入汇总报表
- site_baishi.baishi_download_undelivered_data_impl:进「扫描综合查询→实时扫描率」
  后、下钻未扫前,顺手抓「到/接件扫描→当日」的 应扫(td.nth(10))/已扫(td.nth(11)),
  落库 site_settings(scan_expected_pieces / scan_arrived_pieces);try/except 包裹,
  不影响下载主流程。
- expected_undelivered.process_baishi:应到件/已到件改由 state_store.get_setting 读回
  (空串→None),不再写死 None。
- expected_undelivered.build_summary:百世行有基数时填真实 应到件/已到件/未到率(带热力色),
  无基数仍显示「—」;口径说明文案同步更新。默认不计入合计行/KPI/柱状图(保持 4 站口径)。
2026-07-19 21:18:38 +08:00
Misaka
f1cb0a4d7b 修复百世弹窗关闭严重滞后的问题
- 旧 dismiss_baishi_popups 串行处理四类弹窗,每段 _poll_and_click 在未命中时
  会阻塞等待满 5s 才返回,导致排在后面的「通知」与「配置检查」被卡在前一段
  的等待窗口里,表现为关闭特别慢。
- 改为单条轮询循环:每个 tick(~350ms) 同时检查并关闭所有当前可见的弹窗,
  设 12s 全局上限 + 连续 1s 无命中提前退出,四类互不排队,谁先出现谁即被关。
2026-07-19 20:47:34 +08:00
Misaka
076557aacb 修复百世优惠券广告弹窗从未关闭的问题
- 新增 site_baishi.dismiss_baishi_popups():优惠券广告为全屏居中 Ant Design
  modal,关闭键是 .ant-modal-close(纯图标、无文字)。旧逻辑只识别 '关 闭'/
  '阅读完毕' 等文字按钮,广告关闭键无文字故从未被关闭;现直接点击
  .ant-modal-close 并做重试 + 消失校验,再处理阅读消息与配置检查弹窗。
- runtime._dismiss_initial_popups 百世分支改为调用该 helper。
- 调试模式临时开启 CDP 9223 端口,便于 Playwright CLI 技能挂载已登录页面
  读取内容(仅调试模式生效,服务模式不受影响)。
2026-07-19 19:12:53 +08:00
Misaka
8f79cbc5d7 修复安能实到日期设定 + 站点统计口径重构
- site_anneng.py: set_scan_date 改用 execCommand 全选替换+回车(修复 Ant DatePicker 受控组件未写入导致实到下成当天); wait_scan_form_ready 增加 settle 与 DatePicker 预热,消除首设竞态
- expected_undelivered.py: 应到口径改交接件数、实到改单号去重、未到明细列已到单号(不再编子单号),百世排除
- docs: 补充站点统计逻辑审查报告与韵达/安能计算逻辑梳理
2026-07-19 13:16:01 +08:00
Misaka
f3ff188302 修复任务卡死与安能启动失败
- runtime.py: launch_anneng 拉起前清除 NODE_OPTIONS,避免 Electron 因 --use-system-ca 启动即退出(rc=9)
- server.py: POST /tasks 在 worker 未就绪时返回 409 拦截误操作;worker 就绪后调用 fail_stale_tasks 实现重启自愈
- state_store.py: 新增 fail_stale_tasks(),将遗留 pending/running 任务标记为 failed
2026-07-18 22:22:35 +08:00
Misaka
0be3399165 业务日期端到端:状态库快照 + 下载写入 + 报告读库
- state_store: site_status 加 expected/actual/undelivered_business_date 列(建表+旧库迁移);set_data_state 加可选 business_date(None 时保留,心跳不覆盖下载快照)
- runtime: dispatch_task 下载成功后快照业务日期(today−offset);4 站 undelivered 一并补写 expected/actual/undelivered 三列(_site_undelivered_handler 内部连下绕过 dispatch,否则应到/实到业务日期丢失)
- expected_undelivered: 报告数据日期列改读状态库(_read_business_dates),退役 mtime 反推;KPI 卡 2-3-2-2 等宽对齐 + 柱状图图例 overlay 分行

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-18 14:09:08 +08:00
Misaka
d44521d953 韵达提示弹窗清理改为结构匹配(不依赖文案)
dismiss_audio_prompt 去掉 has_text="音频设备未授权",改为只要出现 .ivu-modal-confirm 就点「确定」。
不同主机/音频状态下文案不同(未授权/未找到…),结构匹配通吃,避免文案不匹配→弹窗未关→遮罩阻塞下载流程。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-18 11:59:27 +08:00
Misaka
ed6d403451 韵达音频授权提示弹窗清理(含重试重载后)+ 服务模式读取 debug 配置
- site_yunda:抽 dismiss_audio_prompt 助手(点「确定」);yunda_reset 重载主页后调用(解决重试重载后弹窗复现)
- runtime:_dismiss_initial_popups 韵达分支改用 dismiss_audio_prompt;launch_and_prepare 自己读 config.yaml 的 debug 段(服务模式也生效,之前仅 main_router 传参)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-18 10:54:10 +08:00
Misaka
6d45a9f5e5 站点配置(百世密码/韵达账密/安能路径)从 config.yaml 移至 state.db
- state_store:加 site_settings(site,key,value) 表 + get_setting/set_setting/get_site_settings
- runtime:seed_legacy_config 启动一次性从 config.yaml 灌入(已存在不覆盖);安能 app_path 改读 state_store
- site_baishi:导出密码改读 get_setting("百世","password")
- site_yunda:登录账密改读 get_setting("韵达",...)
- server:/config PUT 接受 settings;新增 GET /config/{site}/settings(不进 5s 轮询)

debug 仍保留在 config.yaml;站点配置现由前端配置弹窗管理(state.db 为准)。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-18 08:37:15 +08:00
Misaka
96e7018146 定时下载(APScheduler) + 跑比对纯离线 + 偏移拆 expected/actual + 报告下载
定时下载:
- state_store:site_config 加 schedule_enabled/schedule_time(旧库迁移)+ set_schedule/get_all_config
- server:APScheduler BackgroundScheduler,启动按配置注册各站 cron job、改配置重排;job 投 undelivered 任务到队列(错过跳过);/config 扩展 schedule
- runtime:("__compare__","compare") 改纯离线 expected_undelivered.main()(不下载)
- requirements:加 apscheduler

偏移拆分 + 报告下载:
- state_store:date_offset 拆 expected_offset/actual_offset(迁移)+ get_offset(site,kind)/set_offset(site,kind,n)
- 4 站 actual impl 改读 actual 偏移(expected 走默认 kind)
- server:/config 用 expected_offset/actual_offset;新增 GET /report(服务 output/应到未到数据.xlsx)
- paths:加 OUTPUT_DIR

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-18 08:11:18 +08:00
Misaka
82c80fc859 未到数据按站独立 + 百世并入汇总 + 全量跑比对(失败容错)
- state_store:site_status 加 undelivered_ready 字段(旧库 ALTER 迁移);init_db 提早到启动最前(server lifespan + launch_and_prepare 第0步),避免 /api/status 早于迁移报错
- expected_undelivered:重构为 process(name)/process_baishi/write_site_file/build_full_report;build_summary 支持百世(仅未到件、无基数,不计入合计/图表)与失败容错(未成功站保留行无数据)
- runtime:4 站 ("站","undelivered") = 下应到+实到 → 比对写 <站>-未到数据.xlsx;("__compare__","compare") 改 run_all(顺序跑5站、记成功清单 → build_full_report,未登录/失败跳过);DATA_FILENAMES 加 undelivered、心跳探测之;_site_undelivered_handler 用 is not False 与 dispatch 一致
- site_shunxin:shunxin_expected/actual_download 改为 return with_retry 结果(修复返回 None 致调用方误判失败、以及 with_retry 失败被当成功的潜在 bug)
- server:lifespan 启动时 init_db

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 23:19:21 +08:00
Misaka
26a94999dc 按站点指定下载日期偏移(0=今天…30)+ 修复启动陈旧登录态
偏移量功能(前端/API 可配,服务端持久化、多用户共享):
- state_store:新增 site_config 表 + get_offset / set_offset(0..30 钳制) / get_all_offsets
- server:新增 GET/PUT /config(百世锁定当天,不可配置)
- 顺心/中通/韵达/安能(各 expected+actual):日期逻辑由 query_days 范围改为读 offset
  算单日 target=今天-offset,起止同日;百世仍走当日
- config.example.yaml:query_days 标记为已废弃

修复启动时显示上一会话陈旧登录态:
- state_store:新增 reset_login_states(登录态会话级,启动重置为 unknown;数据态会话无关、保留)
- runtime.launch_and_prepare:启动时对各站重置登录态,心跳就绪后重新探测写真实值

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 21:27:52 +08:00
Misaka
66bd8af421 阶段1:服务化骨架(FastAPI + Playwright worker + 任务队列)
- 新增 runtime.py:抽离共享核心 launch_and_prepare / dispatch_task / run_heartbeat /
  probe_* / launch_anneng / RuntimeContext,常量 SITES_CONFIG 等;main_router 复用
- 新增 server.py:FastAPI(主线程)+ Playwright worker(独立线程)+ 任务队列;
  API:POST/GET /tasks、GET /status、GET /data/{file}(防路径穿越)
- state_store:加 task_history 表 + create/update/get/list 接口
- 5 站点 with_retry 改为返回 True/False,供 dispatch_task 判成败
- main_router:重写为复用 runtime 的交互模式(行为不变)
- requirements:加 fastapi、uvicorn
- CLAUDE.md:补充运行模式与共享核心架构说明

线程模型:主线程 FastAPI 不碰 Playwright,worker 线程独占 page,经 Queue + SQLite 通信。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-17 00:04:39 +08:00
Misaka
a5ceb42ac1 阶段0:登录态心跳 + 状态持久化(重构地基)
- 新增 state_store.py:SQLite 状态存储(site_status 表:登录态/数据态/时间戳),
  UPSERT 保留字段,重启不丢
- paths.py:加 STATE_DB_PATH(state/state.db)
- main_router:提取 probe_site_login / probe_data_file;菜单循环改为
  input 后台线程 + 主线程心跳轮询(满足 Playwright sync 线程安全);
  每 30s 探测各站登录态 + 数据文件写库,登录态变化时提示;新增菜单 [12] 站点状态盘
- .gitignore:忽略 state/

为后续 FastAPI 化与 Web 前端提供状态地基。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-16 22:44:15 +08:00
Misaka
5030e5b89b 统一各站成败反馈信号;修复中通报错卡死与提示框误判
成败信号统一化(顺心/中通/韵达/安能/百世):
- 各站 impl 开头清理上次的最终文件,避免无数据/失败时残留旧数据误导比对
- 轮询下载段加汇总校验:下载数 < 预期则 raise,堵住"零/部分下载被判成功"
- 正常完成统一显式 return True(原中通/韵达/安能靠 None 隐式成功)
- 无数据统一不落文件(安能实到不再落空 xlsx);韵达裸 except 改 except Exception
- 中通"票数0无空提示"分支改判失败

中通 bug 修复(site_zto.py):
- poll 加 stall 快速失败(连续4轮无进展即 raise)+ 刷新短超时 + 双失败 raise,
  避免页面被遮挡时干等 5 分钟才触发重试
- "生成离线导出任务成功"提示所在 iframe 提交后被销毁,wait_for 抛
  "Frame was detached" 属正常(提示已随 iframe 消失=任务已建立),改判成功

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-16 21:38:56 +08:00
Misaka
55e9e9e931 顺心站点支持双账号(双归属地)下载与融合
- main_router: 顺心在同一窗口开两个标签页登录两个归属地账号;就绪轮询、
  初始弹窗、菜单 [1][2]、自动化测试均改为按 page 列表处理
- site_shunxin: download 入口改为接收 page 列表;新增 shunxin_belonging
  读归属地、shunxin_merge_final 融合两账号数据;impl 加 out_tag 参数化
  各账号产物文件名;两账号同归属地时去重校验中止以防数据翻倍
- expected_undelivered: 零改动(融合后产物仍为同名文件)
- 更新 CLAUDE.md / README.md 文档说明

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-15 22:36:58 +08:00
Misaka
75abbdf14c Add websocket-client to requirements; sync docs
site_anneng 的 CDP 驱动用到 websocket-client,此前 requirements.txt 缺漏,
需额外手动安装。补进 requirements.txt,并把 README/CLAUDE 里"暂未列入"的
临时提示改为"已包含"。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-14 22:48:07 +08:00
Misaka
c6a23febc0 Add README and CLAUDE.md project docs
- README.md:面向使用者的项目说明(站点/流程、环境、配置、运行、架构、输出)。
- CLAUDE.md:面向后续开发的关键命令与非显而易见的跨文件架构与约定。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-14 22:38:21 +08:00
Misaka
a71924ca1a Cap export-task poll loops at 5min to avoid infinite hang
顺心/中通/韵达/安能 的导出任务轮询循环原本只在"本批任务全部完成"时
退出,无任何时间上限:任务卡在生成中、或因时钟偏移没落进 40s 容差窗口
匹配不上时,会无限轮询、流程永不返回,连 with_retry 都没机会触发。

给这 5 处轮询加 300s deadline,超时即 raise → 流程返回 False → 触发
已有的重置重试机制。把"永久挂死"变成"有界失败→自动重试"。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-14 22:24:41 +08:00
Misaka
2d49c3d6f1 Add per-site exception fallback (reset + retry), router stays pure dispatch
每个站点模块自带"失败→重置回初始态→重试"兜底(最多 3 次,含首次;每次
失败都重置,含最终放弃那次清场),对外只暴露 xxx_download 公开入口,路由
层无感、只负责调度。

- 各站点模块新增:HOME_URL、xxx_reset、内联 with_retry;原流程拆为
  xxx_download(带重试) + xxx_download_impl(单次,供自动化测试探测原始失败)。
- 网页站重置=goto 首页 URL;安能(Electron)重置=关业务 tab+收菜单(实测
  reload 会留下杀不掉的僵尸 webContents,故不用 reload)。
- main_router:SITES_CONFIG 改引用各模块 HOME_URL(单一来源);菜单分派回归
  朴素调用;run_automation_test 改走 _impl 以保留"暴露问题"的测试本意。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-08 22:39:38 +08:00
Misaka
3ae94bef0d Fix ZTO actual-download date-cell click blocked by hover tooltip
中通实到流程选日期时,Playwright 的 .click() 会先 hover 日期格子,
触发“范围长度”提示气泡(.date-range-length-tip)盖住格子,导致点击被
判遮挡而超时(query_days=1 时尤甚,起始日==今日,点的是同一格子两次)。

改为直接在日期格子元素上派发 mousedown+mouseup+click 事件(新增
_dom_click helper),不经过坐标命中测试,气泡根本不会出现、也无法拦截。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 23:26:37 +08:00
Misaka
63ee8bbae1 Unify export-task matching to time tolerance only (≤40s)
去掉顺心/中通/韵达/安能(应到) 导出任务认领时的标题/模块名校验,
统一改为仅按提交时间容差 ≤40 秒匹配本批任务。

理由:单账号无并发,时间窗内的任务必为本批所建;而站点可能调整
任务标题名,写死标题会导致匹配失败、任务已创建却一直查不到。
同步清理因此变成死代码的局部变量与函数形参。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 22:04:22 +08:00
Misaka
d05848ee36 Ignore .playwright-cli/ artifacts directory
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 21:26:12 +08:00
Misaka
d55c460c2d Integrate expected-vs-actual (应到未到) comparison into main_router
- 应到未到比对.py -> expected_undelivered.py(英文名);main() 的 sys.exit 改为
  return,使其可被 main_router 安全调用而不杀进程。
- main_router.py:菜单 [9] 改为调用 expected_undelivered.main()——全站点自动比对,
  输出 output/应到未到数据.xlsx(汇总报表 + 中通/顺心/韵达/安能 各站明细);移除被
  取代的旧 task_process_undelivered_data 及其专属 import pandas。
- .gitignore:忽略 output/(比对输出目录)。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 23:39:36 +08:00
Misaka
dea9ad6e07 Add Anneng actual-data (scan) download and wire into automation test
安能站点第二阶段:实到数据下载 + 接入自动化测试。

- site_anneng.py:新增 anneng_actual_download()。网点到件扫描查询页是 Ant Design
  且无导出功能——综合查询→扫描查询→网点到件扫描查询(新)→ 设扫描时间(直接输入
  +回车)→ 选「子单」→ 每页 500(size-changer 用 mousedown 展开)→ 查询 → 直接
  从表格 DOM 抽数、逐页翻页取完 → 存 安能-实到货物数据.xlsx → 关 tab。复用应到的
  CDP/菜单/tab 辅助;扫描专有逻辑单独成段。main() 支持 `actual` 参数独立跑实到。
  修掉复用 tab 时首查读到旧总数的问题(查询后等总数变化再读)。
- main_router.py:菜单加 [11] 实到货物数据下载;run_automation_test 的 flow_table
  加入安能(expected/actual),执行循环对应用类站点(APP_SITES)走无 page 分支。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 22:12:23 +08:00
Misaka
9d3d3cbb78 Add Anneng (Electron) site: expected-data download + main_router integration
安能站点第一阶段:把应到数据(运单信息)下载流程接入主路由。

- site_anneng.py:用页面级 CDP 驱动「安能全网门户」Electron 应用(Playwright
  connect_over_cdp 被 Electron 浏览器域 CDP 拦截,故走页面级)。完整应到流程:
  菜单导航 → 进站交接单查询 → 设日期查询(“带小数点的 0”为加载完成判据)
  → 逐条交接单双击进运单信息并核对单号 → 导出(选中字段+→箭头转移全部列)
  → 关闭查询 tab → 导出下载页轮询 → 免对话框下载(x-auth+TGC 直连 GET)
  → 合并为 安能-应到货物数据.xlsx。CDP 端口可配;anneng_ready() 供路由就绪轮询。
- main_router.py:以调试模式启动安能(动态空闲端口,避免端口冲突),经 CDP
  判断首页就绪,菜单加 [10] 应到下载,退出时关闭应用;网页站点逻辑不变。
- config.example.yaml:补充 anneng.query_days 与 anneng.app_path 说明。
- docs/安能门户CDP连接指南.md:连接方案、独立 webContents/tab 处理、免对话框下载。
- .gitignore:忽略 Archive/(早期探查脚本归档)。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 19:38:44 +08:00
Misaka
d4a9fb3b4a Harden Yunda export dialog with two-layer reset and plain wording
- Add a two-layer reset engine to the export dialog (both expected and
  actual arrival): an outer 3x reopen loop with a field-list presence
  check (交接单号 / 扫描类型 as per-page anchor), and an inner 4x
  submit loop that distinguishes a missed field (re-click 全选) from a
  vanished field list (reopen the whole dialog), recovering from
  empty/ghost-cleared export dialogs
- Keep log/comment wording plain and factual (no hyperbole)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 12:19:51 +08:00
Misaka
47d9140dc6 Add Baishi tab cleanup helper and close tab after extraction
- Add _close_tab(page, tab_name) for Baishi's tab close affordance
  (li>span text + i[title='关闭标签页']), matching the other sites
- Close the 扫描综合查询 tab both on the empty-data early return and
  after a successful download; return True on completion

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 10:55:56 +08:00
Misaka
5b7a024fa8 Parameterize Shunxin query window and document it in example config
- site_shunxin: replace the hardcoded 1天 radio with config-driven
  shunxin.query_days, set the date range via the ant-picker, and add a
  two-pass query/retry to clear stale "暂无数据" DOM state; return True
  on completion
- config.example.yaml: add the shunxin section (query_days) so the
  example matches what the code reads

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-21 10:43:55 +08:00
Misaka
f4f9f520e1 Document yunda config and remove dead main_task.py
- config.example.yaml: add the yunda section (username/password for
  auto-login, query_days) and list 韵达 as a valid debug target_site
- Delete main_task.py: early single-site (顺心-only) prototype whose
  every function is superseded by main_router.py + site_shunxin.py;
  nothing imports it, and it predates paths.py / multi-site / the
  automation test

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 23:35:42 +08:00
Misaka
229e474b58 Add cross-flow automation test and tone down log/comment wording
- main_router: add run_automation_test that runs each site's download
  flows in a cross sequence (expected/actual) plus a pass/fail report
  with timings; expose it as menu [8]
- Site download functions now return False on their exception paths so
  the test harness can record failures (baishi also flags config-read
  failure)
- Replace overblown log/comment phrasing across all modules with plain
  statements; trim docstrings

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 23:33:22 +08:00
Misaka
ab3f25a10e Centralize paths in paths.py and harden waybill-number handling
- Add paths.py: BASE_DIR/DOWNLOAD_DIR/CONFIG_PATH anchored on __file__
  so paths resolve regardless of the launch cwd
- Route main_router and all site modules through DOWNLOAD_DIR/CONFIG_PATH,
  replacing os.getcwd()-based download dirs and the "config.yaml" literal
- main_router compare engine: read Excel as str with keep_default_na,
  strip/normalize 运单号, drop blanks, and isin against a set so
  int/float-vs-str mismatches no longer produce false "undelivered"
- Shunxin: give downloaded files unique microsecond temp names and read
  Excel as str to preserve long-waybill precision
- Normalize bare except: to except Exception: across affected files

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 22:28:30 +08:00
Misaka
5d13c13f40 Close Shunxin tabs after each stage via shared _close_tab helper
- Add _close_tab(page, tab_name) for closing Ant Design Tabs by label
- Close 运单列表 / 车辆点到 in the expected-arrival flow after export
  submission, and 数据导出 after download/merge
- Close 卸车扫描记录 and 数据导出 likewise in the actual-arrival flow
- Renumber step comments

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 21:47:05 +08:00
Misaka
a67b64f8f8 Auto-login Yunda and switch router to ready-guard polling
- Add yunda_login: probe the login page, fill credentials from
  config.yaml (yunda.username/password), submit; no-op if already
  logged in
- main_router: register per-site READY_SELECTORS and replace the
  blocking input() with a ready-guard poll loop; invoke Yunda
  auto-login up front; simplify site-init cleanup
- site_yunda: drop the custom frame probes in favor of native
  frame_locator("section iframe")
- Skip handover rows already marked 已绑定; abort the export harvest
  when no tasks were submitted
- Fix NameError from undefined username/password fallback

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 21:33:09 +08:00
Misaka
e8b789e906 Add anti-collapse menu nav and fix multi-tab Strict Mode leakage
- Rewrite yunda_smart_menu_click to check each parent submenu's
  is-opened state before clicking (skip if already open), using XPath
  parent-child isolation to avoid collapsing accordion menus
- Scope the expected/actual-arrival loading mask, #sum panel, and
  empty-record hint to #tab-1 and add .first to fix Playwright Strict
  Mode (5 elements found) leaks across multiple open tabs
- Move _yunda_poll_and_download_tasks above yunda_actual_download for
  definition order

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 19:22:01 +08:00
Misaka
604d1822e4 Implement Yunda actual-arrival download and harden query loading
- Implement yunda_actual_download via the 报表管理 -> 扫描记录查询
  menu: set date range, switch scan type to 到件, then reuse the
  shared poll/download/merge engine
- Add a three-stage query buffer (yield, wait for the Bootstrap Table
  loading mask, settle) before the data/empty verdict to avoid stale
  DOM reads in the expected-arrival flow
- Trim the offline-export retry state machine's verbose logs
- Fix a typo (研编 -> 研判) and drop the 待开发 tag on menu [7]

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 18:59:49 +08:00
Misaka
194fbc7cda Integrate Yunda site module and rewire router menu numbering
- Add site_yunda.py: Yunda expected-arrival download (进站交接单查询)
  with Layui date-range setup, row double-click export, a retry state
  machine for the offline-export dialog, and a shared poll/download/
  merge engine; actual-arrival is a placeholder
- Wire Yunda into main_router: site URL, login detection, menu items,
  and read yunda.query_days from config.yaml
- Rework the hub menu: drop the standalone Shunxin local-compare entry,
  renumber modules (百世 3, 中通 4/5, 韵达 6/7), keep global compare [9]
- Streamline site-init cleanup and log wording

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 18:20:46 +08:00
Misaka
eb9bbe9b70 Harden ZTO export flows and close site tabs after each stage
- Poll #inEwbCount until it updates before reading the result, then
  branch on zero-count (confirm empty-data area vs missing flag)
- Wait for real grid rows to render in the expected-arrival flow
- Close the active site tab (进站交接单查询 / 到件扫描监控 / 导出任务管理)
  after each stage completes to reset the page state
- Tighten export-task time-match window from 60s to 20s
- Trim verbose log messages and stale comments

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 15:24:48 +08:00
Misaka
358b8327b9 Implement ZTO actual-arrival download and extract shared poll engine
- Implement zto_actual_download via the 到件扫描监控 menu with 子单
  number-type filter, reusing the common download flow
- Extract the export-task poll/verify/download/merge steps into
  _zto_poll_and_download_tasks so both ZTO flows share it
- Tighten task time-match window from 120s to 60s and add a
  refresh-and-recheck loop when expected tasks are missing or pending
- Drop unused _ensure_menu_expanded helper
- Update menu [6] label since actual-arrival is now implemented

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 13:48:28 +08:00
Misaka
9375b6efd6 Integrate ZTO site module and add debug/single-site mode
- Add site_zto.py: ZTO expected-arrival data download with menu
  navigation, export polling, and Excel merge
- Wire ZTO into main_router: site URL, login detection, menu items
  [5]/[6], and a global offline Left Anti-Join compare entry [9]
- Read config.yaml debug flags to launch only the target site for
  focused debugging; guard each site init and route handler behind
  site-loaded checks via is_site_ready()
- Expand config.example.yaml with documented debug, baishi, and zto
  config keys

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-20 12:46:46 +08:00
Misaka_Company
7f8d81af21 Add zero-count validation for Baishi undelivered data extraction 2026-06-17 15:27:51 +08:00
Misaka_Company
7cae24fae3 Add PyYAML dependency to fix missing yaml module 2026-06-17 15:11:03 +08:00
Misaka
b127ef6165 Fix Baishi scan-query selector ambiguity and add password guard
- Scope the 扫描综合查询 click to .nav-level2-wrapper to avoid collision with the right-side tab of the same name
- Warn when config.yaml has no baishi.password before attempting export

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-16 22:32:10 +08:00
Misaka
5c3225388e Add multi-round popup cleanup engine for Baishi site initialization
- Loop up to 5 rounds to dismiss 阅读完毕 / 配置检查 / 通知提示 / 广告 popups until the page is clean
- Revert undelivered-data docstring to dynamic-prefix wording

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-16 22:20:07 +08:00
Misaka
ec1a9d9fbc Implement Baishi undelivered-data extraction and restructure menu by site
- Replace Baishi placeholders with full download flow (scan-rate query, export modal, password auth via config.yaml)
- Reorganize router menu into 顺心 / 百世 modules; move undelivered-data compare into 顺心 flow
- Add config.example.yaml and ignore config.yaml to keep credentials out of VCS

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-16 21:58:02 +08:00
Misaka
497fd11674 Add per-site file prefix for undelivered-data comparison
- task_process_undelivered_data now accepts site_name (default 顺心)
  and reads/writes {site}-{应到/实到/应到未到}货物数据.xlsx
- Menu option 5 prompts for site name before running reconciliation
- Update 顺心 site outputs to 顺心-应到/实到货物数据.xlsx naming
- Remove redundant inline comments in site_shunxin.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-15 22:09:12 +08:00
Misaka
411a2487fc Refactor monolithic script into multi-site module architecture
- Introduce main_router.py as multi-site controller with interactive menu
- Extract 顺心 site logic into site_shunxin.py (expected & actual download)
- Add site_baishi.py stub for upcoming 百世 integration
- Add global 应到未到 data reconciliation via Left Anti-Join on 运单号

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-15 22:02:51 +08:00
39 changed files with 13635 additions and 482 deletions

9
.gitignore vendored
View File

@@ -40,6 +40,9 @@ desktop.ini
# Project specific
downloads/
output/
state/
logs/
*.xlsx
*.xls
*.log
@@ -48,8 +51,12 @@ downloads/
.env
.env.local
# Local secrets / credentials
config.yaml
# Playwright
.playwright/
.playwright-cli/
ms-playwright/
# Claude
@@ -58,3 +65,5 @@ ms-playwright/
# Memory
memory/
MEMORY.md
Archive/

144
CLAUDE.md Normal file
View File

@@ -0,0 +1,144 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## 项目概览
自动登录 5 家物流承运商工作台,下载"应到 / 实到"货物数据,离线比对出**应到未到**
异常运单并汇总成 Excel。4 个网页站点 + 1 个 Electron 应用(安能)。
**两种运行模式**阶段1 起):
- **交互模式** `inbound_verify.cli.router`:人工调试 / 操作,交互菜单(登录、触发下载、[12] 状态盘)。
- **服务模式** `inbound_verify.cli.server`:常驻 + FastAPI客户端经 HTTP 触发任务、查状态、下载数据API 文档 `/docs`)。
- 两者共享 `inbound_verify.runtime`(启动 / 就绪 / 任务派发 / 心跳)与 `inbound_verify.state_store`SQLite 状态持久化)。
## 常用命令
所有 Python 一律在项目虚拟环境 `.venv` 中运行Windows 下可直接用
`.venv/Scripts/python.exe`,无需激活)。首次 / 拉取新代码后需
`.venv/Scripts/python.exe -m pip install -e .`(以可编辑模式注册
`inbound-verify` 等命令)。
```bash
# 安装依赖(含安能 CDP 驱动所需的 websocket-client+ 以可编辑模式注册命令
pip install -r requirements.txt
pip install -e .
playwright install chromium
# 运行主程序(交互式菜单,详见 inbound_verify.cli.router 的 run_multi_site_daemon
.venv/Scripts/python.exe -m inbound_verify.cli.router
# 装包后也可直接用命令inbound-verify
# 服务模式(常驻 + FastAPI客户端经 HTTP 触发;默认 :8000API 文档见 /docs
.venv/Scripts/python.exe -m inbound_verify.cli.server
# 或inbound-verify-server
# DB CLI建库 / 初始化 / 灌数据 / 全流程 / 单站单类
.venv/Scripts/python.exe -m inbound_verify.store createdb # 或 init | ingest | ingest-one <site> <kind> | all
# 或inbound-verify-db createdb|init|ingest|ingest-one|all
# 单站点联调:在 config.yaml 设 debug.enabled=true + debug.target_site=顺心|百世|中通|韵达|安能
# 网页站:只挂载该站;安能:只启动 Electron 应用。
# 安能独立运行(需先以 --remote-debugging-port=9222 启动「安能全网门户.exe」并手动登录
.venv/Scripts/python.exe -m inbound_verify.sites.anneng expected # 或 actual
# 格式化(全局规范:改完 Python 必须 Black
.venv/Scripts/python.exe -m black inbound_verify
# 语法自检
.venv/Scripts/python.exe -m py_compile inbound_verify
```
**没有 pytest 测试套件。** "测试"指 `inbound_verify.cli.router` 菜单 **[8] 自动化测试**
`run_automation_test`,按 `CROSS_TEST_SEQUENCE` 交叉跑通各站点流程)。
## 架构big picture
### 运行模式与共享核心阶段0/1 重构)
- **`inbound_verify.runtime`**:两种模式共享的核心——`launch_and_prepare`(启动 Playwright + 各站就绪 + 弹窗 + 心跳初值,阻塞至就绪)、`dispatch_task(ctx, {site,kind})`(派发任务,掉登录直接判 failed`run_heartbeat``probe_site_login/probe_data_file``RuntimeContext.stop()`。常量 `SITES_CONFIG/READY_SELECTORS/APP_SITES/HEARTBEAT_INTERVAL/DATA_FILENAMES` 在此。
- **`inbound_verify.state_store`**SQLite 状态持久化(`state/state.db`)。`site_status`(登录态 + 数据态 + 时间戳,心跳刷新)、`task_history`(任务记录)。重启不丢。
- **`inbound_verify.cli.router`**交互模式菜单循环input 后台线程 + `_await_command` + `dispatch_task` + 心跳)。
- **`inbound_verify.cli.server`**:服务模式。**FastAPI主线程+ Playwright worker独立线程**——主线程处理 HTTP绝不碰 Playwrightworker 独占 page 操作,经 `task_queue` + `state_store` 通信。API`POST/GET /tasks``GET /status``GET /data/{file}`
- **关键线程约束**Playwright sync 对象绑定创建它的线程;`launch_and_prepare`(含 `sync_playwright().start()`)必须在持有 Playwright 的线程调用(交互=主线程,服务=worker 线程。FastAPI 路由绝不访问 page。
- 站点模块 `inbound_verify.sites.*``with_retry` 返回 `True/False`(成功 / 放弃),供 `dispatch_task` 判成败。
### 两套驱动模态 —— 这是理解全局的关键
- **网页 4 站**(顺心/百世/中通/韵达):`inbound_verify.cli.router` 用 Playwright 开 chromium
每站一个 `page`,流程函数签名为 `xxx_download_impl(page)`。**例外:顺心是双账号**
——同一窗口开两个标签页(两个归属地账号),`pages_map["顺心"]` 存为 page **列表**
`shunxin_download(pages)` 接收列表(详见下文「顺心双账号」)。
- **安能**Electron 桌面应用,**不走 Playwright**。`inbound_verify.cli.router`
`--remote-debugging-port=<动态空闲端口>` 启动 exe`launch_anneng`
通过 `inbound_verify.sites.anneng.set_cdp_port` 告知模块;`inbound_verify.sites.anneng` 用裸 CDPwebsocket
驱动,业务 tab 是独立 webContents。这也是 `playwright-cli` 接管不了安能的原因
Electron 19 / Chrome 102 不支持 Playwright 要的 setDownloadBehavior
### 分层:路由纯调度,站点模块自洽
- `inbound_verify.cli.router` **只调度**:启动浏览器/安能、就绪轮询、登录检测、菜单分发。
菜单项直接调 `sites.xxx_download(page)`(顺心传 page 列表),**不关心**重试/重置。
- 每个 `inbound_verify.sites.*` 模块对外只暴露"把任务做了"的入口,内部自洽:
- `xxx_download(...)` —— 公开入口,= `with_retry(站点, 标签, xxx_download_impl, xxx_reset)`
- `xxx_download_impl(...)` —— 单次执行、**无重试**(自动化测试刻意调它以探测原始失败)
- `xxx_reset(...)` —— 重置回初始态(网页 = `page.goto(HOME_URL)`;安能 = 关业务 tab + 收菜单)
- `with_retry(...)` —— 重试逻辑**内联在每个站点模块**(不抽公共组件,现阶段刻意不优化结构);
失败→重置→重试,最多 3 次(含首次),每次失败都重置(含最终放弃那次清场)
- `HOME_URL` —— 站点首页 URL`runtime.SITES_CONFIG` 引用它(单一来源)
### 导出任务队列模式(顺心/中通/韵达/安能-应到 共用)
这些站的下载是异步的:提交导出(记 `export_times` 时间戳)→ 跳"导出任务管理"页轮询 →
匹配本批任务 → 下载 → 合并多个临时 xlsx。
- **匹配只按时间容差 ≤40s****不校验任务标题/模块名**(标题可能被站点改动;单账号无并发,
时间窗内的任务必为本批所建)。**不要把标题校验加回来。**
- 轮询循环都有 **300s deadline**:超时 `raise` → 流程返回 False → 触发 `with_retry` 重试。
- 例外:**百世**是同步下载(`expect_download`,无队列);**安能-实到**页面无导出按钮,
直接抓表格 DOM 翻页。
### 顺心双账号(双归属地)
顺心业务上要同时处理**两个归属地网点**(两个账号)。程序在同一窗口开两个标签页,
人工分别登录两个账号(顺心站点支持同浏览器双账号并存,无需独立 context/窗口)。
- `runtime` 启动时为顺心开 2 个 `context.new_page()``pages_map["顺心"]` 为列表;
就绪轮询要求**两个标签页都进主页**才算就绪;初始弹窗对两个标签页各处理一遍。
- `shunxin_expected_download(pages)` / `shunxin_actual_download(pages)` 接收 page 列表:
先用 `shunxin_belonging(page)` 读各账号归属地(首页「切换网点」控件 `.site___3o7nH`
**去重校验**(两账号同归属地则报错中止,防数据翻倍),再顺序对各账号跑一遍
`xxx_download_impl(page, out_tag=归属地)`(产物 `顺心-{归属}-{应到/实到}货物数据.xlsx`
最后 `shunxin_merge_final` 把两份 `pd.concat` 成统一的 `顺心-{应到/实到}货物数据.xlsx`
并删中间文件。比对层 `compare` **零改动**(仍读同名文件)。
- 导出队列不串扰:双账号**顺序执行**,账号 A 走完完整下载流程(远超 40s后 B 才提交,
配合每账号独立 `export_times` + ≤40s 容差B 不会误匹配 A 的任务。
### 比对
`inbound_verify.compare`(菜单 [9])纯离线:读 `downloads/` 下各站应到/实到 xlsx
比对生成 `output/应到未到数据.xlsx`(汇总 + 各站明细)。
### 自动入库(下载成功后 → PostgreSQL
`dispatch_task` 下载成功后,在 `_record_business_date` 旁挂一个**尽力而为**钩子
`_persist_to_db(site, kind)``runtime`):懒导入 `store`,调 `store.ingest_task(site, kind)`
按 kind 幂等 UPSERT 进 PostgreSQLexpected/actual 各入其列undelivered——百世入未到、
4 站连入 expected+actual。**绝不影响下载任务的成功判定**:所有写库/写状态都包 try/except
失败只告警。
- 开关 `postgres.auto_ingest`(默认开)+ `connect_timeout_seconds`(兜 cpolar 抖动)。
- 可见性:结果写 `state_store.ingest_state`(每站每类 ok/count/ingested_at/error
`GET /status``ingest` 字段暴露。
- 手动:`store` CLI `ingest-one <site> <kind>` 用同套路由单测(脱离下载)。
- 注意:跑在 Playwright 线程、同步阻塞cpolar 慢/断靠超时 + try/except 降级,不重试不补入。
### 路径
`inbound_verify.paths``DOWNLOAD_DIR` / `CONFIG_PATH` / `BASE_DIR` 全部锚定到项目目录,
**不依赖运行时 cwd**——别用相对路径或 `os.getcwd()`
## 重要约定 / 易踩坑
- **登录是手动的**`inbound_verify.cli.router` 启动后会停在就绪轮询(`READY_SELECTORS` / `anneng_ready`
直到检测到所有站点进入工作台才进菜单。仅韵达支持凭 `config.yaml` 凭据自动登录。
**顺心需登录两个账号**:同一窗口的两个标签页分别登录两个不同归属地账号,两个标签页
都进主页后才算就绪(顺心站点支持同浏览器双账号并存,故用同 context 标签页而非独立窗口)。
- **安能单实例**:启动前必须先关闭已打开的安能窗口,否则调试端口起不来。
- **安能重置绝不能用 `Page.reload`**reload 会让已开业务 webContents 失去 app 引用、
变成 `Target.closeTarget` / `window.close` 都杀不掉的僵尸。重置只能走 tab 条 X 关 tab + 收菜单。
- **`config.yaml` 已 gitignore**,存放凭据 / `query_days` / `debug` / `anneng.app_path`
切勿提交,也别把真实凭据写进 `config.example.yaml`
- **不要自动提交 / 推送**:本仓库约定改动后等用户明确说"提交"再 commit/push
(覆盖全局 CLAUDE.md 的 auto-push 默认)。
- 改完 Python 文件**必须跑 Black**(全局规范)。

190
README.md Normal file
View File

@@ -0,0 +1,190 @@
# 物流到货数据自动下载与应到未到核对工具
自动登录 5 家物流承运商的工作台,下载"应到 / 实到"货物数据,并离线比对出
**应到未到**(该到没到)的异常运单,最终汇总成一份 Excel 报表。
适用场景:网点每日核对进站货物是否到齐。
---
## 一、覆盖站点与流程
5 个站点中4 个是网页、1 个是 Electron 桌面应用:
| 站点 | 形态 | 应到 | 实到 |
|---|---|---|---|
| 顺心捷达 | 网页 | ✅ 运单信息 | ✅ 卸车扫描记录 |
| 百世快运 | 网页 | — | —(直接提取"应到未到/当日未扫",单流程) |
| 中通快运 | 网页 | ✅ 运单信息 | ✅ 到件扫描 |
| 韵达快运 | 网页 | ✅ 进站主单 | ✅ 扫描记录 |
| 安能全网门户 | Electron 应用 | ✅ 运单信息 | ✅ 网点到件扫描 |
- **应到** = 进站交接单下的运单明细(这批货"应该"到);
- **实到** = 到件 / 卸车扫描记录(实际扫到了哪些);
-**9 个下载流程**(顺心/中通/韵达/安能 各 2 个 + 百世 1 个)。
每个站点产出独立的 `{站点}-应到货物数据.xlsx` / `{站点}-实到货物数据.xlsx`
(百世为 `百世-应到未到货物数据.xlsx`),落在 `downloads/`
---
## 二、目录结构
```
InboundVerify/
├── pyproject.toml # 打包 + 依赖 + console_scriptsinbound-verify 等)
├── inbound_verify/ # 源码包
│ ├── paths.py # 统一路径锚点(以项目目录为基准,不依赖 cwd
│ ├── runtime.py # 两种模式共享核心(启动 / 就绪 / 任务派发 / 心跳)
│ ├── state_store.py # SQLite 状态持久化state/state.db
│ ├── domain.py # 站点 / 文件名 / 列映射共享配置单一来源leaf
│ ├── compare.py # 全站点应到未到离线比对,输出 output/应到未到数据.xlsx
│ ├── store.py # DB CLI 入口createdb|init|ingest|ingest-one|all
│ ├── sites/ # 各站点模块(流程 + 重置 + 重试,自洽)
│ │ ├── shunxin.py # 顺心(含双账号)
│ │ ├── baishi.py # 百世
│ │ ├── zto.py # 中通
│ │ ├── yunda.py # 韵达(含自动登录)
│ │ └── anneng.py # 安能Electron + CDP 驱动)
│ └── cli/ # 命令行入口
│ ├── router.py # 交互菜单(调度层:启动 / 就绪轮询 / 登录检测 / 菜单分发)
│ └── server.py # FastAPI 服务模式(常驻 + HTTP 触发)
├── config.example.yaml # 配置模板
├── config.yaml # 真实配置(自行创建,已被 .gitignore 忽略)
├── schema.sql # 数据库表结构store.py createdb / init 使用)
├── requirements.txt # pyproject 依赖的静态镜像
├── downloads/ # 各站点下载的原始数据
├── output/ # 比对报表输出
├── state/ # 运行状态持久化state.db
└── docs/ # 说明文档
```
---
## 三、环境准备
需 Python 3.10+。
```bash
# 1. 创建并激活虚拟环境
python -m venv .venv
.venv\Scripts\activate # Windows
# source .venv/bin/activate # macOS / Linux
# 2. 安装依赖(含安能 CDP 驱动所需的 websocket-client
pip install -r requirements.txt
# 3. 以可编辑模式安装本包(注册 inbound-verify 等命令)
pip install -e .
# 4. 安装 Playwright 浏览器内核(网页站点用)
playwright install chromium
# 5. 由模板创建本地配置并填入真实凭据
cp config.example.yaml config.yaml
```
> 安能是 Electron 应用,还需在 `config.yaml` 里填 `anneng.app_path`
> 指向本机的「安能全网门户.exe」路径。
---
## 四、配置说明config.yaml
`config.yaml` 存放凭据与各站参数,**已被 .gitignore 忽略,不会提交**。
所有项都有默认值,留空不会报错(但凭据留空会导致对应站点登录/导出失败)。
| 配置项 | 说明 |
|---|---|
| `debug.enabled` / `debug.target_site` | 调试模式:仅挂载启动指定单个站点(顺心/百世/中通/韵达/安能) |
| `shunxin.query_days` | 顺心查询时间范围(向前回溯 N 天至今天) |
| `baishi.password` | 百世导出授权密码(必填,否则导出失败) |
| `zto.query_days` | 中通查询时间范围 |
| `yunda.username` / `yunda.password` | 韵达自动登录凭据(留空则需手动登录) |
| `yunda.query_days` | 韵达查询时间范围 |
| `anneng.query_days` | 安能查询时间范围 |
| `anneng.app_path` | 安能 Electron 可执行文件路径 |
---
## 五、运行
```bash
# 交互菜单(任选其一)
python -m inbound_verify.cli.router
# 或装包后直接用命令:
inbound-verify
```
程序会:
1. 用 Playwright 打开各网页站点(**顺心为双账号**:同一窗口开两个标签页,分别登录两个
不同归属地账号;韵达若配了凭据会尝试自动登录,其余手动登录);
2. 以调试模式启动安能 Electron 应用(动态空闲端口),**需在应用内手动登录**
3. **轮询各站点登录就绪状态**——全部登录完成后自动进入主菜单(顺心需两个标签页都进主页);
4. 弹出主菜单,按编号选择任务。
主菜单:
```
[1][2] 顺心 应到 / 实到
[3] 百世 应到未到(当日未扫)
[4][5] 中通 应到 / 实到
[6][7] 韵达 应到 / 实到
[10][11] 安能 应到 / 实到
[8] 全站点自动化测试(交叉跑通校验)
[9] 应到未到比对(全站点汇总 → output/应到未到数据.xlsx
[0] 退出
```
### DB CLI入库
```bash
# DB CLI建库 / 初始化 / 灌数据 / 全流程 / 单站单类
.venv/Scripts/python.exe -m inbound_verify.store createdb # 或 init | ingest | ingest-one <site> <kind> | all
# 注下载成功后会自动入库postgres.auto_ingest默认开ingest-one 用于手动重灌指定站/类。
```
---
## 六、架构
**分层原则:路由层只调度,站点模块自洽。**
- **`inbound_verify.cli.router`(调度层)**:负责启动浏览器 / 安能、就绪轮询、登录检测、
菜单分发。**不关心**"任务能否完成、失败怎么办"——只调
`inbound_verify.sites.xxx_download(page)` 然后等结果。
- **各 `inbound_verify.sites.*`(站点模块)**:每个模块对外只暴露一个"把任务做了"的入口
`xxx_download(...)`,内部自行处理一切:
- `HOME_URL`:站点首页 URL也供路由层 `SITES_CONFIG` 引用,单一来源);
- `xxx_reset(...)`:异常兜底的重置(网页 = 跳首页 URL安能 = 关业务 tab + 收菜单);
- `with_retry(...)`:内联在本模块的重试逻辑;
- `xxx_download(...)`**公开入口** = `with_retry(站点, 标签, xxx_download_impl, xxx_reset)`
- `xxx_download_impl(...)`:单次执行、无重试(供自动化测试探测原始失败)。
**异常兜底(失败 → 重置 → 重试)**:任一流程失败(返回 False 或抛异常)→ 调对应
站点的 `xxx_reset` 回到初始态 → 重试,**最多 3 次(含首次)**;每次失败都重置
(含最终放弃那次),确保环境不残留脏状态。
> **顺心是双账号特例**:同一窗口开两个标签页(两个归属地账号),`shunxin_download`
> 接收 page 列表,内部读各账号归属地 → 去重校验 → 顺序下载 → `pd.concat` 融合成
> 统一的 `顺心-{应到/实到}货物数据.xlsx`,比对层无感(仍读同名文件)。
> 安能是 Electron由 CDP远程调试端口驱动而非 Playwright page。其重置**不用
> `Page.reload`**——reload 会让已开的业务 webContents 失去引用、变成无法清理的僵尸,
> 故改用"走 tab 条 X 关 tab + 收起菜单"。
---
## 七、输出
- `downloads/{站点}-应到货物数据.xlsx``{站点}-实到货物数据.xlsx`:各站点原始数据;
- `output/应到未到数据.xlsx`:全站点比对报表(汇总 + 各站明细),由菜单 [9] 生成。
---
## 八、备注
- 首次运行需手动登录各站点(程序会停在就绪轮询,直到检测到所有站点进入工作台);
- 安能为单实例 Electron 应用:启动前请先关闭已打开的安能窗口;
- 所有下载/输出路径以项目目录为基准(见 `inbound_verify.paths`),与从哪个目录启动无关。

93
config.example.yaml Normal file
View File

@@ -0,0 +1,93 @@
# config.example.yaml
# ============================================================================
# 复制本文件为 config.yaml 并填入真实凭据后使用:
# cp config.example.yaml config.yaml
#
# 说明:
# - config.yaml 已被 .gitignore 忽略,不会提交到仓库,可安全存放凭据。
# - 所有配置项均设有默认值,未填写时会自动回退到默认行为,不会报错。
# ============================================================================
# ----------------------------------------------------------------------------
# 调试模式:用于单网点联调,仅挂载并启动指定网点,其余网点不加载。
# ----------------------------------------------------------------------------
debug:
# 是否启用调试模式true=仅启动 target_site 一个网点false=全量启动所有网点)。
enabled: false
# 调试模式下要单独启动的网点名称。
# 仅在 enabled: true 时生效。可选值:顺心 / 百世 / 中通 / 韵达 / 安能。
# 留空或不匹配时将回退为全量模式。
target_site: ""
# ----------------------------------------------------------------------------
# 顺心捷达 (https://sxne.sxjdfreight.com)
# ----------------------------------------------------------------------------
shunxin:
# 【已废弃】下载日期改由 Web 前端/API 按站点设偏移0=今天1=昨天…,存 state.db此项不再生效。
query_days: 1
# ----------------------------------------------------------------------------
# 百世快运 (https://v5.800best.com)
# ----------------------------------------------------------------------------
baishi:
# 应到未到数据导出时需要填写的“登录密码”。
# 留空会导致导出授权校验失败,无法下载数据。
password: "YOUR_PASSWORD_HERE"
# ----------------------------------------------------------------------------
# 中通快运 (https://ws.zto56.com/)
# ----------------------------------------------------------------------------
zto:
# 【已废弃】下载日期改由 Web 前端/API 按站点设偏移0=今天1=昨天…,存 state.db此项不再生效。
query_days: 1
# ----------------------------------------------------------------------------
# 韵达快运 (https://ky-sso.yunda56.com)
# ----------------------------------------------------------------------------
yunda:
# 自动登录的账号与密码(由 inbound_verify.sites.yunda.yunda_login 读取并填入登录表单)。
# 留空时自动登录会填入空串导致登录失败,届时可在浏览器中改为手动登录。
# 真实凭据仅写进被 .gitignore 忽略的 config.yaml切勿提交示例值。
username: "YOUR_USERNAME_HERE"
password: "YOUR_PASSWORD_HERE"
# 【已废弃】下载日期改由 Web 前端/API 按站点设偏移0=今天1=昨天…,存 state.db此项不再生效。
query_days: 1
# ----------------------------------------------------------------------------
# 安能全网门户Electron 桌面应用,非网页)
# ----------------------------------------------------------------------------
# 与其他站点不同:安能不由浏览器打开,而是以调试模式启动其
# Electron 可执行文件(自动选取一个空闲端口作为 --remote-debugging-port避免端口冲突
# 启动后请在应用内手动登录runtime 会自动轮询判断是否进入主页。
anneng:
# 【已废弃】下载日期改由 Web 前端/API 按站点设偏移0=今天1=昨天…,存 state.db此项不再生效。
query_days: 1
# 安能 Electron 应用的可执行文件路径。
# 路径含反斜杠/空格/@用单引号包裹即可YAML 单引号串按字面解析)。
app_path: 'D:\SoftWare\SoftWare Installation\@ane-electron-uiapp\安能全网门户.exe'
# ----------------------------------------------------------------------------
# PostgreSQL 数据持久化(到货核销数据入库,详见 store.py
# ----------------------------------------------------------------------------
# createdb 会连接名为 postgres 的维护库来创建下方 dbname 指定的数据库。
# 命令行:
# python -m inbound_verify.store createdb 创建数据库(幂等)
# python -m inbound_verify.store init 建表(幂等)
# python -m inbound_verify.store ingest [site] 入库全站或单站(幂等 UPSERT
# python -m inbound_verify.store all createdb → init → 全站 ingest
postgres:
host: 127.0.0.1
port: 5432
user: postgres
# 连接密码config.yaml 已 gitignore真实凭据只写在那里切勿提交示例值。
password: "YOUR_PASSWORD_HERE"
dbname: CQHXDB
# 承载到货核销表的专用 schema隔离 public表建在此 schema 下。
schema: inbound_verify
# 下载成功后自动入库(钩子,见 runtime._persist_to_dbfalse=跳过(无 PG/cpolar 的开发机)。
auto_ingest: true
# PG 连接超时cpolar 抖动时快速失败,不拖垮下载 worker。
connect_timeout_seconds: 5

View File

@@ -0,0 +1,102 @@
# 应到数据「提交导出任务前」去重 — 实现总结
> 日期2026-07-29
> 范围:顺心 / 中通 / 韵达 / 安能 4 站**应到expected**数据;百世与实到不在本次范围。
## 一、运行机制
在周期 / 手动触发下载时,于**提交导出任务之前**按交接单号判断该批应到数据是否已落库,已落库则跳过,从源头消除重复下载与重复落库。
```mermaid
flowchart TD
TRIG[周期调度 / 手动触发<br/>task_spec: site, kind, force] --> DISP[dispatch_task → 站点 download_impl]
DISP --> LOAD{force 强制重下?}
LOAD -- 是 --> EMPTY[existing = 空集]
LOAD -- 否 --> QRY[查 PG expected_record.handover_no]
QRY -- cpolar 失败 --> EMPTY
QRY -- 成功 --> SET[existing = 已落库交接单号集合]
EMPTY --> LOOP[遍历本次查询到的班次/交接单号]
SET --> LOOP
LOOP --> JUDGE{交接单号 ∈ existing?}
JUDGE -- 是 → 已落库 --> SKIP[⏭️ 跳过:不提交导出<br/>不 append export_times]
JUDGE -- 否 → 新单 --> EXP[提交导出任务 → 轮询下载 → 入库 UPSERT]
SKIP --> DONE{全部处理完}
EXP --> DONE
DONE --> FINAL{本次提交了新任务?}
FINAL -- 无 → 全跳过 --> BAIL[空兜底 return不进下载轮询]
FINAL -- 有 --> POLL[轮询导出任务管理页 → 下载 → 入库]
```
**核心要点:**
- **去重数据源**PostgreSQL `expected_record.handover_no`(已落库的权威记录),新增 `store.get_existing_handover_nos(site)` 查询。
- **判断时机**:提交导出任务**之前**(循环内逐单判断),而非下载之后。
- **安全降级**PG 不可用 / `force=true``existing=空集` → 当作未落库 → 继续提交(**宁可重复、绝不漏**UPSERT 兜底)。
- **空兜底**:全部跳过时 `export_times` 为空 → 直接 `return`,不进下载轮询(避免下载数校验失败 / 空转超时)。
- **force 开关**:前端 checkbox默认关`POST /tasks.force` → 一路透传到 impl周期调度恒不 force。
## 二、force 强制重下透传链路
```mermaid
flowchart LR
UI[前端 checkbox<br/>forceRedownload] --> POST["POST /api/tasks<br/>{site,kind,force}"]
POST --> BFF[Next BFF 透传]
BFF --> TS["task_spec<br/>{site,kind,force}"]
TS --> DISP[dispatch_task]
DISP --> HDR["handler(ctx, force)"]
HDR --> DL["download(pg, force)"]
DL --> IMPL["impl(pg, force)"]
IMPL --> DEC{force?}
DEC -- 是 --> EMPTY2["existing = 空集<br/>强制重下,跳过去重"]
DEC -- 否 --> LOAD2[查 PG 加载 existing]
```
> 周期调度(`_enqueue_fetch`)投递任务时不带 `force` → 默认不强制。
## 三、4 站点标识获取
| 站点 | 提交前标识 | 来源 |
| --- | --- | --- |
| 中通 / 韵达 / 安能 | 交接单号(原有代码已读取) | DOM 列 / CDP 复选框 |
| 顺心 | 交接单号 `RTS\d{3}WJ\d+` | 点"运单列表"后从界面读取方式1 |
> 顺心"班次号"业务上等同交接单号4 站统一用交接单号(= DB `handover_no`)作去重键。
## 四、改动概览
**后端 InboundVerify**
| 文件 | 改动 |
| --- | --- |
| `schema.sql` | +`idx_expected_handover` 索引 |
| `inbound_verify/store.py` | +`get_existing_handover_nos(site)`(含 cpolar 降级) |
| `inbound_verify/cli/server.py` | `TaskRequest.force` + 透传到 task_spec |
| `inbound_verify/runtime.py` | `dispatch_task` + 所有 handler 透传 `force``download(impl)` |
| `inbound_verify/sites/{zto,yunda,anneng,shunxin}.py` | `impl``force` + 提交导出前注入去重 + 空兜底 |
| `inbound_verify/sites/baishi.py` | `force` 形参兼容 |
**前端 dashboard**
| 文件 | 改动 |
| --- | --- |
| `app/page.tsx` | `forceRedownload` state + checkbox + `trigger`/`triggerPrimary`/`onTrigger` 透传 force |
> BFF `app/api/tasks/route.ts` 是 generic 透传,无需改动。
## 五、验证结论
4 站去重 + force 开关均实测通过:
| 站点 | 二次触发行为 | 结果 |
| --- | --- | --- |
| 中通 | `...801` 已入库 → 跳过 → 空兜底 return | 20svs 首次 59s ✅ |
| 顺心 | 双账号 4 班次全跳过(`RTS023WJ375478` 等) | ✅ |
| 韵达 | `...82001` 已入库 → 跳过 → 原有空兜底 | ✅ |
| 安能 | `4008242619171180544` 已入库 → 跳过 | **5svs 4 分钟)** ✅ |
| force | `[去重] 强制重下,跳过去重` + 已入库的重新导出 | ✅ |
实施过程中的两个问题均已解决:
1. **顺心 RTS 正则**`RTS\d+` 遇字母 W 停(只抓 `RTS023`)→ 改 `RTS[A-Z0-9]+` 抓完整 `RTS023WJ375320`
2. **韵达 force 偶发失败**:韵达站点自身 UI 不稳定(`section iframe` 匹配到 2 个 + 弹窗遮挡),与去重/force 无关;换安能验证 force 成功。
静态检查Blackpy310+ compileall + tsc 全绿。

View File

@@ -0,0 +1,399 @@
# 应到数据「提交导出任务前」去重 实施计划 v2
> **执行约定:** 本仓库无 pytest验证靠实跑站点流程 + DB 核对(见第 7 节)。
> 本仓库约定 **不自动提交**;所有改动落地后等用户明确说"提交"再 commit/push。
> 本次会话额外约定:未获用户明确指示前不动代码、不提交。
**Goal** 在周期性自动落库场景下,于「提交导出任务」之前,按**交接单号**(顺心=运单列表界面里的交接单号)判断该批应到数据是否已落库,已落库则跳过提交导出任务,从源头消除重复下载与重复落库;并提供一个"强制重下"开关(默认关)兜底。
**Architecture**
- 去重数据源 = PostgreSQL `expected_record.handover_no`(已存在字段,权威);`store.py` 新增 `get_existing_handover_nos(site)`(含 cpolar 降级)。
- 在 4 站**应到**下载循环内、提交导出动作之前注入"命中已落库则 `continue`"。
- `force` 开关经 `task_spec``dispatch_task` → handler → 各站 `download(page, force)``impl(page, force)` 透传;`force=True` 时跳过去重。周期 job 默认不 force。
**Tech Stack** Python 3.10+ / Playwright网页 3 站)/ 裸 CDP安能/ psycopg / SQLite前端 Next.js 16 + React 19。
## Global Constraints
- Python 一律 `.venv`;改完任何 `.py` 必须跑 `.venv/Scripts/python.exe -m black inbound_verify`
- 改完跑 `.venv/Scripts/python.exe -m py_compile inbound_verify` 自检。
- **不自动提交/推送**(覆盖全局 auto-push 默认)。
- **不动 `export_times` 时间容差≤40s匹配机制**CLAUDE.md 约定)。
- 安能 CDP 驱动,**绝不用 `Page.reload`**。
- `config.yaml` 已 gitignore不提交真实凭据。
- 前端是 **Next.js 16有 breaking changes**,写前端代码前先查 `node_modules/next/dist/docs/`
- 行号基于 2026-07-29 快照,实现时以当前代码为准、就近定位。
## 1. 已定决策v1 审核反馈)
| 决策 | 结论 |
|---|---|
| A. 去重数据源 | **查 PostgreSQL `expected_record.handover_no`** |
| B. 范围 | **本次只做应到expected**;实到/百世不动 |
| C. 强制开关 | **加 force 开关,默认不强制重下** |
| 顺心标识 | **方式1点"运单列表"后、点导出前,从运单列表界面读交接单号**(与其他 3 站统一用交接单号去重) |
## 2. 背景与问题根源
周期链路:`fetch_schedule(IntervalTrigger) → task_queue → worker → dispatch_task → handler → 提交导出+下载 → _persist_to_db(UPSERT)`
- DB 已幂等(`expected_record``(site, waybill_no)` UPSERT
-`dispatch_task` 调 handler 前**无"是否需要下载"判断**,周期触发重复"提交导出→下载→解析"。
- 本方案在「提交导出任务」前按交接单号去重,从源头省掉重复下载。
## 3. 各站探索结论(注入点)
| 站点 | 文件 | 提交导出位置 | 提交前标识 | 来源 |
|---|---|---|---|---|
| 中通 | `sites/zto.py` | `zto_expected_download_impl` L264 | ✅ 已有 `handover_no`L251 | 主表行 `td.nth(3)` 正则 18 位 |
| 韵达 | `sites/yunda.py` | `yunda_expected_download_impl` L342 | ✅ 已有 `raw_no`L291 | 列表行 `td.nth(1)` |
| 安能 | `sites/anneng.py` | 主循环 L927逐条 | ✅ 已有 `ewbs_no`L907-913 | CDP 复选框 `ewbsListNo=` 正则 19 位 |
| 顺心 | `sites/shunxin.py` | L319 点导出 | 🆕 方式1L316 后读运单列表界面交接单号 | DOM 待实勘 |
**顺心方式1关键事实**已探明raw 里「班次号」「交接单号」都有,且一个交接单 = 一个班次 = 多条运单;交接单号即入库 `handover_no`,与另 3 站同键。
## 4. 文件结构
| 文件 | 改动 |
|---|---|
| `schema.sql` | 加 `expected_record(site, handover_no)` 索引 |
| `inbound_verify/store.py` | 新增 `get_existing_handover_nos(site)` |
| `inbound_verify/cli/server.py` | `TaskRequest``force``create_task` 透传 force周期不 force |
| `inbound_verify/runtime.py` | `dispatch_task` 读 force 传 handler所有 handler 加 `force` 形参并透传到 `download_func` |
| `inbound_verify/sites/zto.py` | `download/impl``force`;应到循环注入去重 + 空兜底 |
| `inbound_verify/sites/yunda.py` | 同上(空兜底已存在) |
| `inbound_verify/sites/anneng.py` | 同上 |
| `inbound_verify/sites/shunxin.py` | `download/impl``force`方式1点运单列表后读交接单号去重 + 退回 + 空兜底 |
| `dashboard/app/page.tsx` | 加 `forceRedownload` state + checkbox`trigger/triggerPrimary` 透传 force |
> 4 站的 actual实到`download` 入口也统一加 `force=False` 形参(接收但不用,仅让 `_web_handler` 的统一调用成立actual impl 不改。
## 5. 任务分解
### Task 1schema.sql 加索引
**Files:** Modify `schema.sql``idx_expected_site_date` 之后)
```sql
CREATE INDEX IF NOT EXISTS idx_expected_handover ON expected_record (site, handover_no);
```
**验证:** `.venv/Scripts/python.exe -m inbound_verify.store init`(幂等)。
---
### Task 2store.py 新增查已落库交接单号集合
**Files:** Modify `inbound_verify/store.py``ingest_task` 之后)
**Produces:** `get_existing_handover_nos(site: str) -> set[str]`
```python
def get_existing_handover_nos(site):
"""查该站点已落库的交接单号集合expected_record.handover_no
"提交导出任务前"去重:已落库的不再重复提交导出。
PG 不可用cpolar 抖动等)时返回空集 + 告警,调用方按"未确认存在"处理
继续提交导出UPSERT 兜底,绝不因去重查询失败而漏数据)。"""
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT handover_no FROM expected_record "
"WHERE site=%s AND handover_no IS NOT NULL AND handover_no <> ''",
(site,),
)
return {str(r[0]).strip() for r in cur.fetchall()}
except Exception as e:
print(f">> [去重] 查询已落库交接单号失败({site}),本次不去重: {e}")
return set()
```
**验证:** `.venv/Scripts/python.exe -c "from inbound_verify import store; print(len(store.get_existing_handover_nos('中通')))"` 不抛异常即过。
---
### Task 3force 开关后端骨架server + runtime
`force``task_spec` 一路透传到各站 `download(page, force)``with_retry``flow` 是零参 lambdaforce 经闭包捕获,**with_retry 不动**。
**(a) `server.py` TaskRequest + create_task**
```python
class TaskRequest(BaseModel):
site: str
kind: str
force: bool = False # 新增:强制重下(忽略已落库去重),默认关
```
```python
# create_task 内
task_queue.put((task_id, {"site": req.site, "kind": req.kind, "force": req.force}))
```
> `_enqueue_fetch`L133 周期投递)**保持不变**(不带 force → 默认 False✅。
**(b) `runtime.py` dispatch_task 透传 force**
```python
def dispatch_task(ctx, task_spec):
site = task_spec.get("site")
kind = task_spec.get("kind")
force = bool(task_spec.get("force", False)) # 新增
...
try:
ret = handler(ctx, force) # 改:原 handler(ctx)
```
**(c) `runtime.py` 所有 handler 加 force 形参:**
`_web_handler`
```python
def handler(ctx, force=False):
pg = ctx.pages_map[site]
if isinstance(pg, list): # 顺心双账号
return download_func(pg, foreground=ctx.foreground, force=force)
if ctx.foreground:
pg.bring_to_front()
return download_func(pg, force=force)
```
`_site_undelivered_handler`
```python
def handler(ctx, force=False):
exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force) is not False
act_ok = (TASK_HANDLERS[(site, "actual")](ctx, force) is not False) if exp_ok else False
...
```
安能 expected/actual 与 compare签名兼容即可
```python
("安能", "expected"): lambda ctx, force=False: anneng.anneng_expected_download(force=force),
("安能", "actual"): lambda ctx, force=False: anneng.anneng_actual_download(force=force),
("__compare__", "compare"): lambda ctx, force=False: (compare.main() or True),
```
**验证:** `py_compile` 通过;服务重启后 `POST /tasks {site,kind,force:true}` 不报 TypeError此时各站 `download` 的 force 形参由 Task 4-7 补齐,连续实施)。
---
### Task 4中通 zto.pydownload/impl 加 force + 去重)
**Files:** Modify `inbound_verify/sites/zto.py`
**(a) 入口与 impl 加 force闭包透传with_retry 不动):**
```python
def zto_expected_download(page, force=False):
return with_retry(
"中通", "应到",
lambda: zto_expected_download_impl(page, force=force),
lambda: zto_reset(page),
)
def zto_expected_download_impl(page, force=False):
...
# zto_actual_download / zto_actual_download_impl 同样加 force=False 形参actual 不用 force仅兼容
```
**(b) 循环前加载已落库集合**L241 print 之后、L243 `for` 之前):
```python
# 【去重】加载本站已落库交接单号force=True 或查询失败时 existing=空集(不去重)
if force:
existing = set()
print(">> [去重] 强制重下,跳过去重。")
else:
try:
from inbound_verify import store
existing = store.get_existing_handover_nos("中通")
except Exception as _e:
existing = set()
print(f">> [去重] 加载失败,本次不去重: {_e}")
```
**(c) 循环内命中跳过**L252 print 之后、L254 `row.dblclick()` 之前):
```python
print(f" -> 当前交接单号:{handover_no}")
if handover_no in existing:
print(f" ⏭️ 交接单号 {handover_no} 已落库,跳过提交导出。")
continue
row.dblclick()
```
**(d) 空列表兜底**L305 循环后、进入 `_zto_poll_and_download_tasks` 之前):
```python
if not export_times:
print(">> 本次无新交接单需导出(全部已落库或无数据),结束。")
return
```
---
### Task 5韵达 yunda.py同构
**Files:** Modify `inbound_verify/sites/yunda.py`
(a) `yunda_expected_download(page, force=False)` + impl 加 forceactual 同理加形参);(b) 循环前加载 existing同 Task 4b站点"韵达"
**(c) 循环内命中跳过**L291 `raw_no = ...` 之后、L293 `# 跳过已绑定的交接单` 之前):
```python
raw_no = current_row.locator("td").nth(1).inner_text().strip()
if raw_no in existing:
print(f" ⏭️ 交接单号 {raw_no} 已落库,跳过提交导出。")
continue
# 跳过已绑定的交接单
bind_status = current_row.locator("td").nth(2).inner_text().strip()
```
(d) 空列表兜底 **已存在**L389-392无需新增。
---
### Task 6安能 anneng.pyCDP
**Files:** Modify `inbound_verify/sites/anneng.py`
(a) `anneng_expected_download(force=False)` + impl 加 forceactual 同理);
**(b) 主流程加载 existing**`anneng_expected_download_impl` 内、L907 收集 `target_ids` 前,与 `export_times = []`L882并列同 Task 4b站点"安能"
**(c) 主循环命中跳过**L919 `for` 内、L920 print 之后、L921 `activate_tab` 之前):
```python
for i, ewbs_no in enumerate(target_ids, start=1):
print(f" ⏳ [{i}/{len(target_ids)}] 交接单号 {ewbs_no}")
if ewbs_no in existing:
print(f" ⏭️ 交接单号 {ewbs_no} 已落库,跳过。")
continue
activate_tab(tab_cdp, "交接单信息")
...
```
**(d) 空列表兜底**(主循环后、进入 `poll_and_download_tasks` 之前):
```python
if not export_times:
print(">> 本次无新交接单需导出(全部已落库或无数据),结束。")
return
```
---
### Task 7顺心 shunxin.py方式1 + 实勘)
顺心流程L315 点"运单列表" → L316 等"运单查询"label**运单列表界面** → L319 点"导出"。方式1 在 L316 之后、L319 之前读交接单号。
**Step 1实勘不改业务逻辑** 确认运单列表界面里**交接单号的 DOM 选择器**(哪个元素/列)。实勘方式二选一(待用户同意):
- 方式 A临时在 L316 后加调试打印dump 运单列表界面关键 DOM 文本),跑一次顺心应到,从日志定位选择器,再删调试代码。
- 方式 Bdebug 模式(`config.yaml` debug.target_site=顺心CDP 9223单独挂载用 Playwright CLI 观察。
**Step 2注入选择器 `<HANDOVER_SELECTOR>` 确认后替换):**
(a) 入口与 impl 加 force
```python
def shunxin_expected_download(pages, foreground=True, force=False):
return with_retry(
"顺心", "应到",
lambda: shunxin_expected_download_impl(pages, foreground=foreground, force=force),
lambda: shunxin_reset(pages),
)
def shunxin_expected_download_impl(pages, foreground=True, force=False):
...
# shunxin_actual_download / impl 同样加 force=False 形参actual 不用)
```
(b) 循环前加载 existing同 Task 4b站点"顺心";两账号共享同一 `existing`)。
**(c) 循环内:点运单列表 → 读交接单号 → 命中则退回跳过**L315-316 之后、L319 点导出之前):
```python
for i in range(count):
print(f" ⏳ 正在处理第 {i+1}/{count} 个班次...")
waybill_btns.nth(i).click()
page.locator("label[title='运单查询']").wait_for(state="visible")
# 【方式1】运单列表界面已加载读交接单号 → 已落库则退回列表跳过
handover_no = page.locator("<HANDOVER_SELECTOR>").first.inner_text().strip()
if handover_no in existing:
print(f" ⏭️ 交接单号 {handover_no} 已落库,跳过提交导出。")
page.get_by_role("tab", name="车辆点到").click() # 退回列表(复用 L336
page.wait_for_timeout(500)
continue
# 4. 执行导出流程
page.get_by_role("button", name="export 导出").click()
...
```
(d) 空列表兜底L339 循环后、进入导出任务管理页轮询之前):
```python
if not export_times:
print(">> 本次无新班次需导出(全部已落库或无数据),结束。")
return
```
---
### Task 8前端 force 开关dashboard
**Files:** Modify `dashboard/app/page.tsx`BFF `app/api/tasks/route.ts` 是 generic 透传,**不用改**
**(a) 加 stateL21 附近):**
```tsx
const [forceRedownload, setForceRedownload] = useState(false);
```
**(b) `trigger` 加 force 形参并写入 bodyL27-50**
```tsx
const trigger = useCallback(
async (site: string, kind: string, label: string, force?: boolean) => {
...
body: JSON.stringify({ site, kind, force: !!force }),
...
},
[refreshTasks],
);
```
**(c) `triggerPrimary` 透传L58-63**
```tsx
const triggerPrimary = useCallback(
async (cfg: SiteConfig) => {
await trigger(cfg.name, cfg.primaryKind, `${cfg.name}·获取未到`, forceRedownload);
},
[trigger, forceRedownload],
);
```
**(d) UI在配置区/顶部加 checkbox默认不勾**
```tsx
<label className="inline-flex items-center gap-1 text-xs text-amber-700">
<input
type="checkbox"
checked={forceRedownload}
onChange={(e) => setForceRedownload(e.target.checked)}
/>
</label>
```
> 仅"获取未到"主按钮透传 force周期抓取不经前端、恒不 force。
**验证:** 前端勾选 → 触发 → 网络面板看到 POST `/api/tasks` body 含 `force:true`;后端日志 `强制重下,跳过去重`
## 6. 关键注意事项(陷阱)
- **跳过的交接单号绝不 `export_times.append`**:否则 `len(export_times)` > 实际提交数 → 下载数校验失败。所有 `continue` 都在 append 之前。
- **全部跳过时必须 `return`**`export_times` 为空时不进导出任务管理页轮询。
- **顺心方式1跳过要退回**:点进运单列表后命中已落库,需点"车辆点到"tab 退回再 `continue`(复用 L336
- **PG 降级只防漏不防重**:查询失败 = 空集 = 当作未存在 = 继续提交UPSERT 兜底。
- **不动 `export_times` 时间容差匹配**。
- actual/百世 `download` 只加 `force` 形参兼容impl 不加去重。
## 7. 验证(无 pytest
1. **单站联调**`config.yaml` debug 单站):首次新单正常下载+入库;再触发同范围 → 已入库的全部 `⏭️ 跳过``export_times` 空,直接 return。
2. **force 开关**:勾选"强制重下" → 已落库的也重新提交导出(日志 `强制重下,跳过去重`)。
3. **DB 核对**`SELECT site, handover_no, COUNT(*) FROM expected_record GROUP BY site, handover_no` 无翻倍。
4. **cpolar 降级**:断 PG → `get_existing_handover_nos` 返回空集 + 告警,流程仍正常下载(不漏)。
5. **Black + py_compile**;前端 `npm run build` 或 dev 热更无类型错。
## 8. 实施顺序与依赖
Task 1 → 2 → 3骨架此时各站 download 的 force 形参在 4-7 补)→ 4/5/6/7各站连续做完让链路自洽→ 8前端。顺心 Task 7 的 Step1 实勘需在运行的服务上操作,实施时与用户协调时机。

View File

@@ -0,0 +1,131 @@
# 指定日期下载接口(开发者)— 设计文档
> 日期2026-07-29
> 定位:面向开发者的 HTTP 接口,**不进前端**。提供"指定一个具体日期,下载该日应到 / 实到数据"的能力,用于补下历史数据。
> 前置:中通跨月导航已实现并验证(见 `feat(zto): cross-month calendar navigation`)。
## 一、背景与目标
现状:下载日期由各站 **offset 偏移**0=今天1=昨天…,存 `state.db`,上限 `MAX_DATE_OFFSET=30`)决定,周期调度与手动触发都用 offset。无法指定一个具体日期。
目标:新增"指定日期"入口(开发者用),传一个 `YYYY-MM-DD` 日期,下载该日应到 / 实到数据。不替换 offset 机制,与之并存:传 date 用 date不传走 offset。
## 二、范围
| 站点 | 支持指定日期 | 说明 |
| --- | --- | --- |
| 顺心 / 中通 / 韵达 / 安能 | ✅ | 应到、实到均支持 |
| 百世 | ❌ | 固定下载当天,传 date 返回 400 |
## 三、接口契约
复用 `POST /tasks`body 新增可选字段 `date`(与 `force` 并列):
```json
{ "site": "中通", "kind": "expected", "date": "2026-06-14" }
```
- `date: Optional[str] = None`,格式 `YYYY-MM-DD`
- **优先级**:传 `date` 则本次用 date不传则走站点 offset 配置(默认行为完全不变)。
- `date``force` 可共存(指定日期 + 强制重下)。
### 合法性校验(仅在传了 date 时执行,失败返回 400
1. **格式**`datetime.strptime(date, "%Y-%m-%d")` 解析成功,否则 400。
2. **范围**`今天 - 90 天 ≤ date ≤ 今天`
- `date > 今天` → 400未来日期日历未来格子 `invalid` 物理上点不动,且不应下未来数据)。
- `date < 今天 - 90 天` → 400回溯上限 90 天)。
3. **百世**site=百世 且传 date → 400固定当天
> 合法性校验落在 `POST /tasks``server.py` `create_task`),入队前拦截,非法请求不产生任务。
## 四、透传链路(与现有 `force` 完全对称)
```
POST /tasks {site, kind, force, date}
→ task_queue.put((tid, {site, kind, force, date}))
→ dispatch_task(ctx, task_spec) # 读 task_spec["date"]
→ handler(ctx, force, date) # _web_handler / 安能 lambda / _site_undelivered_handler
→ impl(page, force, date) # 各站 download_impl
```
- `_web_handler``handler(ctx, force=False, date=None)`,透传 `download_func(pg, force, date)`;顺心双账号透传 `(pages, foreground, force, date)`
- `_site_undelivered_handler`(未到):连下 expected + actual**两个子任务共用同一个 date**。
- 安能 lambda`(ctx, force=False, date=None) → anneng_xxx_download(force=force, date=date)`
- **周期调度** `_enqueue_fetch` 投递的 task_spec 只有 `{site, kind}`(不带 date→ 恒走 offset**无需改动**。
## 五、各站 impl 改造(核心)
统一模式:**`target = parse(date) if date else (today offset)`**。
### 中通zto—— 复用跨月算法
把 date 折算成 effective offset复用现有 `target_time = today_time offset*86400000` 与跨月翻页(`_zto_flip_to_target_month`),零额外 UI 逻辑:
```python
def zto_expected_download_impl(page, force=False, date=None):
...
offset = state_store.get_offset("中通")
if date:
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
# 后续 today_time / target_time / 跨月翻页 逻辑完全不变
```
`zto_actual_download_impl` 同理(用 `("中通","actual")` offset。expected / actual 两个 impl 都加 `date=None` 形参,`zto_expected_download` / `zto_actual_download` 公开入口同步加形参并透传。
### 韵达 / 顺心 / 安能 —— date 直接当 target
这三站 offset→日期是 `target = today timedelta(days=offset)` 后填**字符串**到日期控件(非日历格子),指定日期只需替换 target 来源:
```python
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
```
后接的"填起始/截止日期字符串"逻辑完全不变。各站 expected / actual 入口与 impl 都加 `date=None` 形参。
- 韵达:`yunda_expected_download(_impl)` / `yunda_actual_download(_impl)`
- 顺心:`shunxin_expected_download(_impl)` / `shunxin_actual_download(_impl)`(双账号入口透传 date 到各账号 impl
- 安能:`anneng_expected_download` / `anneng_actual_download`
### 百世baishi—— 签名兼容
`baishi_download_undelivered_data(page, date=None)``date=None` 形参(**忽略**),仅为对齐 `_web_handler` 的统一透传签名;百世任务实际不会带 dateserver 已拦截)。
## 六、业务日期快照
`_record_business_date(site, kind, date=None)`:有 date 则业务日期 = date否则维持现状 `today offset``dispatch_task``task_spec["date"]` 透传进去,保证状态盘 / 报告显示的"是哪天的数据"准确(不被 offset 算错)。
## 七、改动文件清单
| 文件 | 改动 |
| --- | --- |
| `inbound_verify/cli/server.py` | `TaskRequest.date` + `create_task` 合法性校验 + task_spec 透传 date |
| `inbound_verify/runtime.py` | `_web_handler` / `_site_undelivered_handler` / 安能 lambda 透传 date`dispatch_task` 读 date 透传给 handler 与 `_record_business_date``_record_business_date` 加 date |
| `inbound_verify/sites/zto.py` | expected/actual 入口+impl 加 `date`date→effective offset 复用跨月 |
| `inbound_verify/sites/yunda.py` | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/sites/shunxin.py` | 同上(双账号透传 date |
| `inbound_verify/sites/anneng.py` | expected/actual 加 `date`date→target |
| `inbound_verify/sites/baishi.py` | 加 `date=None` 形参兼容(忽略) |
## 八、验证计划
1. **接口校验**curl/python urllib
- 合法 date过去某日→ 202任务成功。
- 未来日期 / 超 90 天 / 格式错 → 400。
- 百世 + date → 400。
- 不传 date → 走 offset行为不变
2. **各站实测**(指定一个过去日期触发任务):
- 中通:跨月日期(已知 OK复用已验证的跨月导航
- 顺心 / 韵达 / 安能:实测其日期控件是否接受任意过去日期字符串;若控件是日历选择器需翻月,则按中通同法扩展(本轮发现则记录、必要时追加改动)。
3. **业务日期快照**:下载后 `GET /status``*_business_date` == 指定 date。
4. 改完跑 Black + `py_compile`
## 九、非目标YAGNI
- 前端 UIcheckbox / 日期选择器)——开发者接口,不进前端。
- 周期调度指定日期——周期恒走 offset。
- 批量日期 / 日期范围下载——单次单日。

View File

@@ -0,0 +1,682 @@
# 指定日期下载接口开发者Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:**`POST /tasks` 加可选 `date`YYYY-MM-DD支持顺心/中通/韵达/安能指定一个过去日期下载应到/实到数据;不传 date 时行为不变(走 offset
**Architecture:** 复用现有 `force` 透传链路加一个 `date` 字段:`TaskRequest.date``task_spec["date"]``dispatch_task``handler(ctx, force, date)` → 各站 `impl(page, force, date)`。各站把"算 target 日期"的来源从 `today - offset` 改为"有 date 用 date否则 today - offset"。中通把 date 折算成 effective offset 以复用已验证的跨月翻页。
**Tech Stack:** Python 3.10、FastAPI、Playwright、SQLitestate_store、PostgreSQLstore可选入库
## Global Constraints
- **环境**:所有 Python 用 `D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe`项目虚拟环境无需激活。PYTHONPATH 含 `InboundVerify` 根。
- **格式化/自检(每改一个 .py 必做)**`python -m black <file>`target py310+ `python -m py_compile <file>`。Black 若提示 "Python 3.10 cannot parse code formatted for 3.15",加 `--target-version py310`"left unchanged" 即合格。
- **无 pytest**:本项目无单元测试框架(见 `InboundVerify/CLAUDE.md`)。每个任务的"验证"= Black + py_compile端到端API 校验、各站实测)集中在 Task 8需重启服务加载新代码
- **submodule 工作流**:改动在 `InboundVerify` submodule`dev` 分支)。每个 Task 末尾在 submodule 内 `git add <file> && git commit`。**push 到 origin/dev + 父仓库 bump** 统一在 Task 8项目约定不自动 push等用户确认但 plan 内 commit 步骤照写)。
- **顺序安全**Task 1-5站点只给 impl 加 `date=None` 形参 + date 逻辑,**date 默认 None 时走原 offset 路径,向后兼容**Task 6runtime才把 date 从 task_spec 透传进 implTask 7server才允许 date 入队。任一 Task 完成后系统均可正常运行。
- **合法性边界**(来自 spec`date` 传了才校验——格式 `YYYY-MM-DD``今天-90 ≤ date ≤ 今天`、百世不支持 date。非法返回 HTTP 400。
---
## File Structure
| 文件 | 责任 | 本计划改动 |
| --- | --- | --- |
| `inbound_verify/sites/baishi.py` | 百世下载(固定当天) | 入口加 `date=None` 形参(忽略) |
| `inbound_verify/sites/zto.py` | 中通下载(日历格子,跨月) | 入口+impl 加 `date`date→effective offset 复用跨月 |
| `inbound_verify/sites/yunda.py` | 韵达下载(日期字符串) | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/sites/shunxin.py` | 顺心下载(双账号,日期字符串) | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/sites/anneng.py` | 安能下载CDP日期字符串 | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/runtime.py` | 任务派发/心跳共享核心 | handler/dispatch 透传 date`_record_business_date` 用 date |
| `inbound_verify/cli/server.py` | FastAPI 服务 | `TaskRequest.date` + 合法性校验 + task_spec 透传 |
---
## Task 1: baishi.py — 入口加 date=None 形参(兼容)
**Files:**
- Modify: `inbound_verify/sites/baishi.py:140``baishi_download_undelivered_data`
**Interfaces:**
- Produces: `baishi_download_undelivered_data(page, force=False, date=None)` —— 后续 Task 6 的 `_web_handler` 会以 `download_func(pg, force=force, date=date)` 调用它,必须接受 `date` kwarg百世忽略
- [ ] **Step 1: 改签名**
`def baishi_download_undelivered_data(page, force=False):` 改为:
```python
def baishi_download_undelivered_data(page, force=False, date=None):
"""百世应到未到数据下载固定当天date 形参仅为对齐统一透传签名,忽略)。"""
```
(函数体不动;`date` 不使用。)
- [ ] **Step 2: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/sites/baishi.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
Expected: black "left unchanged" 或 reformat 后通过OK。
- [ ] **Step 3: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/baishi.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(baishi): accept date kwarg (ignored) for unified dispatch signature" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 2: zto.py — date 折算成 effective offset复用跨月
**Files:**
- Modify: `inbound_verify/sites/zto.py``zto_expected_download``zto_expected_download_impl``zto_actual_download``zto_actual_download_impl`
**Interfaces:**
- Produces: `zto_expected_download(page, force=False, date=None)` / `zto_actual_download(page, force=False, date=None)`impl 同签名。Task 6 的 `_web_handler``download_func(pg, force=force, date=date)` 调用。
- [ ] **Step 1: expected 入口加 date 并透传**
`def zto_expected_download(page, force=False):` 及其 `with_retry` 改为:
```python
def zto_expected_download(page, force=False, date=None):
"""中通:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"应到",
lambda: zto_expected_download_impl(page, force=force, date=date),
lambda: zto_reset(page),
)
```
- [ ] **Step 2: expected impl 加 datedate→effective offset**
`def zto_expected_download_impl(page, force=False):` 改签名加 `date=None`。其内"读取服务端日期偏移"段(`offset = state_store.get_offset("中通")` 与随后的 `print(...偏移...)`)改为:
```python
# 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通")
if date:
# 指定日期:折算成相对今天的有效偏移,复用下方 target_time 计算与跨月翻月
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
```
(其后的 `target_time = today_time - offset * 86400000` 与跨月翻月逻辑**不动**——date 经折算后走同一条路径。)
- [ ] **Step 3: actual 入口加 date 并透传**
`def zto_actual_download(page, force=False):` 及其 `with_retry` 改为:
```python
def zto_actual_download(page, force=False, date=None):
"""中通:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"实到",
lambda: zto_actual_download_impl(page, date=date),
lambda: zto_reset(page),
)
```
> 注:`zto_actual_download_impl` 现签名 `(page)`(无 forceactual 不去重Step 4 给它加 `date`。
- [ ] **Step 4: actual impl 加 datedate→effective offset**
`def zto_actual_download_impl(page):` 改为 `def zto_actual_download_impl(page, date=None):`。其内"# 2. 读取服务端日期偏移"段(`offset = state_store.get_offset("中通", "actual")` 与随后的 `print`)改为:
```python
# 2. 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通", "actual")
if date:
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
```
(其后 `target_time = today_time - offset * 86400000` 与跨月翻月不动。)
- [ ] **Step 5: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/sites/zto.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/zto.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(zto): support date arg via effective-offset (reuses cross-month nav)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 3: yunda.py — date→targetexpected + actual
**Files:**
- Modify: `inbound_verify/sites/yunda.py``yunda_expected_download(_impl)``yunda_actual_download(_impl)`
**Interfaces:**
- Produces: `yunda_expected_download(page, force=False, date=None)` / `yunda_actual_download(page, force=False, date=None)`impl 同加 `date=None`
- [ ] **Step 1: expected 入口透传 date**
```python
def yunda_expected_download(page, force=False, date=None):
...
return with_retry(
"韵达",
"应到",
lambda: yunda_expected_download_impl(page, force=force, date=date),
lambda: yunda_reset(page),
)
```
- [ ] **Step 2: expected impl 加 date + date→target**
`def yunda_expected_download_impl(page, force=False):``def yunda_expected_download_impl(page, force=False, date=None):`。其内日期段(`offset = state_store.get_offset("韵达")` 起几行)改为:
```python
offset = state_store.get_offset("韵达")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_ymd = f"{target.year}-{target.month}-{target.day}"
start_date_ymd = target_ymd
today_ymd = target_ymd
print(f">> 设置查询日期: [{target_ymd}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
- [ ] **Step 3: actual 入口透传 date**
```python
def yunda_actual_download(page, force=False, date=None):
...
return with_retry(
"韵达",
"实到",
lambda: yunda_actual_download_impl(page, date=date),
lambda: yunda_reset(page),
)
```
- [ ] **Step 4: actual impl 加 date + date→target**
`def yunda_actual_download_impl(page):``def yunda_actual_download_impl(page, date=None):`。其内日期段改为:
```python
offset = state_store.get_offset("韵达", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
start_date = target # 单日范围:起止同日
today = target # 让下方"截止时间"选择器也指向 target
print(
f">> 设置实到查询日期: [{target.year}-{target.month}-{target.day}]"
f"{'指定 ' + date if date else f'偏移 {offset}0=今天'}"
)
```
- [ ] **Step 5: Black + py_compile**(同 Task 2 命令,文件换 yunda.py
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/yunda.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(yunda): support date arg (date takes precedence over offset)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 4: shunxin.py — date→target双账号透传
**Files:**
- Modify: `inbound_verify/sites/shunxin.py``shunxin_expected_download(_impl)``shunxin_actual_download(_impl)`
**Interfaces:**
- Produces: `shunxin_expected_download(pages, foreground=True, force=False, date=None)` / `shunxin_actual_download(pages, foreground=True, force=False, date=None)`。Task 6 的 `_web_handler``download_func(pg, foreground=ctx.foreground, force=force, date=date)` 调用pg 是 page 列表)。
- [ ] **Step 1: expected 入口加 date 并向 impl 透传**
`def shunxin_expected_download(pages, foreground=True, force=False):` → 加 `date=None`。在函数内调用 `shunxin_expected_download_impl(page, out_tag=..., force=force)` 的位置,加上 `date=date`
```python
def shunxin_expected_download(pages, foreground=True, force=False, date=None):
...
# 对每个账号调用 impl 时透传 date
... shunxin_expected_download_impl(page, out_tag=归属地, force=force, date=date) ...
```
> 执行者:用 Grep 定位 `shunxin_expected_download_impl(page,` 的调用处(在 `shunxin_expected_download` 函数体内,对每个账号调用一次),给每处加 `date=date`。签名行只加 `date=None`其余函数体归属地读取、去重校验、merge不动。
- [ ] **Step 2: expected impl 加 date + date→target**
`def shunxin_expected_download_impl(page, out_tag="", force=False):` → 加 `date=None`。其内日期段(`offset = state_store.get_offset("顺心")` 起几行)改为:
```python
offset = state_store.get_offset("顺心")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}...")
```
- [ ] **Step 3: actual 入口加 date 并透传**
`def shunxin_actual_download(pages, foreground=True, force=False):` → 加 `date=None`;调用 `shunxin_actual_download_impl(page, out_tag=...)` 处加 `date=date`
- [ ] **Step 4: actual impl 加 date + date→target**
`def shunxin_actual_download_impl(page, out_tag=""):``def shunxin_actual_download_impl(page, out_tag="", date=None):`。其内日期段(`offset = state_store.get_offset("顺心", "actual")` 起几行)改为:
```python
offset = state_store.get_offset("顺心", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}...")
```
- [ ] **Step 5: Black + py_compile**(文件 shunxin.py
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/shunxin.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(shunxin): support date arg, propagate to both accounts" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 5: anneng.py — date→targetCDPexpected + actual
**Files:**
- Modify: `inbound_verify/sites/anneng.py``anneng_expected_download(_impl)``anneng_actual_download(_impl)`
**Interfaces:**
- Produces: `anneng_expected_download(force=False, date=None)` / `anneng_actual_download(force=False, date=None)`。Task 6 的 TASK_HANDLERS 安能项以 `lambda ctx, force=False, date=None: anneng.anneng_expected_download(force=force, date=date)` 调用。
- [ ] **Step 1: expected 入口 + impl 加 datedate→target**
```python
def anneng_expected_download(force=False, date=None):
return with_retry(
"安能", "应到", lambda: anneng_expected_download_impl(force=force, date=date), anneng_reset
)
def anneng_expected_download_impl(force=False, date=None):
```
expected impl 内日期段(`offset = state_store.get_offset("安能")` 起几行)改为:
```python
offset = state_store.get_offset("安能")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = f"{target.year}-{target.month:02d}-{target.day:02d}"
start_str = target_str
today_str = target_str
print(f">> 查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
- [ ] **Step 2: actual 入口 + impl 加 datedate→target**
```python
def anneng_actual_download(force=False, date=None):
return with_retry("安能", "实到", lambda: anneng_actual_download_impl(date=date), anneng_reset)
def anneng_actual_download_impl(date=None):
```
actual impl 内日期段(`offset = state_store.get_offset("安能", "actual")` 起几行)改为:
```python
offset = state_store.get_offset("安能", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
start_str = f"{target.year}\{target.month:02d}/{target.day:02d} 00:00:00"
end_str = f"{target.year}\{target.month:02d}/{target.day:02d} 23:59:59"
print(f">> 扫描日期: [{start_str}{end_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
> actual 的 `start_str/end_str` 沿用现有 `{year}\{month}/{day}` 格式(含反斜杠,站点如此),只把 target 来源改成 date。
- [ ] **Step 3: Black + py_compile**(文件 anneng.py
- [ ] **Step 4: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/anneng.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(anneng): support date arg (date takes precedence over offset)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 6: runtime.py — 全链路透传 date + _record_business_date 用 date
**Files:**
- Modify: `inbound_verify/runtime.py``_web_handler``_site_undelivered_handler``TASK_HANDLERS` 安能项、`dispatch_task``_record_business_date`
**Interfaces:**
- Consumes: Task 1-5 产出的各站 `*(..., date=None)` 签名。
- Produces: `dispatch_task``task_spec["date"]` 读 date 透传给 handler 与 `_record_business_date`handler 签名 `(ctx, force=False, date=None)`
- [ ] **Step 1: _web_handler 透传 date**
```python
def _web_handler(site, download_func):
def handler(ctx, force=False, date=None):
pg = ctx.pages_map[site]
if isinstance(pg, list):
# 顺心双账号:置顶与否交给 shunxin_download 在逐账号循环里按 foreground 决定
return download_func(pg, foreground=ctx.foreground, force=force, date=date)
if ctx.foreground:
pg.bring_to_front()
return download_func(pg, force=force, date=date)
return handler
```
- [ ] **Step 2: _site_undelivered_handler 透传 date连下 expected+actual 共用同一 date**
```python
def _site_undelivered_handler(site):
def handler(ctx, force=False, date=None):
exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force, date) is not False
act_ok = (
(TASK_HANDLERS[(site, "actual")](ctx, force, date) is not False)
if exp_ok
else False
)
if exp_ok and act_ok:
return compare.write_site_file(site)
stale = os.path.join(DOWNLOAD_DIR, SITE_UNDELIVERED_FILE.format(name=site))
if os.path.exists(stale):
os.remove(stale)
return False
return handler
```
- [ ] **Step 3: TASK_HANDLERS 安能项透传 date**
```python
("安能", "expected"): lambda ctx, force=False, date=None: anneng.anneng_expected_download(
force=force, date=date
),
("安能", "actual"): lambda ctx, force=False, date=None: anneng.anneng_actual_download(
force=force, date=date
),
```
`__compare__` 项与百世/网页项不动——百世经 `_web_handler` 已透传 datebaishi 忽略。)
- [ ] **Step 4: dispatch_task 透传 date**
`dispatch_task` 内,把 `ret = handler(ctx, bool(task_spec.get("force", False)))` 改为:
```python
ret = handler(
ctx, bool(task_spec.get("force", False)), task_spec.get("date")
)
if ret is False:
return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)")
_record_business_date(site, kind, task_spec.get("date"))
_persist_to_db(site, kind)
return (state_store.TASK_SUCCESS, None)
```
- [ ] **Step 5: _record_business_date 接受 date**
```python
def _record_business_date(site, kind, date=None):
"""下载成功后,把本次数据的业务日期快照写进状态库(供前端/报告显示「是哪天的数据」)。
有 date 用 date否则 = 下载当天 该数据对应的日期偏移。__compare__ 无数据概念,跳过。"""
if site == "__compare__":
return
today = datetime.now().date()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
def _write(k, biz_or_off):
# biz_or_off: int=offsettodayoffsetstr=已确定的业务日期date
biz = (
(today - timedelta(days=biz_or_off)).strftime("%Y-%m-%d")
if isinstance(biz_or_off, int)
else biz_or_off
)
try:
state_store.set_data_state(
site, k, ready=True, generated_at=now, business_date=biz
)
except Exception as e:
print(f">> [状态] 写业务日期失败 {site}/{k}: {e}")
def off(kind_key):
return state_store.get_offset(site, kind_key)
if kind == "expected":
_write("expected", date if date else off("expected"))
elif kind == "actual":
_write("actual", date if date else off("actual"))
elif site == "百世":
_write("undelivered", 0)
else: # 4 站 undelivered连带补写 expected/actual/undelivered 三列
_write("expected", date if date else off("expected"))
_write("actual", date if date else off("actual"))
_write("undelivered", date if date else off("expected"))
```
- [ ] **Step 6: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/runtime.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 7: 冒烟date=None 行为不变)**
服务仍跑旧代码,但 runtime 模块可独立 import 校验:
```bash
PYTHONPATH="D:/projects/LogisticsHubIPA/InboundVerify" "$PY" -c "import inbound_verify.runtime as r; import inspect; print('handler', inspect.signature(r._web_handler('中通', lambda *a, **k: None))); print('rec', inspect.signature(r._record_business_date))"
```
Expected: handler 含 `(ctx, force=False, date=None)`rec 含 `(site, kind, date=None)`
- [ ] **Step 8: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/runtime.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(runtime): propagate date through dispatch chain and business-date snapshot" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 7: server.py — TaskRequest.date + 合法性校验 + task_spec
**Files:**
- Modify: `inbound_verify/cli/server.py` — 顶部 import、`TaskRequest``create_task`
**Interfaces:**
- Produces: `POST /tasks` 接受 `{site, kind, force, date}`;合法 date 入队为 `task_spec["date"]`YYYY-MM-DD非法返回 400。
- [ ] **Step 1: 顶部 import 加 timedelta**
`from datetime import datetime``from datetime import datetime, timedelta`
- [ ] **Step 2: TaskRequest 加 date**
```python
class TaskRequest(BaseModel):
site: str
kind: str
force: bool = False
date: Optional[str] = None # YYYY-MM-DD指定则下载该日数据否则走站点 offset
```
- [ ] **Step 3: create_task 加合法性校验 + task_spec 透传 date**
```python
@app.post("/tasks")
def create_task(req: TaskRequest):
"""提交任务 {site, kind, force, date?} → 入队,返回 task_id。"""
if not worker_state["ready"]:
raise HTTPException(
status_code=409, detail="后端尚未就绪,请等待各站点登录完成后再操作"
)
if (req.site, req.kind) not in TASK_HANDLERS:
raise HTTPException(status_code=400, detail=f"无效任务: {req.site}/{req.kind}")
# 指定日期合法性校验(仅在传了 date 时)
if req.date:
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(
status_code=400, detail=f"date 不可为未来日期: {req.date}"
)
if target_date < today - timedelta(days=90):
raise HTTPException(
status_code=400, detail=f"date 超出 90 天回溯上限: {req.date}"
)
if req.site == "百世":
raise HTTPException(
status_code=400, detail="百世固定下载当天,不支持指定日期"
)
task_id = state_store.create_task(req.site, req.kind)
spec = {"site": req.site, "kind": req.kind, "force": req.force}
if req.date:
spec["date"] = req.date
task_queue.put((task_id, spec))
return {"task_id": task_id}
```
- [ ] **Step 4: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/cli/server.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 5: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/cli/server.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(server): add date field to POST /tasks with legality validation" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 8: 端到端验证 + 收尾push / 父 bump
**Files:** 无代码改动;验证 + 提交推送。
- [ ] **Step 1: 重启服务加载全部新代码**
停掉旧服务进程重启中通调试模式config.yaml 已是中通):
```bash
PYTHONPATH="D:/projects/LogisticsHubIPA/InboundVerify" "D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe" -m inbound_verify.cli.server
```
(后台运行;等中通登录就绪。)
- [ ] **Step 2: 合法性校验端到端API 400/202**
用 Python urllib避免 curl 中文编码问题)逐一验证,期望:
| 请求 | 期望 |
| --- | --- |
| `{site:"中通", kind:"expected", date:"2026-06-14"}` | 202 + task_id |
| `date:"2099-01-01"`(未来) | 400 "不可为未来日期" |
| `date:"2020-01-01"`(超 90 天) | 400 "超出 90 天回溯上限" |
| `date:"2026/06/14"`(格式错) | 400 "格式非法" |
| `{site:"百世", kind:"undelivered", date:"2026-06-14"}` | 400 "百世…不支持指定日期" |
| `{site:"中通", kind:"expected"}`(不传 date | 202走 offset行为不变 |
```bash
"D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe" - <<'PYEOF'
import json, urllib.request, urllib.error
def post(body):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request("http://127.0.0.1:8000/tasks", data=data,
headers={"Content-Type":"application/json"}, method="POST")
try:
print(body, "->", urllib.request.urlopen(req, timeout=10).read().decode())
except urllib.error.HTTPError as e:
print(body, "->", e.code, e.read().decode())
post({"site":"中通","kind":"expected","date":"2099-01-01"})
post({"site":"中通","kind":"expected","date":"2020-01-01"})
post({"site":"中通","kind":"expected","date":"2026/06/14"})
post({"site":"百世","kind":"undelivered","date":"2026-06-14"})
PYEOF
```
- [ ] **Step 3: 中通跨月 date 实测(已知 OK**
触发 `{site:"中通", kind:"expected", date:"2026-06-14"}`,观察 worker 日志:出现 `指定日期 2026-06-14折算偏移 …)` + `偏移日期跨月,正在向前翻月导航` + 查询/下载成功。任务 `success`
- [ ] **Step 4: 顺心/韵达/安能 date 实测**
对顺心/韵达/安能各触发一个 expected `date`(取一个近 1 周内的过去日期,确保站点有数据且控件能接受)。观察日志:`指定日期 …` + 正常查询下载。**若某站日期控件不接受字符串而需日历翻月**记录现象按中通同法_zto_flip 模式)追加改动(可能产生新 Task
- [ ] **Step 5: 业务日期快照校验**
下载后 `curl -s http://127.0.0.1:8000/status`,确认对应站 `expected_business_date` == 指定 date。
- [ ] **Step 6: push submodule dev + 父仓库 bump**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" push origin dev
git -C "D:/projects/LogisticsHubIPA" add InboundVerify
git -C "D:/projects/LogisticsHubIPA" commit -m "chore: bump InboundVerify submodule (date-specific download API)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
git -C "D:/projects/LogisticsHubIPA" push origin master
```
---
## Self-Reviewplan 作者自检)
1. **Spec 覆盖**接口契约→Task 7透传链路→Task 6中通 effective offset→Task 2韵达/顺心/安能 date→target→Task 3/4/5百世兼容→Task 1业务日期快照→Task 6 Step 5周期调度不动→无需 taskspec 明示合法性校验→Task 7验证→Task 8。✅ 全覆盖。
2. **占位符**:无 TBD/TODO每步含完整代码或精确命令。✅
3. **类型/签名一致**`date=None` 贯穿 server→runtime→sitesactual implyunda/shunxin/anneng原本无 force本计划只加 `date`,与 runtime 调用 `download_func(pg, force=, date=)` / 安能 `lambda(ctx,force,date)` 一致shunxin 双账号 `(pages, foreground, force, date)``_web_handler` 的 list 分支一致。✅
4. **顺序安全**Task 1-5 向后兼容date=None 走 offsetTask 6 启用透传但 server 未传 date仍 NoneTask 7 启用 date。✅

View File

@@ -0,0 +1,332 @@
# 四站点差缺对比逻辑审查报告
> 审查日期2026-07-31
> 审查范围:顺心、中通、韵达、安能 四个站点的应到 vs 实到差缺对比逻辑
> 排除:百世(站点直供未到明细,不参与四站比对)
---
## 一、比对算法总览(四站共用)
`compare.py:process()` 对四个站点执行**完全相同**的算法步骤。站点间的差异仅由 `domain.py:STATIONS` 配置注入——列名映射 + 实到单号解析器。
```
步骤1: 读应到Excel → 按运单号去重keep-first → 构建 {运单号 → (交接单号, 交接件数=n)}
步骤2: 读实到Excel → 站点专用解析器 → 构建 {运单基号 → {已到单号集合}}
步骤3: 逐运单比对
arrived_cnt >= n → 足额到货,跳过
arrived_cnt == 0 → 完全未到
0 < arrived < n → 部分未到
步骤4: 产出未到明细(交接单号 | 运单号 | 总件数 | 已到单号1 | 已到单号2 | ...
```
### 核心口径
| 指标 | 口径 |
|------|------|
| 应到件数 | **交接件数**(非录单件数);按运单号去重 keep-first |
| 实到件数 | 单号去重计数(每扫描一件=一个单号) |
| 未到件数 | max(0, 应到件数 实到件数) |
| 未到率 | 未到件数 ÷ 应到件数 |
### 未到明细输出约定
- 仅列出**短少运单**(实到 < 应到)
- 列出该运单**实际已到的单号**已到单号1, 已到单号2, ...
- **不编造缺件子单号**——实到扫描顺序号乱序,无法反推缺了哪个顺序号
### 统计指标
| 指标 | 含义 |
|------|------|
| 运单数 | 应到运单去重数 |
| 应到件 | Σ 交接件数 |
| 已到件 | Σ 实到单号去重数 |
| 未到件 | max(0, 应到件 已到件) |
| 涉及运单 | 存在短少的运单数 |
| 完全未到 | 整单零到货运单数 |
| 部分未到 | 部分缺件运单数 |
---
## 二、四站配置对照
`domain.py:STATIONS` — 所有差异集中于此配置表,比对核心代码不感知站点差异。
| 维度 | 中通 | 顺心 | 韵达 | 安能 |
|------|------|------|------|------|
| 应到文件 | `中通-应到货物数据.xlsx` | `顺心-应到货物数据.xlsx` | `韵达-应到货物数据.xlsx` | `安能-应到货物数据.xlsx` |
| 实到文件 | `中通-实到货物数据.xlsx` | `顺心-实到货物数据.xlsx` | `韵达-实到货物数据.xlsx` | `安能-实到货物数据.xlsx` |
| 应到-运单号列 | `运单号` | `运单号` | `运单号` | `运单号` |
| 应到-件数列 | `交接件数` | `交接件数` | `交接件数` | `交接件数` |
| 应到-交接单号列 | `交接单号` | `交接单号` | `交接单号` | `交接单号` |
| 实到-基号列 | —(从复合串推导) | `运单号` | **`主单号`** | **`所属单号`** |
| 实到-单号列 | `运单号`(复合串) | `子单号` | `子单号` | `扫描单号` |
| 解析器 | `arrived_pieces_zhongtong` | `arrived_pieces_by_cols` | `arrived_pieces_by_cols` | `arrived_pieces_by_cols` |
---
## 三、逐站点详细分析
### 3.1 中通ZTO
#### 业务逻辑
实到货物数据中的「运单号」为复合串,由三部分构成:
```
┌──────────┬────────────┬──────────┐
│ 运单号 │ 录单件数 │ 顺序号 │
│ (12位) │ (4位) │ (4位) │
└──────────┴────────────┴──────────┘
总长 20 位
示例: 330953527953 0001 0001
├─ 运单号 ─┤├录单┤├顺序┤
```
- **运单号(12位)**: 与应到货物数据中的运单号对齐
- **录单件数(4位)**: 该运单在系统中的录单总件数0占位
- **顺序号(4位)**: 0占位`0001`, `0002`, `0003`, `0004`
对比逻辑:
1. 从应到数据取运单号 + 交接件数(**非录单件数**
2. 从实到数据取复合串掐尾8位得运单基号完整串为子运单号
3. 按运单基号分组,子运单号去重得实到件数
4. 实到件数 < 交接件数 → 差缺
> **重要**: 录单件数仅作参考。举例:某运单录单件数=4、交接件数=2实到最多出现2条数据。如果只出现了1条我们只知道差缺了但**无法判断具体差缺了哪一件**(顺序号乱序)。
#### 代码实现
`domain.py:17-26` — 实到解析器:
```python
def arrived_pieces_zhongtong(df):
res = defaultdict(set)
for v in df["运单号"]:
v = str(v).strip()
if len(v) > 8 and v[-4:].isdigit():
res[v[:-8]].add(v) # 基号=前12位, 已到单号=完整20位复合串
return res
```
`domain.py:48-56` — 站点配置:
```python
{
"name": "中通",
"exp_qty": "交接件数", # 应到件数口径:交接件数(非录单件数)
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_zhongtong,
"columns": ["交接单号", "运单号", "总件数"],
}
```
#### 对齐情况:✅ 对齐
代码实现与业务逻辑一致。`v[:-8]` 掐尾8位得12位运单基号保留完整复合串作为已到单号——不解析、不推断录单件数和顺序号的具体含义。
---
### 3.2 安能Anneng
#### 业务逻辑
与中通相同的差缺对比逻辑。
安能实到数据同样为复合串,结构:`运单号(12位) + 录单件数(4位) + 顺序号(4位)`20位
与中通的关键区别:安能实到表有**独立的「所属单号」列**干净运单基号无需像中通那样从复合串掐尾8位推导基号。
#### 代码实现
`domain.py:76-86`
```python
{
"name": "安能",
"arrived_pieces": arrived_pieces_by_cols("所属单号", "扫描单号"),
...
}
```
安能使用 `arrived_pieces_by_cols` 而非 `arrived_pieces_zhongtong`——直接从「所属单号」列读基号、从「扫描单号」列读完整单号,效果等价。
| 差异点 | 中通 | 安能 |
|--------|------|------|
| 实到基号来源 | 从复合串解析(`v[:-8]` | 直接读「所属单号」列 |
| 实到单号来源 | 复合串本身(「运单号」列) | 「扫描单号」列 |
| 解析器 | `arrived_pieces_zhongtong` | `arrived_pieces_by_cols` |
| 最终产出 | `{基号 → {完整单号集合}}` | 相同 |
#### 数据库验证
```
piece_no=61003282264500140014 → waybill_no=610032822645 (12位), total=0014, seq=0014
```
#### 对齐情况:✅ 对齐
---
### 3.3 顺心Shunxin
#### 业务逻辑
顺心站点需区分两类运单:
**A. 非SF开头运单占 97%:**
实到「子单号」结构为两部分:
```
┌──────────┬──────────┐
│ 运单号 │ 顺序号 │
│ (不定长) │ (3位) │
└──────────┴──────────┘
示例: S71623721115 001
├─ 运单号 ──┤├顺序┤
注意:顺心子单号无录单件数部分(仅两部分)
```
对比时从实到取「子单号」列,按「运单号」分组,子单号去重得实到件数。
**B. SF开头运单占 3%:**
SF订单的「子单号」为**随机号码**(非由运单号衍生),不能用于差缺推导。
对比逻辑:
1. 在实到数据中按「运单号」字段查找,统计出现次数
2. 出现次数 < 交接件数 → 差缺
3. 将找到的子单号(虽随机但可以列出来)填入「已到单号」列
SF订单的差缺判定**只基于交接件数与实到运单号出现次数的比较**,不依赖子单号的结构解析。
#### 代码实现
`domain.py:57-65`
```python
{
"name": "顺心",
"arrived_pieces": arrived_pieces_by_cols("运单号", "子单号"),
}
```
**SF 与非 SF 没有任何区分处理。** 所有运单走同一条路径。
#### 数据库验证
**非SF正常:**
```
子单号=S71623721115001 → 运单号=S71623721115 + 后缀=001 ✅
子单号=S71934073996002 → 运单号=S71934073996 + 后缀=002 ✅
```
**SF异常:**
```
运单号=SF1225002296515 的两条实到记录:
子单号=SF2025318183224 (随机SF号码)
子单号=SF1225002296515 (与运单号相同)
```
数据中有 10 个SF运单存在多条实到记录。
#### 对齐情况:⚠️ 部分对齐SF特殊逻辑缺失
| 检查项 | 代码现状 | 业务要求 |
|--------|----------|----------|
| 非SF处理 | ✅ `arrived_pieces_by_cols("运单号", "子单号")` | 一致 |
| 非SF子单号结构 | ✅ 运单号 + 顺序号(两部分) | 一致 |
| SF处理 | ❌ 与非SF完全一致使用子单号去重 | **不能**使用子单号,只按运单号行数计数 |
| 功能影响 | 子单号虽随机但值唯一,按目前逻辑也能正确去重计数 | 但语义不正确——SF子单号不由运单号衍生 |
---
### 3.4 韵达Yunda
#### 业务逻辑
**去重规则:** 韵达实到数据存在重复行(同一子单号出现两次)。去重依据为「交接单号」字段:
- **保留**交接单号为**空**的行
- **丢弃**交接单号**非空**的行
**子单号结构:** 两部分——单号 + 顺序号(无录单件数部分)。
```
┌──────────┬──────────┐
│ 主单号 │ 顺序号 │
│ (不定长) │ (4位) │
└──────────┴──────────┘
示例: 713326603 0003
├─主单号─┤├顺序┤
```
**对比方式:** 与中通/安能同——按「主单号」分组,「子单号」去重得实到件数,与交接件数比对。
#### 代码实现
`store.py:316-320`(入库过滤):
```python
if site == "韵达":
# 韵达业务清洗:抛弃「交接单号」为空的行(派件/签收等其他扫描无交接单号),
# 再按子单号去重一件多扫只留一条清洗后子单号已天然唯一drop 为保险)。
df = df[df["交接单号"].astype(str).str.strip() != ""] # ← 保留非空
df = df.drop_duplicates(subset=[cm["piece"]], keep="last")
```
`domain.py:67-75`(比对配置):
```python
{
"name": "韵达",
"exp_wb": "运单号",
"arrived_pieces": arrived_pieces_by_cols("主单号", "子单号"),
}
```
#### 对齐情况:❌ 交接单号过滤逻辑完全相反
| 检查项 | 代码现状 | 业务要求 |
|--------|----------|----------|
| 交接单号过滤 | 保留 `!= ""`**非空** | 保留 `== ""`**空** |
| 子单号结构 | ✅ `7133266030003` = wb`713326603` + seq`0003` | 一致 |
| 实到解析 | ✅ `arrived_pieces_by_cols("主单号", "子单号")` | 一致 |
| compare.py 过滤 | ❌ **无过滤**,所有行参与比对 | 需要过滤 |
**影响分析:**
1. `store.py` 过滤反了——入库时留下了错误的数据集
2. `compare.py` 完全没有交接单号过滤——如果原始 Excel 中同时存在空和非空行,比对阶段会全部读入导致重复计数
3. 当前数据库中韵达 3483 条记录全部为非空交接单号——说明当前 Excel 数据中空交接单号行偏少或不存在,但这不改变逻辑错误
---
## 四、差异汇总
| # | 站点 | 问题 | 严重程度 | 影响范围 |
|---|------|------|----------|----------|
| 1 | **韵达** | 交接单号过滤反了:`!= ""` 应改为 `== ""` | ❌ 严重 | `store.py:319` + `compare.py` 需新增过滤 |
| 2 | **顺心** | SF运单无特殊处理与非SF混用子单号 | ⚠️ 中等 | `domain.py` 需新增SF判断分支 |
| 3 | **中通** | 录单件数0占位描述与实际数据完全一致 | ✅ 无影响 | 代码不依赖此区分 |
---
## 五、代码位置索引
| 逻辑 | 文件 | 行号 |
|------|------|------|
| 单站比对 `process()` | `compare.py` | 61-143 |
| 站点配置 `STATIONS` | `domain.py` | 46-87 |
| 中通实到解析器 | `domain.py` | 17-26 |
| 通用实到解析器 | `domain.py` | 29-42 |
| 单站未到文件写入 | `compare.py` | 258-273 |
| 全量汇总报告 | `compare.py` | 293-324 |
| 未到触发编排 | `runtime.py` | 478-497 |
| 韵达入库过滤(需修) | `store.py` | 316-320 |
| 顺心实到配置(需修) | `domain.py` | 57-65 |

View File

@@ -0,0 +1,264 @@
# 顺心 DB 差缺对比 — 实施计划
> 日期2026-07-31
> 目标:将顺心站点差缺对比从 Excel 读取改为 PostgreSQL 查询,并修正 SF 运单特殊处理逻辑
---
## 一、背景
### 当前状态Excel 方式)
```
compare.py:process("顺心")
├── 读 downloads/顺心-应到货物数据.xlsx
├── 读 downloads/顺心-实到货物数据.xlsx
├── arrived_pieces_by_cols("运单号", "子单号") ← SF/non-SF 无区分
└── 产出 {站}-未到数据.xlsx + 统计 dict
```
### 需要解决的两个问题
1. **从 Excel 切换到 DB**:数据已持久化到 PostgreSQL比对应直接从 DB 查询
2. **顺心 SF 运单特殊处理**SF 运单的子单号为随机号码,不能用于去重计数,应使用行计数
---
## 二、数据结构
### PostgreSQL 表
**expected_record**(关键列):
| 列 | 类型 | 说明 |
|----|------|------|
| site | TEXT | 站点 |
| waybill_no | TEXT | 运单号唯一键之一SF 以 "SF" 开头) |
| handover_no | TEXT | 交接单号(批次标识) |
| handover_pieces | INTEGER | 交接件数(应到口径) |
| order_pieces | INTEGER | 录单件数(参考) |
| business_date | DATE | 下载目标日期 |
**actual_record**(关键列):
| 列 | 类型 | 说明 |
|----|------|------|
| site | TEXT | 站点 |
| waybill_no | TEXT | 运单基号(关联 expected_record |
| piece_no | TEXT | 扫描单号non-SF运单号+顺序号SF随机号码 |
| scan_time | TIMESTAMPTZ | 扫描时间(可靠,当天数据=当天扫描) |
### SF 数据特征(已验证)
- 顺心 actual_record 中 SF 运单148 条
- `piece_no == waybill_no`86 条58%
- `piece_no != waybill_no`62 条42%)← 随机 SF 号码
- SF 运单 expected99 条,分布在 31 个交接批次中
---
## 三、算法设计
### 核心思路:以实到为锚,通过交接单号反推批次
```
输入: site="顺心", date="2026-07-25"
Step 1 — 取实到锚点
SELECT DISTINCT waybill_no FROM actual_record
WHERE site='顺心' AND scan_time::date = '2026-07-25'
Step 2 — 反推交接批次
SELECT DISTINCT handover_no FROM expected_record
WHERE site='顺心'
AND waybill_no IN (Step 1 的运单集合)
Step 3 — 展开批次全量应到
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site='顺心'
AND handover_no IN (Step 2 的交接单号集合)
Step 4 — 取批次全量实到
SELECT waybill_no, piece_no FROM actual_record
WHERE site='顺心'
AND waybill_no IN (Step 3 的运单集合)
Step 5 — 逐运单比对
for each waybill in Step 3:
if waybill_no LIKE 'SF%':
arrived_cnt = COUNT(*) ← 行计数,不去重
else:
arrived_cnt = COUNT(DISTINCT piece_no) ← 子单号去重
if arrived_cnt < handover_pieces → 差缺
```
### SF vs non-SF 处理差异
| | non-SF | SF |
|------|--------|-----|
| piece_no 含义 | 运单号 + 顺序号(可推导) | 随机 SF 号码(无推导意义) |
| 实到计数方式 | `COUNT(DISTINCT piece_no)` | `COUNT(*)`(行计数) |
| 已到单号列表 | 列出去重后的子单号 | 列出所有 piece_no含重复 |
### 统计指标
| 指标 | 公式 |
|------|------|
| 运单数 | Step 3 去重运单数 |
| 应到件 | Σ handover_pieces |
| 已到件 | Σ arrived_cnt |
| 未到件 | max(0, 应到件 已到件) |
| 涉及运单 | arrived_cnt < handover_pieces 的运单数 |
| 完全未到 | arrived_cnt = 0 的运单数 |
| 部分未到 | 0 < arrived_cnt < handover_pieces 的运单数 |
| 未到率 | 未到件 ÷ 应到件 |
### 边界情况覆盖
| 情况 | 覆盖方式 |
|------|----------|
| 同日多批次 | Step 2 查出全部涉及的 handover_no |
| 跨天到达(延迟) | Step 4 不限 scan_time历史扫描全计入 |
| 溢到(实到 > 应到) | arrived_cnt >= n 跳过,不进差缺表 |
| 完全沉默批次 | 一件未扫 = 实到无锚点,该批次不会被触发——在首次有扫描那天被纳入 |
| SF 子单号重复 | 用 COUNT(*) 而非 COUNT(DISTINCT),不会漏计 |
---
## 四、模块设计
### 新增文件
**`inbound_verify/db_compare.py`** — DB 比对引擎(纯 PostgreSQL + Python
```python
# 核心函数签名
def compare_site_date(site: str, date: str) -> CompareResult | None:
"""对指定站点和日期执行 DB 差缺比对。
返回 CompareResultstats + undelivered_rows
当天无实到数据时返回 None。
"""
def compare_site_batch(site: str, handover_no: str) -> CompareResult | None:
"""按指定交接单号执行全批次比对(不依赖实到锚点)。"""
```
**数据类型**
```python
@dataclass
class CompareResult:
stats: dict # 统计指标
rows: list[dict] # 差缺明细行
batches: list[str] # 涉及的交接批次
@dataclass
class UndeliveredRow:
handover_no: str # 交接单号
waybill_no: str # 运单号
total_pieces: int # 总件数(=交接件数)
arrived_pieces: int # 已到件数
arrived_list: list[str] # 已到单号列表
is_sf: bool # 是否 SF 运单
```
### 修改文件
**`inbound_verify/cli/server.py`** — 新增 API 端点
```python
@app.post("/compare")
def run_compare(req: CompareRequest):
"""DB 比对:{site, date} → 返回差缺结果"""
@app.get("/compare/{site}/{date}")
def get_compare(site: str, date: str):
"""查询某站点某日的差缺结果(缓存)"""
```
### 现有文件保持不动
- `compare.py` — 保留不动Excel 比对继续可用
- `domain.py` — 可能需要新增 DB 版站点配置(或复用现有)
- `runtime.py` — 暂不改动,`_site_undelivered_handler` 仍走 Excel 路径
---
## 五、实施步骤
### Phase 1 — `db_compare.py` 核心引擎
- [ ] 新建 `inbound_verify/db_compare.py`
- [ ] 实现 `compare_site_date("顺心", date)`
- [ ] SF/non-SF 分支处理
- [ ] 返回 `CompareResult`
- [ ] 终端手动验证(直接调函数,打印结果)
### Phase 2 — API 端点
- [ ]`server.py` 新增 `POST /compare`
- [ ] `CompareRequest { site, date }`
- [ ] 调用 `db_compare.compare_site_date()`
- [ ] 返回 JSONstats + undelivered rows
- [ ] HTTP 验证curl 调 `/compare` 对比不同日期结果
### Phase 3 — Excel 输出(可选)
- [ ] `db_compare` 生成 Excel 报告(复用现有 `compare.py` 的 openpyxl 样式)
- [ ] 输出到 `output/顺心-{date}-未到数据.xlsx`
- [ ] 或者只输出 JSON前端自行渲染
### Phase 4 — 替换 undelivered 任务流
- [ ] `runtime.py` 新增 `_db_undelivered_handler`
- [ ] 下载完成后不再调 Excel 比对,改调 DB 比对
- [ ] 逐步替换 `TASK_HANDLERS` 中的顺心 undelivered handler
### Phase 5 — 扩展到中通/韵达/安能
- [ ] 各站适配(主要是 piece_no 去重方式差异)
- [ ] 中通:`COUNT(DISTINCT piece_no)`,无 SF 问题
- [ ] 韵达:同上
- [ ] 安能:同上
---
## 六、测试策略
### 手工验证Phase 1
```python
# 终端直接调
from inbound_verify.db_compare import compare_site_date
result = compare_site_date("顺心", "2026-07-25")
print(result.stats)
# 对比基于 Excel 版的 compare.process("顺心") 结果
```
### API 验证Phase 2
```bash
curl -X POST http://127.0.0.1:8000/compare \
-H "Content-Type: application/json" \
-d '{"site":"顺心","date":"2026-07-25"}'
```
### 回归验证
- 新 DB 比对结果 vs 旧 Excel 比对结果(同一份数据)
- SF 运单的 arrived_cnt 对比DB 版COUNT(*)vs Excel 版COUNT DISTINCT piece_no
- 确认 SF 运单不再被漏计
---
## 七、风险与注意事项
| 风险 | 缓解 |
|------|------|
| DB 连接超时cpolar 隧道) | 加 connect_timeout + try/except 降级 |
| 全表扫描性能 | 依赖 (site, waybill_no) 和 (site, scan_time) 索引 |
| SF 运单数据量小(~1% | 测试覆盖可能不足——需找有 SF 差缺的日期验证 |
| `scan_time` 时区 | 统一用 `::date` cast确认与服务器时区一致 |

View File

@@ -0,0 +1,579 @@
# Tier 1: Package Move Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Move all 12 flat root-level Python modules into an installable `inbound_verify/` package (sites/, cli/ subpackages), rewrite every internal import to package-qualified form, and expose three console_script entry points — with **zero behavior change**.
**Architecture:** Pure mechanical relocation (git mv preserves history) + import rewrite + paths.py anchor fix + entry-point `main()` wrappers + `pyproject.toml` packaging. No logic changes. Module names kept for heavily-referenced modules (`state_store`, `expected_undelivered`, `runtime`, `paths`) to avoid call-site churn; only leaf entries (`db_store→store`, `main_router→cli/router`, `server→cli/server`) and site files (drop `site_` prefix) are renamed.
**Tech Stack:** Python ≥3.10, setuptools (PEP 517/621), pip editable install, psycopg3, FastAPI/uvicorn, Playwright.
## Global Constraints
- **Python ≥ 3.10** (`requires-python = ">=3.10"` in pyproject).
- **Package** import name `inbound_verify`; **distribution** name `inbound-verify`.
- **Dependency floors** (verbatim from spec): `pandas>=2.0.0`, `playwright>=1.40.0`, `openpyxl>=3.1.0`, `PyYAML>=6.0`, `websocket-client>=1.0.0`, `fastapi>=0.110.0`, `uvicorn>=0.27.0`, `apscheduler>=3.10.0`, `psycopg[binary]>=3.1`.
- **No test suite** (user decision). Verification = `compileall` + import smoke + grep-for-stale-refs + DB connectivity. No pytest.
- **Behavior must not change** in Tier 1 — pure move.
- **No auto-commit/push.** Every commit step below runs ONLY after the user explicitly says "提交/commit". Commit messages in English.
- **Black-format** every changed `.py` (global rule).
- All changes are **inside the `InboundVerify` git submodule**; the parent repo pointer bump is a separate parent-repo step, out of scope.
- All commands run from the `InboundVerify/` directory using the venv interpreter `.venv/Scripts/python.exe` (Windows; no activation needed).
**Reference spec:** `docs/superpowers/specs/2026-07-23-package-restructure-design.md` (§3 mapping table, §4 paths anchor, §5 import rules, §6 entry/packaging, §7 verification gate).
---
## File Structure (what each file becomes responsible for)
```
inbound_verify/
├── __init__.py # empty (package marker)
├── paths.py # path anchors → PROJECT ROOT (one dir above package)
├── runtime.py # orchestration core (unchanged logic)
├── state_store.py # SQLite state (unchanged; name kept)
├── expected_undelivered.py# offline compare (unchanged; name kept — Tier 2 renames to compare)
├── store.py # PostgreSQL persist + main() (was db_store.py)
├── sites/
│ ├── __init__.py # empty
│ ├── shunxin.py # (was site_shunxin.py)
│ ├── baishi.py # (was site_baishi.py)
│ ├── zto.py # (was site_zto.py)
│ ├── yunda.py # (was site_yunda.py)
│ └── anneng.py # (was site_anneng.py)
└── cli/
├── __init__.py # empty
├── router.py # interactive menu + main() (was main_router.py)
└── server.py # FastAPI service + main() (was server.py)
```
Root keeps: `pyproject.toml` (new), `config.yaml`, `config.example.yaml`, `schema.sql`, `requirements.txt`, `README.md`, `CLAUDE.md`, `docs/`, `downloads/`, `output/`, `state/`.
---
## Task 1: Package scaffold + pyproject + editable install
**Files:**
- Create: `inbound_verify/__init__.py`, `inbound_verify/sites/__init__.py`, `inbound_verify/cli/__init__.py`
- Create: `pyproject.toml`
**Interfaces:**
- Produces: an importable (near-empty) `inbound_verify` package + console_script registration. The flat root scripts remain 100% functional after this task (untouched).
- [ ] **Step 1: Create package marker files**
Create three empty files:
- `inbound_verify/__init__.py`
- `inbound_verify/sites/__init__.py`
- `inbound_verify/cli/__init__.py`
Each is a single comment line:
```python
# inbound_verify package
```
- [ ] **Step 2: Write pyproject.toml**
Create `pyproject.toml`:
```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "inbound-verify"
version = "0.1.0"
description = "物流到货数据自动下载与应到未到核对工具"
requires-python = ">=3.10"
dependencies = [
"pandas>=2.0.0",
"playwright>=1.40.0",
"openpyxl>=3.1.0",
"PyYAML>=6.0",
"websocket-client>=1.0.0",
"fastapi>=0.110.0",
"uvicorn>=0.27.0",
"apscheduler>=3.10.0",
"psycopg[binary]>=3.1",
]
[project.scripts]
inbound-verify = "inbound_verify.cli.router:main"
inbound-verify-server = "inbound_verify.cli.server:main"
inbound-verify-db = "inbound_verify.store:main"
[tool.setuptools.packages.find]
include = ["inbound_verify*"]
```
- [ ] **Step 3: Editable-install into the existing venv**
Run:
```bash
.venv/Scripts/python.exe -m pip install -e .
```
Expected: `Successfully installed inbound-verify-0.1.0` (deps already satisfied from earlier install — no network needed).
- [ ] **Step 4: Verify package imports**
Run:
```bash
.venv/Scripts/python.exe -c "import inbound_verify, inbound_verify.sites, inbound_verify.cli; print('package OK')"
```
Expected output: `package OK`
- [ ] **Step 5: Commit (only after user confirms)**
```bash
git add inbound_verify/__init__.py inbound_verify/sites/__init__.py inbound_verify/cli/__init__.py pyproject.toml
git commit -m "chore: scaffold inbound_verify package and pyproject
Co-Authored-By: Claude <noreply@anthropic.com>"
```
> Do NOT commit until the user says to.
---
## Task 2: Atomic move — relocate, rewrite imports, fix anchor, wire mains, verify
**Files:**
- Move (git mv): all 12 root `.py` modules → package locations (see Step 1)
- Modify: `inbound_verify/paths.py` (anchor), and import lines + call sites in every moved module
- Modify: entry `main()` in `cli/router.py`, `cli/server.py`, `store.py`
**Interfaces:**
- Consumes: the scaffold from Task 1 (package importable + editable install active).
- Produces: a fully functional `inbound_verify` package invokable via `python -m inbound_verify.cli.router`, `python -m inbound_verify.cli.server`, `python -m inbound_verify.store`, or the three console_scripts. Old root `.py` files are gone. The flat root scripts no longer exist — invocation switches to package form.
> **Why this is one task:** in a flat-import codebase, moving `paths.py` (imported by everyone) immediately breaks every importer until ALL moves + rewrites are complete. There is no intermediate state that imports cleanly, so the whole move is one atomic unit verified by the gate at the end. Each file-edit step below is followed by `py_compile` of that file to catch syntax errors as we go.
- [ ] **Step 1: Relocate all 12 modules with git mv (history preserved)**
From the `InboundVerify/` directory:
```bash
git mv paths.py inbound_verify/paths.py
git mv runtime.py inbound_verify/runtime.py
git mv state_store.py inbound_verify/state_store.py
git mv expected_undelivered.py inbound_verify/expected_undelivered.py
git mv db_store.py inbound_verify/store.py
git mv main_router.py inbound_verify/cli/router.py
git mv server.py inbound_verify/cli/server.py
git mv site_shunxin.py inbound_verify/sites/shunxin.py
git mv site_baishi.py inbound_verify/sites/baishi.py
git mv site_zto.py inbound_verify/sites/zto.py
git mv site_yunda.py inbound_verify/sites/yunda.py
git mv site_anneng.py inbound_verify/sites/anneng.py
```
After this the tree is temporarily broken (imports unresolved) — expected. Continue.
- [ ] **Step 2: Fix paths.py anchor to point at project root**
In `inbound_verify/paths.py`, replace:
```python
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
```
with:
```python
# __file__ = <root>/inbound_verify/paths.py → 上两级 = 项目根
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
```
(`DOWNLOAD_DIR`/`OUTPUT_DIR`/`CONFIG_PATH`/`STATE_DB_PATH` lines stay unchanged — they derive from BASE_DIR.)
Verify syntax:
```bash
.venv/Scripts/python.exe -m py_compile inbound_verify/paths.py
```
Expected: no output (success).
- [ ] **Step 3: Rewrite imports in state_store.py**
In `inbound_verify/state_store.py`, replace:
```python
from paths import STATE_DB_PATH
```
with:
```python
from inbound_verify.paths import STATE_DB_PATH
```
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/state_store.py` → no output.
- [ ] **Step 4: Rewrite imports + call sites in runtime.py**
In `inbound_verify/runtime.py`, replace the import block:
```python
from paths import DOWNLOAD_DIR, CONFIG_PATH
import state_store
import site_shunxin
import site_baishi
import site_zto
import site_yunda
import site_anneng
import expected_undelivered # dispatch 的 compare 任务用
```
with:
```python
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng
from inbound_verify import expected_undelivered # dispatch 的 compare 任务用
```
Then **drop the `site_` prefix at every call site** (28 references — 18 in this file). Apply these 5 replacements (all occurrences):
- `site_shunxin.``shunxin.`
- `site_baishi.``baishi.`
- `site_zto.``zto.`
- `site_yunda.``yunda.`
- `site_anneng.``anneng.`
Affected lines in runtime.py (for reference, all must change): 36, 37, 38, 39, 118, 139, 336, 408, 417, 463, 464, 467, 469, 470, 472, 473, 475, 476.
`state_store.` and `expected_undelivered.` call sites stay UNCHANGED (names kept). Confirm the key handler block now reads:
```python
TASK_HANDLERS = {
("顺心", "expected"): _web_handler("顺心", shunxin.shunxin_expected_download),
("顺心", "actual"): _web_handler("顺心", shunxin.shunxin_actual_download),
("顺心", "undelivered"): _site_undelivered_handler("顺心"),
("百世", "undelivered"): _web_handler(
"百世", baishi.baishi_download_undelivered_data
),
("中通", "expected"): _web_handler("中通", zto.zto_expected_download),
("中通", "actual"): _web_handler("中通", zto.zto_actual_download),
("中通", "undelivered"): _site_undelivered_handler("中通"),
("韵达", "expected"): _web_handler("韵达", yunda.yunda_expected_download),
("韵达", "actual"): _web_handler("韵达", yunda.yunda_actual_download),
("韵达", "undelivered"): _site_undelivered_handler("韵达"),
("安能", "expected"): lambda ctx: anneng.anneng_expected_download(),
("安能", "actual"): lambda ctx: anneng.anneng_actual_download(),
("安能", "undelivered"): _site_undelivered_handler("安能"),
("__compare__", "compare"): lambda ctx: (expected_undelivered.main() or True),
}
```
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/runtime.py` → no output.
- [ ] **Step 5: Rewrite imports in expected_undelivered.py**
This module has TWO lazy `import state_store` statements (inside functions). Replace each occurrence of:
```python
import state_store
```
with:
```python
from inbound_verify import state_store
```
(There is no `from paths import` here — the module has its own `BASE/DOWNLOADS/OUTPUT` constants; that dedup is Tier 2, not now.)
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/expected_undelivered.py` → no output.
- [ ] **Step 6: Rewrite imports + rename _cli→main in store.py**
In `inbound_verify/store.py`, replace:
```python
from paths import BASE_DIR, CONFIG_PATH, DOWNLOAD_DIR
```
with:
```python
from inbound_verify.paths import BASE_DIR, CONFIG_PATH, DOWNLOAD_DIR
```
Replace:
```python
import expected_undelivered as eu # 复用站点 / 文件名 / 列映射 / 基号口径(单一来源)
```
with:
```python
from inbound_verify import expected_undelivered as eu # 复用站点 / 文件名 / 列映射 / 基号口径(单一来源)
```
(`eu.` call sites stay unchanged.)
Rename the CLI entry: replace the function definition:
```python
def _cli():
```
with:
```python
def main():
```
And at the bottom replace:
```python
if __name__ == "__main__":
_cli()
```
with:
```python
if __name__ == "__main__":
main()
```
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/store.py` → no output.
- [ ] **Step 7: Rewrite imports in all 5 site modules**
In each of `inbound_verify/sites/{shunxin,baishi,zto,yunda,anneng}.py`, replace:
```python
from paths import DOWNLOAD_DIR, CONFIG_PATH
import state_store
```
with:
```python
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
```
(Each site module has exactly these two internal imports; `state_store.` call sites unchanged.)
Verify all five:
```bash
.venv/Scripts/python.exe -m py_compile inbound_verify/sites/shunxin.py inbound_verify/sites/baishi.py inbound_verify/sites/zto.py inbound_verify/sites/yunda.py inbound_verify/sites/anneng.py
```
Expected: no output.
- [ ] **Step 8: Rewrite imports + call sites + add main() in cli/router.py**
In `inbound_verify/cli/router.py`, replace the import block (lines ~1431):
```python
from paths import CONFIG_PATH
from runtime import (
APP_SITES,
HEARTBEAT_INTERVAL,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
import state_store
# 各站点模块(自动化测试 + 比对用;任务派发在 runtime
import site_shunxin
import site_baishi
import site_zto
import site_yunda
import site_anneng
import expected_undelivered
```
with:
```python
from inbound_verify.paths import CONFIG_PATH
from inbound_verify.runtime import (
APP_SITES,
HEARTBEAT_INTERVAL,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
from inbound_verify import state_store
# 各站点模块(自动化测试 + 比对用;任务派发在 runtime
from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng
from inbound_verify import expected_undelivered
```
Drop the `site_` prefix at the 10 call sites (apply the same 5 replacements as Step 4). Affected lines: 60, 61, 64, 65, 68, 69, 72, 73, 86, 105. (`expected_undelivered.main()` at line 38 stays unchanged.)
Add an entry function and update the `__main__` guard. Replace:
```python
if __name__ == "__main__":
run_multi_site_daemon()
```
with:
```python
def main():
"""交互菜单模式入口。"""
run_multi_site_daemon()
if __name__ == "__main__":
main()
```
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/cli/router.py` → no output.
- [ ] **Step 9: Rewrite imports + add main() in cli/server.py**
In `inbound_verify/cli/server.py`, replace:
```python
from paths import DOWNLOAD_DIR, OUTPUT_DIR
import state_store
from runtime import (
HEARTBEAT_INTERVAL,
TASK_HANDLERS,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
```
with:
```python
from inbound_verify.paths import DOWNLOAD_DIR, OUTPUT_DIR
from inbound_verify import state_store
from inbound_verify.runtime import (
HEARTBEAT_INTERVAL,
TASK_HANDLERS,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
```
Replace the bottom entry block:
```python
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
```
with:
```python
def main():
"""服务模式入口。传字符串导入路径(规范写法;不开 reload/workers 时进程内 import行为等价"""
uvicorn.run("inbound_verify.cli.server:app", host="0.0.0.0", port=8000)
if __name__ == "__main__":
main()
```
Verify: `.venv/Scripts/python.exe -m py_compile inbound_verify/cli/server.py` → no output.
- [ ] **Step 10: Clean up stale filename comments**
Cosmetic but keeps grep clean (Step 12 depends on it). In each `inbound_verify/sites/*.py`, update the line-1 header `# site_xxx.py``# sites/xxx.py`. In `inbound_verify/sites/anneng.py`, update the standalone-run comment near the top:
```python
# .venv/Scripts/python.exe site_anneng.py
```
```python
# python -m inbound_verify.sites.anneng expected # 或 actual
```
And the comment at the `CDP_PORT` line referencing "独立运行 site_anneng.py" — update to "独立运行python -m inbound_verify.sites.anneng".
- [ ] **Step 11: Black-format all changed files**
```bash
.venv/Scripts/python.exe -m black inbound_verify
```
Expected: `reformatted ...` / `left unchanged` lines, exit 0.
- [ ] **Step 12: VERIFICATION GATE — run all five checks**
**12a. compileall (syntax across whole package):**
```bash
.venv/Scripts/python.exe -m compileall inbound_verify
```
Expected: no errors.
**12b. Import smoke (catches every wrong import path / missed rewrite):**
```bash
.venv/Scripts/python.exe -c "import inbound_verify.cli.router, inbound_verify.cli.server, inbound_verify.store, inbound_verify.runtime, inbound_verify.state_store; print('import smoke OK')"
```
Expected: `import smoke OK`. (The three entries transitively import sites + expected_undelivered.)
**12c. No stale `site_` references:**
```bash
grep -rn "site_shunxin\|site_baishi\|site_zto\|site_yunda\|site_anneng" inbound_verify || echo "no stale site_ refs OK"
```
Expected: `no stale site_ refs OK`.
**12d. No stale bare flat imports:**
```bash
grep -rnE "^from paths import|^from runtime import|^import site_|^import state_store$|^import expected_undelivered$" inbound_verify || echo "no stale flat imports OK"
```
Expected: `no stale flat imports OK`.
**12e. paths anchor points at project root + DB still connects:**
```bash
.venv/Scripts/python.exe -c "from inbound_verify.paths import BASE_DIR; print('BASE_DIR', BASE_DIR)"
.venv/Scripts/python.exe -c "from inbound_verify.store import _connect, _load_pg_config; c=_load_pg_config(); conn=_connect(c['dbname']); print('DB OK', conn.info.server_version); conn.close()"
```
Expected: `BASE_DIR` prints the `InboundVerify` project root (the dir containing `config.yaml`); `DB OK <pg version>`.
> If 12b fails with ModuleNotFoundError for a site module, run `.venv/Scripts/python.exe -m pip install -e .` again (editable finder refresh) and retry. If 12d still shows a line, that import was missed — rewrite it per Step 4/8 rules.
- [ ] **Step 13: Commit (only after user confirms)**
```bash
git add -A inbound_verify
git commit -m "refactor: move flat modules into inbound_verify package (Tier 1, behavior-identical)
- relocate 12 root .py into inbound_verify/ (sites/, cli/ subpackages)
- rewrite all internal imports to package-qualified
- fix paths.py BASE_DIR to anchor at project root
- add main() entry wrappers; register console_scripts
- drop site_ prefix on site modules; keep state_store/expected_undelivered names
Co-Authored-By: Claude <noreply@anthropic.com>"
```
> Do NOT commit until the user says to. This commit is the **safety baseline**; the manual end-to-end test (spec §7 gate) runs against this state before any Tier 2/Tier 3 work.
---
## Task 3: Docs sync (README + CLAUDE.md)
**Files:**
- Modify: `README.md` (§二 directory tree, §三 env prep, §五 run)
- Modify: `CLAUDE.md` (常用命令 section)
**Interfaces:**
- Consumes: the completed package from Task 2 (docs must describe the real new layout/commands).
- Produces: documentation matching the new invocation model. No code impact.
- [ ] **Step 1: Update README §二 directory tree**
Replace the tree block (README lines ~3349) with the actual new layout:
```
InboundVerify/
├── pyproject.toml # 打包 + 依赖 + console_scripts
├── inbound_verify/ # 源码包
│ ├── paths.py runtime.py state_store.py expected_undelivered.py store.py
│ ├── sites/ shunxin / baishi / zto / yunda / anneng
│ └── cli/ router交互菜单/ serverFastAPI 服务)
├── config.example.yaml / config.yaml
├── schema.sql
├── requirements.txt # pyproject 的静态镜像
├── downloads/ output/ state/
└── docs/
```
- [ ] **Step 2: Update README §三 env prep — add editable install**
In the env-prep command block (README lines ~5871), after `pip install -r requirements.txt`, add:
```bash
# 4. 以可编辑模式安装本包(注册 inbound-verify 等命令)
pip install -e .
```
(renumber the subsequent `cp config.example.yaml config.yaml` step).
- [ ] **Step 3: Update README §五 run — new commands**
Replace `python main_router.py` with:
```bash
# 交互菜单(任选其一)
python -m inbound_verify.cli.router
# 或装包后inbound-verify
```
- [ ] **Step 4: Update CLAUDE.md 常用命令**
In the 常用命令 section, change every `.venv/Scripts/python.exe <module>.py` to the package form:
- `main_router.py``python -m inbound_verify.cli.router` (or `inbound-verify`)
- `server.py``python -m inbound_verify.cli.server` (or `inbound-verify-server`)
- `db_store.py createdb|init|ingest|all``python -m inbound_verify.store createdb|init|ingest|all` (or `inbound-verify-db ...`)
- `site_anneng.py expected|actual``python -m inbound_verify.sites.anneng expected|actual`
- `black`/`py_compile` targets → package paths (e.g. `-m black inbound_verify`)
Add a one-liner near the top of that section: `首次/拉取新代码后需 .venv/Scripts/python.exe -m pip install -e .`
- [ ] **Step 5: Verify docs render + commands are real**
```bash
grep -n "main_router.py\|python server.py\|python db_store.py\|site_anneng.py\|site_shunxin" README.md CLAUDE.md || echo "no stale old-path commands OK"
```
Expected: `no stale old-path commands OK` (every old invocation updated).
- [ ] **Step 6: Commit (only after user confirms)**
```bash
git add README.md CLAUDE.md
git commit -m "docs: update README and CLAUDE.md for package layout and console_scripts
Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Self-Review (completed)
- **Spec coverage:** spec §3 (layout) → Task 1+2; §4 (paths anchor) → Task 2 Step 2; §5 (import rules) → Task 2 Steps 39; §6 (entry/packaging) → Task 1 Step 2 + Task 2 Steps 6/8/9; §7 (verification gate) → Task 2 Step 12; §10 (docs) → Task 3. All covered.
- **Placeholder scan:** none — every step has exact code or an exact command with expected output. The 28 call-site rewrites are given as a deterministic prefix-drop rule + enumerated line numbers + verification grep (complete, not a placeholder).
- **Type/name consistency:** kept-module names (`state_store`, `expected_undelivered`, `runtime`, `paths`) used consistently across all import rewrites and call sites; renamed entries (`store`, `cli/router`, `cli/server`) consistent with pyproject `[project.scripts]`. `main()` signature consistent across router/server/store and console_scripts.

View File

@@ -0,0 +1,403 @@
# Tier 2: domain extract + rename + path dedup — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Three targeted, behavior-preserving cleanups of the compare/config layer: (1) make `expected_undelivered` use `paths.DOWNLOAD_DIR/OUTPUT_DIR` instead of its own duplicate anchor; (2) extract shared site/file/column config into a new `domain` module; (3) rename `expected_undelivered``compare`.
**Architecture:** Pure refactor — move definitions, rewire imports, no logic change. Removes the duplicate path anchor that caused the Tier 1 hotfix bug (`17293be`), and the `db_store → expected_undelivered` coupling where the DB layer imported the whole compare engine just to read site config.
**Tech Stack:** Python ≥3.10, package `inbound_verify`, pandas, openpyxl, psycopg3.
## Global Constraints
- **Python ≥ 3.10**, package import name `inbound_verify`.
- **No behavior change** — pure refactor. Any logic change is a defect.
- **No test suite** (user decision). Per-task verification = `compileall` + **import smoke** (fresh process, no backend needed) + grep-for-stale-refs. NOT pytest. (The running backend is unaffected until a restart; a final end-to-end re-confirm is done once at the end of Tier 2.)
- **No auto-commit** (project rule). Each task's commit step runs only after the user says "提交". Commit messages in English.
- **Black-format** every changed `.py` (global rule).
- All changes inside the `InboundVerify` submodule; commits local on `dev`.
- Venv interpreter: `.venv/Scripts/python.exe` (absolute: `D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe`). Run commands from `D:/projects/LogisticsHubIPA/InboundVerify/`.
**Reference spec:** `docs/superpowers/specs/2026-07-23-package-restructure-design.md` §8 Tier 2.
**Out of scope (deferred):** `config.py` centralization (consolidating the 6 `open(CONFIG_PATH)+yaml.safe_load` sites in store/runtime×2/anneng/router). It is the churniest Tier 2 item (6 files) for the least marginal value — pure DRY, no pain addressed — and under no-tests more churn = more risk. Reconsider after the rest of Tier 2 lands.
---
## File Structure
```
inbound_verify/
├── domain.py # NEW (Task 2): shared site/file/column config — leaf module
├── paths.py # unchanged (already the single anchor)
├── expected_undelivered.py# Task 1: use paths.* ; Task 2: import config from domain ; Task 3: renamed → compare.py
├── compare.py # (after Task 3) was expected_undelivered.py — compare engine + report
├── store.py # Task 2: import config from domain (not eu); Task 3: import _read_business_dates from compare
├── runtime.py # Task 2: SITE_UNDELIVERED_FILE from domain; Task 3: write_site_file from compare
└── cli/router.py # Task 3: compare.main()
```
**`domain.py` responsibility:** pure data — `ALL_REPORT_SITES`, `SITE_UNDELIVERED_FILE`, `BAISHI_FILE`, `BAISHI_COLUMNS`, `arrived_pieces_zhongtong`, `arrived_pieces_by_cols`, `STATIONS`, `_site_cfg`. No `state_store` dependency, no file I/O. Leaf module.
---
## Task 1: Path dedup — expected_undelivered uses paths.DOWNLOAD_DIR/OUTPUT_DIR
**Files:**
- Modify: `inbound_verify/expected_undelivered.py` (lines 43-48 defs; usages at 146,147,298,338,374,400,669,675,676,690)
**Interfaces:**
- Consumes: `paths.DOWNLOAD_DIR`, `paths.OUTPUT_DIR` (already exist, anchored at project root).
- Produces: `expected_undelivered` no longer defines `BASE/DOWNLOADS/OUTPUT`; keeps `OUTFILE` (now `= join(OUTPUT_DIR, "应到未到数据.xlsx")`). The 6 `DOWNLOADS` and 1 `OUTPUT` usages point at the `paths.*` constants — same resolved values as the current hotfixed anchor, so behavior identical.
- [ ] **Step 1: Add the paths import to the top import block**
In `inbound_verify/expected_undelivered.py`, after the existing `import os` (top of file), add a line importing the two path constants. (The file currently has no `paths` import — it used its own BASE.)
Add:
```python
from inbound_verify.paths import DOWNLOAD_DIR, OUTPUT_DIR
```
(place it alongside the other imports near the top, e.g. right after `import os`.)
- [ ] **Step 2: Replace the self-anchored BASE/DOWNLOADS/OUTPUT/OUTFILE block**
Replace this block (currently lines ~43-48):
```python
# 包内文件:上两级 = 项目根(与 paths.BASE_DIR 一致;站点下载落在 <root>/downloads
# 注:本模块自带锚点是 paths.py 的重复Tier 2 计划改为直接引用 paths.DOWNLOAD_DIR/OUTPUT_DIR。
BASE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DOWNLOADS = os.path.join(BASE, "downloads")
OUTPUT = os.path.join(BASE, "output")
OUTFILE = os.path.join(OUTPUT, "应到未到数据.xlsx")
```
with:
```python
# 比对报表输出文件(路径锚定统一走 paths.py
OUTFILE = os.path.join(OUTPUT_DIR, "应到未到数据.xlsx")
```
- [ ] **Step 3: Replace all `DOWNLOADS` usages with `DOWNLOAD_DIR`**
Apply `DOWNLOADS``DOWNLOAD_DIR` (6 occurrences, at lines 146, 147, 298, 338, 669, 675, 676). Use replace-all on the token `DOWNLOADS`.
- [ ] **Step 4: Replace the remaining `OUTPUT` usage (the makedirs line)**
At line ~374, replace:
```python
os.makedirs(OUTPUT, exist_ok=True)
```
with:
```python
os.makedirs(OUTPUT_DIR, exist_ok=True)
```
(`OUTFILE` at lines 400/690 is unchanged — it's a different token, already redefined in Step 2.)
- [ ] **Step 5: Verify syntax + no stale BASE/DOWNLOADS/OUTPUT**
```bash
cd /d/projects/LogisticsHubIPA/InboundVerify
.venv/Scripts/python.exe -m py_compile inbound_verify/expected_undelivered.py
grep -nE "\b(BASE|DOWNLOADS|OUTPUT)\b" inbound_verify/expected_undelivered.py || echo "no stale BASE/DOWNLOADS/OUTPUT OK"
```
Expected: py_compile silent; grep prints `no stale BASE/DOWNLOADS/OUTPUT OK` (OUTFILE is a different token, won't match).
- [ ] **Step 6: Import smoke + paths equivalence**
```bash
.venv/Scripts/python.exe -c "from inbound_verify import expected_undelivered as eu; from inbound_verify.paths import DOWNLOAD_DIR, OUTPUT_DIR; import os; print('OUTFILE dir matches OUTPUT_DIR:', os.path.dirname(eu.OUTFILE)==OUTPUT_DIR)"
.venv/Scripts/python.exe -c "import inbound_verify.cli.router, inbound_verify.cli.server, inbound_verify.store, inbound_verify.runtime; print('import smoke OK')"
```
Expected: `OUTFILE dir matches OUTPUT_DIR: True` and `import smoke OK`.
- [ ] **Step 7: Black + commit (after user confirms)**
```bash
.venv/Scripts/python.exe -m black inbound_verify/expected_undelivered.py
git add inbound_verify/expected_undelivered.py
git commit -m "refactor: expected_undelivered uses paths.DOWNLOAD_DIR/OUTPUT_DIR (Tier 2)
Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 2: Extract `domain.py` (shared site/file/column config)
**Files:**
- Create: `inbound_verify/domain.py`
- Modify: `inbound_verify/expected_undelivered.py` (remove the moved block, import from domain)
- Modify: `inbound_verify/store.py` (config from domain, not eu)
- Modify: `inbound_verify/runtime.py` (SITE_UNDELIVERED_FILE from domain)
**Interfaces:**
- Consumes: nothing new (moves existing definitions verbatim).
- Produces: `inbound_verify.domain` exposing `ALL_REPORT_SITES`, `SITE_UNDELIVERED_FILE`, `BAISHI_FILE`, `BAISHI_COLUMNS`, `arrived_pieces_zhongtong(df)`, `arrived_pieces_by_cols(wb_col, piece_col)`, `STATIONS` (list of dicts), `_site_cfg(name)`. These are the exact same objects that lived in `expected_undelivered` lines 50-135.
- [ ] **Step 1: Create `inbound_verify/domain.py`**
Create `inbound_verify/domain.py` with this exact content (moved verbatim from expected_undelivered.py lines 50-135, plus the `defaultdict` import it needs):
```python
# -*- coding: utf-8 -*-
"""domain.py — 站点 / 文件名 / 列映射的共享配置(单一来源)。
比对compare与入库store都依赖这套配置抽出独立 leaf 模块,
让 store 不必为读配置而依赖整个比对引擎。纯数据,无 state_store / 文件 IO 依赖。
"""
from collections import defaultdict
# 汇总报表覆盖的全部站点4 站在前、百世在末;汇总页图表只取 4 站)
ALL_REPORT_SITES = ["顺心", "中通", "韵达", "安能", "百世"]
# 4 站单站未到明细文件名(百世未到文件由站点直接产出,名为 BAISHI_FILE
SITE_UNDELIVERED_FILE = "{name}-未到数据.xlsx"
BAISHI_FILE = "百世-应到未到货物数据.xlsx"
BAISHI_COLUMNS = ["类型", "子单号", "运单号", "最新扫描记录"]
def arrived_pieces_zhongtong(df):
"""中通实到「运单号」为复合串H + 运单号(12) + 总数(4) + 顺序(4))。
基号 = v[:-8](与应到表运单号对齐),单件 = 整串(每串即一件)。"""
res = defaultdict(set)
for v in df["运单号"]:
v = str(v).strip()
if len(v) > 8 and v[-4:].isdigit():
res[v[:-8]].add(v) # 以完整复合串作为“已到单号”存入
return res
def arrived_pieces_by_cols(wb_col, piece_col):
"""顺心 / 韵达 / 安能:按干净运单列分组,单件 = 子单号 / 扫描单号。
wb_col实到表中与应到运单号对齐的干净列
(顺心=运单号 / 韵达=主单号 / 安能=所属单号)
piece_col实到表中每件货物的单号列子单号 / 扫描单号)"""
def parse(df):
res = defaultdict(set)
for m, s in zip(df[wb_col], df[piece_col]):
m, s = str(m).strip(), str(s).strip()
if m and s:
res[m].add(s)
return res
return parse
STATIONS = [
{
"name": "中通",
"exp": "中通-应到货物数据.xlsx",
"act": "中通-实到货物数据.xlsx",
"exp_qty": "交接件数", # 应到件数口径:交接件数(非录单件数)
"exp_wb": "运单号", # 应到表运单号列(兼作去重键)
"exp_jd": "交接单号", # 未到数据需展示的交接单号
"arrived_pieces": arrived_pieces_zhongtong,
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "顺心",
"exp": "顺心-应到货物数据.xlsx",
"act": "顺心-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("运单号", "子单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "韵达",
"exp": "韵达-应到货物数据.xlsx",
"act": "韵达-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("主单号", "子单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "安能",
"exp": "安能-应到货物数据.xlsx",
"act": "安能-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("所属单号", "扫描单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
]
def _site_cfg(name):
"""按名称取 4 站配置(百世不在 STATIONS返回 None"""
return next((c for c in STATIONS if c["name"] == name), None)
```
- [ ] **Step 2: Remove the moved block from expected_undelivered.py and import from domain**
In `inbound_verify/expected_undelivered.py`:
- Add to the top import block:
```python
from inbound_verify.domain import (
ALL_REPORT_SITES,
BAISHI_COLUMNS,
BAISHI_FILE,
SITE_UNDELIVERED_FILE,
STATIONS,
_site_cfg,
arrived_pieces_by_cols,
arrived_pieces_zhongtong,
)
```
- Delete the now-duplicated definitions (the block from `ALL_REPORT_SITES = ...` through the end of `_site_cfg`, i.e. old lines ~50-135 — the comment header `# 汇总报表...` through `return next(...)`). These now live in domain.py. The `from collections import defaultdict` import in expected_undelivered.py can stay (harmless) or be removed if unused — check with grep after.
- [ ] **Step 3: store.py — take config from domain, _read_business_dates still from eu**
In `inbound_verify/store.py`:
- Add to imports:
```python
from inbound_verify.domain import BAISHI_FILE, _site_cfg
```
- Replace `cfg = eu._site_cfg(site)` (2 occurrences, lines 264 and 296) → `cfg = _site_cfg(site)`.
- Replace `eu.BAISHI_FILE` (2 occurrences, lines 335 and 337) → `BAISHI_FILE`.
- Leave `eu._read_business_dates(...)` (line 213) unchanged — that's compare behavior, stays accessed via the compare module (`eu`). The `from inbound_verify import expected_undelivered as eu` import stays for this.
- [ ] **Step 4: runtime.py — SITE_UNDELIVERED_FILE from domain**
In `inbound_verify/runtime.py`:
- Add to imports:
```python
from inbound_verify.domain import SITE_UNDELIVERED_FILE
```
- Replace (line ~448):
```python
DOWNLOAD_DIR, expected_undelivered.SITE_UNDELIVERED_FILE.format(name=site)
```
with:
```python
DOWNLOAD_DIR, SITE_UNDELIVERED_FILE.format(name=site)
```
- Leave `expected_undelivered.write_site_file(site)` (line ~446) and `expected_undelivered.main()` (line ~476) unchanged (compare behavior).
- [ ] **Step 5: Verify — compileall + import smoke + domain is leaf**
```bash
cd /d/projects/LogisticsHubIPA/InboundVerify
.venv/Scripts/python.exe -m compileall -q inbound_verify
.venv/Scripts/python.exe -c "import inbound_verify.domain as d; print('STATIONS:', [s['name'] for s in d.STATIONS]); print('_site_cfg(中通):', d._site_cfg('中通')['exp']); print('BAISHI_FILE:', d.BAISHI_FILE)"
.venv/Scripts/python.exe -c "import inbound_verify.cli.router, inbound_verify.cli.server, inbound_verify.store, inbound_verify.runtime, inbound_verify.expected_undelivered; print('import smoke OK')"
```
Expected: STATIONS lists `['中通', '顺心', '韵达', '安能']`; `_site_cfg('中通')['exp']` = `中通-应到货物数据.xlsx`; `BAISHI_FILE` = `百世-应到未到货物数据.xlsx`; `import smoke OK`.
- [ ] **Step 6: Black + commit (after user confirms)**
```bash
.venv/Scripts/python.exe -m black inbound_verify/domain.py inbound_verify/expected_undelivered.py inbound_verify/store.py inbound_verify/runtime.py
git add inbound_verify/domain.py inbound_verify/expected_undelivered.py inbound_verify/store.py inbound_verify/runtime.py
git commit -m "refactor: extract domain.py (shared site/file/colmap config) from expected_undelivered
Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 3: Rename `expected_undelivered.py` → `compare.py`
**Files:**
- Move: `inbound_verify/expected_undelivered.py``inbound_verify/compare.py` (git mv)
- Modify: `inbound_verify/store.py`, `inbound_verify/runtime.py`, `inbound_verify/cli/router.py` (import + call-site rewrites)
**Interfaces:**
- Consumes: Task 2's `domain` (compare still uses it).
- Produces: module is `inbound_verify.compare`; all public names (`main`, `write_site_file`, `_read_business_dates`) unchanged. `expected_undelivered` no longer exists as a module name.
- [ ] **Step 1: git mv the module (history preserved)**
```bash
cd /d/projects/LogisticsHubIPA/InboundVerify
git mv inbound_verify/expected_undelivered.py inbound_verify/compare.py
```
- [ ] **Step 2: store.py — import compare instead of expected_undelivered**
In `inbound_verify/store.py`, replace:
```python
from inbound_verify import expected_undelivered as eu
```
with:
```python
from inbound_verify import compare
```
and replace the one call site (line ~213):
```python
return eu._read_business_dates(ALL_SITES + ["百世"]) or {}
```
with:
```python
return compare._read_business_dates(ALL_SITES + ["百世"]) or {}
```
(`_site_cfg` and `BAISHI_FILE` already come from `domain` after Task 2 — no change there.)
- [ ] **Step 3: runtime.py — import compare, fix write_site_file/main**
In `inbound_verify/runtime.py`, replace:
```python
from inbound_verify import expected_undelivered
```
with:
```python
from inbound_verify import compare
```
Replace `expected_undelivered.write_site_file(site)` (line ~446) → `compare.write_site_file(site)`.
Replace `(expected_undelivered.main() or True)` (line ~476) → `(compare.main() or True)`.
- [ ] **Step 4: cli/router.py — import compare, fix main()**
In `inbound_verify/cli/router.py`, replace:
```python
from inbound_verify import expected_undelivered
```
with:
```python
from inbound_verify import compare
```
Replace `expected_undelivered.main()` (line ~34, inside `run_undelivered_compare`) → `compare.main()`.
- [ ] **Step 5: Verify — no stale references + import smoke**
```bash
cd /d/projects/LogisticsHubIPA/InboundVerify
.venv/Scripts/python.exe -m compileall -q inbound_verify
grep -rn "expected_undelivered" inbound_verify || echo "no stale expected_undelivered refs OK"
.venv/Scripts/python.exe -c "import inbound_verify.compare, inbound_verify.cli.router, inbound_verify.cli.server, inbound_verify.store, inbound_verify.runtime; print('import smoke OK')"
```
Expected: compileall silent; grep prints `no stale expected_undelivered refs OK`; `import smoke OK`.
- [ ] **Step 6: Black + commit (after user confirms)**
```bash
.venv/Scripts/python.exe -m black inbound_verify
git add -A inbound_verify
git commit -m "refactor: rename expected_undelivered to compare
Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Final end-to-end re-confirm (once, after all 3 tasks; requires backend restart + re-login)
After Task 3 commits, the running backend still has the old modules loaded. To confirm end-to-end behavior is unchanged on the refactored code:
- [ ] **Restart backend** (TaskStop current → clean ms-playwright/anneng orphans → `python -m inbound_verify.cli.server`), re-login all sites.
- [ ] **Re-run the gate from the Tier 1 manual test:** trigger `undelivered` for 顺心/中通/韵达/安能 via API → expect 4× success + 4× `*-未到数据.xlsx`; trigger `__compare__` → expect `output/应到未到数据.xlsx`; run `python -m inbound_verify.store ingest` → expect rows UPSERTed. All green = Tier 2 behavior-identical, done.
> If you want to skip the re-login cost: the per-task import-smoke gates already prove the import graph is correct and the changes are behavior-preserving moves. The e2e re-confirm is belt-and-suspenders.
---
## Self-Review (completed)
- **Spec coverage:** spec §8 Tier 2 — path dedup → Task 1; domain extract → Task 2; rename → compare → Task 3. config.py explicitly deferred (noted with rationale). All spec items addressed or consciously deferred.
- **Placeholder scan:** none — every step has exact code or exact commands with expected output. The domain.py content is the verbatim extracted block.
- **Type/name consistency:** `_site_cfg`, `BAISHI_FILE`, `SITE_UNDELIVERED_FILE`, `STATIONS`, `write_site_file`, `main`, `_read_business_dates` referenced consistently across tasks. Task 2 routes config to `domain` and leaves behavior (`_read_business_dates`, `write_site_file`, `main`) in compare — verified against store.py/runtime.py/router.py usages. Task 3 renames the module but preserves all public names.

View File

@@ -0,0 +1,522 @@
# 下载后自动入库钩子 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 把 PostgreSQL 入库动作挂到 `runtime.dispatch_task` 下载成功分支使每次下载成功后自动、尽力而为、kind 级地把刚下载的数据 UPSERT 进 PG。
**Architecture:** 新增 `store.ingest_task(site, kind)`kind 级路由,复用现有 `_ingest_*`+ `runtime._persist_to_db(site, kind)`(同步内联钩子,懒导入 store绝不外抛+ `state_store.ingest_state` 表(结果可查,经 `/status` 暴露)+ 配置开关/超时。详见 spec `docs/superpowers/specs/2026-07-24-ingest-hook-design.md`
**Tech Stack:** Python 3.10+、psycopg(v3)、sqlite3、Playwright(不动)、FastAPI(仅 `/status` 加字段)、pyyaml。包以可编辑模式安装`pip install -e .`),命令走 `.venv/Scripts/python.exe -m inbound_verify...`
## Global Constraints
- **Python 一律走项目虚拟环境**`.venv/Scripts/python.exe`(全局 CLAUDE.md 协议)。命令默认在 `InboundVerify/` 根目录执行。
- **无 pytest 测试套件**(本仓库约定,覆盖 writing-plans 默认的 TDD-pytest 步骤):每个任务的验证用 `py_compile` + `black` + 导入冒烟 + 功能/手动校验,不写 pytest。
- **改完任何 .py 必须跑 Black**`.venv/Scripts/python.exe -m black inbound_verify`
- **不自动提交/推送**(本仓库约定,覆盖 writing-plans 默认的"每任务即提交"):每个任务的 Commit 步骤**仅在用户明确说"提交"/"commit"时执行**;否则完成任务后停在待提交态,告知用户。
- **commit message 一律英文**(全局 CLAUDE.md
- **不改下载流程/比对逻辑**4 站 `-未到数据.xlsx` 不入库(既有设计);不上连接池/后台线程/补入重试。
- 导入约束:`runtime` 已 import `compare``store` 也 import `compare`;钩子里对 `store` **懒导入**以回避成环。
---
## File Structure
| 文件 | 责任 | 本计划改动 |
|---|---|---|
| `inbound_verify/store.py` | PG 持久化(建库/建表/入库 CLI | 改 `_load_pg_config`/`_connect`;加 `ingest_enabled()``ingest_task()`CLI 加 `ingest-one` |
| `inbound_verify/state_store.py` | SQLite 状态持久化 | `init_db``ingest_state` 表;加 `set_ingest_state()``get_all_ingest_state()` |
| `inbound_verify/runtime.py` | 两种模式共享核心(含 dispatch_task | 加 `_persist_to_db()`;在 `dispatch_task` 成功分支调用 |
| `inbound_verify/cli/server.py` | FastAPI 服务模式 | `/status` 返回加 `ingest` 字段 |
| `config.example.yaml` | 配置模板(提交) | `postgres` 段加 `auto_ingest`/`connect_timeout_seconds`;修过期 `db_store.py``store.py` 注释 |
| `config.yaml` | 真实配置gitignored | 同上两键 + 修注释 |
| `README.md` | 用户文档 | DB CLI 命令列表加 `ingest-one` |
任务依赖Task 2 依赖 Task 1Task 4 依赖 Task 1+2+3Task 5 依赖 Task 3。Task 3 独立。建议顺序 1→2→3→4→5。
---
### Task 1: store.py 配置键 + 连接超时层
**Files:**
- Modify: `inbound_verify/store.py:48-78``_load_pg_config``_connect`
- Modify: `config.example.yaml:72-89``config.yaml`postgres 段)
**Interfaces:**
- Consumes: 无(配置层根基)
- Produces: `_load_pg_config()` 返回新增 `auto_ingest: bool``connect_timeout_seconds: int``_connect(dbname)` 连接带 `connect_timeout` 且会话级 `statement_timeout=30s`;新函数 `ingest_enabled() -> bool`。后续任务依赖 `ingest_enabled()` 与超时连接。
- [ ] **Step 1: 改 `_load_pg_config` 加两键**
`inbound_verify/store.py``_load_pg_config` 返回 dict`schema` 之后追加两键:
```python
return {
"host": pg.get("host", "127.0.0.1"),
"port": int(pg.get("port", 5432)),
"user": pg.get("user", "postgres"),
"password": pg.get("password", ""),
"dbname": pg.get("dbname", "CQHXDB"),
"schema": pg.get("schema", "inbound_verify"),
"auto_ingest": bool(pg.get("auto_ingest", True)),
"connect_timeout_seconds": int(pg.get("connect_timeout_seconds", 5)),
}
```
- [ ] **Step 2: 改 `_connect` 加 connect_timeout + statement_timeout**
`_connect` 替换为(用 `options` 一次性设 search_path + statement_timeout等价于 spec 的 SET 但不引入额外事务):
```python
def _connect(dbname):
"""用关键字参数连接(避开 conninfo 对密码特殊字符的解析)。
options 设 search_path 到专用 schema + 会话级 statement_timeout=30s
cpolar 隧道上防失控查询connect_timeout 守连接阶段)。"""
c = _load_pg_config()
return psycopg.connect(
host=c["host"],
port=c["port"],
dbname=dbname,
user=c["user"],
password=c["password"],
options=f"-c search_path={c['schema']} -c statement_timeout=30s",
connect_timeout=c["connect_timeout_seconds"],
)
```
- [ ] **Step 3: 加 `ingest_enabled()` 薄封装**
`_connect` 之后、`# 建库 / 建表` 分节注释之前插入:
```python
def ingest_enabled():
"""是否启用下载后自动入库config.yaml postgres.auto_ingest默认 True
供 runtime 钩子判定开关,避免它伸手进 _load_pg_config。"""
return _load_pg_config()["auto_ingest"]
```
- [ ] **Step 4: 更新 store.py 模块 docstring 的命令列表**
把文件顶部 docstring 的命令行小节,在 `ingest` 行后补一行 `ingest-one`Task 2 会实现该命令docstring 先行):
```
python -m inbound_verify.store ingest [site] 入库全站或单站(幂等 UPSERT
python -m inbound_verify.store ingest-one <site> <kind> 仅入库指定站/类(钩子同款路由)
python -m inbound_verify.store all createdb → init → 全站 ingest 一条龙
```
- [ ] **Step 5: config.example.yaml 加两键 + 修过期注释**
`config.example.yaml` 的 postgres 段:把 `# PostgreSQL 数据持久化(到货核销数据入库,详见 db_store.py` 改为 `... 详见 store.py`;命令注释里的 `python db_store.py ...` 改为 `python -m inbound_verify.store ...`;在 `schema: inbound_verify` 之后追加:
```yaml
schema: inbound_verify
# 下载成功后自动入库(钩子,见 runtime._persist_to_dbfalse=跳过(无 PG/cpolar 的开发机)。
auto_ingest: true
# PG 连接超时cpolar 抖动时快速失败,不拖垮下载 worker。
connect_timeout_seconds: 5
```
- [ ] **Step 6: config.yaml 同步gitignored 本地文件)**
`config.yaml` 做同样两键追加,并把注释里的 `db_store.py` 改为 `store.py`。保留其真实凭据不动。
- [ ] **Step 7: 编译 + Black**
```
.venv/Scripts/python.exe -m py_compile inbound_verify/store.py
.venv/Scripts/python.exe -m black inbound_verify/store.py
```
Expected: py_compile 无输出black 报 `reformatted``left unchanged`
- [ ] **Step 8: 回归既有 CLI 不破**
```
.venv/Scripts/python.exe -c "from inbound_verify import store; c=store._load_pg_config(); assert c['auto_ingest'] is True and c['connect_timeout_seconds']==5; assert store.ingest_enabled() is True; print('config ok')"
```
Expected: `config ok`
- [ ] **Step 9: Commitgated**
仅当用户说"提交"时执行:
```bash
git add inbound_verify/store.py config.example.yaml
git commit -m "feat(store): add auto_ingest config + pg connect/statement timeouts"
```
`config.yaml` 已 gitignore不入提交。
---
### Task 2: store.ingest_task + ingest-one CLI
**Files:**
- Modify: `inbound_verify/store.py`(加 `ingest_task``main()``ingest-one` 分支)
**Interfaces:**
- Consumes: Task 1 的 `_connect`(超时)、`_load_pg_config`、既有 `_ingest_expected/_ingest_actual/_ingest_undelivered_baishi``_read_business_dates``ALL_SITES``_site_cfg`
- Produces: `ingest_task(site: str, kind: str) -> int`(返回入库总条数;`__compare__`/不支持组合返回 0
- [ ] **Step 1: 加 `ingest_task`**
`ingest(site)` 函数之后、`# 命令行` 分节之前插入:
```python
def ingest_task(site, kind):
"""按 (site, kind) 入库本次刚下载的文件(幂等 UPSERT返回总条数。
与 ingest(site) 的区别:只入本次刷新的那一类,避免重读写另一类文件(同步钩子里减少阻塞)。
kind 路由:
expected/actual 各入其列;
undelivered 百世 入未到;
undelivered 4 站 _site_undelivered_handler 内部连带下了 expected+actual故入两者
__compare__ / 其它组合 返回 0。
"""
if site == "__compare__":
return 0
dates = _read_business_dates()
total = 0
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
if kind == "expected":
total += _ingest_expected(cur, site, dates.get(site))
elif kind == "actual":
total += _ingest_actual(cur, site)
elif kind == "undelivered":
if site == "百世":
total += _ingest_undelivered_baishi(cur)
else: # 顺心/中通/韵达/安能
total += _ingest_expected(cur, site, dates.get(site))
total += _ingest_actual(cur, site)
# 其它组合(如 百世/expected正常不经钩子触发防御性返回 0
conn.commit()
return total
```
- [ ] **Step 2: `main()` 加 `ingest-one` 分支**
`main()``elif cmd == "all":` 分支之后、`else:` 之前插入:
```python
elif cmd == "ingest-one":
kind = sys.argv[3] if len(sys.argv) > 3 else None
if not site or kind not in ("expected", "actual", "undelivered"):
print("用法: python -m inbound_verify.store ingest-one <site> <expected|actual|undelivered>")
sys.exit(1)
total = ingest_task(site, kind)
print(f">> [ingest-one] {site}/{kind} 入库 {total}")
```
- [ ] **Step 3: 编译 + Black**
```
.venv/Scripts/python.exe -m py_compile inbound_verify/store.py
.venv/Scripts/python.exe -m black inbound_verify/store.py
```
Expected: 无编译错误。
- [ ] **Step 4: 路由正确性(手动,需 PG 已 createdb+init 且 downloads/ 有数据)**
逐条验证 kind 级只入对应文件:
```
.venv/Scripts/python.exe -m inbound_verify.store ingest-one 韵达 expected
```
Expected: stdout 只出现 `[应到] 韵达N 条运单`**不**出现 `[实到] 韵达` 行;结尾 `[ingest-one] 韵达/expected 入库 N 条`
```
.venv/Scripts/python.exe -m inbound_verify.store ingest-one 顺心 undelivered
```
Expected: stdout 同时出现 `[应到] 顺心``[实到] 顺心`undelivered→两者
```
.venv/Scripts/python.exe -m inbound_verify.store ingest-one 百世 undelivered
```
Expected: stdout 出现 `[未到] 百世`
> 若某站 downloads/ 无文件:`_ingest_*` 打印 `[跳过] ... 文件不存在`ingest_task 返回 0属正常非错误
- [ ] **Step 5: 无 PG 时的降级(手动,可选)**
临时把 `config.yaml``host` 改成不可达地址,重跑 Step 4 任一命令:应在 `connect_timeout_seconds`(默认 5s内报 psycopg 连接错误并退出(非 hang。验证后改回真实 host。
- [ ] **Step 6: Commitgated**
```bash
git add inbound_verify/store.py
git commit -m "feat(store): add kind-level ingest_task + ingest-one CLI subcommand"
```
---
### Task 3: state_store ingest_state 表 + 读写函数
**Files:**
- Modify: `inbound_verify/state_store.py:32-113``init_db` 加表)、`:219-247`(加两函数)
**Interfaces:**
- Consumes: 无(独立叶子,仅依赖 paths + sqlite3 + datetime
- Produces: `set_ingest_state(site, kind, ok, count=0, error=None) -> None``get_all_ingest_state() -> dict`(形状 `{site: {kind: {ok, ingested_at, count, error}}}`。Task 4 与 Task 5 依赖这两个。
- [ ] **Step 1: `init_db` 加 `ingest_state` 表**
`init_db` 内、`site_settings` 表 CREATE 之后(`conn.commit()` 之前)插入:
```python
conn.execute("""
CREATE TABLE IF NOT EXISTS ingest_state (
site TEXT,
kind TEXT,
ok INTEGER,
ingested_at TEXT,
count INTEGER,
error TEXT,
PRIMARY KEY (site, kind)
)
""")
```
- [ ] **Step 2: 加 `set_ingest_state` 与 `get_all_ingest_state`**
`get_all_status()` 函数之后插入:
```python
def set_ingest_state(site, kind, ok, count=0, error=None):
"""记录一次入库结果UPSERT。ok: boolcount: 入库条数error: 失败原因或 None。"""
with sqlite3.connect(STATE_DB_PATH) as conn:
conn.execute(
"INSERT INTO ingest_state (site, kind, ok, ingested_at, count, error) "
"VALUES (?, ?, ?, ?, ?, ?) "
"ON CONFLICT(site, kind) DO UPDATE SET "
"ok=excluded.ok, ingested_at=excluded.ingested_at, "
"count=excluded.count, error=excluded.error",
(site, kind, 1 if ok else 0, _now(), int(count or 0), error or ""),
)
conn.commit()
def get_all_ingest_state():
"""返回 {site: {kind: {ok, ingested_at, count, error}}};库不存在返回 {}"""
if not os.path.exists(STATE_DB_PATH):
return {}
with sqlite3.connect(STATE_DB_PATH) as conn:
rows = conn.execute(
"SELECT site, kind, ok, ingested_at, count, error FROM ingest_state"
).fetchall()
out = {}
for site, kind, ok, ingested_at, count, error in rows:
out.setdefault(site, {})[kind] = {
"ok": bool(ok),
"ingested_at": ingested_at or "",
"count": int(count or 0),
"error": error or "",
}
return out
```
- [ ] **Step 3: 编译 + Black**
```
.venv/Scripts/python.exe -m py_compile inbound_verify/state_store.py
.venv/Scripts/python.exe -m black inbound_verify/state_store.py
```
Expected: 无错误。
- [ ] **Step 4: set/get 往返冒烟**
```
.venv/Scripts/python.exe -c "from inbound_verify import state_store as s; s.init_db(); s.set_ingest_state('韵达','expected',True,count=42); s.set_ingest_state('韵达','expected',False,error='boom'); d=s.get_all_ingest_state(); r=d['韵达']['expected']; assert r['ok'] is False and r['count']==42 and r['error']=='boom' and r['ingested_at']; print('ingest_state ok')"
```
Expected: `ingest_state ok`(验证 UPSERT 覆盖:第二次写把 ok 改 Falsecount 保留 42error 写入)。
- [ ] **Step 5: Commitgated**
```bash
git add inbound_verify/state_store.py
git commit -m "feat(state_store): add ingest_state table + set/get helpers"
```
---
### Task 4: runtime._persist_to_db 钩子 + 挂到 dispatch_task
**Files:**
- Modify: `inbound_verify/runtime.py`(加 `_persist_to_db`,位置在 `_record_business_date` 之后、`dispatch_task` 之前;改 `dispatch_task` 成功分支 `:552-557`
**Interfaces:**
- Consumes: Task 1 `store.ingest_enabled()`、Task 2 `store.ingest_task(site, kind)`、Task 3 `state_store.set_ingest_state(...)`;既有 `state_store`runtime 已 import
- Produces: `_persist_to_db(site, kind)``dispatch_task` 在下载成功后调用;下载任务的成功判定**不变**。
- [ ] **Step 1: 加 `_persist_to_db`**
`_record_business_date` 函数之后(`def dispatch_task` 之前)插入:
```python
def _persist_to_db(site, kind):
"""下载成功后把本次数据入库 PostgreSQL尽力而为绝不外抛不影响任务判定
- __compare__ 无源数据,跳过。
- auto_ingest=false 时跳过(无 PG/cpolar 的开发机)。
- 懒导入 store 以回避 import 顺序store↔compare 与 runtime↔compare 共存)。
- 结果写 state_store.ingest_state供 /api/status 反映入库健康。
所有写库/写状态都包 try/except失败仅告警绝不改变 dispatch_task 的 SUCCESS 判定。"""
if site == "__compare__":
return
try:
from inbound_verify import store # 懒导入:冷路径(每下载一次),回避成环
except Exception as e:
print(f">> [warn] 入库模块不可用: {e}")
return
if not store.ingest_enabled():
print(">> [入库] 已关闭 (auto_ingest=false),跳过")
return
try:
count = store.ingest_task(site, kind)
state_store.set_ingest_state(site, kind, ok=True, count=count)
print(f">> [入库] {site}/{kind} 成功,{count}")
except Exception as e:
print(f">> [warn] 入库失败 {site}/{kind}: {e}")
try:
state_store.set_ingest_state(site, kind, ok=False, error=str(e))
except Exception as e2:
print(f">> [warn] 写入库状态也失败: {e2}")
```
- [ ] **Step 2: 挂到 `dispatch_task` 成功分支**
`dispatch_task` 内的成功分支:
```python
_record_business_date(site, kind)
return (state_store.TASK_SUCCESS, None)
```
改为:
```python
_record_business_date(site, kind)
_persist_to_db(site, kind)
return (state_store.TASK_SUCCESS, None)
```
`_persist_to_db` 绝不外抛,故不会被外层 `except Exception` 误判为任务失败。)
- [ ] **Step 3: 编译 + Black + 导入冒烟**
```
.venv/Scripts/python.exe -m py_compile inbound_verify/runtime.py
.venv/Scripts/python.exe -m black inbound_verify/runtime.py
.venv/Scripts/python.exe -c "from inbound_verify.runtime import dispatch_task, _persist_to_db; print('runtime import ok')"
```
Expected: `runtime import ok`
- [ ] **Step 4: 端到端(手动,需站点已登录)**
任选一种模式触发一次真实下载并观察钩子:
- 服务模式:`POST /tasks` `{"site":"韵达","kind":"expected"}`(或经 dashboard 触发),下载完成后看 worker stdout
- 成功:`>> [入库] 韵达/expected 成功N 条`
- 失败(如 PG 未 init`>> [warn] 入库失败 ...`,且任务本身仍 `success``GET /tasks/{id}` 验证)。
- 交互模式:菜单 `[6]` 韵达应到,完成后看同样的 `[入库]` 行。
并查 `state/state.db`
```
.venv/Scripts/python.exe -c "from inbound_verify import state_store as s; print(s.get_all_ingest_state())"
```
Expected: 含 `韵达`/`expected` 的记录,`ok` 与 stdout 一致。
- [ ] **Step 5: 关开关降级(手动)**
`config.yaml``auto_ingest``false`再触发一次下载stdout 应出现 `>> [入库] 已关闭 (auto_ingest=false),跳过`,且不连 PG任务仍 SUCCESS。验证后改回 `true`
- [ ] **Step 6: Commitgated**
```bash
git add inbound_verify/runtime.py
git commit -m "feat(runtime): auto-ingest hook after successful download"
```
---
### Task 5: /status 暴露 ingest 态 + README
**Files:**
- Modify: `inbound_verify/cli/server.py:189-196``get_status`
- Modify: `README.md`DB CLI 命令列表)
**Interfaces:**
- Consumes: Task 3 `state_store.get_all_ingest_state()`
- Produces: `GET /status` 返回体新增 `ingest` 字段。
- [ ] **Step 1: `/status` 加 `ingest` 字段**
`get_status` 的返回 dict
```python
return {
"worker_ready": worker_state["ready"],
"worker_error": worker_state["error"],
"sites": state_store.get_all_status(),
}
```
改为:
```python
return {
"worker_ready": worker_state["ready"],
"worker_error": worker_state["error"],
"sites": state_store.get_all_status(),
"ingest": state_store.get_all_ingest_state(),
}
```
并把 docstring 顺手补一句(可选):`"""各站登录态 + 数据态 + 入库态(前端状态盘用),另含 worker 就绪状态。"""`
- [ ] **Step 2: README DB CLI 命令列表加 `ingest-one`**
`README.md` 第五章"运行"下、DB CLI 注释行(`# 或inbound-verify-db createdb|init|ingest|all` 附近)补一行说明自动入库 + 新命令:
```
# DB CLI建库 / 初始化 / 灌数据 / 全流程 / 单站单类
.venv/Scripts/python.exe -m inbound_verify.store createdb # 或 init | ingest | ingest-one <site> <kind> | all
# 注下载成功后会自动入库postgres.auto_ingest默认开ingest-one 用于手动重灌指定站/类。
```
- [ ] **Step 3: 编译 + Black**
```
.venv/Scripts/python.exe -m py_compile inbound_verify/cli/server.py
.venv/Scripts/python.exe -m black inbound_verify/cli/server.py
```
Expected: 无错误。
- [ ] **Step 4: `/status` 含 ingest 字段(手动,服务模式已启动)**
```
curl -s http://127.0.0.1:8000/status | python -m json.tool
```
(或浏览器 `http://127.0.0.1:8000/docs``/status`Expected: 返回体含 `"ingest": {...}` 键(无入库记录时为 `{}`Task 4 跑过后会有值)。经 dashboard 的 `GET /api/status`(代理)同样可见。
- [ ] **Step 5: 全量回归冒烟**
```
.venv/Scripts/python.exe -m py_compile inbound_verify
.venv/Scripts/python.exe -m black inbound_verify
.venv/Scripts/python.exe -c "import inbound_verify.store, inbound_verify.state_store, inbound_verify.runtime, inbound_verify.cli.server; print('all imports ok')"
```
Expected: `all imports ok`black 无 diff。
- [ ] **Step 6: Commitgated**
```bash
git add inbound_verify/cli/server.py README.md
git commit -m "feat(server): expose ingest state in /status; doc auto-ingest in README"
```
---
## Self-Review 结论
- **Spec 覆盖**spec §4.1 → Task 1+2§4.2 → Task 4§4.3 → Task 3§4.4 → Task 5§4.5 配置 → Task 1config 两文件§7 测试手段 → 各任务手动步ingest-one 路由、端到端、降级、回归)。无遗漏。
- **占位符**:无 TBD/TODO每步含完整代码与确切命令。
- **类型/命名一致**`ingest_task(site, kind)``ingest_enabled()``set_ingest_state(site, kind, ok, count=0, error=None)``get_all_ingest_state()``_persist_to_db(site, kind)` 在各任务间签名一致;`/status` 字段名 `ingest``get_all_ingest_state` 返回一致。
- **已标注偏离**(a) 无 pytest → 用 py_compile/black/冒烟/手动代替Global Constraints(b) 不自动提交 → Commit 步骤 gatedGlobal Constraints + 每任务注明);(c) `_connect``options` 设 statement_timeout 取代 spec 的 `SET`等价、无额外事务Task 1 Step 2 注释说明)。

View File

@@ -0,0 +1,274 @@
# InboundVerify 包化重构设计
- **日期**:2026-07-23
- **方案**:C(规范)—— `pyproject.toml` + `[project.scripts]` + 根级包 `inbound_verify/`,根目录不留 `.py` 薄壳
- **力度**:彻底(建包 + 定点小改进 + 大文件拆分),但**大文件拆分(Tier 3)前置手动测试闸门**
- **状态**:已与用户对齐,待 spec 评审
---
## 1. 背景与目标
当前 13 个 Python 模块(共约 6500 行)全部平铺在仓库根目录,随脚本量增长结构混乱。本设计将其重组为一个规范的、可 `pip install -e .` 安装的 Python 包。
**目标**
1. 扁平脚本 → `inbound_verify/` 包(`sites/``cli/` 两个子包,其余平铺包根,避免一层只放一两个文件的过度嵌套)。
2. 全部内部 import 改为包内绝对引用。
3. `pyproject.toml` 打包,`[project.scripts]` 暴露命令入口;**根目录不留 `.py` 薄壳**(规范要求)。
4. 顺手抽离共享配置、消重(定点小改进,Tier 2)。
5. 大文件拆分作为**最后、可选、风险隔离**的阶段(Tier 3),且必须先过手动测试闸门。
**已确认约束**
- InboundVerify 是 git **子模块**;改动在子模块内提交,父仓库 `LogisticsHubIPA` 仅跟踪子模块指针。
- **不加测试套件**(用户决定);验证靠 `compileall` + 导入冒烟 + grep 查残留引用 + 手动端到端测试。
- 入口走规范:无根薄壳;一次性 `pip install -e .` 后用命令或 `python -m` 启动。
- **不做 src/ 布局**(内部工具、不发 PyPI、无测试,边际价值有限)。
- 本仓库约定:**不自动提交 / 不自动推送**;改动等用户明确说"提交"再 commit/push(本约定覆盖 brainstorming 默认的"写完即提交")。
---
## 2. 现状分析
### 2.1 依赖分层(自底向上)
```
paths ← 万物之基(被所有模块 import)
state_store ← SQLite 状态持久化
expected_undelivered ← 离线比对(兼职存站点/文件名/列映射共享配置,被 db_store 复用 —— 耦合点)
site_*.py(×5) ← 各依赖 paths + state_store
runtime ← 编排核心(启动/派发/心跳,依赖上面全部)
main_router / server / db_store ← 三入口(均有 __main__/CLI)
```
### 2.2 关键发现
1. **`paths.py` 是地雷**:`BASE_DIR = dirname(abspath(__file__))`——paths.py 在哪,根就在哪。搬进子目录后必须改为上跳一级,否则 `downloads/``output/``config.yaml``state/state.db``schema.sql` 全部跑偏。
2. **import 全是扁平顶层**(`import site_shunxin``from paths import …`),且**没有任何地方按字符串名引用模块**(`TASK_HANDLERS` 用函数引用、`dispatch_task` 用 site/kind 字典),改写纯机械。
3. **重复代码**:`with_retry``_remove_if_exists` 在 5 个站点逐字重复(CLAUDE.md 注明"现阶段刻意不优化结构")。
4. **路径常量重复**:`expected_undelivered.py` 自带 `BASE/DOWNLOADS/OUTPUT`,与 `paths.py` 重复。
5. **耦合**:`db_store` 为读站点配置而 `import expected_undelivered`(为了配置而依赖整个比对引擎)。
---
## 3. 目标目录结构
```
InboundVerify/
├── pyproject.toml # 新增:打包 + 依赖 + console_scripts
├── config.example.yaml
├── config.yaml # 留根(gitignored)
├── schema.sql # 留根(store 按绝对路径读)
├── requirements.txt # 保留为静态镜像(pyproject 为准)
├── README.md / CLAUDE.md / .gitignore
├── downloads/ output/ state/ # 运行时数据,留根
├── docs/
└── inbound_verify/ # ← 包
├── __init__.py
├── __main__.py # 可选:python -m inbound_verify → 交互菜单
├── paths.py # 锚点改为指向项目根(§4)
├── config.py # 新增(Tier2):集中 load_config()
├── domain.py # 新增(Tier2):从 expected_undelivered 抽出的共享站点/文件/列映射
├── runtime.py # ← runtime.py(编排核心,整体保留不拆)
├── state_store.py # ← state_store.py(保留名,仅搬运)
├── expected_undelivered.py # ← 搬运;Tier2 改名 compare.py + 抽 domain.py
├── store.py # ← db_store.py(叶子,改名无 churn)
├── sites/
│ ├── __init__.py
│ ├── shunxin.py baishi.py zto.py yunda.py # ← site_*.py(去 site_ 前缀)
│ └── anneng.py # ← site_anneng.py(Tier3 可选再拆成子包)
└── cli/
├── __init__.py
├── router.py # ← main_router.py(叶子,改名无 churn)
└── server.py # ← server.py(叶子)
```
### 3.1 搬运映射表(全部 `git mv` 保历史)
| 现在 | Tier 1 后 | 调用点改动 |
|---|---|---|
| `paths.py` | `inbound_verify/paths.py` | 无(仅改 import 行 + 锚点) |
| `runtime.py` | `inbound_verify/runtime.py` | 无(保留名) |
| `state_store.py` | `inbound_verify/state_store.py` | 无(**保留名**,仅改 import 行) |
| `expected_undelivered.py` | `inbound_verify/expected_undelivered.py` | 无(Tier1 保留名;Tier2 改名 compare) |
| `db_store.py` | `inbound_verify/store.py` | 无(叶子,无人 import) |
| `main_router.py` | `inbound_verify/cli/router.py` | 无(叶子) |
| `server.py` | `inbound_verify/cli/server.py` | 无(叶子) |
| `site_{shunxin,baishi,zto,yunda,anneng}.py` | `inbound_verify/sites/{…}.py`(去 `site_` 前缀) | 改 `runtime` + `main_router` 调用点;**grep 查残留引用兜底** |
> **命名策略(降低无测试下的风险)**:Tier 1 只对**被多处裸名引用**的模块(`state_store`、`expected_undelivered`、`runtime`、`paths`)**保留原名**,做到"仅改 import 行、零调用点改动";只对**叶子入口**(`db_store`/`main_router`/`server`)和**站点文件**(去 `site_` 前缀)做改名。`expected_undelivered` 的改名(`→ compare.py`)推迟到 Tier 2——那时本就要为抽 `domain.py` 重做该文件及其调用方,把改名 churn 并入一个已经在改的批次。这样 Tier 1 的纯搬运可被 `compileall` + 导入冒烟 + grep 充分验证,不依赖手测。
---
## 4. `paths.py` 锚点修正(唯一地雷,必须改对)
包搬进 `inbound_verify/` 后,`__file__` 多降一级。要把 `BASE_DIR` 继续指回项目根:
```python
# inbound_verify/paths.py
import os
# __file__ = .../InboundVerify/inbound_verify/paths.py
# 上两级 = .../InboundVerify (= 项目根,config.yaml/downloads/schema.sql 所在)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DOWNLOAD_DIR = os.path.join(BASE_DIR, "downloads")
OUTPUT_DIR = os.path.join(BASE_DIR, "output")
CONFIG_PATH = os.path.join(BASE_DIR, "config.yaml")
STATE_DB_PATH = os.path.join(BASE_DIR, "state", "state.db")
```
editable 安装不会移动文件,`__file__` 仍指向源码树,两级上跳稳定指向项目根。`store.py``SCHEMA_PATH = join(BASE_DIR, "schema.sql")` 自动跟着对。
---
## 5. import 改写规则
```python
from paths import ... from inbound_verify.paths import ...
import state_store from inbound_verify import state_store # 保留名,调用点 state_store.X 不变
from runtime import (...) from inbound_verify.runtime import (...)
import site_shunxin from inbound_verify.sites import shunxin # 5 站同理,调用点改 shunxin.X
import expected_undelivered from inbound_verify import expected_undelivered # Tier1 保留名
import expected_undelivered as eu from inbound_verify import expected_undelivered as eu # eu.X 不变
```
(Tier 2 后,`expected_undelivered` 改名 `compare`,`db_store` 的共享配置改 `from inbound_verify import domain`。)
---
## 6. 入口与打包
### 6.1 `main()` 包装(根目录不留薄壳)
```python
# inbound_verify/cli/server.py 末尾
def main():
uvicorn.run("inbound_verify.cli.server:app", host="0.0.0.0", port=8000)
if __name__ == "__main__":
main()
```
> `uvicorn.run` 改传**字符串** `"inbound_verify.cli.server:app"`(规范写法;不开 `reload`/`workers` 时仍在当前进程 import,行为等价)。`router.py` 套 `main()` 调 `run_multi_site_daemon()`;`store.py` 已有 `_cli`,改名为 `main`。
### 6.2 `pyproject.toml`(骨架)
```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "inbound-verify"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["pandas>=2.0.0", "playwright>=1.40.0", "openpyxl>=3.1.0",
"PyYAML>=6.0", "websocket-client>=1.0.0", "fastapi>=0.110.0",
"uvicorn>=0.27.0", "apscheduler>=3.10.0", "psycopg[binary]>=3.1"]
[project.scripts]
inbound-verify = "inbound_verify.cli.router:main"
inbound-verify-server = "inbound_verify.cli.server:main"
inbound-verify-db = "inbound_verify.store:main"
[tool.setuptools.packages.find]
include = ["inbound_verify*"]
```
### 6.3 装包后三种启动方式(都汇到同一个 `main()`)
```bash
pip install -e . # 一次性
inbound-verify-server # 命令
python -m inbound_verify.cli.server # 兜底
uvicorn inbound_verify.cli.server:app # 生产最标准(端口/worker 命令行控)
```
---
## 7. 执行顺序与验证闸门(为"无测试"量身)
**安全原则**:Tier 1 必须先独立完成并验证为**行为等价**,再做任何动逻辑的改动。每段后必验证。验证职责分清——自动化部分我跑,端到端部分需你跑(真实登录/凭据/安能 Electron 只有你能提供)。
```
Tier 1 纯搬运(行为零改变)
├─ 我的自动验证: ① compileall 全过 ② 导入冒烟 ③ grep 查残留 site_ 引用 ④ DB 连通
└─ [等用户说"提交"] commit ← 安全基线
Tier 2 domain 抽取 / config 集中 / 路径消重 / expected_undelivered 改名 compare
├─ 我的自动验证: 同上
└─ [等用户说"提交"] commit
═══════ 手动测试闸门(你跑)═══════
全链路端到端:
起 inbound-verify → 登录 5 站(顺心双账号)→ 各站下载 → 比对(菜单 9)
→ inbound-verify-db ingest → 核对 output/应到未到数据.xlsx 与 PostgreSQL 三张表
通过? 否 → 修到通过
是 ↓
Tier 3 此时再定:anneng 拆不拆 / with_retry 抽不抽 base.py
└─ 每项后重跑我的自动验证,你按需复测
```
### 7.1 自动验证命令清单(我每个 Tier 后都跑)
```bash
.venv/Scripts/python.exe -m compileall inbound_verify # ① 语法
.venv/Scripts/python.exe -c "import inbound_verify.cli.router, \
inbound_verify.cli.server, inbound_verify.store, inbound_verify.runtime, \
inbound_verify.state_store" # ② 导入冒烟(三个入口会传递导入 sites/expected_undelivered 等;
# Tier2 改名后 expected_undelivered → compare,无需单独显式导入)
grep -rn "site_shunxin\|site_baishi\|site_zto\|site_yunda\|site_anneng" \
inbound_verify || echo "无残留 site_ 引用 ✓" # ③ 改名残留
.venv/Scripts/python.exe -c "from inbound_verify.store import _connect, _load_pg_config; \
c=_load_pg_config(); conn=_connect(c['dbname']); print('DB OK', conn.info.server_version)" # ④ DB
```
---
## 8. 各 Tier 内容
### Tier 1 — 纯搬运(行为零改变)
1. 建包骨架 + 空白 `__init__.py`(`sites/``cli/`)。
2. `git mv` §3.1 表中所有文件到新位置。
3. 按 §5 规则机械改写全部 import;站点改名后改 `runtime` + `main_router` 调用点(`shunxin.shunxin_expected_download(...)` 等)。
4.`paths.py` 锚点(§4)。
5. 三个入口加 `main()`(`store``_cli` 改名 `main`)。
6.`pyproject.toml`,`.venv``pip install -e .`
7. 跑 §7.1 四项自动验证。
8. 等用户说"提交"→ commit(子模块内)。
### Tier 2 — 定点小改进(每项后跑 §7.1)
- **抽 `domain.py`**:把 `expected_undelivered.py` 里的 `STATIONS`/`_site_cfg`/`ALL_REPORT_SITES`/`SITE_UNDELIVERED_FILE`/`BAISHI_FILE`/`BAISHI_COLUMNS`/`arrived_pieces_*` 移到 `inbound_verify/domain.py`;`store.py` 从依赖整个比对引擎改为 `from inbound_verify import domain`。**接缝最干净、收益明确,推荐做。**
- **`expected_undelivered.py``compare.py`** 改名,更新调用方(`store``as eu``runtime``cli/router`)。
- **加 `config.py`**:集中 `load_config()`(带缓存),各站点把自家的 `yaml.safe_load(open(CONFIG_PATH))` 换掉。
- **消重路径常量**:`compare.py` 自带那份 `BASE/DOWNLOADS/OUTPUT` 改成引用 `paths.py`
### Tier 3 — 大文件拆分(手动测试闸门之后;具体决策推迟到闸门)
- **`anneng.py``sites/anneng/` 子包**(`cdp.py`/`nav.py`/`expected.py`/`actual.py`):接缝分层清晰,但共享可变状态多(`CDP_PORT` 全局被 `set_cdp_port` 改、各种 URL hint、僵尸 tab 逻辑),CLAUDE.md 标注为脚gun。**倾向不拆**(1375 行虽大但是内聚的 CDP 驱动,强拆无测试网兜底风险高)——最终在闸门后定。
- **抽 `sites/base.py`**:`with_retry` / `_remove_if_exists` 在 5 站逐字重复;原作者在 CLAUDE.md 写"现阶段刻意不优化结构"。**抽不抽,闸门后定。**
---
## 9. 待决策(推迟到手动测试闸门)
| 决策 | 默认倾向 | 何时定 |
|---|---|---|
| `anneng.py` 拆不拆子包 | **不拆** | 手动测试通过后 |
| `with_retry`/`_remove_if_exists``base.py` | 待定 | 手动测试通过后 |
| `requirements.txt` 留还是删 | **留静态镜像**(注明 pyproject 为准) | 随时可改 |
---
## 10. 文档同步(Tier 1 必做)
- **README.md**:第二节目录树、第三节环境准备(加 `pip install -e .`)、第五节运行(改新命令)。
- **CLAUDE.md**:常用命令段全部改 `python -m inbound_verify…` / `inbound-verify…`,补 `pip install -e .``playwright install chromium``black`/`py_compile` 的包内路径写法。
---
## 11. 范围外
- **不做** src/ 布局。
- **不做** 给站点流程加 mock 测试。
- **不自动** commit/push(等用户明确指示)。
- 父仓库 `LogisticsHubIPA` 的子模块指针更新,是父仓库的单独一步,不在本 spec 范围。

View File

@@ -0,0 +1,290 @@
# 下载后自动入库钩子设计
- **日期**:2026-07-24
- **方案**:在 `runtime.dispatch_task` 下载成功分支挂一个**同步、kind 级、尽力而为**的入库钩子,调用新增的 `store.ingest_task(site, kind)`,仅入库本次刚下载的文件
- **力度**:后端最小闭环(钩子 + `ingest_task` + `ingest_state` 表 + `/status` 字段 + 配置开关/超时);不碰 dashboard UI、不上连接池/后台线程
- **状态**:已与用户对齐(4 个关键决策已逐项确认),待 spec 评审
---
## 1. 背景与目标
各站点的下载流程与 PostgreSQL 持久化模块(`store.py`)**都已实现**,但入库动作目前只能靠 CLI
(`inbound-verify-db ingest` / `python -m inbound_verify.store ingest`)手动触发,**没有挂到任何自动钩子上**。
本设计把入库动作挂到"数据完成下载之后"——具体挂在所有下载任务的唯一汇聚点
`runtime.dispatch_task` 的成功分支,与既有的"下载后写业务日期"钩子 `_record_business_date` 并列。
**目标**
1. 下载成功后**自动**把刚下载的那份数据 UPSERT 进 PostgreSQL无需手动跑 CLI。
2. 入库是**尽力而为**:失败只告警、绝不影响下载任务的成功判定(下载成功 = 任务成功)。
3. 入库结果可查:写入 `state_store`,经 `/api/status` 暴露,便于发现"入库坏了好几天"。
4. 可门控、可降级:配置开关 + PG 连接/语句超时,无 PG/cpolar 的开发机可整体跳过。
**非目标(本 spec 不做)**
- dashboard 前端展示 ingest 态(Next.js 侧API 字段已就绪待消费,单独小任务)。
- 4 站 `-未到数据.xlsx` 入库(既有设计仅百世未到入库4 站未到只给汇总报表)。
- 连接池、后台入库线程、失败补入重试(YAGNI)。
- 任何与下载流程本身、比对逻辑相关的改动。
---
## 2. 现状(决策依据)
### 2.1 汇聚点与既有钩子先例
- `runtime.dispatch_task(ctx, {"site","kind"})` 是全部 9 个下载任务 + 比对任务的**唯一执行入口**;
交互模式(`cli/router`)与服务模式(`cli/server` worker)都走它。
- 它的成功分支**已经有一个"下载后"钩子**:`_record_business_date(site, kind)`——写业务日期到
`state.db`best-effort,失败仅告警。新钩子天然挂在它旁边house style 完全一致。
- 4 站 `undelivered` 任务的 handler(`_site_undelivered_handler`)内部**直接调**
`TASK_HANDLERS[(site,"expected"/"actual")](ctx)`(不经 dispatch_task),故只触发**一次**外层
dispatch_task 钩子——`ingest_task(site,"undelivered")` 需据此同时入 expected+actual。
### 2.2 持久化模块现状
- `store.ingest(site=None)`:读 `downloads/` 现有 xlsx,幂等 UPSERT 到 PG。**站点级**:对顺心/中通/
韵达/安能同时入该站 expected+actual;百世只入 undelivered。**不感知 kind**。
- 三张表:`expected_record`(运单级) / `actual_record`(扫描件级) / `undelivered_record`(仅百世)。
- `_ingest_expected(cur, site, business_date)` / `_ingest_actual(cur, site)` /
`_ingest_undelivered_baishi(cur)` 为内部助手,直接复用,不改。
- `_connect(dbname)`:每次新建一条连接,`options=-c search_path=<schema>`。**经 cpolar 隧道**
(`5.tcp.cpolar.top:10364`),潜在延迟/抖动。
### 2.3 线程模型
- dispatch_task 跑在 Playwright 所属线程(router=主线程;server=单 worker 线程,串行消费
task_queue、空闲跑心跳)。**同步做 PG I/O 会阻塞这条线程**——故入库必须快、可超时、可降级。
### 2.4 数据流定位
- dashboard **不直接读 PG**:全部经 Next.js `/api/*` 代理到 InboundVerify FastAPI(读 state.db +
文件 + 汇总报表)。故自动入库的直接受益者是 **PG 这个下游数仓**(BI/长期归档/未来报表),
不是当前前端。这支撑了"同步内联、不上复杂调度"的判断。
### 2.5 state_store 现状(决定表设计)
- `site_status` 是**每站一行**、按 kind 展开列(`{kind}_ready/_generated_at/_business_date`)。
- `_upsert` 是**手写枚举列**的 read-modify-write(脆)。往里塞 ingest 列(4 列 × 3 kind = 12 列)
会很丑且易错——故选**独立 `ingest_state` 表**(用户选项里也提过"或一张很小的入库记录表")。
---
## 3. 四个关键决策(均已与用户确认)
| 决策 | 选定 | 理由 |
|---|---|---|
| 执行模型 | **同步内联** | 与 `_record_business_date` 一致、零新线程,贴合"刻意不优化结构"风格;cpolar 风险用超时+try/except 兜底 |
| 入库粒度 | **kind 级**(`ingest_task(site,kind)`) | 只入本次刚下载的文件,阻塞最小;"下了什么入什么";CLI `ingest(site)` 保留不动 |
| 配置门控 | **开关 + 连接超时** | `auto_ingest`(默认开)+ `connect_timeout_seconds`(默认 5);无 PG 开发机可关 |
| 失败可见性 | **stdout + state_store** | 仅 stdout 会在服务模式静默失败多天;落库后 `/api/status` 可查 |
---
## 4. 组件设计
### 4.1 `store.py` — 新增 `ingest_task(site, kind)` + 连接/配置增强
**新函数 `ingest_task(site, kind) -> int`**
单连接、单事务,复用现有 `_ingest_*`,返回总条数。路由表:
| site | kind | 调用 |
|---|---|---|
| `__compare__` | * | 无 → 返回 0 |
| 顺心/中通/韵达/安能 | `expected` | `_ingest_expected(cur, site, dates.get(site))` |
| 顺心/中通/韵达/安能 | `actual` | `_ingest_actual(cur, site)` |
| 顺心/中通/韵达/安能 | `undelivered` | `_ingest_expected` + `_ingest_actual` |
| 百世 | `undelivered` | `_ingest_undelivered_baishi(cur)` |
| 百世 | `expected`/`actual` | (无此任务)防御性返回 0 |
`dates = _read_business_dates()`(既有)。骨架:
```python
def ingest_task(site, kind):
"""按 (site, kind) 入库本次刚下载的文件(幂等 UPSERT)。返回总条数。
与 ingest(site) 的区别:只入本次刷新的那一类,避免重读写另一类文件。"""
if site == "__compare__":
return 0
dates = _read_business_dates()
total = 0
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
if kind == "expected":
total += _ingest_expected(cur, site, dates.get(site))
elif kind == "actual":
total += _ingest_actual(cur, site)
elif kind == "undelivered":
if site == "百世":
total += _ingest_undelivered_baishi(cur)
else: # 4 站:handler 内部连带下了 expected+actual
total += _ingest_expected(cur, site, dates.get(site))
total += _ingest_actual(cur, site)
# 其它组合(如 百世/expected):防御性 0
conn.commit()
return total
```
**`_load_pg_config()` 增键**:`auto_ingest`(默认 `True`)、`connect_timeout_seconds`(默认 `5`)。
**`_connect(dbname)` 增强**:
- `psycopg.connect(..., connect_timeout=c["connect_timeout_seconds"])`
- 连上后 `cur.execute("SET statement_timeout = '30s'")`(固定值,带注释说明可按需 knob 化;
connect_timeout 守隧道死连,statement_timeout 守失控查询,小 UPSERT 极少触发)。
- `create_database` / `init_schema` / `ingest` 共用此连接,30s 对它们无影响。
**新薄封装 `ingest_enabled() -> bool`**:读 `_load_pg_config()["auto_ingest"]`
供 runtime 钩子判定开关,避免钩子伸手进 store 私有函数。
**CLI `ingest-one`**(便于脱离下载单测路由):`python -m inbound_verify.store ingest-one 韵达 expected`
→ 直接调 `ingest_task(site, kind)``main()` 增该分支。
**不改**:既有 `ingest(site)` / `_ingest_*` / SQL / `domain`
### 4.2 `runtime.py` — 新增 `_persist_to_db(site, kind)`,挂到 `dispatch_task`
**`dispatch_task` 成功分支**(`_record_business_date` 之后)新增一行:
```python
ret = handler(ctx)
if ret is False:
return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)")
_record_business_date(site, kind)
_persist_to_db(site, kind) # 新增:尽力而为,绝不外抛,不影响任务判定
return (state_store.TASK_SUCCESS, None)
```
**新函数 `_persist_to_db(site, kind) -> None`**:
- `store` **懒导入**(函数内 `from inbound_verify import store`),彻底回避 import 顺序/成环,冷路径无性能影响。
- `if site == "__compare__": return`
- 读开关:`if not store.ingest_enabled(): print(">> [入库] 已关闭,跳过"); return`
- 主逻辑(所有写库/写状态都包 try/except,绝不外抛):
```python
try:
count = store.ingest_task(site, kind)
state_store.set_ingest_state(site, kind, ok=True, count=count)
print(f">> [入库] {site}/{kind} 成功,{count}")
except Exception as e:
print(f">> [warn] 入库失败 {site}/{kind}: {e}")
try:
state_store.set_ingest_state(site, kind, ok=False, error=str(e))
except Exception as e2:
print(f">> [warn] 写入库状态也失败: {e2}")
```
### 4.3 `state_store.py` — 新增独立表 `ingest_state` + 读写
**`init_db()` 增表(幂等)**:
```sql
CREATE TABLE IF NOT EXISTS ingest_state (
site TEXT,
kind TEXT,
ok INTEGER, -- 1=成功 0=失败
ingested_at TEXT,
count INTEGER, -- 入库条数
error TEXT, -- 失败原因(成功则 '')
PRIMARY KEY (site, kind)
)
```
**新函数**:
- `set_ingest_state(site, kind, ok, count=0, error=None)`:UPSERT(短连接,`ingested_at=_now()`,
与现有 `set_*` 风格一致)。
- `get_all_ingest_state() -> {site: {kind: {ok, ingested_at, count, error}}}`(库不存在返回 `{}`)。
**心跳不刷 `ingest_state`**(只由钩子写)。澄清:复用的是 state_store 管线 + `/api/status` 通道,
不是心跳循环。**不碰 `site_status` / `_upsert`**。
### 4.4 `cli/server.py` — `/status` 暴露 ingest 态
`GET /status` 现有返回 `get_all_status()`;追加一个 `ingest` 键 = `state_store.get_all_ingest_state()`
一行改动,前端可按需消费。
### 4.5 配置 — `config.example.yaml` + `config.yaml`
`postgres` 段新增(两文件都加;`config.yaml` 已 gitignore):
```yaml
postgres:
# ...既有 host/port/user/password/dbname/schema...
auto_ingest: true # 下载成功后自动入库;false=跳过(无 PG/cpolar 的开发机)
connect_timeout_seconds: 5 # PG 连接超时(秒);cpolar 抖动兜底
```
顺手把 `config.yaml` 注释里过期的 `db_store.py` 改为 `store.py`(小清理)。
---
## 5. 数据流
```
dispatch_task 成功
→ _record_business_date(site, kind) # 既有:SQLite 业务日期,best-effort
→ _persist_to_db(site, kind) # 新增
auto_ingest? ─ no ─→ print 跳过,return
└ yes ─→ 懒导 store
store.ingest_task(site, kind)
_connect(connect_timeout) → SET statement_timeout
路由 _ingest_* → commit → 返回 count
成功 → set_ingest_state(ok,count) + print
异常 → set_ingest_state(fail,error) + warn
→ return TASK_SUCCESS(始终)
```
---
## 6. 错误处理矩阵
| 情形 | 行为 | 任务判定 |
|---|---|---|
| `auto_ingest=false` | print 跳过,return | SUCCESS(不变) |
| 文件缺失(下载刚成功却无文件,罕见) | `_ingest_*``[跳过]` 返回 0;ingest_task 返回 0;记 ok/count=0 | SUCCESS |
| PG 不可达 / connect_timeout / statement_timeout | psycopg 异常 → 捕获 → fail + warn | SUCCESS |
| 表未初始化(UndefinedTable) | 捕获 → fail + warn(提示"请先 `inbound-verify-db init`") | SUCCESS |
| `set_ingest_state` 自身失败 | 再包 try/except,绝不外抛(呼应 `_record_business_date`) | SUCCESS |
**核心不变式:入库的任何失败都不改变下载任务的成功判定。**
---
## 7. 测试
无 pytest(house 约定)。验证手段:
1. **路由单测(新 CLI)**:`.venv/Scripts/python.exe -m inbound_verify.store ingest-one 韵达 expected`
→ 确认只入韵达应到,不动韵达实到;`ingest-one 百世 undelivered` → 只入百世未到;
`ingest-one 顺心 undelivered` → 入顺心 expected+actual。
2. **端到端**:dashboard/API 触发一次下载 → 看 stdout `[入库]` 行 → 查 PG 行数 →
`GET /api/status``ingest` 字段反映 ok/count/ingested_at。
3. **降级**:`auto_ingest=false` → 下载仍 SUCCESS、无 `[入库]` 行;临时封掉 cpolar 端口 →
下载仍 SUCCESS、`[warn] 入库失败``ingest_state` 记 fail、`/status` 可见。
4. **回归**:`py_compile inbound_verify` + `black inbound_verify` + `compileall` 导入冒烟;
确认未改 `ingest(site)` 行为(手动跑一次 `store ingest` 全站入库仍正常)。
---
## 8. 已确认约束
- InboundVerify 是 git **子模块**;改动在子模块内,父仓库仅跟踪指针。
- **不加测试套件**;验证靠编译/导入冒烟 + 手动端到端。
- 改完 Python **必须跑 Black**
- **不自动提交/推送**:本仓库约定改动后等用户明确说"提交"再 commit/push(覆盖全局 auto-push 默认,
亦覆盖 brainstorming 默认的"写完即提交")。
---
## 9. 实现顺序提示(供 writing-plans 展开)
1. `store.py`:`_load_pg_config` 加键 + `_connect` 加超时/语句超时 → `ingest_task` → CLI `ingest-one`
2. `state_store.py`:`ingest_state` 建表 + `set_ingest_state` + `get_all_ingest_state`
3. `runtime.py`:`_persist_to_db` + 挂到 `dispatch_task`
4. `cli/server.py`:`/status``ingest` 字段。
5. 配置:`config.example.yaml` + `config.yaml` 加键 + 修过期注释。
6. Black + 编译/导入冒烟 + 手动端到端验证。

View File

@@ -0,0 +1,430 @@
# 安能全网门户 CDP 连接与驱动指南
> 目标:连接到一个**已经以调试模式启动**的「安能全网门户」Electron 应用,像 Playwright 一样查看和操作界面元素。
>
> 实测环境Windows 11、Electron 19 / Chrome 102、Python 3.13、`playwright 1.60`、`websocket-client 1.9`。
---
## 1. 启动应用
用 Electron/Chromium 的远程调试参数启动可执行文件:
```bash
"安能全网门户.exe" --remote-debugging-port=9222
```
启动后,应用会在 `http://localhost:9222` 暴露一套 **Chrome DevTools Protocol (CDP)** 的 HTTP + WebSocket 接口。
快速自检(任意浏览器或 curl
```bash
curl http://localhost:9222/json/version # 确认端口在线、看 Electron/Chrome 版本
curl http://localhost:9222/json # 列出所有「页面目标」
```
`/json/version``User-Agent` 会带 `Electron/19.0.4 Chrome/102.0.5005.63`,据此判断 Chromium 内核版本(影响后续兼容性判断)。
---
## 2. 技术路线选型(重要)
### ❌ 路线 APlaywright `connect_over_cdp` —— 不可用
直觉上最省事,但实测在这个 Electron 构建上**握手后立刻断开**
```
playwright._impl._errors.Error: BrowserType.connect_over_cdp:
Protocol error (Browser.setDownloadBehavior): Browser context management is not supported.
```
**原因**Playwright 在 CDP 连接建立后,会调用浏览器域的 `Browser.setDownloadBehavior`
来初始化默认下载上下文;而这个 Electron 19 / Chrome 102 构建的 CDP **只部分实现了浏览器域**
对“浏览器上下文管理”一类命令直接返回 not supported于是连接被关闭。
**为什么不能靠降级 Playwright 绕过**:该行为由来已久,需要降到 2022 年的 Playwright 才可能规避;
但本机 venv 是 **Python 3.13**,而 `playwright < 1.48` 没有提供 3.13 的 wheel装不上。
### ✅ 路线 B页面级 CDP本项目采用
关键观察:**Electron 的浏览器域受限,但页面域(`Page` / `DOM` / `Runtime`)完全正常。**
因此绕开 Playwright 的高层连接,直接:
1. `GET http://localhost:9222/json` 拿到所有页面目标的 `webSocketDebuggerUrl`
2. 用 WebSocket 直连某个**页面目标**(不是 browser 目标);
3. 在该 WebSocket 上收发 CDP 消息(`Runtime.evaluate` / `DOM.*` / `Input.*` …)。
在上面再封装一层 Playwright 风格的薄封装(`Page.eval` / `Page.query_all` / `Page.click` …),
用起来接近 Playwright又不受 Electron 浏览器域限制。
> 备选:也可用 [`pychrome`](https://github.com/mineking0175/pychrome) 这类 CDP 客户端,它同样是页面级、不做上下文管理,
> 同样不会踩这个坑。本项目为减少依赖,直接用已安装的 `websocket-client` 手写。
---
## 3. 连接原理
### 3.1 发现目标
`GET /json` 返回一个数组,每个元素形如:
```json
{
"id": "4E4584692AF9A17D12E70E024F6DC938",
"type": "page",
"title": "安能全网门户",
"url": "file:///.../app.asar/website/index.html#/ai-button",
"webSocketDebuggerUrl": "ws://localhost:9222/devtools/page/4E458469..."
}
```
- `type == "page"` 的才是普通页面(另有 `background_page``service_worker``webview` 等,按需过滤)。
- **`id` / `webSocketDebuggerUrl` 每次启动都会变**,所以一定要动态发现,**绝不能硬编码**(早期 `sites/anneng.py` 硬编码 WS URL重启就失效
-`title``url` 来挑选你真正要操作的那一页。
### 3.2 CDP 消息往返
CDP 是 JSON-RPC 风格:客户端发 `{id, method, params}`,服务端回 `{id, result}``{id, error}`
服务端也会**主动推送事件** `{method, params}`(没有 `id`)。
所以收消息时要**过滤掉事件**,只取 `id` 与本次请求匹配的那条回复:
```python
while True:
msg = json.loads(ws.recv())
if msg.get("id") == my_id:
return msg.get("result", {})
# 否则是事件,忽略
```
### 3.3 取值:`Runtime.evaluate` + `returnByValue`
直接执行 JS让浏览器把结果序列化成 JSON 带回来:
```python
result = call("Runtime.evaluate",
expression="document.title",
returnByValue=True, awaitPromise=True)
title = result["result"]["value"]
```
- `returnByValue=True`:把 JS 返回值按值序列化为 JSON适合结构化数据
- `awaitPromise=True`:表达式中若返回 Promise 会被自动 await写普通表达式也无副作用
绝大多数“查询/读取”需求都能用一段 JS + `Runtime.evaluate` 解决,比走 `DOM.*` 更直观。
---
## 4. 可运行的最小封装
下面是项目里 `_probe_anneng.py` 的核心结构(精简版),可直接复用:
```python
"""页面级 CDP连接 安能全网门户(:9222Playwright 风格的薄封装。"""
import json
import sys
import urllib.request
import websocket
sys.stdout.reconfigure(encoding="utf-8") # Windows 控制台是 GBK否则打印中文/emoji 会崩
CDP_URL = "http://localhost:9222"
def list_pages():
with urllib.request.urlopen(f"{CDP_URL}/json") as resp:
data = json.load(resp)
return [p for p in data if p.get("type") == "page"]
class CDP:
"""绑定到单个页面目标的同步 CDP 客户端。"""
def __init__(self, ws_url):
self.ws = websocket.create_connection(ws_url, timeout=15) # 15s socket 超时:防 Electron 业务 tab 偶发不回包时 recv 无限阻塞
self._id = 0
def call(self, method, **params):
self._id += 1
self.ws.send(json.dumps({"id": self._id, "method": method, "params": params}))
while True:
msg = json.loads(self.ws.recv())
if msg.get("id") == self._id:
if "error" in msg:
raise RuntimeError(f"{method} failed: {msg['error']}")
return msg.get("result", {})
def eval(self, expression):
res = self.call("Runtime.evaluate", expression=expression,
returnByValue=True, awaitPromise=True)
return res.get("result", {}).get("value")
def close(self):
self.ws.close()
class Page:
def __init__(self, cdp):
self.cdp = cdp
cdp.call("Page.enable")
cdp.call("Runtime.enable")
@property
def title(self):
return self.cdp.eval("document.title")
def query_all(self, selector, limit=20):
return self.cdp.eval(
"(() => [...document.querySelectorAll(%r)].slice(0,%d).map(e => ({"
"tag:e.tagName.toLowerCase(), text:(e.innerText||'').trim().slice(0,40),"
"id:e.id||'', cls:(e.className||'').toString().slice(0,40),"
"title:e.getAttribute('title')||'' })))()" % (selector, limit)
) or []
```
用法:
```python
pages = list_pages()
# 挑你要的那一页(示例:取标题里含“鱼洞”的那页,即应用外壳)
target = next(p for p in pages if "鱼洞" in p["title"])
page = Page(CDP(target["webSocketDebuggerUrl"]))
print(page.title)
print(page.query_all("div.rc-menu-submenu-title"))
```
---
## 5. 关键发现:界面结构与定位
实测两个页面目标:
| 页面 title | url 片段 | 说明 |
|---|---|---|
| `安能全网门户` | `…/index.html#/ai-button` | 仅 AI 按钮落地页,几乎是空的,**不是主界面** |
| `重庆鱼洞镇` | `…/index.html` | **真正的应用外壳**,左侧整棵导航菜单都在这里 |
主界面(“重庆鱼洞镇”页)的左侧一级菜单(实测抓取):
```
基础数据 / 客服 / 运单管理 / 运营管理 / 扫描操作 / 物料管理 /
网点金融 / 自营客户管理 / 综合查询 / 新闻与问卷 / 问题反馈
```
**定位特征**React + styled-components + rc-menu 技术栈):
- 一级菜单项:`div.rc-menu-submenu-title[title='运营管理']`
- 站点信息(顶部):`div[title='02330019400049']`
- 菜单搜索框:`input[placeholder*='菜单搜索']`
- 折叠按钮:`div[title='折叠菜单']`
> 这是 React 应用,**class 名是 styled-components 生成的散列**(如 `sc-dntSTA ginXyJ`
> **不要**用这些散列 class 做定位——换一次构建就变。优先用 `[title=…]`、`[role=…]`、
> 稳定的 `rc-menu-*` 类名或文本内容来定位。
---
## 6. 独立 webContentstab 页(独立网页)的元素操作
应用里有些功能(如「导出下载」)点击导航菜单后,会在右侧打开一个 **tab 页**
这类 tab 的内容是**独立的 webContents一个远程网页**,而不是主页面里的普通 iframe。
**判据**(出现以下现象即说明是独立 webContents
- 用主窗口 DevToolsCtrl+Shift+I的元素拾取能抓导航栏却**抓不到 tab 内的元素**
- 在 tab 内点右键只有「检查」一项,点开后会**另开一个 DevTools 窗口**,在那里才能拾取 tab 元素。
**关键结论**:这种 tab 会作为 `/json` 里一个**独立的 page 目标**出现(`url` 是远程 https 地址)。
所以**不需要任何特殊 API** —— 把它当成“又一个页面”:按 URL 从 `/json` 里挑出来、直连它的
`webSocketDebuggerUrl`,用同一套 `Runtime.evaluate` / `DOM` 操作即可,和主页面毫无区别。
**打开并定位 tab 目标**的通用做法(已打开就按 URL 复用,没打开就点菜单触发后轮询新目标):
```python
import time
def find_or_open_tab(url_hint, menu_text=None, timeout=20):
"""按 url_hint 找到 tab 目标;没打开就点 menu_text 菜单触发,再轮询新目标。"""
# 1) 已打开则直接复用(不必重复点击)
for p in list_pages():
if url_hint in p.get("url", ""):
return CDP(p["webSocketDebuggerUrl"])
# 2) 未打开 → 点导航菜单触发,再轮询新出现的目标
assert menu_text, "tab 未打开且未提供 menu_text"
main = next(CDP(p["webSocketDebuggerUrl"]) for p in list_pages()
if p.get("title") and "鱼洞" in p["title"]) # 主应用外壳页
before = {p["id"] for p in list_pages()}
main.eval(
"[...document.querySelectorAll('li.rc-menu-item')]"
f".find(e => e.textContent.trim() === {json.dumps(menu_text)})?.click()"
)
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
for p in list_pages():
if url_hint in p.get("url", "") and p["id"] not in before:
return CDP(p["webSocketDebuggerUrl"])
time.sleep(0.5)
raise TimeoutError("tab 目标未出现")
```
> 实测「导出下载」tab 的 URL 含 `exportAllRecords`,打开后即是一个远程 https 页面目标,
> 连上后读表格、点按钮等都与主页面完全一样。
---
## 7. 实战:免对话框下载导出文件
目标:触发导出下载并自动存到默认下载目录,**不弹 Windows 保存对话框**。
**核心经验:下载不要靠点页面按钮,直接 GET 下载接口即可。**
(点按钮会走 Electron 主进程的下载流程并弹原生保存对话框CDP 无法从外部抑制;
而直接发 HTTP 请求取回文件字节,根本不经过那条流程,自然无对话框。)
**下载接口与鉴权**实测自「导出下载」tab
| 项 | 值 |
|---|---|
| 接口 | `GET https://uep.ane56.com/uep/foreign/api/fileDownloadRecord/download` |
| 参数 | `fileId=<文件路径>``appId=LB``aneFile=false``index=<同 fileId>` |
| 鉴权头 | `x-auth` = `sessionStorage["x-auth"]`base64 JWT |
| cookie | `TGC`httpOnly`Network.getCookies` 读取;`document.cookie` 取不到) |
| 响应 | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`xlsx |
**文件名**:用表格「文件名」列的可读名(如 `交接单明细-20260621_153527.xlsx`
服务端 `Content-Disposition` 里给的是 UUID 名,不要用。
**完整步骤**CDP 取鉴权 + Python `urllib` 下载):
```python
import urllib.parse
import urllib.request
# 1) 连到「导出下载」tab 目标,取出鉴权信息
cdp = find_or_open_tab("exportAllRecords", menu_text="导出下载")
x_auth = cdp.eval('sessionStorage.getItem("x-auth")')
ua = cdp.eval("navigator.userAgent")
referer = cdp.eval("location.href")
tgc = next(c["value"] for c in
cdp.call("Network.getCookies", urls=["https://uep.ane56.com"])["cookies"]
if c["name"] == "TGC")
# 2) 直接 GET 下载接口(不点按钮 → 不弹对话框)
params = {"fileId": file_id, "appId": "LB", "aneFile": "false", "index": file_id}
url = ("https://uep.ane56.com/uep/foreign/api/fileDownloadRecord/download?"
+ urllib.parse.urlencode(params))
req = urllib.request.Request(url, headers={
"x-auth": x_auth, "Cookie": f"TGC={tgc}",
"Referer": referer, "User-Agent": ua, "Accept": "*/*",
})
data = urllib.request.urlopen(req, timeout=30).read()
assert data[:2] == b"PK", "非 xlsx" # ZIP 魔数校验
open(r"C:\Users\pengq\Downloads\交接单明细-xxx.xlsx", "wb").write(data)
```
实测GET 返回 `200` / 21508 字节 / `PK` 头合法,全程**不点按钮、不弹对话框、不留 `<uuid>.tmp`**。
完整可运行脚本见项目 `download_clean.py`
> **关于 `fileId`**:下载按钮的 `<a>` 没有 href是 JS 点击),`fileId` 不在可见 DOM 里。
> `download_clean.py` 当前用抓包所得的 `fileId` 验证通过;要批量下载多行,需从表格行的
> React record 里取每行的 `fileId`(路径模式 `/task/YYYYMMDD/<uuid>.xlsx`)。
---
## 8. 常见坑
1. **页面 ID 会变**:每次重启应用,`/json` 里的 `id` / `webSocketDebuggerUrl` 都不同 → 必须运行时动态发现。
2. **Windows 控制台编码**:默认 GBK打印中文/emoji 抛 `UnicodeEncodeError` → 脚本里加
`sys.stdout.reconfigure(encoding="utf-8")`,或设环境变量 `PYTHONUTF8=1` 运行。
3. **必须过滤 CDP 事件**recv 循环里要跳过没有匹配 `id` 的事件消息,否则会把事件当成回复解析出错。
4. **直连 page 目标,不是 browser 目标**`/json/version` 里的 `webSocketDebuggerUrl` 是 browser 级 WS
连它同样会触发上下文管理类命令的坑;要用 `/json` 里各 page 的 WS。
5. **不要关掉用户的进程**:我们只是“附加”到外部进程,`ws.close()` 只断开自己的连接,不会杀掉 Electron。
---
## 9. 扩展:点击与输入
### 9.1 点击JS 触发,最简单)
```python
def click(self, selector):
self.cdp.eval(
"document.querySelector(%r)?.click() || false" % selector
)
```
适合菜单展开、按钮提交这类纯逻辑点击。
### 9.2 点击(真实鼠标事件,需要坐标时)
`Input.dispatchMouseEvent` 模拟真实移动/按下/抬起,坐标来自元素包围盒:
```python
def click_real(self, selector):
box = self.cdp.eval(
"(() => {const r=document.querySelector(%r).getBoundingClientRect();"
"return {x:r.x+r.width/2, y:r.y+r.height/2};})()" % selector
)
for t in ("mouseMoved", "mousePressed", "mouseReleased"):
self.cdp.call("Input.dispatchMouseEvent", type=t,
x=box["x"], y=box["y"], button="left", clickCount=1)
```
### 9.3 文本输入React 应用特别注意)
React 受控组件**不会响应**直接赋的 `el.value = '...'`。要触发它认得的 `input` 事件:
```python
def fill(self, selector, text):
self.cdp.eval(
"(() => {const el=document.querySelector(%r);"
"const s=Object.getOwnPropertyDescriptor(el.constructor.prototype,'value').set;"
"s.call(el, %r);"
"el.dispatchEvent(new Event('input',{bubbles:true}));"
"})()" % (selector, text)
)
```
### 9.4 等待元素出现
CDP 没有现成的 `wait_for_selector`,轮询最稳:
```python
import time
def wait_for(self, selector, timeout=10, interval=0.3):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if self.cdp.eval(f"!!document.querySelector({selector!r})"):
return True
time.sleep(interval)
raise TimeoutError(selector)
```
---
## 10. 运行环境约定
- 所有 Python 脚本在项目 `.venv` 内运行(`python -m venv .venv`,激活后安装依赖)。
- 依赖:`websocket-client`(已装)、`black`(格式化,已装)、可选 `pychrome`
- 相关脚本:`inbound_verify/sites/anneng.py`(菜单导航 + CDP 驱动;早期探查脚本 `_probe_anneng.py`、免对话框下载脚本 `download_clean.py` 已并入此模块,不再单独存在)。
- 运行:`.venv/Scripts/python.exe -m inbound_verify.sites.anneng expected|actual`
---
## 11. 速查清单
| 任务 | 做法 |
|---|---|
| 确认应用在线 | `curl http://localhost:9222/json/version` |
| 列出页面 | `curl http://localhost:9222/json`(或 `list_pages()` |
| 选目标页 | 按 `title` / `url` 过滤,**别用 id** |
| 连接 | 直连该页的 `webSocketDebuggerUrl` |
| 读取元素 | `Runtime.evaluate` + 一段 `querySelector` JS |
| 点击 | JS `.click()``Input.dispatchMouseEvent` |
| 输入React | 走原型 setter + `dispatchEvent('input')` |
| 操作 tab 页(独立网页) | 它是 `/json` 里独立 page 目标,按 URL 取出直连即可 |
| 免对话框下载 | **别点按钮**;用 `x-auth`(sessionStorage)+`TGC`(Network.getCookies) 直接 GET 下载接口 |
| 格式化代码 | `black` |
| 终端乱码 | `sys.stdout.reconfigure(encoding="utf-8")` |

View File

@@ -0,0 +1,120 @@
# 后端各站点「应到件数 / 已到件数」统计逻辑审查报告
> 审查范围:`InboundVerify` 后端
> 核心模块:`compare.py`(比对统计)、`domain.py`(站点/文件/列配置)、`runtime.py`(调度/下载路由)、`sites/*.py`(各站数据采集)、`cli/server.py`(对外接口)
> 审查重点:每个站点的「应到件数」和「已到件数」如何统计、数据来源、口径与潜在歧义。
---
## 一、总体架构(两层)
统计逻辑分两层,前端看到的「应到/已到/未到」计数最终都来自**第二层**,且只在跑比对时才生成。
| 层 | 模块 | 职责 | 产物 |
|---|---|---|---|
| **① 数据采集层** | `sites/{顺心,中通,韵达,安能,百世}.py` | 用 Playwright安能用 CDP登录各承运商后台导出 Excel 到 `downloads/` | `downloads/<站>-应到货物数据.xlsx``downloads/<站>-实到货物数据.xlsx``downloads/<站>-未到数据.xlsx` |
| **② 比对统计层** | `compare.py`(站点/文件/列配置取自 `domain.py` | 读 `downloads/` 源文件,两表比对,算出应到/实到/未到件数并出报表 | `output/应到未到数据.xlsx`(汇总+各站明细) |
调度入口:`runtime.py``TASK_HANDLERS`
- 4 站:`expected` 下载 + `actual` 下载 + `undelivered`(先下应到+实到,再调 `write_site_file` 算出未到)。
- 百世:只有 `undelivered`(直接导「当日未扫」明细,无应到/实到两份基表)。
- `__compare__`(菜单[9]):调 `compare.main()`,用 `downloads/` 现有文件生成全站汇总报表。
---
## 二、核心统计公式4 站统一口径)
`compare.process(name)` 是 4 站统计的唯一实现:
```
应到(按运单号去重,保留首条):
应到件数 N = 应到数据中的「交接件数」 # 注意:用「交接件数」,不是「录单件数」
(录单件数只是该单号总录单量,并非真正到站量)
应到件数 += N # 站点级应到 = Σ N
实到(直接数,不再由「应到−未到」倒推):
按「运单号」把实到表的「单号/子单号/扫描单号」分组,组内去重计数
→ 每个运单的实到件数;站点级实到 = Σ 各运单实到件数
未到:
未到件数 = max(0, 应到件数 实到件数) # 站点级
未到率 = 未到件数 ÷ 应到件数
短少运单 = 实到件数 < 应到件数 N 的运单
(未到明细 downloads/<站>-未到数据.xlsx 只列这些短少运单,
每行:交接单号 | 运单号 | 总件数(=N) | 已到单号1 | 已到单号2 | …)
```
**关键事实**`实到件数` 是直接数实到表「单号」、按运单号分组去重得到的(**不再由「应到−未到」倒推**`未到件数 = 应到件数 实到件数`。前提是实到单号能正确按运单号分组(分组规则见下表各站 `arrived_pieces_*`)。
---
## 三、各站点统计明细
| 站点 | 应到件数来源 | 实到件数来源 | 单号→运单号 分组规则(`arrived_pieces_*` | 源文件(`downloads/` |
|---|---|---|---|---|
| **中通** | `中通-应到货物数据.xlsx` 的「交接件数」 | 实到表单号去重(直接数) | 实到`运单号`是复合串 = 运单号 + 总数(4) + 顺序(4);按 `v[:-8]` 归并到应到运单号,每条复合串即 1 件 | `中通-应到货物数据.xlsx` / `中通-实到货物数据.xlsx` |
| **顺心** | `顺心-应到货物数据.xlsx` 的「交接件数」 | 实到表单号去重(直接数) | 按`运单号`分组,`子单号`=每件(一件一个子单号) | `顺心-应到货物数据.xlsx` / `顺心-实到货物数据.xlsx` |
| **韵达** | `韵达-应到货物数据.xlsx` 的「交接件数」 | 实到表单号去重(直接数) | 按`主单号`分组,`子单号`=每件 | `韵达-应到货物数据.xlsx` / `韵达-实到货物数据.xlsx` |
| **安能** | `安能-应到货物数据.xlsx` 的「交接件数」 | 实到表单号去重(直接数) | 按`所属单号`分组,`扫描单号`=每件 | `安能-应到货物数据.xlsx` / `安能-实到货物数据.xlsx` |
| **百世** | **无**(站点只给未到) | **无(显示「—」)** | 不适用(无实到基表) | 仅 `百世-应到未到货物数据.xlsx`= 当日未扫明细,本身就是未到结果) |
> 4 站合计/图表口径:`compare.build_summary` 只累加 4 站(`应到−实到`口径),**百世不计入合计**(无应到基数)。百世在表中单列,未到件数 = 其明细行数。
---
## 四、百世的特殊口径(务必注意)
百世是唯一「无应到/实到基数」的站点:
- 它的数据来自后台「扫描综合查询 → 到/接件扫描 → 当日 → 未扫」,导出即「当日未扫」明细(`sites/baishi.py`)。
- 因此 `process_baishi()` 只能给 `未到件数 = 行数``应到件数 = None``已到件数 = None`、完全/部分未到 = None。
- 报表里百世的应到/已到列显示「—」,未到率无法计算。
- **含义**:百世统计的是「今天还没扫到的件」,不是「相对应到总量的缺件率」。与 4 站口径不可直接相加比较。
---
## 五、数据日期与偏移(潜在口径不一致风险)
`runtime._record_business_date` 在下载成功后把业务日期写进状态库:
- `业务日期 = 下载当天 日期偏移`
- 各站 `expected_offset` / `actual_offset` 独立配置(前端 `/config` 可改;百世偏移恒 0锁定当天
- 韵达默认 `expected_offset=1`(取前一日应到)。
**风险点**4 站的「应到」和「实到」是**两次独立下载**,各自可能带不同偏移。若 `expected_offset ≠ actual_offset`,则「应到件数」和「已到件数」来自**不同业务日期**,比对会变成「拿昨天的应到对比今天的实到」,未到率失真。报表「数据日期」列分别标注各站,但汇总合计不标注,肉眼难发现。
---
## 六、统计结果如何暴露给前端
| 接口 | 返回内容 | 是否含应到/已到计数 |
|---|---|---|
| `GET /status` | 各站 `login_state` + `expected/actual/undelivered_ready` + `business_date` + `worker_ready` + `ingest`(每站每类入库 ok/count/时间) | **不含**应到/已到件数计数(`ingest` 是入库条数,非核销件数) |
| `GET /report` | `FileResponse(output/应到未到数据.xlsx)` | 计数只在 xlsx 里 |
| `GET /data/{filename}` | 下载 `downloads/` 下某源文件 | 原始数据,非统计值 |
**结论**:后端**没有**把应到/已到件数以 JSON 形式实时返回前端。计数仅物化在 `output/应到未到数据.xlsx`。任何前端界面显示的应到/已到数字,都是解析这份 xlsx 得到的——即**「截至上次跑比对」的快照**,不是实时值。
---
## 七、潜在歧义与风险点(审查结论)
1. **实到依赖单号→运单号分组正确**:实到件数靠把实到表「单号」按运单号分组去重计数(`arrived_pieces_*`)。一旦某站单号格式与分组规则不匹配(如复合串切分错),该件归不到对应运单 → 实到被低估、未到率虚高。规则硬编码,承运商改版号段即失准。
2. **应到件数依赖「交接件数」**:若应到数据某运单 `交接件数` 缺失/为 0/非数字,该运单被跳过,既不计入应到也不计入未到 → 静默漏统(应到总量被低估;该运单即便出现在实到中也因不在应到循环而无处抵扣)。
3. **应到按运单号去重keep first**:同一运单多条交接记录只取首条 `录单件数`。若重复行的件数不同,取首条,可能与实际不符。
4. **百世不可并入合计**4 站合计的「已到总件数」不含百世;跨站看「已到」时别把百世当成有应到基数的站。
5. **应到/实到业务日期可能错位**(见第五节):两表不同步下载或偏移不一致时,比对口径失真。
6. **计数是比对产物,非实时**:前端若要「实时件数」需先触发比对任务;`/status``ready` 仅表示「下载成功」,不代表「已比对出数」。
7. **子单号是程序现拼的**4 站缺件的「子单号/扫描单号」由 `code_*` 按各站编号规则生成(`code_shunxin` 等),并非实到原始记录——明细里的缺件单号是推算值,用于人工核对,不是系统回执。
---
## 八、关键文件索引
| 文件 | 角色 |
|---|---|
| `compare.py` | 统计核心:`process()`4站比对`process_baishi()`(百世)、`build_summary()`(汇总报表) |
| `domain.py` | 站点/文件名/列映射配置:`STATIONS`(各站解析配置)、`arrived_pieces_*`(实到单号→运单号分组)、`_site_cfg` |
| `runtime.py` | `TASK_HANDLERS`(下载/比对路由)、`_site_undelivered_handler`4站先下应到+实到再算未到)、`_record_business_date`(业务日期/偏移写入) |
| `sites/{中通,顺心,韵达,安能}.py` | 各站 `expected_download` / `actual_download`Playwright/CDP 导出源表) |
| `sites/baishi.py` | `baishi_download_undelivered_data`(直接导「当日未扫」) |
| `state_store.py` | `site_status`ready/business_date`site_config`offset/schedule`get_offset` |
| `cli/server.py` | `/status``/report``/data/{filename}` 接口 |

View File

@@ -0,0 +1,124 @@
# 韵达 / 安能 应到未到计算逻辑梳理(基于 inbound_verify/compare.py 重构版代码)
> 用途:供审核两站「应到件数 / 实到件数 / 未到件数」的实现逻辑与关联键。
> 代码基准:`InboundVerify/inbound_verify/compare.py`(重构版)。
---
## 、两站共用的计算骨架process 函数)
无论韵达还是安能,最终都走同一个 `process(name)`
1. 读应到表 + 实到表(第 148149 行)。
2. 应到表按关联键列去重、保留首条(第 152153 行)。
3. **应到件数** = 应到表「交接件数」之和(第 158168 行)。
4. **实到件数** = 实到表单件号列的**全局去重数**(第 176177 行,`act_pieces = sum(len(s) for s in arrived.values())`)。
5. 逐运单比较:实到扫到数 `<` 应到件数 的运单,才进未到明细(第 183201 行)。
6. **未到件数** = `max(0, 应到件数 实到件数)`(第 210 行)。
两站差异**只在配置**(关联键列、单件号列不同),算法完全一致。
---
## 一、韵达
### 1.1 配置STATIONS 第 106115 行)
| 配置项 | 值 | 含义 |
|---|---|---|
| exp应到文件 | `韵达-应到货物数据.xlsx` | 源 |
| act实到文件 | `韵达-实到货物数据.xlsx` | 源 |
| exp_qty | `交接件数` | 应到件数的取值列 |
| exp_wb | `运单号` | 应到表去重键 **+ 关联键** |
| exp_jd | `交接单号` | 未到明细展示字段 |
| arrived_pieces | `arrived_pieces_by_cols("主单号", "子单号")` | 实到解析 |
|  ├ wb_col | `主单号`(实到表) | 实到侧**关联基号**列 |
|  └ piece_col | `子单号`(实到表) | 实到侧**单件号**列 |
### 1.2 两套表的字段角色
- **应到表**`运单号`(去重+关联)、`交接件数`(该单应到件数)、`交接单号`(展示)。
- **实到表**`主单号`(关联基号,须与应到`运单号`同值域)、`子单号`(每件货物的单号,一个单号=一件)。
### 1.3 关联Join条件
```
应到表.运单号 == 实到表.主单号
```
机制:`arrived_pieces_by_cols` 以「主单号」为 key 建 dict第 7681 行);`process` 第 185 行 `arrived.get(wb)` 用**应到运单号 wb** 去查该 dict。两列值必须相等才对得上。
### 1.4 计算口径
- 应到件数 = Σ 交接件数(按运单号去重后)。
- 实到件数 = 实到表「子单号」全局去重数(不同主单号下即使子单号文本相同也只计一次)。
- 未到件数 = `max(0, 应到 实到)`
- 未到明细(每行一个短少运单):`交接单号 | 运单号 | 总件数(=应到件数) | 已到单号1..k`
「已到单号1..k」= 该主单号下所有子单号排序后填入k = 实际扫到件数(第 199200 行)。
仅当「实到扫到数 < 应到件数」才列入(第 187188 行 `arrived_cnt >= n` 跳过)。
### 1.5 当前数据状态实测2026-07-19 10:50
- 应到运单数 **57**,实到基号数 **58**,交集命中 **57**
- 应到件数 = 140实到单号去重 = 141
- 样本:`应到 617936212` == `实到主单号 617936212`,完全对齐
- **结论:韵达关联键现已对齐,逻辑可正确产出结果。**(此前 2/57 为旧数据,已失效)
---
## 二、安能
### 2.1 配置STATIONS 第 116125 行)
| 配置项 | 值 | 含义 |
|---|---|---|
| exp应到文件 | `安能-应到货物数据.xlsx` | 源 |
| act实到文件 | `安能-实到货物数据.xlsx` | 源 |
| exp_qty | `交接件数` | 应到件数取值列 |
| exp_wb | `运单号` | 应到表去重键 **+ 关联键** |
| exp_jd | `交接单号` | 未到明细展示字段 |
| arrived_pieces | `arrived_pieces_by_cols("所属单号", "扫描单号")` | 实到解析 |
|  ├ wb_col | `所属单号`(实到表) | 实到侧**关联基号**列 |
|  └ piece_col | `扫描单号`(实到表) | 实到侧**单件号**列 |
### 2.2 两套表的字段角色
- **应到表**`运单号`(去重+关联)、`交接件数`(应到件数)、`交接单号`(展示)。
- **实到表**`所属单号`(关联基号,须与应到`运单号`同值域)、`扫描单号`(每件单号,格式=`所属单号`+总数4位+顺序4位一个=一件)。
### 2.3 关联Join条件
```
应到表.运单号 == 实到表.所属单号
```
机制同韵达:`arrived_pieces_by_cols` 以「所属单号」建 key dict`process` 用应到运单号去查。
### 2.4 计算口径(与韵达结构完全一致)
- 应到件数 = Σ 交接件数。
- 实到件数 = 实到表「扫描单号」全局去重数。
- 未到件数 = `max(0, 应到 实到)`
- 未到明细:`交接单号 | 运单号 | 总件数 | 已到单号1..k`k=该所属单号下扫描单号数)。
### 2.5 当前数据状态实测2026-07-19 10:50
- 应到运单数 **97**,实到基号数 **140**,交集命中 **4**
- 应到件数 = 236实到单号去重 = 279
- 样本:`应到 750101236894` vs `实到所属单号 760237639793` —— 两套不同编号体系
- **后果**:实到 279 件几乎全无法对应到任何应到运单 → 明细把 93 个运单列「完全未到」,但汇总层 `实到=279 > 应到=236` → 未到净额被 `max(0,…)` 截断为 **0**,于是「明细列 96 行短少」与「汇总未到=0」自相矛盾。
- **结论:安能关联键未对齐,当前产出不可信。**
---
## 三、两站逻辑差异对照
| 维度 | 韵达 | 安能 |
|---|---|---|
| 应到关联键列 | `运单号` | `运单号` |
| 实到关联基号列 | `主单号` | `所属单号` |
| 实到单件号列 | `子单号` | `扫描单号` |
| 单件号是否带后缀 | 是(主单号+顺序号) | 是(所属单号+总数+顺序) |
| 关联键是否对齐(当前数据) | ✅ 57/57 | ❌ 4/97 |
| 计算结果是否可信 | 可信 | 不可信(自相矛盾) |
> 两站**算法完全相同**,唯一区别是实到侧的「基号列 / 单件号列」列名不同。因此问题不在代码逻辑,而在**安能的关联键取值域对不上**。
---
## 四、审核要点(请你判断)
1. **韵达**:关联键 `应到.运单号 == 实到.主单号` 是否符合业务实际?当前数据已对齐,似乎正确;若你确认,韵达逻辑可定稿。
2. **安能**:关联键 `应到.运单号 == 实到.所属单号` 是否正确实测两套编号不重合97 个应到运单仅 4 个能在实到找到)。可能的方向:
- (a) 实到表存在另一列能与应到`运单号`对应(需确认列名,可能改 `wb_col`
- (b) 应到/实到文件是按不同条件/批次拉的,需要按同一天、同线路重新拉取;
- (c) 暂时把安能排除(同百世),等数据对齐再加回。
3. **共同结构问题**:汇总「未到件数」用净额 `max(0, 应到−实到)`,而明细按「逐运单短少」列——当关联键断裂时二者会矛盾(安能现例)。关联键对齐后此矛盾自然消失;是否需要在代码里对「净额 vs 明细」做一致性校验/告警,也请你定。

View File

@@ -0,0 +1 @@
# inbound_verify package

View File

@@ -0,0 +1 @@
# inbound_verify.cli — 入口(交互菜单 / 服务)

View File

@@ -0,0 +1,348 @@
# inbound_verify/cli/router.py
#
# 交互菜单模式入口(调试 / 人工操作)。核心 Playwright 管理、任务派发、心跳
# 已抽到 runtime.py 共享;本文件只保留交互菜单与自动化测试。
# 服务模式(常驻 + FastAPI 接收指令)见 cli/server.py。
import os
import queue
import threading
import time
import yaml
from inbound_verify.paths import CONFIG_PATH
from inbound_verify.runtime import (
APP_SITES,
HEARTBEAT_INTERVAL,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
from inbound_verify import state_store
# 各站点模块(自动化测试 + 比对用;任务派发在 runtime
from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng
from inbound_verify import compare
def run_undelivered_compare():
"""应到未到比对(全站点):调用 compare读 downloads/ 下的应到/实到
数据,生成 output/应到未到数据.xlsx汇总报表 + 各站明细)。"""
print("\n▶ 开始执行【应到未到比对(全站点)】任务 ...")
compare.main()
# ====================================================================
# 自动化测试入口
# ====================================================================
# 每个(非百世)站点的交叉测试序列:覆盖两种下载流程之间的全部 4 种相邻转换,
# 用于验证无论上一个流程把页面留在什么状态,下一个流程都能正常运行:
# 应到->实到、实到->应到、应到->应到、实到->实到
CROSS_TEST_SEQUENCE = ["expected", "actual", "expected", "expected", "actual", "actual"]
def run_automation_test(pages_map):
"""自动化测试入口:按交叉序列逐个跑通各站点的下载流程,结束后打印统计报告。
判定规则:流程函数返回 False 或抛出异常记为 FAIL其余记为 PASS。
"""
# 站点 -> {流程键: (中文名, 流程函数)};百世为单流程,单独处理
# 自动化测试刻意走各站点的 _impl单次执行、无兜底重试以便探测原始失败、
# 不被模块内部"失败→重置→重试"机制掩盖。
flow_table = {
"顺心": {
"expected": ("应到", shunxin.shunxin_expected_download_impl),
"actual": ("实到", shunxin.shunxin_actual_download_impl),
},
"中通": {
"expected": ("应到", zto.zto_expected_download_impl),
"actual": ("实到", zto.zto_actual_download_impl),
},
"韵达": {
"expected": ("应到", yunda.yunda_expected_download_impl),
"actual": ("实到", yunda.yunda_actual_download_impl),
},
"安能": {
"expected": ("应到", anneng.anneng_expected_download_impl),
"actual": ("实到", anneng.anneng_actual_download_impl),
},
}
# 构建测试计划:[(站点, 流程中文名, 流程函数, 绑定page, 归属标签), ...]
# 顺心为双账号:两个 page 各自读归属地后跑一遍交叉序列;其余站点单 page。
plan = []
for site_name, flows in flow_table.items():
if site_name == "顺心":
if "顺心" not in pages_map:
continue
for sx_idx, sx_page in enumerate(pages_map["顺心"], start=1):
try:
tag = shunxin.shunxin_belonging(sx_page)
except Exception:
tag = f"账号{sx_idx}" # 读不到归属地时用序号占位,不阻断测试
for flow_key in CROSS_TEST_SEQUENCE:
label, func = flows[flow_key]
plan.append((f"顺心·{tag}", label, func, sx_page, tag))
continue
if site_name not in pages_map:
continue
bound_page = pages_map[site_name]
for flow_key in CROSS_TEST_SEQUENCE:
label, func = flows[flow_key]
plan.append((site_name, label, func, bound_page, ""))
# 百世:单流程,跑一次即可
if "百世" in pages_map:
plan.append(
(
"百世",
"应到未到",
baishi.baishi_download_undelivered_data_impl,
pages_map["百世"],
"",
)
)
if not plan:
print("\n⚠️ 当前没有已就绪的站点,无法执行自动化测试。")
return
total = len(plan)
print("\n====================================================")
print(f"自动化测试开始,共 {total} 个步骤。")
print("(双流程站点按应到/实到交叉序列执行,覆盖全部相邻转换)")
print("====================================================")
results = [] # [(站点, 流程, 状态, 耗时秒, 错误信息)]
for idx, (site_name, label, func, bound_page, out_tag) in enumerate(plan, start=1):
print("\n----------------------------------------------------")
print(f"[步骤 {idx}/{total}] 站点【{site_name}】流程【{label}")
print("----------------------------------------------------")
start = time.time()
status = "PASS"
err = ""
try:
if site_name in APP_SITES:
# 安能Electron 应用,无 Playwright page函数不收 page 参数
ret = func()
else:
bound_page.bring_to_front()
# 顺心 _impl 带 out_tag归属地其余站点 _impl 仅收 page
ret = func(bound_page, out_tag=out_tag) if out_tag else func(bound_page)
if ret is False:
status = "FAIL"
err = "流程返回失败状态"
except Exception as e:
status = "FAIL"
err = str(e)
elapsed = time.time() - start
results.append((site_name, label, status, elapsed, err))
print(f">> 步骤结果: {status} (耗时 {elapsed:.1f}s)")
_print_test_report(results)
def _print_test_report(results):
"""打印自动化测试统计报告。"""
passed = sum(1 for r in results if r[2] == "PASS")
failed = len(results) - passed
print("\n====================================================")
print("自动化测试统计报告")
print("====================================================")
for i, (site_name, label, status, elapsed, err) in enumerate(results, start=1):
mark = "" if status == "PASS" else ""
print(
f" {i:>2}. {mark} {status} {site_name} - {label} (耗时 {elapsed:.1f}s)"
)
if err:
note = err if len(err) <= 60 else err[:57] + "..."
print(f" 说明: {note}")
print("----------------------------------------------------")
print(f" 合计 {len(results)} 步:通过 {passed},失败 {failed}")
if failed == 0:
print(" ✅ 全部流程跑通。")
else:
print(" ❌ 存在失败流程,请结合上方说明与运行日志排查。")
print("====================================================")
# ====================================================================
# 交互菜单模式
# ====================================================================
def _read_debug_config():
"""读 config.yaml 的 debug 段,返回 (debug_mode, debug_target)。"""
debug_mode = False
debug_target = ""
try:
if os.path.exists(CONFIG_PATH):
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
config = yaml.safe_load(f) or {}
debug_mode = (config.get("debug", {}) or {}).get("enabled", False)
debug_target = (config.get("debug", {}) or {}).get("target_site", "")
except Exception as e:
print(f"⚠️ 读取 config.yaml 异常,将使用全量模式启动: {e}")
return debug_mode, debug_target
# 菜单编号 → 任务规格dispatch_task 消费)
CHOICE_TO_TASK = {
"1": {"site": "顺心", "kind": "expected"},
"2": {"site": "顺心", "kind": "actual"},
"3": {"site": "百世", "kind": "undelivered"},
"4": {"site": "中通", "kind": "expected"},
"5": {"site": "中通", "kind": "actual"},
"6": {"site": "韵达", "kind": "expected"},
"7": {"site": "韵达", "kind": "actual"},
"10": {"site": "安能", "kind": "expected"},
"11": {"site": "安能", "kind": "actual"},
"9": {"site": "__compare__", "kind": "compare"},
}
def _interactive_menu_loop(ctx):
"""交互菜单循环input 后台线程 + _await_command + dispatch_task + 心跳 + 状态盘。
所有 page 操作经 runtime主线程满足 Playwright sync 线程安全。
"""
pages_map = ctx.pages_map
sites_to_watch = ctx.sites_to_watch
def is_site_ready(site_name):
if site_name not in pages_map:
print(f"\n🚫 站点 [{site_name}] 未加载(当前为调试模式),已跳过。")
return False
return True
last_heartbeat = 0.0
command_queue = queue.Queue()
def _await_command():
nonlocal last_heartbeat
while True:
try:
return command_queue.get(timeout=0.5)
except queue.Empty:
if time.monotonic() - last_heartbeat >= HEARTBEAT_INTERVAL:
run_heartbeat(ctx)
last_heartbeat = time.monotonic()
def _print_status_board():
print("\n====================== 站点状态盘 ======================")
status = state_store.get_all_status()
if not status:
print(" (暂无状态记录)")
print("======================================================")
return
login_text = {
state_store.LOGIN_IN: "✅ 已登录",
state_store.LOGIN_OUT: "❌ 未登录",
state_store.LOGIN_UNKNOWN: "❔ 未知",
}
for site_name in sites_to_watch:
s = status.get(site_name)
if not s:
continue
login_mark = login_text.get(s["login_state"], s["login_state"])
exp = f"应到{'' if s['expected_ready'] else ''} {s['expected_generated_at'] or ''}"
act = f"实到{'' if s['actual_ready'] else ''} {s['actual_generated_at'] or ''}"
print(
f"{site_name}{login_mark} | {exp} | {act} "
f"| 探测于 {s['login_checked_at']}"
)
print("======================================================")
def _input_loop():
while True:
try:
command_queue.put(input())
except EOFError:
return
threading.Thread(target=_input_loop, daemon=True).start()
while True:
print("\n====================================================")
print(" 物流数据下载主菜单 ")
if ctx.debug_mode:
print(f" [ 调试模式,仅加载: {ctx.debug_target} ]")
print("====================================================")
print(" 模块一:【顺心】数据处理流")
print(" [1] 执行 - 应到货物数据下载")
print(" [2] 执行 - 实到货物数据下载")
print("-" * 52)
print(" 模块二:【百世】数据处理流")
print(" [3] 执行 - 一键提取应到未到异常数据")
print("-" * 52)
print(" 模块三:【中通】数据处理流")
print(" [4] 执行 - 应到货物数据下载")
print(" [5] 执行 - 实到货物数据下载")
print("-" * 52)
print(" 模块四:【韵达】数据处理流")
print(" [6] 执行 - 应到货物数据下载")
print(" [7] 执行 - 实到货物数据下载")
print("-" * 52)
print(" 模块五【安能】数据处理流Electron 应用)")
print(" [10] 执行 - 应到货物数据下载(运单信息)")
print(" [11] 执行 - 实到货物数据下载(网点到件扫描)")
print("-" * 52)
print(" 自动化测试")
print(" [8] 执行 - 全站点下载流程自动化测试 (交叉跑通校验)")
print("-" * 52)
print(" 全局离线数据处理")
print(" [9] 执行 - 应到未到比对(全站点汇总,输出 output/应到未到数据.xlsx")
print("-" * 52)
print(" 站点状态")
print(" [12] 查看 - 各站登录态 / 数据就绪状态")
print("-" * 52)
print(" [0] 退出系统")
print("====================================================")
print("请输入任务编号并回车: ", end="", flush=True)
choice = _await_command()
try:
if choice in CHOICE_TO_TASK:
task = CHOICE_TO_TASK[choice]
site = task["site"]
if site == "__compare__" or is_site_ready(site):
status, error = dispatch_task(ctx, task)
if status == state_store.TASK_FAILED:
print(f"❌ 任务失败: {error}")
elif choice == "8":
run_automation_test(pages_map)
elif choice == "12":
_print_status_board()
elif choice == "0":
break
elif choice.strip() != "":
print("\n⚠️ 无效输入,请查证后回车。")
except Exception as e:
print(f"❌ 任务调度异常: {e}")
def run_multi_site_daemon():
"""多站点自动化主控流程(交互菜单模式)。
启动 → 等待各站登录就绪 → 进入交互菜单;退出时关闭浏览器与安能。
"""
debug_mode, debug_target = _read_debug_config()
ctx = launch_and_prepare(debug_mode, debug_target)
try:
_interactive_menu_loop(ctx)
finally:
print("\n正在关闭浏览器并退出...")
ctx.stop()
print("程序已退出。")
def main():
"""交互菜单模式入口。"""
run_multi_site_daemon()
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,435 @@
# server.py
#
# 服务模式入口FastAPI主线程uvicorn/asyncio+ Playwright worker独立线程
# 客户端经 HTTP 触发任务、查状态、下载数据worker 串行执行任务并跑心跳。
#
# 线程模型(关键):
# - 主线程uvicorn + FastAPI。路由【绝不】访问 Playwright 对象,只经
# task_queue投递任务+ state_store查状态/任务)+ 文件系统(下载数据)。
# - worker 线程runtime.launch_and_preparesync_playwright 在此)+ 任务循环,
# 独占所有 page 操作;与主线程仅经 Queue + SQLite 通信。
# 违反"路由不碰 Playwright"会崩sync 对象跨线程访问)。
#
# 运行python -m inbound_verify.cli.server (或 inbound-verify-server默认监听 0.0.0.0:8000
import os
import queue
import threading
import time
from contextlib import asynccontextmanager
from datetime import datetime, timedelta
from typing import Dict, Optional
import uvicorn
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.interval import IntervalTrigger
from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse
from pydantic import BaseModel
from inbound_verify.paths import OUTPUT_DIR
from inbound_verify import state_store
from inbound_verify.runtime import (
HEARTBEAT_INTERVAL,
TASK_HANDLERS,
dispatch_task,
launch_and_prepare,
run_heartbeat,
)
from inbound_verify import db_compare
# 全部站点;百世固定下载当天,不可配置偏移
ALL_SITES = ["顺心", "百世", "中通", "韵达", "安能"]
CONFIGURABLE_SITES = {"顺心", "中通", "韵达", "安能"}
# 任务队列:元素 (task_id, task_spec)。主线程投递worker 消费。
task_queue: "queue.Queue" = queue.Queue()
# worker 运行状态主线程只读worker 写)
worker_state = {
"ctx": None,
"stop": False,
"thread": None,
"ready": False, # launch_and_prepare 完成(各站就绪,可接任务)
"error": None, # worker 启动失败原因
}
# 每日定时下载调度器进程内job 只往 task_queue 投任务,不碰 Playwright
scheduler = BackgroundScheduler(daemon=True)
def _worker_loop():
"""worker 线程:启动 Playwright + 等就绪 + 任务循环(执行任务 + 心跳)。"""
try:
ctx = launch_and_prepare(foreground=False)
worker_state["ctx"] = ctx
worker_state["ready"] = True
# 【P1-2 重启自愈】worker 就绪后清理上轮遗留的 pending/running 僵尸任务
cleaned = state_store.fail_stale_tasks()
if cleaned:
print(
f">> [worker] 自愈:清理 {cleaned} 条遗留任务pending/running → failed"
)
print(">> [worker] 各站就绪,开始接收任务 ...")
except Exception as e:
worker_state["error"] = str(e)
print(f"❌ [worker] 启动失败: {e}")
return
last_heartbeat = 0.0
while not worker_state["stop"]:
try:
task_id, task_spec = task_queue.get(timeout=1)
except queue.Empty:
# 空闲时跑心跳
if time.monotonic() - last_heartbeat >= HEARTBEAT_INTERVAL:
run_heartbeat(ctx)
last_heartbeat = time.monotonic()
continue
state_store.update_task(task_id, state_store.TASK_RUNNING)
print(f">> [worker] 执行任务 #{task_id}: {task_spec}")
status, error = dispatch_task(ctx, task_spec)
state_store.update_task(task_id, status, error)
print(f">> [worker] 任务 #{task_id} 完成: {status} {error or ''}")
try:
ctx.stop()
except Exception:
pass
worker_state["ready"] = False
print(">> [worker] 已退出。")
def _in_active_window(active_start, active_end):
"""当前本地时间是否落在激活时段内(避免半夜空跑)。
- start/end 任一为空 → 不限时段24h 活跃)
- start == end非空→ 视为全天活跃
- start < end → 半开区间 [start, end)
- start > end → 跨午夜(如 22:00-06:00now >= start 或 now < end
"""
if not active_start or not active_end:
return True
now = datetime.now().strftime("%H:%M")
if active_start == active_end:
return True
if active_start < active_end:
return active_start <= now < active_end
return now >= active_start or now < active_end
def _enqueue_fetch(site, kind):
"""周期 job 回调worker 就绪 + 在激活时段内 + 该(site,kind)无未完成任务时,投递一次抓取任务。
跑在 APScheduler 线程池线程——绝不碰 Playwright只经 SQLite + task_queue 通信。"""
if not worker_state["ready"]:
return # worker 未就绪:与 POST /tasks 的 409 同义,下个周期补抓
cfg = state_store.get_fetch_schedule(site, kind)
if not cfg or not cfg["enabled"]:
return # 已禁用job 本应已注销,双重保险)
if not _in_active_window(cfg["active_start"], cfg["active_end"]):
return # 不在激活时段,跳过本次 fire
try:
target_date = state_store.resolve_target_date(site, kind)
tid = state_store.create_task_if_idle(
site, kind, trigger="auto", target_date=target_date
)
if tid is None:
return # 上一次同类任务还没跑完,跳过避免堆积
task_queue.put((tid, {"site": site, "kind": kind}))
print(f">> [周期] 投递 {site}/{kind} 任务 #{tid}")
except Exception as e:
print(f">> [周期] 投递 {site}/{kind} 失败: {e}")
def _reschedule_fetch(site, kind):
"""按持久化配置(重新)注册或取消该 (site,kind) 的周期抓取 jobIntervalTrigger"""
job_id = f"{site}_{kind}"
try:
scheduler.remove_job(job_id)
except Exception:
pass
cfg = state_store.get_fetch_schedule(site, kind)
if not cfg or not cfg["enabled"] or cfg["interval_minutes"] < 1:
return # 未启用或频率非法:不注册(等同取消)
try:
scheduler.add_job(
_enqueue_fetch,
IntervalTrigger(minutes=cfg["interval_minutes"]),
args=[site, kind],
id=job_id,
replace_existing=True,
)
print(
f">> [周期] 已注册 {site}/{kind}:每 {cfg['interval_minutes']} 分钟"
f"(激活 {cfg['active_start'] or '不限'}~{cfg['active_end'] or '不限'}"
)
except Exception as e:
print(f">> [周期] 注册 {site}/{kind} 失败: {e}")
@asynccontextmanager
async def lifespan(_app):
"""服务启停:起 worker 线程 / 通知 worker 停。"""
state_store.init_db() # 先建表/迁移状态库,确保早于 worker 就绪的 /api/status 可用
for site in ALL_SITES: # 按持久化配置注册各站各 kind 的周期抓取 job
for kind in state_store.allowed_kinds(site):
_reschedule_fetch(site, kind)
scheduler.start()
print(">> [定时] 调度器已启动")
t = threading.Thread(target=_worker_loop, daemon=True)
worker_state["thread"] = t
t.start()
yield
scheduler.shutdown(wait=False)
print(">> [定时] 调度器已停止")
worker_state["stop"] = True
t.join(timeout=10)
app = FastAPI(title="InboundVerify 服务端", lifespan=lifespan)
class TaskRequest(BaseModel):
site: str
kind: str
force: bool = (
False # 强制重下:忽略已落库去重,重新提交所有班次/交接单的导出任务(默认关)
)
date: Optional[str] = None # YYYY-MM-DD指定则下载该日数据否则走站点 offset
@app.post("/tasks")
def create_task(req: TaskRequest):
"""提交任务 {site, kind, force, date?} → 入队,返回 task_id。"""
# 【P0】后端未就绪时直接拒绝避免任务在 worker 启动前入队卡死
if not worker_state["ready"]:
raise HTTPException(
status_code=409, detail="后端尚未就绪,请等待各站点登录完成后再操作"
)
if (req.site, req.kind) not in TASK_HANDLERS:
raise HTTPException(status_code=400, detail=f"无效任务: {req.site}/{req.kind}")
# 指定日期合法性校验(仅在传了 date 时)
if req.date:
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(
status_code=400, detail=f"date 不可为未来日期: {req.date}"
)
if target_date < today - timedelta(days=31):
raise HTTPException(
status_code=400, detail=f"date 超出 31 天回溯上限: {req.date}"
)
if req.site == "百世":
raise HTTPException(
status_code=400, detail="百世固定下载当天,不支持指定日期"
)
task_id = state_store.create_task(
req.site,
req.kind,
trigger="manual",
target_date=state_store.resolve_target_date(req.site, req.kind, req.date),
force=req.force,
)
spec = {"site": req.site, "kind": req.kind, "force": req.force}
if req.date:
spec["date"] = req.date
task_queue.put((task_id, spec))
return {"task_id": task_id}
@app.get("/tasks/{task_id}")
def get_task(task_id: int):
t = state_store.get_task(task_id)
if not t:
raise HTTPException(status_code=404, detail="任务不存在")
return t
@app.get("/tasks")
def list_tasks(limit: int = 20):
return state_store.list_tasks(limit)
# ── DB 比对(基于 PostgreSQL不依赖 Excel 文件)──
class CompareRequest(BaseModel):
site: str
date: str # YYYY-MM-DD
@app.post("/compare")
def run_compare(req: CompareRequest):
"""DB 差缺比对:以实到扫描日期为锚点,反推交接批次,展开全量比对。
返回统计指标 + 差缺明细。
"""
# 合法性校验
if req.site not in db_compare.SITE_COMPARE_CONFIG:
raise HTTPException(
status_code=400,
detail=f"不支持的站点: {req.site}(支持: {list(db_compare.SITE_COMPARE_CONFIG.keys())}",
)
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(status_code=400, detail=f"date 不可为未来日期: {req.date}")
result = db_compare.compare_site_date(req.site, req.date)
if result is None:
raise HTTPException(
status_code=404,
detail=f"{req.site} {req.date}: 当天无实到数据,无法比对",
)
return {
"site": result.site,
"date": result.date,
"batches": result.batches,
"stats": {
"waybill_count": result.stats.waybill_count,
"sf_wb_count": result.stats.sf_wb_count,
"expected_pieces": result.stats.expected_pieces,
"arrived_pieces": result.stats.arrived_pieces,
"undelivered_pieces": result.stats.undelivered_pieces,
"undelivered_wb": result.stats.undelivered_wb,
"full_miss": result.stats.full_miss,
"part_miss": result.stats.part_miss,
"sf_undelivered": result.stats.sf_undelivered,
},
"rows": [
{
"handover_no": r.handover_no,
"waybill_no": r.waybill_no,
"total_pieces": r.total_pieces,
"arrived_pieces": r.arrived_pieces,
"arrived_list": r.arrived_list,
"is_sf": r.is_sf,
}
for r in result.rows
],
}
@app.get("/status")
def get_status():
"""各站登录态 + 数据态 + 入库态(前端状态盘用),另含 worker 就绪状态。"""
return {
"worker_ready": worker_state["ready"],
"worker_error": worker_state["error"],
"sites": state_store.get_all_status(),
"ingest": state_store.get_all_ingest_state(),
}
class FetchScheduleSpec(BaseModel):
enabled: bool
active_start: str = ""
active_end: str = ""
interval_minutes: int
class ConfigRequest(BaseModel):
expected_offset: Optional[int] = None
actual_offset: Optional[int] = None
# DEPRECATED旧"每日定点"字段,保留一个发布周期兼容旧前端请求(收到即忽略)
schedule_enabled: Optional[bool] = None
schedule_time: Optional[str] = None
settings: Optional[Dict[str, str]] = None
fetch_schedules: Optional[Dict[str, FetchScheduleSpec]] = None
def _default_cfg():
return {
"expected_offset": 0,
"actual_offset": 0,
"fetch_schedules": {},
}
@app.get("/config")
def get_config():
"""各站配置:应到/实到偏移 + 每日定时(百世偏移恒 0"""
cfg = state_store.get_all_config()
return {site: cfg.get(site, _default_cfg()) for site in ALL_SITES}
@app.put("/config/{site}")
def set_config(site: str, req: ConfigRequest):
if site not in ALL_SITES:
raise HTTPException(status_code=400, detail=f"未知站点: {site}")
# 应到/实到偏移(百世锁定当天)
for kind, val in (
("expected", req.expected_offset),
("actual", req.actual_offset),
):
if val is not None:
if site == "百世":
raise HTTPException(
status_code=400, detail="百世固定下载当天,不可配置偏移"
)
state_store.set_offset(site, kind, val)
# 周期抓取调度(应到/实到各自独立;百世只允许 undelivered
if req.fetch_schedules:
allowed = state_store.allowed_kinds(site)
for kind, spec in req.fetch_schedules.items():
if kind not in allowed:
raise HTTPException(
status_code=400,
detail=f"站点 {site} 不支持抓取类型 {kind}(允许: {list(allowed)}",
)
interval = max(1, min(1440, int(spec.interval_minutes)))
state_store.set_fetch_schedule(
site,
kind,
spec.enabled,
spec.active_start,
spec.active_end,
interval,
)
_reschedule_fetch(site, kind)
# 站点专属配置(密码/账号/路径…)
if req.settings:
for k, v in req.settings.items():
state_store.set_setting(site, k, v)
cfg = state_store.get_all_config().get(site, _default_cfg())
cfg["settings"] = state_store.get_site_settings(site)
return cfg
@app.get("/config/{site}/settings")
def get_site_settings_api(site: str):
"""单站专属配置(密码/账号/路径…;不进 5s 轮询)。"""
if site not in ALL_SITES:
raise HTTPException(status_code=400, detail=f"未知站点: {site}")
return state_store.get_site_settings(site)
REPORT_FILE = "应到未到数据.xlsx"
@app.get("/report")
def download_report():
"""下载 output/应到未到数据.xlsx跑比对生成未生成 404"""
path = os.path.join(OUTPUT_DIR, REPORT_FILE)
if not os.path.isfile(path):
raise HTTPException(status_code=404, detail="报告尚未生成,请先跑比对")
return FileResponse(path, filename=REPORT_FILE)
def main():
"""服务模式入口。传字符串导入路径(规范写法;不开 reload/workers 时进程内 import行为等价"""
uvicorn.run("inbound_verify.cli.server:app", host="0.0.0.0", port=8000)
if __name__ == "__main__":
main()

623
inbound_verify/compare.py Normal file
View File

@@ -0,0 +1,623 @@
# -*- coding: utf-8 -*-
"""
应到未到数据比对(重构版)
目的:对中通 / 顺心 / 韵达 / 安能 四个站点,比对各自的「应到货物数据」与
「实到货物数据」,找出应到却未到的运单,汇总到 output/应到未到数据.xlsx。
(百世为站点直供未到明细,不参与 4 站比对;其应到/实到基数取自「扫描综合查询」应扫/已扫,见 process_baishi。
核心口径(四站点统一,重构后):
1. 应到件数 = 应到表「交接件数」之和(按运单号去重 keep-first
—— 录单件数 只是该单号的总录单量,实际只有“交接件数”会真正到站,
故应到必须按交接件数统计,不能用录单件数。
2. 实到件数 = 实到表「单号」的去重数量(直接数,不再由“应到−未到”倒推)。
—— 每扫描一件,系统生成该件的单号(一个单号=一件);后缀含总数/顺序号,
但计数时无视后缀,仅对单号去重即得实到件数。
3. 未到件数 = max(0, 应到件数 实到件数)。
4. 未到明细downloads/<站>-未到数据.xlsx仅列“短少”运单实到 < 应到),
每行:交接单号 | 运单号 | 总件数(=应到/交接件数) | 已到单号1 | 已到单号2 | …。
—— 实到扫描的顺序号是乱序的,缺件的“顺序号”无法反推,故不再编造子单号;
改为把该运单“实际扫到的单号”依次填到后续单元格,便于核对到了哪几件。
各站实到单号列 / 运单基号:
中通:单号列=运单号(复合串 H+运单号+总数+顺序),基号=v[:-8]
顺心:单号列=子单号,基号=运单号
韵达:单号列=子单号,基号=主单号
安能:单号列=扫描单号,基号=所属单号
目录约定:
源数据放在脚本同级目录的 downloads/ 下;结果写入 output/(不存在则自动创建)。
"""
import os
from datetime import datetime
from collections import defaultdict
import pandas as pd
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.chart import BarChart, Reference
from openpyxl.worksheet.page import PageMargins
from openpyxl.worksheet.properties import PageSetupProperties
from inbound_verify.paths import DOWNLOAD_DIR, OUTPUT_DIR
from inbound_verify.domain import (
ALL_REPORT_SITES,
BAISHI_COLUMNS,
BAISHI_FILE,
SITE_UNDELIVERED_FILE,
STATIONS,
_site_cfg,
arrived_pieces_by_cols,
arrived_pieces_zhongtong,
)
# 比对报表输出文件(路径锚定统一走 paths.py
OUTFILE = os.path.join(OUTPUT_DIR, "应到未到数据.xlsx")
# 站点 / 文件名 / 列映射配置ALL_REPORT_SITES / STATIONS / _site_cfg / BAISHI_FILE 等)见 domain.py。
def process(name):
"""4 站单站比对(重构版):返回 (列名list, 明细行list[dict], 统计dict)。
源文件缺失或非 4 站返回 None。
新口径:应到=交接件数;实到=直接数单号去重;未到=应到−实到;
未到明细行仅含「交接单号|运单号|总件数|+已到单号…」,不再编造子单号。"""
cfg = _site_cfg(name)
if cfg is None:
return None
exp_path = os.path.join(DOWNLOAD_DIR, cfg["exp"])
act_path = os.path.join(DOWNLOAD_DIR, cfg["act"])
if not os.path.exists(exp_path) or not os.path.exists(act_path):
print(f"[跳过] {cfg['name']}downloads 下缺少 {cfg['exp']}{cfg['act']}")
return None
df_exp = pd.read_excel(exp_path, dtype=str).fillna("")
df_act = pd.read_excel(act_path, dtype=str).fillna("")
if name == "韵达":
# 韵达实到数据有重复行(同子单号出现两次),保留交接单号为空的(到/接件扫描),
# 丢弃交接单号不为空的(派件/签收等),再按子单号去重。
df_act = df_act[df_act["交接单号"].astype(str).str.strip() == ""]
df_act = df_act.drop_duplicates(subset=["子单号"], keep="last")
# 同一运单可能有多条交接记录,按运单号去重、保留首条
dup = int(df_exp[cfg["exp_wb"]].duplicated().sum())
df_exp = df_exp.drop_duplicates(subset=[cfg["exp_wb"]], keep="first")
# 应到件数(新口径)= 交接件数 之和;记录 运单 -> (交接单号, 应到件数)
exp_by_wb = {}
exp_pieces = 0
for _, r in df_exp.iterrows():
wb = str(r[cfg["exp_wb"]]).strip()
if not wb:
continue
try:
n = int(float(r[cfg["exp_qty"]]))
except (TypeError, ValueError, KeyError):
n = 0
if n <= 0:
continue
exp_pieces += n
if wb not in exp_by_wb:
exp_by_wb[wb] = {
"jd": str(r.get(cfg["exp_jd"], "")).strip(),
"n": n,
}
# 实到件数(新口径)= 实到表单号去重数量(分组 运单->已到单号集合)
arrived = cfg["arrived_pieces"](df_act)
act_pieces = sum(len(s) for s in arrived.values()) # 全局去重单号数
# 未到:逐运单比较,列出实际已到的单号(顺序号乱序,无法反推缺件序号)
rows = []
full_miss = part_miss = 0
max_arrived = 0
for wb, info in exp_by_wb.items():
n = info["n"]
arrived_set = arrived.get(wb, set())
arrived_cnt = len(arrived_set)
if arrived_cnt >= n:
continue # 足额或溢到,不进未到表
if arrived_cnt == 0:
full_miss += 1
else:
part_miss += 1
max_arrived = max(max_arrived, arrived_cnt)
row = {
cfg["exp_jd"]: info["jd"],
cfg["exp_wb"]: wb,
"总件数": n,
}
for i, piece in enumerate(sorted(arrived_set, key=lambda x: str(x))):
row[f"已到单号{i+1}"] = piece
rows.append(row)
# 动态列:基础 3 列 + 已到单号1..max_arrived
columns = list(cfg["columns"]) + [f"已到单号{i+1}" for i in range(max_arrived)]
stats = {
"运单数": len(exp_by_wb),
"应到件": exp_pieces,
"已到件": act_pieces,
"未到件": max(0, exp_pieces - act_pieces),
"涉及运单": full_miss + part_miss,
"完全未到": full_miss,
"部分未到": part_miss,
"重复运单": dup,
}
return columns, rows, stats
# ============================ 样式常量 ============================
FONT = "微软雅黑"
NAVY = "1F3864" # 标题栏
BLUE = "305496" # 表头
LIGHTBLUE = "D6DCE5" # 合计行
CARD_BG = "F2F6FC" # 指标卡底
RED = "C00000" # 未到
GREEN = "548235" # 已到
GRAY = "808080"
ZEBRA = "F4F7FC"
LINE = "D9D9D9"
TILE = "BFBFBF"
THIN = Side(style="thin", color=LINE)
BORDER = Border(left=THIN, right=THIN, top=THIN, bottom=THIN)
def heat(rate):
"""未到率热力底色:绿(低) / 黄(中) / 红(高)。"""
if rate >= 0.50:
return "FFC7CE"
if rate >= 0.15:
return "FFEB9C"
return "C6EFCE"
# ============================ 写明细表 ============================
HEADER_FILL = PatternFill("solid", fgColor=BLUE)
HEADER_FONT = Font(name=FONT, bold=True, color="FFFFFF", size=11)
BODY_FONT = Font(name=FONT, size=10)
def write_station(ws, columns, rows):
ws.sheet_view.showGridLines = False
ws.append(columns)
for c in range(1, len(columns) + 1):
cell = ws.cell(row=1, column=c)
cell.fill = HEADER_FILL
cell.font = HEADER_FONT
cell.alignment = Alignment(horizontal="center", vertical="center")
cell.border = BORDER
for row in rows:
ws.append([row.get(c, "") for c in columns])
for r in range(2, ws.max_row + 1):
for c, col in enumerate(columns, start=1):
cell = ws.cell(row=r, column=c)
cell.font = BODY_FONT
cell.border = BORDER
if col == "总件数":
cell.number_format = "#,##0"
cell.alignment = Alignment(horizontal="right", vertical="center")
else:
cell.number_format = "@" # 文本,避免长单号被转科学计数
for c, col in enumerate(columns, start=1):
body = [len(str(row.get(col, ""))) for row in rows] if rows else []
width = min(max([len(str(col))] + body) + 4, 36)
ws.column_dimensions[ws.cell(row=1, column=c).column_letter].width = max(
width, 12
)
ws.freeze_panes = "A2"
ws.page_setup.orientation = "landscape"
ws.page_setup.fitToWidth = 1
ws.page_setup.fitToHeight = 0
ws.sheet_properties.pageSetUpPr = PageSetupProperties(fitToPage=True)
ws.print_title_rows = "1:1"
# ============================ 单站 / 全量产出 ============================
def process_baishi():
"""百世:读站点直供的未到明细,返回 (columns, rows, stats);文件缺失返回 None。
百世文件本身即未到结果(无应到/已到基数),统计只能给出未到件数。"""
path = os.path.join(DOWNLOAD_DIR, BAISHI_FILE)
if not os.path.exists(path):
return None
df = pd.read_excel(path, dtype=str).fillna("")
rows = df.to_dict("records")
wb_count = df["运单号"].nunique() if "运单号" in df.columns else len(rows)
# 应到/实到基数取自「扫描综合查询」应扫/已扫(到/接件扫描→当日),
# 由 baishi_download_undelivered_data_impl 在同次导航里抓取并落 site_settings。
# 未抓取过则 get_setting 返回 "" → 视为无基数(报表显示「—」)。
from inbound_verify import (
state_store,
) # 与 _read_business_dates 一致:比对模块纯离线,懒加载
def _to_int(v):
v = (v or "").strip().replace(",", "")
try:
return int(float(v)) if v not in ("", "-") else None
except (TypeError, ValueError):
return None
exp_n = _to_int(state_store.get_setting("百世", "scan_expected_pieces"))
arr_n = _to_int(state_store.get_setting("百世", "scan_arrived_pieces"))
stats = {
"运单数": wb_count,
"应到件": exp_n,
"已到件": arr_n,
"未到件": len(rows),
"涉及运单": wb_count,
"完全未到": None,
"部分未到": None,
"重复运单": 0,
}
return (BAISHI_COLUMNS, rows, stats)
def write_site_file(name):
"""4 站:把该站未到明细写到 downloads/<站>-未到数据.xlsx。
应到/实到缺process 返回 None→ 删旧文件、返回 False成功返回 True。"""
path = os.path.join(DOWNLOAD_DIR, SITE_UNDELIVERED_FILE.format(name=name))
out = process(name)
if out is None:
if os.path.exists(path):
os.remove(path)
return False
columns, rows, _stats = out
wb = Workbook()
wb.remove(wb.active)
ws = wb.create_sheet(name)
write_station(ws, columns, rows)
wb.save(path)
return True
def _read_business_dates(include):
"""从状态库读各站业务日期dispatch 下载成功时快照写入),供报告「数据日期」列。
4 站取 expected_business_date报告按应到口径百世取 undelivered_business_date。
从未下过的站返回空串(诚实留空,不反推)。"""
from inbound_verify import state_store # lazy import比对模块本身保持纯离线
status = state_store.get_all_status()
dates = {}
for name in include:
s = status.get(name, {})
if name == "百世":
dates[name] = s.get("undelivered_business_date", "")
else:
dates[name] = s.get("expected_business_date", "")
return dates
def build_full_report(include, dates=None):
"""生成全站汇总报表 output/应到未到数据.xlsx。
include: 本次成功的站点集合;未成功站点在汇总里保留行、无数据(不影响他站)。
返回 {站点: 未到件或None} 供日志。"""
os.makedirs(OUTPUT_DIR, exist_ok=True)
wb = Workbook()
wb.remove(wb.active)
summary_ws = wb.create_sheet("汇总报表") # 首页占位
summary = [] # (name, stats_or_None)顺序4 站 + 百世
for name in ALL_REPORT_SITES:
if name == "百世":
out = process_baishi() if "百世" in include else None
columns = BAISHI_COLUMNS
else:
out = process(name) if name in include else None
cfg = _site_cfg(name)
columns = cfg["columns"] if cfg else []
stats = out[2] if out is not None else None
rows = out[1] if out is not None else []
summary.append((name, stats))
ws = wb.create_sheet(name)
write_station(ws, columns, rows)
build_summary(
summary_ws,
summary,
datetime.now().strftime("%Y-%m-%d %H:%M"),
dates=dates or {},
)
wb.save(OUTFILE)
return {n: (s["未到件"] if s else None) for (n, s) in summary}
# ============================ 写汇总报表 ============================
def build_summary(ws, results, generated_at, dates=None):
dates = dates or {}
center = Alignment(horizontal="center", vertical="center")
left = Alignment(horizontal="left", vertical="center", indent=1)
# 合计/KPI 只算 4 站中本次成功的(百世无应到基数、失败站无数据,均不计入)
four = [(n, s) for (n, s) in results if n != "百世"]
ok = [s for _, s in four if s]
t_wb = sum(s["运单数"] for s in ok)
t_exp = sum(s["应到件"] for s in ok)
t_arr = sum(s["已到件"] for s in ok)
t_miss = sum(s["未到件"] for s in ok)
t_full = sum(s["完全未到"] for s in ok)
t_part = sum(s["部分未到"] for s in ok)
rate = (t_miss / t_exp) if t_exp else 0
ws.sheet_view.showGridLines = False
ws.column_dimensions["A"].width = 2.5
# 列宽按「4 个 KPI 卡等宽」设计B+C = D+E+F = G+H = I+J = 26
for col, w in {
"B": 12,
"C": 14,
"D": 9,
"E": 9,
"F": 8,
"G": 12,
"H": 14,
"I": 13,
"J": 13,
}.items():
ws.column_dimensions[col].width = w
ws.row_dimensions[1].height = 6
# —— 标题栏 ——
ws.merge_cells("B2:J2")
t = ws["B2"]
t.value = "应到未到比对 · 汇总报表"
t.fill = PatternFill("solid", fgColor=NAVY)
t.font = Font(name=FONT, bold=True, size=18, color="FFFFFF")
t.alignment = center
for row in ws["B2:J2"]:
for c in row:
c.fill = PatternFill("solid", fgColor=NAVY)
ws.row_dimensions[2].height = 34
ws.merge_cells("B3:J3")
sub = ws["B3"]
sub.value = f"数据快照 · 生成于 {generated_at}"
sub.font = Font(name=FONT, size=10, color=GRAY)
sub.alignment = Alignment(horizontal="right", vertical="center")
ws.row_dimensions[3].height = 18
# —— KPI 指标卡 ——
cards = [
("应到总件数", t_exp, NAVY, "#,##0"),
("已到总件数", t_arr, GREEN, "#,##0"),
("未到总件数", t_miss, RED, "#,##0"),
("总体未到率", rate, RED, "0.0%"),
]
# 2-3-2-2 分布填满 B-J9 列),配合上方列宽使 4 卡视觉等宽
spans = [
("B5:C5", "B6:C6"),
("D5:F5", "D6:F6"),
("G5:H5", "G6:H6"),
("I5:J5", "I6:J6"),
]
card_bg = PatternFill("solid", fgColor=CARD_BG)
thin = Side(style="thin", color=TILE)
for (lab, val, acc, fmt), (lrng, vrng) in zip(cards, spans):
ws.merge_cells(lrng)
ws.merge_cells(vrng)
acctop = Side(style="medium", color=acc)
for row in ws[lrng]:
for c in row:
c.fill = card_bg
c.font = Font(name=FONT, size=10, color=GRAY)
c.alignment = center
c.border = Border(left=thin, right=thin, top=acctop, bottom=thin)
for row in ws[vrng]:
for c in row:
c.fill = card_bg
c.font = Font(name=FONT, bold=True, size=20, color=acc)
c.alignment = center
c.border = Border(left=thin, right=thin, top=thin, bottom=thin)
ws[lrng.split(":")[0]].value = lab
vc = ws[vrng.split(":")[0]]
vc.value = val
vc.number_format = fmt
ws.row_dimensions[5].height = 18
ws.row_dimensions[6].height = 38
ws.row_dimensions[7].height = 8
# —— 小节标题 ——
ws.merge_cells("B8:J8")
sec = ws["B8"]
sec.value = "各站点明细统计"
sec.font = Font(name=FONT, bold=True, size=12, color=NAVY)
sec.alignment = Alignment(horizontal="left", vertical="center")
for row in ws["B8:J8"]:
for c in row:
c.border = Border(bottom=Side(style="medium", color=BLUE))
ws.row_dimensions[8].height = 22
# —— 统计表头 ——
headers = [
"站点",
"应到运单数",
"应到件数",
"已到件数",
"未到件数",
"未到率",
"完全未到运单",
"部分未到运单",
"数据日期",
]
head_align = Alignment(horizontal="center", vertical="center", wrap_text=True)
for i, h in enumerate(headers):
col = chr(ord("B") + i)
cell = ws[f"{col}9"]
cell.value = h
cell.fill = HEADER_FILL
cell.font = HEADER_FONT
cell.alignment = head_align
cell.border = BORDER
ws.row_dimensions[9].height = 30
# —— 各站数据行4 站 + 百世)——
r = 10
for idx, (name, s) in enumerate(results):
is_baishi = name == "百世"
srate = 0
if s is None:
vals = [f"{name}(无数据)", 0, 0, 0, 0, 0, 0, 0]
elif is_baishi:
# 百世:未到明细已知;若已抓取应到/实到基数(扫描综合查询应扫/已扫)则填真实值
if s["应到件"] is not None and s["已到件"] is not None:
srate = (s["未到件"] / s["应到件"]) if s["应到件"] else 0
vals = [
name,
s["运单数"],
s["应到件"],
s["已到件"],
s["未到件"],
srate,
"",
"",
]
else:
vals = [name, s["运单数"], "", "", s["未到件"], "", "", ""]
else:
srate = (s["未到件"] / s["应到件"]) if s["应到件"] else 0
vals = [
name,
s["运单数"],
s["应到件"],
s["已到件"],
s["未到件"],
srate,
s["完全未到"],
s["部分未到"],
]
vals.append(dates.get(name, "")) # 末列:该站业务日期
for i, v in enumerate(vals):
col = chr(ord("B") + i)
cell = ws[f"{col}{r}"]
cell.value = v
cell.font = BODY_FONT
cell.border = BORDER
cell.alignment = left if i == 0 else center
if s is None:
cell.fill = PatternFill("solid", fgColor="EFEFEF")
elif not is_baishi and idx % 2 == 1 and i != 5:
cell.fill = PatternFill("solid", fgColor=ZEBRA)
if isinstance(v, (int, float)):
cell.number_format = "0.0%" if i == 5 else "#,##0"
if (
i == 5
and s is not None
and isinstance(v, (int, float))
and not isinstance(v, bool)
):
cell.fill = PatternFill("solid", fgColor=heat(srate))
ws.row_dimensions[r].height = 19
r += 1
# —— 合计行 ——
tot_fill = PatternFill("solid", fgColor=LIGHTBLUE)
tot_font = Font(name=FONT, bold=True, size=10)
totals = ["合计", t_wb, t_exp, t_arr, t_miss, rate, t_full, t_part, ""]
for i, v in enumerate(totals):
col = chr(ord("B") + i)
cell = ws[f"{col}{r}"]
cell.value = v
cell.fill = tot_fill
cell.font = tot_font
cell.border = BORDER
cell.alignment = left if i == 0 else center
if i in (1, 2, 3, 4, 6, 7):
cell.number_format = "#,##0"
if i == 5:
cell.number_format = "0.0%"
ws.row_dimensions[r].height = 20
last_data_row = 9 + len(four) # 图表只取 4 站(百世无应到/已到基数,不绘图)
chart_anchor = r + 2
# —— 堆叠柱状图:各站已到 / 未到 ——
chart = BarChart()
chart.type = "col"
chart.grouping = "stacked"
chart.overlap = 100
chart.title = "各站点到货构成(已到 / 未到 件数)"
data = Reference(
ws, min_col=5, max_col=6, min_row=9, max_row=last_data_row
) # E已到 F未到
chart.add_data(data, titles_from_data=True)
cats = Reference(ws, min_col=2, min_row=10, max_row=last_data_row)
chart.set_categories(cats)
chart.series[0].graphicalProperties.solidFill = GREEN
chart.series[1].graphicalProperties.solidFill = RED
chart.y_axis.title = "件数"
chart.x_axis.delete = False
chart.y_axis.delete = False
chart.legend.position = "b"
chart.legend.overlay = False # 不覆盖绘图区:图例独占底部一行,与 X 轴站点名错开
chart.height = 9
chart.width = 20
ws.add_chart(chart, f"B{chart_anchor}")
# —— 口径说明 ——
note_row = chart_anchor + 19
notes = [
"指标口径:未到率 未到件数 ÷ 应到件数;完全未到运单 整单零到货;部分未到运单 部分到货、部分缺件。",
"合计 / 图表仅含 4 站(顺心/中通/韵达/安能,应到−实到口径);百世应到/实到取自「扫描综合查询」应扫/已扫(到/接件扫描→当日),已填入百世行,但为保持 4 站口径一致、不计入合计与图表。",
"本次下载失败的站点标注为(无数据)并计 0不影响其余站点统计。",
"明细见各站点工作表未到明细仅列短少运单并列出该运单实际扫到的单号已到单号1…缺件不再编造子单号。",
"数据日期:各站本次纳入数据对应的业务日期(=应到数据下载日 日期偏移;韵达偏移 1 为前一日);合计为多站混合、不标注。",
]
for k, text in enumerate(notes):
rr = note_row + k
ws.merge_cells(f"B{rr}:J{rr}")
cell = ws[f"B{rr}"]
cell.value = text
cell.font = Font(name=FONT, size=9, color=GRAY)
cell.alignment = Alignment(horizontal="left", vertical="center", wrap_text=True)
ws.page_setup.orientation = "landscape"
ws.page_setup.fitToWidth = 1
ws.page_setup.fitToHeight = 0
ws.sheet_properties.pageSetUpPr = PageSetupProperties(fitToPage=True)
ws.page_margins = PageMargins(left=0.4, right=0.4, top=0.5, bottom=0.5)
ws.print_area = f"A1:J{note_row + 1}"
# ============================ 主流程 ============================
def main():
"""菜单 [9] / 离线入口:用 downloads/ 下现有文件生成全站汇总报告(有文件的站即纳入)。"""
print("应到未到比对(全站汇总)")
print("-" * 56)
include = set()
for name in ALL_REPORT_SITES:
if name == "百世":
if os.path.exists(os.path.join(DOWNLOAD_DIR, BAISHI_FILE)):
include.add(name)
else:
cfg = _site_cfg(name)
if (
cfg
and os.path.exists(os.path.join(DOWNLOAD_DIR, cfg["exp"]))
and os.path.exists(os.path.join(DOWNLOAD_DIR, cfg["act"]))
):
include.add(name)
if not include:
print("未处理任何站点:请确认 downloads/ 下存在源数据文件。")
return
dates = _read_business_dates(include)
undel = build_full_report(include, dates=dates)
print("-" * 56)
for name in ALL_REPORT_SITES:
if name in include:
print(f"{name}:未到 {undel.get(name)}")
else:
print(f"{name}:无数据,跳过")
print(f"已输出:{OUTFILE}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,658 @@
# -*- coding: utf-8 -*-
"""
db_compare.py — 基于 PostgreSQL 的应到未到差缺比对引擎。
与 compare.pyExcel 版)并行:本模块直接从 DB 查询数据进行比对,
不依赖 downloads/ 下的 Excel 文件。
核心思路:以实到扫描日期为锚点 → 反推交接批次 → 展开批次全量比对。
每个站点只需提供配置waybill 列名 / piece 列名 / 是否有 SF 特殊处理),
核心比对逻辑完全通用。
顺心站点 SF 运单特殊处理SF 运单的子单号piece_no为随机号码不能用
COUNT(DISTINCT piece_no) 去重计数,改为 COUNT(*) 行计数。
用法:
from inbound_verify.db_compare import compare_site_date, SITE_COMPARE_CONFIG
result = compare_site_date("顺心", "2026-07-25")
if result:
print(result.stats)
for row in result.rows:
print(row)
"""
import os
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta
import psycopg
import yaml
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from inbound_verify.paths import CONFIG_PATH, OUTPUT_DIR, DOWNLOAD_DIR
from inbound_verify.domain import _site_cfg, ALL_REPORT_SITES, BAISHI_COLUMNS
# ============================== 结果类型 ==============================
@dataclass
class CompareStats:
"""单站点/单批次比对统计。"""
waybill_count: int = 0 # 应到运单数
expected_pieces: int = 0 # 应到件数
arrived_pieces: int = 0 # 实到件数
undelivered_pieces: int = 0 # 未到件数
undelivered_wb: int = 0 # 差缺运单数
full_miss: int = 0 # 完全未到
part_miss: int = 0 # 部分未到
sf_wb_count: int = 0 # SF 运单数
sf_undelivered: int = 0 # SF 差缺数
@dataclass
class UndeliveredRow:
"""单条差缺明细。"""
handover_no: str = "" # 交接单号
waybill_no: str = "" # 运单号
total_pieces: int = 0 # 总件数(交接件数)
arrived_pieces: int = 0 # 已到件数
arrived_list: list = field(default_factory=list) # 已到单号列表
is_sf: bool = False # 是否 SF 运单
@dataclass
class CompareResult:
"""一次比对的完整结果。"""
site: str = ""
date: str = ""
batches: list = field(default_factory=list) # 涉及的交接批次
stats: CompareStats = field(default_factory=CompareStats)
rows: list = field(default_factory=list) # UndeliveredRow 列表
# ============================== 站点比对配置 ==============================
@dataclass
class SiteCompareConfig:
"""DB 比对的站点参数。"""
name: str # 站点名
has_sf: bool = False # 是否需要区分 SF 运单
# 四站点 DB 比对配置(百世不参与 4 站比对)
SITE_COMPARE_CONFIG: dict[str, SiteCompareConfig] = {
"顺心": SiteCompareConfig(name="顺心", has_sf=True),
"中通": SiteCompareConfig(name="中通", has_sf=False),
"韵达": SiteCompareConfig(name="韵达", has_sf=False),
"安能": SiteCompareConfig(name="安能", has_sf=False),
}
# ============================== DB 连接 ==============================
def _load_pg_config():
"""从 config.yaml 读 postgres 段。与 store.py 共用同一配置源。"""
if not os.path.exists(CONFIG_PATH):
raise FileNotFoundError(
f"未找到配置文件 {CONFIG_PATH}(请参考 config.example.yaml 创建 config.yaml"
)
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
pg = cfg.get("postgres") or {}
return {
"host": pg.get("host", "127.0.0.1"),
"port": int(pg.get("port", 5432)),
"user": pg.get("user", "postgres"),
"password": pg.get("password", ""),
"dbname": pg.get("dbname", "CQHXDB"),
"schema": pg.get("schema", "inbound_verify"),
"connect_timeout_seconds": int(pg.get("connect_timeout_seconds", 5)),
}
def _connect():
c = _load_pg_config()
return psycopg.connect(
host=c["host"],
port=c["port"],
dbname=c["dbname"],
user=c["user"],
password=c["password"],
options=f"-c search_path={c['schema']} -c statement_timeout=30s",
connect_timeout=c["connect_timeout_seconds"],
)
# ============================== 核心比对逻辑 ==============================
def compare_site_date(site: str, target_date: str) -> CompareResult | None:
"""对指定站点和日期执行 DB 差缺比对。
算法:
1. 取 scan_time::date = target_date 的实到运单(锚点)
2. 反推这些运单所属的交接批次handover_no
3. 展开批次全量应到运单
4. 查询批次全量实到扫描
5. 逐运单比对差缺SF/non-SF 分支处理)
Args:
site: 站点名("顺心"/"中通"/"韵达"/"安能"
target_date: 日期 "YYYY-MM-DD"
Returns:
CompareResult 或 None当天无实到数据时返回 None
"""
cfg = SITE_COMPARE_CONFIG.get(site)
if cfg is None:
print(f"[db_compare] 不支持的站点: {site}")
return None
try:
conn = _connect()
cur = conn.cursor()
# ── Step 1: 取实到锚点 ──
cur.execute(
"""
SELECT DISTINCT waybill_no FROM actual_record
WHERE site = %s AND scan_time::date = %s
""",
(site, target_date),
)
anchor_wbs = [r[0] for r in cur.fetchall()]
if not anchor_wbs:
print(f"[db_compare] {site} {target_date}: 当天无实到数据")
conn.close()
return None
# ── Step 2: 反推交接批次 ──
cur.execute(
"""
SELECT DISTINCT e.handover_no FROM expected_record e
WHERE e.site = %s AND e.waybill_no = ANY(%s)
""",
(site, anchor_wbs),
)
batches = [r[0] for r in cur.fetchall()]
# ── Step 3: 展开批次全量应到 ──
cur.execute(
"""
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site = %s AND handover_no = ANY(%s)
ORDER BY handover_no, waybill_no
""",
(site, batches),
)
exp_rows = cur.fetchall() # [(waybill_no, handover_no, handover_pieces), ...]
if not exp_rows:
conn.close()
return None
all_wbs = [r[0] for r in exp_rows]
# ── Step 4: 取批次全量实到 ──
cur.execute(
"""
SELECT waybill_no, piece_no FROM actual_record
WHERE site = %s AND waybill_no = ANY(%s)
ORDER BY waybill_no, piece_no
""",
(site, all_wbs),
)
act_rows = cur.fetchall() # [(waybill_no, piece_no), ...]
conn.close()
# ── Step 5: 逐运单比对 ──
return _do_compare(site, target_date, batches, exp_rows, act_rows, cfg)
except Exception as e:
print(f"[db_compare] {site} {target_date} 比对异常: {e}")
return None
def compare_site_batch(site: str, handover_no: str) -> CompareResult | None:
"""按指定交接单号执行全批次比对(不依赖实到锚点)。
用于已知交接单号后精确比对某一批次。
"""
cfg = SITE_COMPARE_CONFIG.get(site)
if cfg is None:
print(f"[db_compare] 不支持的站点: {site}")
return None
try:
conn = _connect()
cur = conn.cursor()
cur.execute(
"""
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site = %s AND handover_no = %s
ORDER BY waybill_no
""",
(site, handover_no),
)
exp_rows = cur.fetchall()
if not exp_rows:
conn.close()
return None
all_wbs = [r[0] for r in exp_rows]
cur.execute(
"""
SELECT waybill_no, piece_no FROM actual_record
WHERE site = %s AND waybill_no = ANY(%s)
ORDER BY waybill_no, piece_no
""",
(site, all_wbs),
)
act_rows = cur.fetchall()
conn.close()
return _do_compare(
site,
f"batch:{handover_no}",
[handover_no],
exp_rows,
act_rows,
cfg,
)
except Exception as e:
print(f"[db_compare] {site} batch:{handover_no} 比对异常: {e}")
return None
# ============================== 比对核心 ==============================
def _do_compare(
site: str,
label: str,
batches: list[str],
exp_rows: list[tuple], # [(waybill_no, handover_no, handover_pieces), ...]
act_rows: list[tuple], # [(waybill_no, piece_no), ...]
cfg: SiteCompareConfig,
) -> CompareResult:
"""执行逐运单比对,产出统计 + 差缺明细。
与 compare.py:process() 口径一致:
- 应到件数 = handover_pieces交接件数
- 实到件数 = SF ? COUNT(*) : COUNT(DISTINCT piece_no)
- arrived_cnt >= handover_pieces → 足额到货,跳过
"""
# 构建实到索引: waybill_no → [piece_no, ...](保留所有行,不去重)
act_by_wb: dict[str, list[str]] = {}
for wb, piece in act_rows:
act_by_wb.setdefault(wb, []).append(piece)
stats = CompareStats()
rows: list[UndeliveredRow] = []
max_arrived = 0
for wb, handover_no, handover_pcs in exp_rows:
handover_pcs = handover_pcs or 0
if handover_pcs <= 0:
continue
stats.waybill_count += 1
stats.expected_pieces += handover_pcs
is_sf = cfg.has_sf and wb.startswith("SF")
if is_sf:
stats.sf_wb_count += 1
all_pieces = act_by_wb.get(wb, [])
if is_sf:
# SF: 行计数不去重piece_no 是随机号码)
arrived_cnt = len(all_pieces)
arrived_list = list(all_pieces)
else:
# non-SF: 子单号去重
unique_pieces = list(dict.fromkeys(all_pieces)) # 保序去重
arrived_cnt = len(unique_pieces)
arrived_list = unique_pieces
stats.arrived_pieces += arrived_cnt
if arrived_cnt >= handover_pcs:
continue # 足额或溢到,不进差缺表
if arrived_cnt == 0:
stats.full_miss += 1
else:
stats.part_miss += 1
if is_sf:
stats.sf_undelivered += 1
max_arrived = max(max_arrived, arrived_cnt)
rows.append(
UndeliveredRow(
handover_no=handover_no,
waybill_no=wb,
total_pieces=handover_pcs,
arrived_pieces=arrived_cnt,
arrived_list=arrived_list,
is_sf=is_sf,
)
)
stats.undelivered_pieces = max(0, stats.expected_pieces - stats.arrived_pieces)
stats.undelivered_wb = stats.full_miss + stats.part_miss
result = CompareResult(
site=site,
date=label,
batches=batches,
stats=stats,
rows=rows,
)
# 打印摘要
print(
f"[db_compare] {site} {label}: "
f"batches={len(batches)}, "
f"wb={stats.waybill_count}(SF:{stats.sf_wb_count}), "
f"exp={stats.expected_pieces}, arr={stats.arrived_pieces}, "
f"miss={stats.undelivered_pieces}, "
f"miss_wb={stats.undelivered_wb}(full={stats.full_miss}, part={stats.part_miss})"
)
if stats.sf_undelivered:
print(f" SF 差缺: {stats.sf_undelivered} 个运单")
return result
# ============================== Excel 输出 ==============================
# 样式常量(与 compare.py 对齐)
_FONT = "微软雅黑"
_BLUE = "305496"
_HEADER_FILL = PatternFill("solid", fgColor=_BLUE)
_HEADER_FONT = Font(name=_FONT, bold=True, color="FFFFFF", size=11)
_BODY_FONT = Font(name=_FONT, size=10)
_THIN = Side(style="thin", color="D9D9D9")
_BORDER = Border(left=_THIN, right=_THIN, top=_THIN, bottom=_THIN)
def write_result_excel(result: CompareResult, output_path: str | None = None) -> str:
"""将比对结果写入 Excel 文件。
Args:
result: compare_site_date 或 compare_site_batch 的返回值
output_path: 输出路径,为 None 时自动生成:
output/{站}-{日期}-未到数据.xlsx
Returns:
实际写入的文件路径
"""
if output_path is None:
os.makedirs(OUTPUT_DIR, exist_ok=True)
date_tag = result.date.replace(":", "-").replace("batch:", "batch-")
output_path = os.path.join(
OUTPUT_DIR, f"{result.site}-{date_tag}-未到数据.xlsx"
)
wb = Workbook()
ws = wb.active
ws.title = result.site
_write_sheet(ws, result)
wb.save(output_path)
print(f"[db_compare] Excel 已输出: {output_path}")
return output_path
def _write_sheet(ws, result: CompareResult):
"""写单个站点的差缺明细 sheet。"""
s = result.stats
rows = result.rows
# 动态列: 交接单号 | 运单号 | 总件数 | 已到单号1 | 已到单号2 | ...
max_arrived = max((len(r.arrived_list) for r in rows), default=0)
columns = ["交接单号", "运单号", "总件数"] + [
f"已到单号{i + 1}" for i in range(max_arrived)
]
ws.sheet_view.showGridLines = False
# 表头
ws.append(columns)
for c in range(1, len(columns) + 1):
cell = ws.cell(row=1, column=c)
cell.fill = _HEADER_FILL
cell.font = _HEADER_FONT
cell.alignment = Alignment(horizontal="center", vertical="center")
cell.border = _BORDER
# 数据行
for row in rows:
values = {
"交接单号": row.handover_no,
"运单号": row.waybill_no,
"总件数": row.total_pieces,
}
for i, piece in enumerate(row.arrived_list):
values[f"已到单号{i + 1}"] = piece
ws.append([values.get(c, "") for c in columns])
# 格式
for r in range(2, ws.max_row + 1):
for c, col in enumerate(columns, start=1):
cell = ws.cell(row=r, column=c)
cell.font = _BODY_FONT
cell.border = _BORDER
if col == "总件数":
cell.number_format = "#,##0"
cell.alignment = Alignment(horizontal="right", vertical="center")
else:
cell.number_format = "@"
# 列宽
for c, col in enumerate(columns, start=1):
body_lens = [
len(str(ws.cell(row=r, column=c).value or ""))
for r in range(2, ws.max_row + 1)
]
width = min(max([len(str(col))] + body_lens) + 4, 36)
ws.column_dimensions[ws.cell(row=1, column=c).column_letter].width = max(
width, 12
)
ws.freeze_panes = "A2"
# ============================== 全站汇总报表DB 版)=============================
def _stats_to_dict(s: CompareStats) -> dict:
"""CompareStats -> build_summary 要的中文键 stats dict。"""
return {
"运单数": s.waybill_count,
"应到件": s.expected_pieces,
"已到件": s.arrived_pieces,
"未到件": s.undelivered_pieces,
"完全未到": s.full_miss,
"部分未到": s.part_miss,
}
def _target_date_for(site: str) -> str:
"""4 站比对锚点today - actual_offset以实到扫描日为锚与 _site_undelivered_handler 一致)。"""
from inbound_verify import state_store # 懒导入,避免成环
offset = state_store.get_offset(site, "actual")
return (date.today() - timedelta(days=offset)).strftime("%Y-%m-%d")
def _baishi_from_pg(cur, target: str):
"""查百世当日基数baishi_daily_stats+ 当天未到明细undelivered_record 按 ingested_at 过滤)。
返回 (stats_dict_or_None, rows_or_None);基数与明细均无 → (None, None)。
undelivered_record 是 UPSERT 累积表;按 ingested_at::date = target 取当天入库的未到快照
= 当天下载的当前未到,站点已剔除已到),避免累积偏大。
"""
cur.execute(
"SELECT expected_pieces, arrived_pieces, undelivered_pieces "
"FROM baishi_daily_stats WHERE site = %s AND business_date = %s",
("百世", target),
)
basis = cur.fetchone()
cur.execute(
"SELECT waybill_no, piece_no, biz_type, last_scan FROM undelivered_record "
"WHERE site = %s AND ingested_at::date = %s",
("百世", target),
)
detail = cur.fetchall()
if basis is None and not detail:
return (None, None)
exp = basis[0] if basis else None
arr = basis[1] if basis else None
# 未到件优先取基数差baishi_daily_stats.undelivered_pieces与应到/已到同源自洽);
# 基数缺失时退回明细行数。
undel = basis[2] if (basis and basis[2] is not None) else len(detail)
wb_count = len({r[0] for r in detail if r[0]}) # 运单号去重
rows = [
{
"类型": r[2] or "",
"子单号": r[1] or "",
"运单号": r[0] or "",
"最新扫描记录": r[3] or "",
}
for r in detail
]
stats = {
"运单数": wb_count,
"应到件": exp,
"已到件": arr,
"未到件": undel,
"完全未到": None,
"部分未到": None,
}
return (stats, rows)
def build_full_report(date=None) -> str:
"""DB 版全站汇总报表4 站走 DB 比对、百世走 PG复用 compare.build_summary 渲染。
产出 output/应到未到数据.xlsx/report 下载。date=None 时各站按 actual_offset 算锚点(以实到扫描日为锚)。
返回输出路径。"""
from inbound_verify import compare # 复用 build_summary / write_station / OUTFILE
print("[db_compare] 开始生成全站汇总报表 ...")
wb = Workbook()
wb.remove(wb.active)
summary_ws = wb.create_sheet("汇总报表")
results = [] # [(name, stats_dict_or_None)],顺序 ALL_REPORT_SITES
site_targets = {} # name -> target_date汇总表"数据日期"列)
conn = _connect()
cur = conn.cursor()
try:
for name in ALL_REPORT_SITES:
if name == "百世":
target = date or datetime.now().strftime("%Y-%m-%d")
site_targets[name] = target
stats, rows = _baishi_from_pg(cur, target)
results.append((name, stats))
if rows is not None:
compare.write_station(wb.create_sheet(name), BAISHI_COLUMNS, rows)
continue
if name not in SITE_COMPARE_CONFIG:
results.append((name, None))
continue
target = date or _target_date_for(name)
site_targets[name] = target
result = compare_site_date(name, target)
if result is not None:
results.append((name, _stats_to_dict(result.stats)))
_write_sheet(wb.create_sheet(name), result)
else:
results.append((name, None))
finally:
conn.close()
compare.build_summary(
summary_ws,
results,
datetime.now().strftime("%Y-%m-%d %H:%M"),
dates=site_targets,
)
os.makedirs(OUTPUT_DIR, exist_ok=True)
wb.save(compare.OUTFILE)
print(f"[db_compare] 全站汇总已输出: {compare.OUTFILE}")
for name, s in results:
print(f" {name}:未到 {s['未到件']}" if s else f" {name}:无数据,跳过")
return compare.OUTFILE
# ============================== 终端验证入口 ==============================
def main():
"""命令行验证入口:
python -m inbound_verify.db_compare 顺心 2026-07-25
"""
import sys
site = sys.argv[1] if len(sys.argv) > 1 else "顺心"
target_date = sys.argv[2] if len(sys.argv) > 2 else "2026-07-25"
result = compare_site_date(site, target_date)
if result is None:
print(f"{site} {target_date}: 无结果")
return
print(f"\n=== {result.site} {result.date} 差缺明细 ===")
print(f"涉及批次: {result.batches}")
print(f"应到运单: {result.stats.waybill_count} (SF: {result.stats.sf_wb_count})")
print(f"应到件数: {result.stats.expected_pieces}")
print(f"实到件数: {result.stats.arrived_pieces}")
print(f"未到件数: {result.stats.undelivered_pieces}")
print(
f"差缺运单: {result.stats.undelivered_wb} (完全未到: {result.stats.full_miss}, 部分未到: {result.stats.part_miss})"
)
if result.stats.sf_undelivered:
print(f"SF 差缺: {result.stats.sf_undelivered}")
if result.rows:
print(f"\n--- 差缺明细 (共 {len(result.rows)} 条) ---")
for row in result.rows[:20]:
sf = "[SF]" if row.is_sf else ""
arrived_preview = row.arrived_list[:5]
print(
f" {sf} {row.waybill_no}: "
f"应到{row.total_pieces}件, 实到{row.arrived_pieces}"
f" {f'已到: {arrived_preview}' if arrived_preview else ''}"
)
if len(result.rows) > 20:
print(f" ... 还有 {len(result.rows) - 20}")
# 输出 Excel
path = write_result_excel(result)
print(f"\n结果文件: {path}")
if __name__ == "__main__":
main()

92
inbound_verify/domain.py Normal file
View File

@@ -0,0 +1,92 @@
# -*- coding: utf-8 -*-
"""domain.py — 站点 / 文件名 / 列映射的共享配置(单一来源)。
比对compare与入库store都依赖这套配置抽出独立 leaf 模块,
让 store 不必为读配置而依赖整个比对引擎。纯数据,无 state_store / 文件 IO 依赖。
"""
from collections import defaultdict
# 汇总报表覆盖的全部站点4 站在前、百世在末;汇总页图表只取 4 站)
ALL_REPORT_SITES = ["顺心", "中通", "韵达", "安能", "百世"]
# 4 站单站未到明细文件名(百世未到文件由站点直接产出,名为 BAISHI_FILE
SITE_UNDELIVERED_FILE = "{name}-未到数据.xlsx"
BAISHI_FILE = "百世-应到未到货物数据.xlsx"
BAISHI_COLUMNS = ["类型", "子单号", "运单号", "最新扫描记录"]
def arrived_pieces_zhongtong(df):
"""中通实到「运单号」为复合串H + 运单号(12) + 总数(4) + 顺序(4))。
基号 = v[:-8](与应到表运单号对齐),单件 = 整串(每串即一件)。"""
res = defaultdict(set)
for v in df["运单号"]:
v = str(v).strip()
if len(v) > 8 and v[-4:].isdigit():
res[v[:-8]].add(v) # 以完整复合串作为“已到单号”存入
return res
def arrived_pieces_by_cols(wb_col, piece_col):
"""顺心 / 韵达 / 安能:按干净运单列分组,单件 = 子单号 / 扫描单号。
wb_col实到表中与应到运单号对齐的干净列
(顺心=运单号 / 韵达=主单号 / 安能=所属单号)
piece_col实到表中每件货物的单号列子单号 / 扫描单号)"""
def parse(df):
res = defaultdict(set)
for m, s in zip(df[wb_col], df[piece_col]):
m, s = str(m).strip(), str(s).strip()
if m and s:
res[m].add(s)
return res
return parse
STATIONS = [
{
"name": "中通",
"exp": "中通-应到货物数据.xlsx",
"act": "中通-实到货物数据.xlsx",
"exp_qty": "交接件数", # 应到件数口径:交接件数(非录单件数)
"exp_wb": "运单号", # 应到表运单号列(兼作去重键)
"exp_jd": "交接单号", # 未到数据需展示的交接单号
"arrived_pieces": arrived_pieces_zhongtong,
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "顺心",
"exp": "顺心-应到货物数据.xlsx",
"act": "顺心-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("运单号", "子单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "韵达",
"exp": "韵达-应到货物数据.xlsx",
"act": "韵达-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("主单号", "子单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
{
"name": "安能",
"exp": "安能-应到货物数据.xlsx",
"act": "安能-实到货物数据.xlsx",
"exp_qty": "交接件数",
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_by_cols("所属单号", "扫描单号"),
"columns": ["交接单号", "运单号", "总件数"],
},
]
def _site_cfg(name):
"""按名称取 4 站配置(百世不在 STATIONS返回 None"""
return next((c for c in STATIONS if c["name"] == name), None)

23
inbound_verify/paths.py Normal file
View File

@@ -0,0 +1,23 @@
# paths.py
# 统一的路径锚点:所有路径都以本项目所在目录为基准,避免依赖运行时的工作目录(cwd)。
# 这样无论从哪个目录启动脚本IDE / 命令行 / 计划任务 / 双击),
# 下载目录与配置文件都能稳定定位,不会出现“文件落到别处”或“读不到密码”的隐蔽故障。
import os
# 项目根目录(以本文件所在位置为基准,与从哪个目录启动脚本无关)
# __file__ = <root>/inbound_verify/paths.py → 上两级 = 项目根
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# 统一的下载 / 输出目录
DOWNLOAD_DIR = os.path.join(BASE_DIR, "downloads")
OUTPUT_DIR = os.path.join(BASE_DIR, "output")
# 统一的配置文件路径注意config.yaml 需与本项目脚本放在同一目录下)
CONFIG_PATH = os.path.join(BASE_DIR, "config.yaml")
# 状态存储SQLite阶段0心跳 / 登录态 / 数据态持久化,重启不丢)
STATE_DB_PATH = os.path.join(BASE_DIR, "state", "state.db")
# 错误截图目录(下载流程失败时自动截取,供问题排查)
SCREENSHOT_DIR = os.path.join(BASE_DIR, "logs", "screenshots")

767
inbound_verify/runtime.py Normal file
View File

@@ -0,0 +1,767 @@
# runtime.py
# 阶段1服务化共享核心。把"启动浏览器 + 各站就绪 + 弹窗 + 心跳初值"
# (launch_and_prepare)、"执行一条任务"(dispatch_task)、"一轮心跳"(run_heartbeat)
# 抽出来,供
# - cli/router.py交互菜单模式调试 / 人工操作)
# - cli/server.pyFastAPI 服务模式:常驻 + 接收 API 指令)
# 共同复用,避免两处重复维护。
#
# 线程模型launch_and_prepare 内 sync_playwright().start() 必须在"持有 Playwright 的
# 线程"调用(交互模式=主线程;服务模式=worker 线程)。该线程独占所有 page 操作;
# 其他线程(如 FastAPI 路由)只能经 task_queue + state_store 与之通信,绝不跨线程
# 访问 page。
import os
import socket
import subprocess
import time
import urllib.request
from datetime import date, datetime, timedelta
import yaml
from playwright.sync_api import sync_playwright
from inbound_verify.paths import CONFIG_PATH, SCREENSHOT_DIR
from inbound_verify import state_store
from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng
from inbound_verify import compare # dispatch 的 compare 任务用
# 各网页站点首页 URL单一来源取自各站点模块 HOME_URL
SITES_CONFIG = {
"顺心": shunxin.HOME_URL,
"百世": baishi.HOME_URL,
"中通": zto.HOME_URL,
"韵达": yunda.HOME_URL,
}
# 站点就绪特征:登录成功进入工作台后的标志性控件
READY_SELECTORS = {
"顺心": 'h1:has-text("盟商门户网")',
"百世": 'h1[title="百世快运"]',
"中通": '.logo:has-text("网点版")',
"韵达": '.el-menu-item:has-text("首页")',
}
# 安能是 Electron 桌面应用(不是 Playwright 网页),单独启动
APP_SITES = {"安能"}
# 心跳间隔(秒)
HEARTBEAT_INTERVAL = 30
# ============================ 安能启动CDP============================
def _find_free_port():
"""让操作系统分配一个空闲端口,避免固定端口冲突。"""
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
def _wait_cdp_up(port, timeout=60.0):
"""轮询直到 CDP 调试端口就绪(应用启动需要时间)。"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(
f"http://localhost:{port}/json/version", timeout=2
) as resp:
if resp.status == 200:
return True
except Exception:
pass
time.sleep(1)
return False
# 【环境兼容】宿主 shellCodex/VS Code 插件、WorkBuddy 等)会向子进程注入一批与业务
# 无关的变量,实测会让安能应用登录后反复弹出“获取试用网点接口报错”:
# - HTTP(S)_PROXY=http://127.0.0.1:8800QuickQ 加速器代理):安能的 wnp.ane56.com
# 接口请求被塞进第三方代理后返回 400/用户未登录;
# - VSCODE_* / CODEX_* / EFC_*VS Code 扩展宿主注入IPC、PID、NLS、ESM 等);
# - NODE_TLS_REJECT_UNAUTHORIZED / DEBUG / RUST_LOG 等宿主调试变量。
# 另ELECTRON_RUN_AS_NODE=1 会把安能当作纯 Node 运行(拒绝 Chromium 参数、启动即
# 退出 rc=9NODE_OPTIONS 同样会干扰。拉起前全部摘掉,尽量还原终端手动启动环境。
_ANNENG_STRIP_PREFIXES = ("VSCODE_", "CODEX_", "EFC_")
_ANNENG_STRIP_EXACT = {
"NODE_OPTIONS",
"ELECTRON_RUN_AS_NODE",
"HTTP_PROXY",
"HTTPS_PROXY",
"ALL_PROXY",
"NO_PROXY",
"NODE_TLS_REJECT_UNAUTHORIZED",
"NODEFAULTCURRENTDIRECTORYINEXEPATH",
"DEBUG",
"RUST_LOG",
"APPLICATION_INSIGHTS_NO_STATSBEAT",
}
def launch_anneng(app_path):
"""以调试模式启动安能 Electron 应用(自动选取空闲端口),返回子进程对象。"""
anneng_env = {
key: value
for key, value in os.environ.items()
if key not in _ANNENG_STRIP_EXACT and not key.startswith(_ANNENG_STRIP_PREFIXES)
}
port = _find_free_port()
print(f">> 以调试模式启动【安能】应用(端口 {port}{app_path}")
proc = subprocess.Popen(
[app_path, f"--remote-debugging-port={port}"], env=anneng_env
)
anneng.set_cdp_port(port)
if not _wait_cdp_up(port):
raise RuntimeError(
f"安能应用调试端口 {port} 未就绪——可能应用已在运行(单实例),"
"请先关闭已有的安能窗口再试"
)
return proc
# ============================ 探测函数 ============================
def probe_site_login(site_name, pages_map):
"""探测单站是否登录(复用就绪判据)。任何异常一律返回 False。
只在 Playwright 所属线程调用。
"""
try:
if site_name not in pages_map:
return False
if site_name == "安能":
return anneng.anneng_ready()
if site_name == "顺心":
return all(
pg.locator(READY_SELECTORS["顺心"]).is_visible(timeout=500)
for pg in pages_map["顺心"]
)
return (
pages_map[site_name]
.locator(READY_SELECTORS[site_name])
.is_visible(timeout=500)
)
except Exception:
return False
# ============================ 运行上下文 ============================
class RuntimeContext:
"""launch_and_prepare 的返回值,持有 Playwright 运行所需对象。"""
def __init__(
self,
pw,
browser,
pages_map,
ready_status,
anneng_proc,
sites_to_watch,
debug_mode,
debug_target,
foreground=True,
):
self.pw = pw
self.browser = browser
self.pages_map = pages_map
self.ready_status = ready_status
self.anneng_proc = anneng_proc
self.sites_to_watch = sites_to_watch
self.debug_mode = debug_mode
self.debug_target = debug_target
# True=任务执行时把 page 置顶交互调试False=后台静默不置顶(服务模式,避免抢焦点)
self.foreground = foreground
def stop(self):
"""关闭 browser + 安能 + Playwright。退出时调用。"""
try:
self.browser.close()
except Exception:
pass
if self.anneng_proc is not None:
try:
self.anneng_proc.terminate()
print("已关闭安能应用。")
except Exception:
pass
try:
self.pw.stop()
except Exception:
pass
# ============================ 启动 + 就绪 + 弹窗 ============================
def seed_legacy_config():
"""一次性:把 config.yaml 里的站点配置(百世密码 / 韵达账密 / 安能 exe 路径)
灌入 state.db。已存在的值不覆盖前端改过的不动"""
if not os.path.exists(CONFIG_PATH):
return
try:
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
except Exception as e:
print(f"⚠️ 读取 config.yaml 做 seed 失败: {e}")
return
items = [
("百世", "password", (cfg.get("baishi", {}) or {}).get("password", "")),
("韵达", "username", (cfg.get("yunda", {}) or {}).get("username", "")),
("韵达", "password", (cfg.get("yunda", {}) or {}).get("password", "")),
("安能", "app_path", (cfg.get("anneng", {}) or {}).get("app_path", "")),
]
seeded = []
for site, key, val in items:
if val and not state_store.get_setting(site, key):
state_store.set_setting(site, key, str(val))
seeded.append(f"{site}.{key}")
if seeded:
print(f">> [seed] 从 config.yaml 灌入站点配置: {', '.join(seeded)}")
def launch_and_prepare(debug_mode=False, debug_target="", foreground=True):
"""启动 Playwright + 各站 page + 就绪轮询 + 弹窗清理 + 心跳初值,返回 RuntimeContext。
必须在"持有 Playwright 的线程"调用(交互模式主线程 / 服务模式 worker 线程)。
阻塞至所有站点登录就绪才返回。
foregroundTrue=任务执行时把 page 置顶交互调试False=后台静默不置顶(服务模式,
避免抢用户焦点)。仅控制任务执行阶段的 bring_to_front启动登录/初始弹窗清理的置顶
不受影响(启动时窗口需对用户可见以便登录)。
"""
# 0. 状态库建表/迁移 + 从 config.yaml 灌入站点配置(须在 reset_login_states 等之前)
state_store.init_db()
seed_legacy_config()
# 1. 读 debug 配置config.yaml服务模式也生效+ 安能路径state.dbseed 已灌入)
if os.path.exists(CONFIG_PATH):
try:
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
_cfg = yaml.safe_load(f) or {}
_dbg = _cfg.get("debug", {}) or {}
if _dbg.get("enabled"):
debug_mode = True
debug_target = str(_dbg.get("target_site", "") or "")
except Exception as e:
print(f"⚠️ 读取 config.yaml(debug) 失败: {e}")
anneng_app_path = state_store.get_setting("安能", "app_path")
# 2. 确定要挂载的网页站点 + 安能标记
anneng_active = False
if debug_mode:
if debug_target in SITES_CONFIG:
print(f"\n🛠️ 【调试模式】仅加载目标站点: [{debug_target}]")
active_sites = {debug_target: SITES_CONFIG[debug_target]}
elif debug_target == "安能":
print(f"\n🛠️ 【调试模式】仅加载目标站点: [安能]")
active_sites = {}
anneng_active = True
else:
active_sites = dict(SITES_CONFIG)
anneng_active = True
else:
active_sites = dict(SITES_CONFIG)
anneng_active = True
if anneng_active and not anneng_app_path:
print("⚠️ 已启用安能但 config.yaml 未配置 anneng.app_path将跳过安能。")
anneng_active = False
# 2.5 重置各站登录态为 unknown登录态是会话级的避免显示上一会话残留的陈旧登录态。
# 数据态(文件就绪)会话无关、保留不动;心跳就绪后会重新探测写真实值。
reset_sites = list(active_sites.keys())
if anneng_active:
reset_sites.append("安能")
state_store.reset_login_states(reset_sites)
# 3. 启动 Playwright不用 with改 .start(),由 RuntimeContext.stop() 收尾)
pw = sync_playwright().start()
# 调试模式临时开启 CDP 端口,便于外部 Playwright如 Playwright CLI 技能)
# 通过 connectOverCDP 挂载到已登录的页面读取内容。仅调试模式生效,不影响服务模式。
_launch_args = ["--remote-debugging-port=9223"] if debug_mode else []
browser = pw.chromium.launch(headless=False, args=_launch_args)
context = browser.new_context(viewport={"width": 1920, "height": 1080})
# 默认禁用麦克风/摄像头:在每个页面/iframe 加载前覆盖 getUserMedia 为“直接拒绝”,
# 这样站点(如韵达登录/工作台会请求麦克风)调用时立即 NotAllowedErrorChromium 不再
# 弹出系统授权窗,且麦克风被真正挡住(不是授权给它)。物流工作台无需音视频采集。
context.add_init_script(
"(()=>{const d=()=>Promise.reject(new DOMException('Permission disabled','NotAllowedError'));"
"if(navigator.mediaDevices)navigator.mediaDevices.getUserMedia=d;"
"for(const k of ['getUserMedia','webkitGetUserMedia','mozGetUserMedia']){"
"if(typeof navigator[k]==='function')navigator[k]=function(){return d();};}})();"
)
pages_map = {}
print("\n====================================================")
print("【启动】正在打开各站点页面...")
print("====================================================")
def _open_page(label, attempts=3):
"""开一个页面并 goto(url, domcontentloaded);容忍瞬时 DNS/超时抖动重试,全失败才抛。
domcontentloadedDOM 就绪即返回,不等慢资源(广告/图片)的 load 事件,
避免某站 load 超 30s / 瞬时 DNS 失败导致整个 worker 启动失败。就绪轮询自会判登录态。"""
last = None
for i in range(1, attempts + 1):
pg = context.new_page()
try:
pg.goto(url, wait_until="domcontentloaded")
return pg
except Exception as e:
last = e
try:
pg.close()
except Exception:
pass
print(f"⚠️ 打开【{label}】第 {i}/{attempts} 次失败: {e}")
if i < attempts:
time.sleep(2)
raise RuntimeError(f"打开【{label}】连续 {attempts} 次失败: {last}")
for site_name, url in active_sites.items():
if site_name == "顺心":
# 顺心:两个归属地账号在同一窗口各开一个标签页
sx_pages = []
for acct in range(1, 3):
print(f">> 正在启动【顺心】账号{acct}标签页: {url}")
sx_pages.append(_open_page(f"顺心账号{acct}"))
pages_map["顺心"] = sx_pages
else:
print(f">> 正在启动【{site_name}】页面: {url}")
pages_map[site_name] = _open_page(site_name)
# 4. 安能 Electron
anneng_proc = None
if anneng_active:
try:
anneng_proc = launch_anneng(anneng_app_path)
pages_map["安能"] = True # 哨兵:已启动(无 Playwright page
except Exception as e:
print(f"⚠️ 启动安能应用失败,已跳过安能:{e}")
anneng_active = False
# 5. 韵达前置自动登录
print("\n====================================================")
print("【登录检测】正在准备各站点登录...")
print("====================================================")
if "韵达" in pages_map:
try:
pages_map["韵达"].bring_to_front()
yunda.yunda_login(pages_map["韵达"])
except Exception as e:
print(f" ⚠️ 韵达前置自动登录模块发生波动: {e}")
# 6. 就绪轮询(复用 probe_site_login
ready_status = {site: False for site in active_sites.keys()}
if anneng_active:
ready_status["安能"] = False
print("\n>> 正在轮询各站点就绪状态 (自动登录或手动登录均可)...")
while not all(ready_status.values()):
for site_name in list(ready_status.keys()):
if ready_status[site_name]:
continue
if probe_site_login(site_name, pages_map):
ready_status[site_name] = True
print(f" ✅ 【{site_name}】已检测到主页,登录就绪。")
pending = [s for s, r in ready_status.items() if not r]
if pending:
print(
f" ⏳ 等待以下站点完成登录: [{', '.join(pending)}] ... "
"(请在浏览器/应用中操作)"
)
time.sleep(3)
# 7. 初始弹窗清理
print("\n====================================================")
print("【准备】所有站点已就绪,正在清理初始弹窗...")
print("====================================================")
_dismiss_initial_popups(pages_map)
for s in ("中通", "韵达", "安能"):
if s in pages_map:
print(f" ✅ 【{s}】已就绪。")
# 8. 心跳初值(就绪轮询刚通过 → 各站视为已登录init_db 已在启动时完成)
sites_to_watch = list(ready_status.keys())
for _site in sites_to_watch:
state_store.set_login_state(_site, True)
return RuntimeContext(
pw,
browser,
pages_map,
ready_status,
anneng_proc,
sites_to_watch,
debug_mode,
debug_target,
foreground,
)
def _dismiss_initial_popups(pages_map):
"""顺心 / 百世 / 韵达 初始弹窗清理。"""
if "顺心" in pages_map:
for acct, sx_page in enumerate(pages_map["顺心"], start=1):
try:
sx_page.bring_to_front()
print(f">> 正在处理【顺心】账号{acct}弹窗与遮罩...")
sx_page.locator("a").nth(4).click(timeout=2000)
sx_page.wait_for_timeout(500)
sx_page.get_by_role("button", name="Close").click(timeout=2000)
sx_page.wait_for_timeout(500)
sx_page.get_by_role("button", name="不再询问").click(timeout=2000)
print(f" ✅ 【顺心】账号{acct}初始弹窗处理完成。")
except Exception:
pass
if "百世" in pages_map:
try:
bs_page = pages_map["百世"]
bs_page.bring_to_front()
print(">> 正在处理【百世】初始弹窗(阅读消息 / 配置检查 / 优惠券广告)...")
# 委托给 baishi 的专用清理:优惠券广告是全屏居中 modal
# 关闭键为 .ant-modal-close纯图标无文字必须点它才能真正关掉。
baishi.dismiss_baishi_popups(bs_page)
print(" ✅ 【百世】初始弹窗处理完成。")
except Exception:
pass
if "韵达" in pages_map:
try:
yd_page = pages_map["韵达"]
yd_page.bring_to_front()
print(">> 正在检查【韵达】音频设备授权提示...")
yunda.dismiss_audio_prompt(yd_page)
except Exception:
pass
# ============================ 任务派发 ============================
def _web_handler(site, download_func):
"""构造网页站任务 handler。
foregroundctx控制任务执行时是否把 page 置顶:服务模式后台跑不置顶,避免抢用户
焦点;交互模式置顶便于调试。顺心是 page 列表,置顶标志透传给 shunxin_download。
"""
def handler(ctx, force=False, date=None):
pg = ctx.pages_map[site]
if isinstance(pg, list):
# 顺心双账号:置顶与否交给 shunxin_download 在逐账号循环里按 foreground 决定
return download_func(pg, foreground=ctx.foreground, force=force, date=date)
if ctx.foreground:
pg.bring_to_front()
return download_func(pg, force=force, date=date)
return handler
def _site_undelivered_handler(site):
"""4 站未到:下应到+实到 → DB 比对 → 写 output/<站>-<日期>-未到数据.xlsx。
应到全量去重(已落库则跳过导出),因此比对不依赖 Excel 文件,走数据库查询。
下载成功则返回 True比对失败不影响任务判定数据已入库"""
def handler(ctx, force=False, date=None):
exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force, date) is not False
act_ok = (
(TASK_HANDLERS[(site, "actual")](ctx, force, date) is not False)
if exp_ok
else False
)
if not exp_ok or not act_ok:
return False
# ── 先入库再比对(修复时序:比对须读到本次下载的数据,
# 否则首次/force 时 PG 无当天数据,比对返回 None、不产出 Excel──
try:
_record_business_date(site, "undelivered", date)
except Exception:
pass
try:
from inbound_verify import store # 懒导入,避免成环
if store.ingest_enabled():
store.ingest_task(
site, "undelivered"
) # 4 站 = ingest expected + actual
print(f">> [入库] {site} 前置入库完成")
except Exception as e:
print(f">> [入库] {site} 前置入库失败(不影响比对尝试): {e}")
# ── DB 比对(替代旧 Excel 比对)──
try:
from inbound_verify import db_compare # 懒导入,避免成环
if date:
target_date = date
else:
offset = state_store.get_offset(site, "actual")
target_date = (datetime.now().date() - timedelta(days=offset)).strftime(
"%Y-%m-%d"
)
result = db_compare.compare_site_date(site, target_date)
if result is not None:
db_compare.write_result_excel(result)
else:
print(f">> [未到] {site} {target_date}: 当天无实到数据,跳过比对")
except Exception as e:
print(f">> [未到] {site} DB 比对异常(不影响下载结果): {e}")
return True # 下载成功即返回 True比对失败不影响任务判定
return handler
# 「跑比对」= DB 版全站汇总报表(替代旧 compare.main Excel 路径;下载交由各站定时/手动)。
def _run_db_full_report(date=None):
"""生成 DB 版全站汇总报表output/应到未到数据.xlsx
懒导入 db_comparebest-effort失败只告警返回 True与旧 lambda 契约一致)。"""
try:
from inbound_verify import db_compare
db_compare.build_full_report(date)
except Exception as e:
print(f">> [跑比对] DB 汇总报表生成失败: {e}")
return True
TASK_HANDLERS = {
("顺心", "expected"): _web_handler("顺心", shunxin.shunxin_expected_download),
("顺心", "actual"): _web_handler("顺心", shunxin.shunxin_actual_download),
("顺心", "undelivered"): _site_undelivered_handler("顺心"),
("百世", "undelivered"): _web_handler(
"百世", baishi.baishi_download_undelivered_data
),
("中通", "expected"): _web_handler("中通", zto.zto_expected_download),
("中通", "actual"): _web_handler("中通", zto.zto_actual_download),
("中通", "undelivered"): _site_undelivered_handler("中通"),
("韵达", "expected"): _web_handler("韵达", yunda.yunda_expected_download),
("韵达", "actual"): _web_handler("韵达", yunda.yunda_actual_download),
("韵达", "undelivered"): _site_undelivered_handler("韵达"),
(
"安能",
"expected",
): lambda ctx, force=False, date=None: anneng.anneng_expected_download(
force=force, date=date
),
(
"安能",
"actual",
): lambda ctx, force=False, date=None: anneng.anneng_actual_download(
force=force, date=date
),
("安能", "undelivered"): _site_undelivered_handler("安能"),
("__compare__", "compare"): lambda ctx, force=False, date=None: _run_db_full_report(
date
),
}
def _record_business_date(site, kind, date=None):
"""下载成功后,把本次数据的业务日期快照写进状态库(供前端/报告显示「是哪天的数据」)。
有 date 用 date否则 = 下载当天 该数据对应的日期偏移。__compare__ 无数据概念,跳过。
kind → 写入:
expected/actual各写自己一列。
undelivered百世直供恒当天写 undelivered4 站未到由 _site_undelivered_handler
内部连带下了 expected+actual不经 dispatch无业务日期写入故此处一并补写
expected/actual/undelivered 三列——actual 用 actual 偏移、未到跟随 expected 偏移。
只写业务日期ready 语义已移交「入库成功」_persist_to_db 置位),此处不再碰 ready。"""
if site == "__compare__":
return
today = datetime.now().date()
def _write(k, biz_or_off):
# biz_or_off: int=偏移todayoffstr=已确定业务日期date
biz = (
(today - timedelta(days=biz_or_off)).strftime("%Y-%m-%d")
if isinstance(biz_or_off, int)
else biz_or_off
)
try:
state_store.set_business_date(site, k, biz)
except Exception as e:
print(f">> [状态] 写业务日期失败 {site}/{k}: {e}")
def off(kind_key):
return state_store.get_offset(site, kind_key)
if kind == "expected":
_write("expected", date if date else off("expected"))
elif kind == "actual":
_write("actual", date if date else off("actual"))
elif site == "百世":
_write("undelivered", 0)
else: # 4 站 undelivered连带补写 expected/actual/undelivered 三列
_write("expected", date if date else off("expected"))
_write("actual", date if date else off("actual"))
_write("undelivered", date if date else off("expected"))
def _ready_flags(site):
"""从 PG 业务表派生单站三就绪态ready = DB 数据真相)。
expected/actual = PG 中存在对应 target_datetoday offset的数据
百世 undelivered = baishi_daily_stats 中存在 target_date 的数据;
4 站 undelivered = expected_ready ∧ actual_ready派生
PG 不可达时返回全 False降级安全不阻塞心跳
返回 (flags: {kind: bool}, dates: {kind: target_date_str})。
dates 与 flags 同源——ready=True 时 business_date 即该 target_date
彻底消除 ready 与 business_date 不同源导致的日期标签漂移。"""
from inbound_verify import store # 懒导入:避免模块级循环
today = date.today()
today_str = today.isoformat()
if site == "百世":
has_und, _ = store.has_data(site, "undelivered", today_str)
return (
{"expected": False, "actual": False, "undelivered": has_und},
{"undelivered": today_str},
)
exp_off = state_store.get_offset(site, "expected")
act_off = state_store.get_offset(site, "actual")
exp_date = (today - timedelta(days=exp_off)).isoformat()
act_date = (today - timedelta(days=act_off)).isoformat()
has_exp, _ = store.has_data(site, "expected", exp_date)
has_act, _ = store.has_data(site, "actual", act_date)
return (
{"expected": has_exp, "actual": has_act, "undelivered": has_exp and has_act},
{"expected": exp_date, "actual": act_date, "undelivered": exp_date},
)
def _apply_ready(site, flags, dates=None):
"""写入单站就绪态 + 业务日期同源ready 与 business_date 均据 PG + offset 派生)。
ready=True 时同步写入 target_date 作为 business_date消除不同源导致的日期标签漂移。
失败仅告警。"""
for k, rdy in flags.items():
try:
state_store.set_ready(site, k, rdy)
if rdy and dates and dates.get(k):
state_store.set_business_date(site, k, dates[k])
except Exception as e:
print(f">> [状态] 置就绪态失败 {site}/{k}: {e}")
def capture_error_screenshot(page, site, kind, attempt, error):
"""流程失败时截取当前页面,保存到 logs/screenshots/。
page: Playwright Page 对象(安能传 None 走 CDP 分支,调用方自行处理)。
截图失败绝不外抛——只打告警,不干扰任务重试/清场流程。"""
try:
os.makedirs(SCREENSHOT_DIR, exist_ok=True)
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
err_short = (error or "unknown")[:40].replace("/", "_").replace("\\", "_")
fname = f"{site}_{kind}_{ts}_attempt{attempt}_{err_short}.png"
path = os.path.join(SCREENSHOT_DIR, fname)
page.screenshot(path=path, full_page=False)
print(f"📸 【{site}-{kind}】错误截图已保存: {path}")
except Exception as se:
print(f"📸 【{site}-{kind}】截图失败(不影响任务): {se}")
def _refresh_ready(site):
"""入库后立即据 PG 派生并写入该站就绪态(省 30s 心跳等待,与心跳同源)。"""
flags, dates = _ready_flags(site)
_apply_ready(site, flags, dates)
def _persist_to_db(site, kind):
"""下载成功后把本次数据入库 PostgreSQL尽力而为绝不外抛不影响任务判定
- __compare__ 无源数据,跳过。
- auto_ingest=false 时跳过(无 PG/cpolar 的开发机)。
- 懒导入 store 以回避 import 顺序store↔compare 与 runtime↔compare 共存)。
- 结果写 state_store.ingest_state供 /api/status 反映入库健康。
所有写库/写状态都包 try/except失败仅告警绝不改变 dispatch_task 的 SUCCESS 判定。"""
if site == "__compare__":
return
try:
from inbound_verify import store # 懒导入:冷路径(每下载一次),回避成环
except Exception as e:
print(f">> [warn] 入库模块不可用: {e}")
return
try:
if not store.ingest_enabled(): # 移入 tryconfig.yaml 缺失/损坏时也不外抛
print(">> [入库] 已关闭 (auto_ingest=false),跳过")
return
count = store.ingest_task(site, kind)
# 4 站 undelivered 连带入了 expected+actual按实际入库的类补记 ingest_state
# 否则心跳派生 readyexpected ∧ actual → undelivered会读到陈旧值。
logged = (
["expected", "actual", "undelivered"]
if kind == "undelivered" and site != "百世"
else [kind]
)
for k in logged:
state_store.set_ingest_state(site, k, ok=True, count=count)
_refresh_ready(site) # 入库成功 → 立即据 ingest_state 派生就绪态(与心跳同源)
print(f">> [入库] {site}/{kind} 成功,{count}")
except Exception as e:
print(f">> [warn] 入库失败 {site}/{kind}: {e}")
try:
state_store.set_ingest_state(site, kind, ok=False, error=str(e))
except Exception as e2:
print(f">> [warn] 写入库状态也失败: {e2}")
def dispatch_task(ctx, task_spec):
"""执行一条任务。task_spec = {"site", "kind"}。返回 (status, error)。
掉登录的站点直接判 failed呼应"掉登录则该站任务全停")。
只在 Playwright 所属线程调用。
"""
site = task_spec.get("site")
kind = task_spec.get("kind")
if site != "__compare__":
if site not in ctx.pages_map:
return (state_store.TASK_FAILED, f"站点 {site} 未加载")
if not probe_site_login(site, ctx.pages_map):
return (
state_store.TASK_FAILED,
f"站点 {site} 未登录,已跳过(需人工重登)",
)
handler = TASK_HANDLERS.get((site, kind))
if handler is None:
return (state_store.TASK_FAILED, f"未知任务: {site}/{kind}")
try:
ret = handler(ctx, bool(task_spec.get("force", False)), task_spec.get("date"))
if ret is False:
return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)")
_record_business_date(site, kind, task_spec.get("date"))
_persist_to_db(site, kind)
return (state_store.TASK_SUCCESS, None)
except Exception as e:
return (state_store.TASK_FAILED, str(e))
# ============================ 心跳 ============================
def run_heartbeat(ctx):
"""一轮心跳:探测各站登录态 + 据 PG 业务表派生数据就绪态;登录态变化时提示。
ready 直接查询 PG 业务表expected_record / actual_record / baishi_daily_stats
以「目标业务日期是否有数据」为唯一依据,彻底消除 ingest_state 日期比对带来的每日零点重置。
_refresh_ready 在入库瞬间即据 PG 派生(省 30s 等待),心跳同源复核。
只在 Playwright 所属线程调用。
"""
prev = state_store.get_all_status()
for site_name in ctx.sites_to_watch:
logged_in = probe_site_login(site_name, ctx.pages_map)
prev_login = prev.get(site_name, {}).get("login_state")
state_store.set_login_state(site_name, logged_in)
now_login = state_store.LOGIN_IN if logged_in else state_store.LOGIN_OUT
if prev_login and prev_login not in (now_login, state_store.LOGIN_UNKNOWN):
print(f"\n ⚠️【{site_name}】登录态变化: {prev_login}{now_login}")
flags, dates = _ready_flags(site_name)
_apply_ready(site_name, flags, dates)

View File

@@ -0,0 +1 @@
# inbound_verify.sites — 各承运商站点驱动

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,291 @@
# sites/baishi.py
import os
import yaml
from playwright.sync_api import sync_playwright
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
"""
for attempt in range(1, max_attempts + 1):
try:
ret = flow()
if ret is False:
raise RuntimeError("流程返回失败状态")
if attempt > 1:
print(f">> 【{site_name}-{label}】第 {attempt} 次尝试成功 ✅")
return True
except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try:
reset()
except Exception as re:
print(f" ⚠️ 重置异常: {re}")
if attempt < max_attempts:
continue
print(
f"❌ 【{site_name}-{label}】已达最大尝试次数 {max_attempts},放弃(环境已清理)。"
)
return False
# 站点首页 URL异常兜底重置用也供 runtime 的 SITES_CONFIG 引用(单一来源)
HOME_URL = "https://v5.800best.com"
def baishi_reset(page):
"""异常兜底:重置百世到初始态(跳首页 URL丢弃当前页面状态登录态保留"""
page.goto(HOME_URL)
page.wait_for_timeout(1500)
def dismiss_baishi_popups(page):
"""清理百世首页杂乱弹窗:优惠券广告 / 通知消息 / 配置检查面板 / 聊天通知。
所有弹窗都有时序问题——不是登录瞬间就出现,旧代码一登录就判定"无弹窗"而跳过,
导致从未成功关闭过任何一个。修复策略统一为「轮询等待出现 → 关闭」。
经 CDP 端到端验证的选择器2026-07-19
- ① 优惠券广告:全屏居中 Ant Design modal.ant-modal-wrap.ant-modal-centered
关闭键 .ant-modal-close纯图标×、无文字。登录后约 2s 才加载。
- ② Ant Design 通知("您有新的消息".ant-notification 组件,
右下角卡片,关闭键 .ant-notification-notice-close纯图标×
- ③ 配置检查面板:.modal-config-check.react-draggable可拖拽浮动面板
默认 display:none系统检测后展开为 block。底部 footer 有两个 button
"重新检查"(danger) 和 "关 闭"(primary)。点"关 闭"后面板从 DOM 移除。
- ④ 聊天/调查通知("您还有调研单未填写...".chat-notification-popover-wrapper
z-index:1000关闭键 .chat-notification-close图标×
注意:左侧首页轮播/广告横幅是页面正常内容区(无关闭键),不在此处处理。
"""
import time as _time
# 单条轮询循环:每个 tick短间隔同时检查并关闭「所有当前可见」的弹窗。
#
# 旧实现是串行四段,每段 _poll_and_click 在「未命中」时会阻塞等待整段
# max_poll_s=5s 才返回 False。于是广告段跑完(~7s)才开始处理通知、配置,
# 而广告/通知其实登录后 1~2s 就出现了,却被卡在前面段的等待窗口里,表现为
# 「检查配置 / 消息通知关得特别慢」。改为单循环后,谁先出现谁在下一个 tick
# ~350ms就被关掉四类互不排队。
#
# 选择器均经 CDP 端到端验证(见函数 docstring
targets = [
(".ant-modal-wrap.ant-modal-centered .ant-modal-close", "优惠券广告"),
(".ant-notification-notice-close", "通知消息"),
(".modal-config-check button:has-text('关 闭')", "配置检查面板"),
(".chat-notification-close", "聊天调查通知"),
]
deadline = _time.monotonic() + 12.0 # 最长处理 12s兜底正常几秒内结束
tick_ms = 350
idle_rounds = 0 # 连续无处理的轮数
while _time.monotonic() < deadline:
handled_any = False
for sel, label in targets:
try:
loc = page.locator(sel)
if loc.count() > 0 and loc.first.is_visible(timeout=120):
loc.first.click(timeout=2000)
handled_any = True
page.wait_for_timeout(150) # 留一点关闭动画时间
except Exception:
pass
if handled_any:
idle_rounds = 0
else:
idle_rounds += 1
if idle_rounds >= 3: # 连续 ~1.05s 无任何弹窗 → 提前结束
break
page.wait_for_timeout(tick_ms)
def _remove_if_exists(path):
"""删除文件(若存在):流程开头清理上次的最终文件,避免无数据/失败时残留旧数据。"""
try:
if os.path.exists(path):
os.remove(path)
except Exception:
pass
def _close_tab(page, tab_name):
"""关闭指定名称的百世标签页。
通过 li > span 中的文本定位标签,并点击其内部 title 为 "关闭标签页" 的图标。
"""
try:
tab = page.locator("li").filter(has=page.locator("span", has_text=tab_name))
if tab.count() == 0:
print(f" 未找到标签页【{tab_name}】(可能尚未打开或已关闭),跳过。")
return
# 点击百世特有的关闭按钮
tab.first.locator("i[title='关闭标签页']").click()
print(f" 🗙 已关闭标签页【{tab_name}")
page.wait_for_timeout(300)
except Exception as e:
print(f" ⚠️ 关闭标签页【{tab_name}】时出错: {e}")
def baishi_download_undelivered_data(page, force=False, date=None):
"""百世:一键提取应到未到(当日未扫)数据(内部含异常兜底重试,路由层无感)。
date 形参仅为对齐统一透传签名(百世固定下载当天),忽略。"""
return with_retry(
"百世",
"应到未到",
lambda: baishi_download_undelivered_data_impl(page),
lambda: baishi_reset(page),
page=page,
)
def baishi_download_undelivered_data_impl(page):
"""百世:一键提取应到未到(当日未扫)数据(单次执行,无重试;供自动化测试用)。"""
print("\n▶ 开始执行【百世 - 一键提取应到未到数据】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次的最终文件,避免本次无数据/失败时残留旧数据误导比对
_remove_if_exists(os.path.join(download_dir, "百世-应到未到货物数据.xlsx"))
try:
# 1. 导航与页面加载
print(">> 正在进入【扫描综合查询】界面...")
# 打开基础服务菜单面板
page.locator("div.nav-level1", has_text="基础服务").click()
# 限制在菜单面板nav-level2-wrapper内查找避免命中右侧同名标签页
page.locator(".nav-level2-wrapper").locator(
"a", has_text="扫描综合查询"
).click()
# 验证表格主界面加载完毕
page.get_by_role("tab", name="实时扫描率").wait_for(state="visible")
print("✅ 扫描综合查询界面已加载")
# 1.5 顺手抓取「到/接件扫描 → 当日」的 应扫/已扫(=应到/实到基数),
# 供汇总报表填写百世行的应到件/实到件。
# 实时扫描率表头为 14 列 leaf发/交件[昨日×3, 当日×4] +
# 到/接件[昨日×3, 当日×4],到/接件当日四列索引为
# 10(应扫) 11(已扫) 12(未扫) 13(率)。(与下方 nth(12) 未扫同源)
try:
_first_row = page.locator(".ant-table-tbody > tr").first
_exp_txt = (
_first_row.locator("td").nth(10).inner_text().strip().replace(",", "")
)
_arr_txt = (
_first_row.locator("td").nth(11).inner_text().strip().replace(",", "")
)
def _to_int(v):
try:
return int(float(v)) if v not in ("", "-") else 0
except (TypeError, ValueError):
return 0
_exp_n, _arr_n = _to_int(_exp_txt), _to_int(_arr_txt)
if _exp_n > 0:
state_store.set_setting("百世", "scan_expected_pieces", str(_exp_n))
state_store.set_setting("百世", "scan_arrived_pieces", str(_arr_n))
from inbound_verify import (
store,
) # 直接落库 PG一步不绕 state_store→store
store.upsert_baishi_daily_stats(_exp_n, _arr_n)
print(f" 已记录百世应到/实到基数:应扫 {_exp_n} / 已扫 {_arr_n}")
except Exception as _e:
# 抓取失败绝不影响未到明细下载主流程
print(f" ⚠️ 抓取百世应到/实到基数失败(不影响未到明细下载):{_e}")
# 2. 精准定位并点击【到/接件扫描 -> 当日 -> 未扫】的数字控件
print(">> 正在解析表格,提取当日到件未扫明细...")
# 定位第一行数据的第 13 列(索引 12
target_cell = page.locator(".ant-table-tbody > tr").first.locator("td").nth(12)
# 特殊情况处理检查未扫数量是否为0
cell_text = target_cell.inner_text().strip()
if cell_text == "0" or cell_text == "":
print(
f" 注意:当日未扫数量为【{cell_text}】,无数据需要提取,任务结束。"
)
# 数据为空时,提前结束前也需清理环境
_close_tab(page, "扫描综合查询")
return True
# 有未扫数据,继续点击操作
target_cell.locator("a").click()
# 等待下方弹出的明细表格区域加载完毕
detail_section = page.locator(".m-query-all-scanRateDetail")
detail_section.wait_for(state="visible")
page.wait_for_timeout(1000)
# 3. 触发导出设置模态框
print(">> 正在打开导出配置面板...")
# 使用明细区域内的导出按钮,避免误点主表导出
detail_section.locator(".export-wrap a[title='导出']").click()
# 验证模态框弹出
page.locator(".ant-modal-title", has_text="导出设置").wait_for(state="visible")
# 4. 读取百世导出密码(存 state.db由前端站点配置
print(">> 正在读取配置并填写密码...")
password = state_store.get_setting("百世", "password")
if not password:
print(" ⚠️ 警告:未设置百世导出密码(前端站点配置),可能导致导出失败。")
page.get_by_placeholder("请输入登录密码").fill(password)
# 5. 执行最终下载
print(">> 正在下载...")
with page.expect_download() as download_info:
# 点击模态框底部的“导 出”按钮
page.locator(".ant-modal-footer").get_by_role(
"button", name="导 出"
).click()
download = download_info.value
save_path = os.path.join(download_dir, "百世-应到未到货物数据.xlsx")
# 如果之前已经有同名文件,覆盖保存
if os.path.exists(save_path):
os.remove(save_path)
download.save_as(save_path)
print(f"====================================================")
print(f" 提取成功。")
print(f" 已下载: {save_path}")
print(f"====================================================")
# 6. 环境清理
print(">> 任务完成,正在清理环境...")
_close_tab(page, "扫描综合查询")
print("\n【百世 - 应到未到数据提取】流程结束。")
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False

View File

@@ -0,0 +1,884 @@
# sites/shunxin.py
import os
import re
import time
import yaml
from datetime import datetime, timedelta
import pandas as pd
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
"""
for attempt in range(1, max_attempts + 1):
try:
ret = flow()
if ret is False:
raise RuntimeError("流程返回失败状态")
if attempt > 1:
print(f">> 【{site_name}-{label}】第 {attempt} 次尝试成功 ✅")
return True
except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try:
reset()
except Exception as re:
print(f" ⚠️ 重置异常: {re}")
if attempt < max_attempts:
continue
print(
f"❌ 【{site_name}-{label}】已达最大尝试次数 {max_attempts},放弃(环境已清理)。"
)
return False
# 站点首页 URL异常兜底重置用也供 runtime 的 SITES_CONFIG 引用(单一来源)
HOME_URL = "https://sxne.sxjdfreight.com"
def shunxin_reset(page):
"""异常兜底:重置顺心到初始态(跳首页 URL丢弃当前页面状态登录态保留"""
page.goto(HOME_URL)
page.wait_for_timeout(1500)
def _close_tab(page, tab_name):
"""关闭指定名称的标签页Ant Design Tabs
通过标签文字定位标签容器,点击其右侧的关闭(×)按钮。
- tab_name 采用包含匹配,对“ 运单列表”这类带空格的标签同样有效。
- 关闭失败不会影响主流程,仅打印提示信息。
"""
try:
tab = page.locator(".ant-tabs-tab").filter(
has=page.get_by_role("tab", name=tab_name)
)
if tab.count() == 0:
print(
f" 未找到标签页【{tab_name.strip()}】(可能尚未打开或已关闭),跳过。"
)
return
tab.first.locator(".ant-tabs-tab-remove").click()
print(f" 🗙 已关闭标签页【{tab_name.strip()}")
page.wait_for_timeout(300)
except Exception as e:
print(f" ⚠️ 关闭标签页【{tab_name.strip()}】时出错: {e}")
def _sanitize_for_filename(name):
"""剔除 Windows 文件名非法字符,避免归属地名含特殊字符导致落盘失败。"""
return re.sub(r'[\\/:*?"<>|]', "", str(name)).strip()
def _remove_if_exists(path):
"""删除文件(若存在):清理上次的本账号中间文件/最终文件,避免残留旧数据。"""
try:
if os.path.exists(path):
os.remove(path)
except Exception:
pass
def _shunxin_navigate_picker_to_target(page, date_str, max_flips=24):
"""顺心 Ant Design 单月面板跨月导航(车辆点到 / 卸车扫描记录 共用同一组件)。
读面板头部 .ant-picker-year-btn / .ant-picker-month-btn 得当前显示的年月,按与目标
年月的差值点 .ant-picker-header-prev-btn上月/ .ant-picker-header-next-btn下月
翻到目标月视窗。返回 True 表示当前视窗已是目标月(目标格子随后可见可点)。
调用前提:开始/结束时间输入已点开,.ant-picker-dropdown:visible 已就绪。
"""
try:
ty, tm = (int(x) for x in date_str.split("-")[:2])
except Exception:
return True # 解析不出就不翻,交由后续 cell.click 自行成败
drop = page.locator(".ant-picker-dropdown:visible")
for _ in range(max_flips):
try:
cur_y = int(
re.search(
r"\d+", drop.locator(".ant-picker-year-btn").first.inner_text()
).group()
)
cur_m = int(
re.search(
r"\d+", drop.locator(".ant-picker-month-btn").first.inner_text()
).group()
)
except Exception:
return False
cur = cur_y * 12 + (cur_m - 1)
tgt = ty * 12 + (tm - 1)
if cur == tgt:
return True
btn_sel = (
".ant-picker-header-prev-btn"
if tgt < cur
else ".ant-picker-header-next-btn"
)
drop.locator(btn_sel).first.click()
page.wait_for_timeout(300)
return False
def _shunxin_pick_date(page, date_str):
"""在已打开的顺心 Ant Design 日期浮层上选中指定日期格子(含跨月翻月)。
目标格子不在当前月视窗(跨月)时,先调 _shunxin_navigate_picker_to_target 翻到目标月,
再点格子;同月则直接点。与中通 _zto_flip_to_target_month 思路对称,适配 Ant Design 面板。
"""
cell = page.locator(f".ant-picker-dropdown:visible td[title='{date_str}']").first
if not cell.is_visible():
print(f" 目标日期 {date_str} 不在当前月视窗,正在翻月导航 ...")
if not _shunxin_navigate_picker_to_target(page, date_str):
raise RuntimeError(f"翻月后仍无法定位目标日期格子 {date_str}")
cell = page.locator(
f".ant-picker-dropdown:visible td[title='{date_str}']"
).first
cell.click()
def shunxin_belonging(page):
"""读取顺心当前账号的归属网点名(仅在首页可见,须在导航离开首页前调用)。
取首页「切换网点」下拉框选中项的 title形如「【SX】重庆巴南龙海大道店」
去掉【XX】前缀得到归属地如「重庆巴南龙海大道店」并做文件名安全处理。
读取失败时抛异常,交由上层处理。
"""
item = page.locator(".site___3o7nH .ant-select-selection-item").first
title = (item.get_attribute("title") or item.inner_text() or "").strip()
if not title:
raise RuntimeError("未能读取顺心归属网点(首页「切换网点」控件为空)")
tag = re.sub(r"^【[^】]*】", "", title).strip() or title
return _sanitize_for_filename(tag)
def shunxin_merge_final(kind, tags):
"""把各归属地的中间产物融合成统一的「顺心-{kind}货物数据.xlsx」。
kind ∈ {"应到","实到"}tags 为各账号归属地列表。逐个读取
「顺心-{tag}-{kind}货物数据.xlsx」缺失则跳过容错空数据账号pd.concat
后写出统一文件,并删除中间带 tag 的文件;全部缺失则仅提示、不产出。
"""
final_name = f"顺心-{kind}货物数据.xlsx"
final_path = os.path.join(DOWNLOAD_DIR, final_name)
frames = []
mid_paths = []
for tag in tags:
mid_name = f"顺心-{tag}-{kind}货物数据.xlsx"
mid_path = os.path.join(DOWNLOAD_DIR, mid_name)
if not os.path.exists(mid_path):
print(f" 归属【{tag}】无{kind}中间文件(可能本次无数据),跳过。")
continue
mid_paths.append(mid_path)
try:
df = pd.read_excel(mid_path, dtype=str)
if not df.empty:
frames.append(df)
except Exception as e:
print(f" ⚠️ 读取中间文件 {mid_name} 失败: {e}")
if not frames:
print(f">> ⚠️ 所有归属地均无{kind}数据,删除残留的 {final_name}(不写空表)。")
_remove_if_exists(final_path)
return
combined = pd.concat(frames, ignore_index=True)
combined.to_excel(final_path, index=False)
print("====================================================")
print(
f" {kind}数据融合完成(共 {len(combined)} 行),输出: downloads/{final_name}"
)
print("====================================================")
for mid_path in mid_paths:
try:
os.remove(mid_path)
except Exception:
pass
def shunxin_expected_download(pages, foreground=True, force=False, date=None):
"""顺心:应到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。
pages 为该站点的 page 列表(双账号在同一窗口的各一个标签页)。
先在首页读取各账号归属地并去重校验(两账号登同一归属地则中止,防数据翻倍),
再顺序对各账号跑一遍下载impl 用归属地作输出文件后缀),最后融合成统一的
「顺心-应到货物数据.xlsx」。路由层只需传入 page 列表,对双账号无感。
"""
tags = []
for idx, pg in enumerate(pages, start=1):
tag = shunxin_belonging(pg)
print(f">> 【顺心】账号{idx} 归属网点:{tag}")
tags.append(tag)
if len(set(tags)) != len(tags):
raise RuntimeError(
f"顺心两个账号归属地相同({tags}),疑似登录了同一账号,已中止以防数据翻倍。"
)
for idx, (pg, tag) in enumerate(zip(pages, tags), start=1):
if foreground:
pg.bring_to_front()
print(f"\n========== 顺心 · 账号{idx}{tag})应到数据下载 ==========")
ok = with_retry(
f"顺心-{tag}",
"应到",
lambda p=pg, t=tag, f=force, d=date: shunxin_expected_download_impl(
p, out_tag=t, force=f, date=d
),
lambda p=pg: shunxin_reset(p),
page=pg,
)
if not ok:
return False # 某账号重试耗尽 → 整体失败,不融合(避免部分数据)
shunxin_merge_final("应到", tags)
return True
def shunxin_expected_download_impl(page, out_tag="", force=False, date=None):
"""顺心:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。
out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-应到货物数据.xlsx」
作为双账号融合前的各账号中间文件;为空时退化为「顺心-应到货物数据.xlsx」。
"""
print("\n▶ 开始执行【顺心 - 应到货物数据下载】任务...")
# 初始化并创建下载目录
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
print(f">> 已创建下载目录: {download_dir}")
# 清理上次本账号的中间文件,避免本次无数据/失败时残留旧数据被 merge_final 误读
_mid_suffix = f"-{out_tag}" if out_tag else ""
_remove_if_exists(
os.path.join(download_dir, f"顺心{_mid_suffix}-应到货物数据.xlsx")
)
export_times = []
target_task_timestamps = []
try:
# 1. 导航与页面加载判断
print(">> 正在进入【车辆点到】界面...")
page.locator("span.ant-pro-menu-item-title:has-text('派件管理')").click()
page.locator("div.ant-pro-menu-item:has-text('车辆点到')").click()
page.get_by_role("button", name="点到").wait_for(state="visible")
page.get_by_role("button", name="打印交接单").wait_for(state="visible")
page.get_by_role("button", name="强卸").wait_for(state="visible")
print("✅ 车辆点到界面加载完毕")
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("顺心")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 正在设置查询日期: [{target_str}]{src}...")
# 分两步精准呼出和点击时间控件
print(" >> 设置起始时间...")
page.get_by_placeholder("开始时间").click()
page.wait_for_timeout(500)
_shunxin_pick_date(page, start_date_str)
page.wait_for_timeout(300)
print(" >> 设置截止时间...")
page.get_by_placeholder("结束时间").click()
page.wait_for_timeout(500)
_shunxin_pick_date(page, today_str)
page.wait_for_timeout(300)
# 确认日期
page.locator(".ant-picker-dropdown:visible button", has_text="确 定").click()
page.wait_for_timeout(500)
print(">> 正在展开【状态】下拉菜单...")
page.locator(
".ant-select-selection-item", has_text=re.compile(r"已发|已到")
).click()
print(">> 正在选择状态为【已到】...")
page.locator(".ant-select-item-option", has_text="已到").click()
page.wait_for_timeout(500)
# ====================================================================
# 🛡️ 双重校验兜底机制:破除顺心表格“暂无数据”的旧状态遗留陷阱
# ====================================================================
print(">> 正在发起查询与数据状态研判...")
has_data = False
waybill_btns = None
for attempt in range(2):
print(f" -> 第 {attempt + 1} 次点击【查询】按钮...")
page.get_by_role("button", name="search 查询").click()
# 尝试在极短时间内捕获加载小菊花
loading_spinner = page.locator(".ant-spin-dot-spin").first
try:
loading_spinner.wait_for(state="visible", timeout=800)
print(" ⏳ 捕捉到加载动画,等待数据渲染完成...")
loading_spinner.wait_for(state="hidden", timeout=15000)
except Exception:
print(" ⚡ 加载动画闪过过快或未出现强制安全缓冲1秒...")
page.wait_for_timeout(1000)
# 解析查询结果
empty_desc = page.locator(
".ant-empty-description", has_text="暂无数据"
).first
waybill_btns = page.get_by_role("button", name="运单列表")
if waybill_btns.count() > 0:
has_data = True
print(" ✅ 数据已成功加载。")
break
elif empty_desc.is_visible():
print(" ⚠️ 当前表格显示【暂无数据】。")
if attempt == 0:
print(" -> 疑似前端 DOM 旧状态未刷新,触发二次兜底查询...")
page.wait_for_timeout(500)
else:
print(" -> 已二次确认为空数据环境。")
else:
# 没出现暂无数据,也没出现按钮,稳妥判定缓冲完毕
pass
if not has_data:
print(">> ⚠️ 本次查询区间内没有数据记录,提前结束流程。")
_close_tab(page, "车辆点到")
return True
# ====================================================================
count = waybill_btns.count()
print(f">> 共发现 {count} 个班次需要导出。")
# 【去重】加载本站已落库交接单号force=True 或查询失败时 existing=空集(不去重)。
# 两账号共享同一集合(班次号/交接单号跨归属地不重叠)。
if force:
existing = set()
print(">> [去重] 强制重下,跳过去重。")
else:
try:
from inbound_verify import store
existing = store.get_existing_handover_nos("顺心")
except Exception as _e:
existing = set()
print(f">> [去重] 加载已落库交接单号失败,本次不去重: {_e}")
for i in range(count):
print(f" ⏳ 正在处理第 {i+1}/{count} 个班次...")
waybill_btns.nth(i).click()
page.locator("label[title='运单查询']").wait_for(state="visible")
# 【方式1】运单列表界面已加载读交接单号RTS 开头)→ 已落库则退回列表跳过。
# 交接单号格式 RTS\d{3}WJ\d+(如 RTS023WJ374837用 [A-Z0-9]+ 连续匹配整段。
# 读不到DOM 变动/未渲染)则 handover_no 为空 → 不跳过(安全降级,继续导出)。
handover_no = ""
try:
_txt = page.locator("text=/RTS\\d+/").first.inner_text(timeout=3000)
_m = re.search(r"RTS[A-Z0-9]+", _txt)
if _m:
handover_no = _m.group(0)
except Exception:
pass
print(f" -> 运单列表交接单号:{handover_no or '(未读到,不去重)'}")
if handover_no and handover_no in existing:
print(f" ⏭️ 交接单号 {handover_no} 已落库,跳过提交导出。")
page.get_by_role("tab", name="车辆点到").click()
page.wait_for_timeout(500)
continue
# 4. 执行导出流程
page.get_by_role("button", name="export 导出").click()
page.locator(
"span.ant-transfer-list-header-title:has-text('待选导出列')"
).wait_for(state="visible")
page.locator(".ant-transfer-list").first.locator(
".ant-transfer-list-header label"
).click()
page.get_by_label("导出").get_by_role("button", name="right").click()
page.get_by_role("button", name="export 导出数据").click()
export_times.append(datetime.now())
page.locator("text=任务添加成功!").wait_for(state="visible")
page.get_by_role("button", name="知道了").click()
page.get_by_role("tab", name="车辆点到").click()
page.wait_for_timeout(500)
print("✅ 所有班次的导出任务已成功提交!")
# 5. 关闭标签页
_close_tab(page, "运单列表")
_close_tab(page, "车辆点到")
# 【去重兜底】全部已落库/无数据 → 无导出任务,标签页已关,跳过下载轮询
if not export_times:
print(">> 本次无新班次需导出(全部已落库或无数据),结束。")
return True
# 6. 前往数据导出页面去下载
print(">> 正在前往【数据导出】界面...")
page.locator("a[href='/dataExport']").click()
page.get_by_role("columnheader", name="任务标题").wait_for(state="visible")
page.wait_for_timeout(2000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 7. 轮询任务状态
print(">> 列表已加载,开始匹配并检查任务状态...")
poll_deadline = (
time.monotonic() + 300
) # 5 分钟上限:任务卡死/匹配不上时超时失败,交由上层重置重试
while True:
if time.monotonic() > poll_deadline:
raise RuntimeError(
"轮询导出任务超时5 分钟未全部完成/匹配),疑似任务卡死"
)
rows = page.locator(".ant-table-tbody > tr.ant-table-row")
row_count = rows.count()
pending_tasks = 0
current_ready_timestamps = []
# 单账号无并发:仅按提交时间容差(40s)认领本批任务,不再校验任务标题
# (站点可能调整标题名,写死标题会导致匹配失败、任务一直查不到)。
for i in range(row_count):
tds = rows.nth(i).locator("td")
if tds.count() < 9:
continue
submit_time_str = tds.nth(2).inner_text().strip()
status_str = tds.nth(6).inner_text().strip()
try:
row_time = datetime.strptime(submit_time_str, "%Y-%m-%d %H:%M:%S")
matched = any(
abs((row_time - et).total_seconds()) <= 40
for et in export_times
)
if matched:
if status_str != "执行完成":
pending_tasks += 1
if submit_time_str not in current_ready_timestamps:
print(
f" ⏳ 任务 [{submit_time_str}] 状态为【{status_str}】,数据生成中..."
)
else:
if submit_time_str not in current_ready_timestamps:
current_ready_timestamps.append(submit_time_str)
except Exception as e:
print(f" ⚠️ 解析时间时出错: {e}")
if pending_tasks > 0:
print(
f">> 共有 {pending_tasks} 个匹配任务还在处理中,等待 5 秒后刷新..."
)
page.wait_for_timeout(5000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
else:
target_task_timestamps = current_ready_timestamps
if len(target_task_timestamps) > 0:
print(">> 所有目标任务已就绪,开始下载...")
break
# 8. 下载逻辑
downloaded_files = []
for time_str in target_task_timestamps:
try:
target_row = page.locator(".ant-table-tbody > tr.ant-table-row").filter(
has=page.locator(f"td:nth-child(3):has-text('{time_str}')")
)
print(f" 开始下载任务 [{time_str}] ...")
with page.expect_download() as download_info:
target_row.locator("td").nth(8).locator(
"button", has_text=re.compile(r"\s*载")
).click()
download = download_info.value
safe_timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
custom_filename = f"顺心_temp_{safe_timestamp}.xlsx"
save_path = os.path.join(download_dir, custom_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" 已下载: downloads/{custom_filename}")
except Exception as e:
print(f" ❌ 下载任务 [{time_str}] 失败: {e}")
# 汇总校验:下载成功数必须等于目标任务数,否则判失败
if len(downloaded_files) < len(target_task_timestamps):
raise RuntimeError(
f"仅成功下载 {len(downloaded_files)}/{len(target_task_timestamps)} 个任务,数据不完整"
)
# 9. 合并数据
if downloaded_files:
print("\n>> 正在合并下载的数据...")
all_data_frames = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path, dtype=str)
if not df.empty:
all_data_frames.append(df)
except Exception as e:
pass
if all_data_frames:
combined_df = pd.concat(all_data_frames, ignore_index=True)
suffix = f"-{out_tag}" if out_tag else ""
final_output_path = os.path.join(
download_dir, f"顺心{suffix}-应到货物数据.xlsx"
)
combined_df.to_excel(final_output_path, index=False)
print(f"====================================================")
print(f" 合并完成,输出文件: {final_output_path}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print("✅ 临时文件已清理。")
# 10. 关闭标签页
_close_tab(page, "数据导出")
print("\n【顺心 - 应到货物数据下载】流程结束。")
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False
def shunxin_actual_download(pages, foreground=True, force=False, date=None):
"""顺心:实到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。
与 shunxin_expected_download 同构:读归属地 → 去重校验 → 顺序各账号下载 →
融合成统一的「顺心-实到货物数据.xlsx」。
"""
tags = []
for idx, pg in enumerate(pages, start=1):
tag = shunxin_belonging(pg)
print(f">> 【顺心】账号{idx} 归属网点:{tag}")
tags.append(tag)
if len(set(tags)) != len(tags):
raise RuntimeError(
f"顺心两个账号归属地相同({tags}),疑似登录了同一账号,已中止以防数据翻倍。"
)
for idx, (pg, tag) in enumerate(zip(pages, tags), start=1):
if foreground:
pg.bring_to_front()
print(f"\n========== 顺心 · 账号{idx}{tag})实到数据下载 ==========")
ok = with_retry(
f"顺心-{tag}",
"实到",
lambda p=pg, t=tag, d=date: shunxin_actual_download_impl(
p, out_tag=t, date=d
),
lambda p=pg: shunxin_reset(p),
page=pg,
)
if not ok:
return False
shunxin_merge_final("实到", tags)
return True
def shunxin_actual_download_impl(page, out_tag="", date=None):
"""顺心:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。
out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-实到货物数据.xlsx」
作为双账号融合前的各账号中间文件;为空时退化为「顺心-实到货物数据.xlsx」。
"""
print("\n▶ 开始执行【顺心 - 实到货物数据下载】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次本账号的中间文件,避免本次无数据/失败时残留旧数据被 merge_final 误读
_mid_suffix = f"-{out_tag}" if out_tag else ""
_remove_if_exists(
os.path.join(download_dir, f"顺心{_mid_suffix}-实到货物数据.xlsx")
)
export_times = []
target_task_timestamps = []
try:
# 1. 导航与页面加载
print(">> 正在进入【卸车扫描记录】界面...")
page.locator("span.ant-pro-menu-item-title:has-text('派件管理')").click()
page.locator("div.ant-pro-menu-item:has-text('卸车扫描记录')").click()
page.get_by_role("radio", name="1天").wait_for(state="visible")
print("✅ 卸车扫描记录界面加载完毕")
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("顺心", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 正在设置查询日期: [{target_str}]{src}...")
# 分两步精准呼出和点击时间控件
print(" >> 设置起始时间...")
page.get_by_placeholder("开始时间").click()
page.wait_for_timeout(500)
_shunxin_pick_date(page, start_date_str)
page.wait_for_timeout(300)
print(" >> 设置截止时间...")
page.get_by_placeholder("结束时间").click()
page.wait_for_timeout(500)
_shunxin_pick_date(page, today_str)
page.wait_for_timeout(300)
# 确认日期
page.locator(".ant-picker-dropdown:visible button", has_text="确 定").click()
page.wait_for_timeout(500)
# ====================================================================
# 🛡️ 卸车扫描记录页面双重校验兜底机制
# ====================================================================
print(">> 正在发起查询与数据状态研判...")
has_data = False
for attempt in range(2):
print(f" -> 第 {attempt + 1} 次点击【查询】按钮...")
page.get_by_role("button", name="search 查询").click()
loading_spinner = page.locator(".ant-spin-dot-spin").first
try:
loading_spinner.wait_for(state="visible", timeout=800)
print(" ⏳ 捕捉到加载动画,等待数据渲染完成...")
loading_spinner.wait_for(state="hidden", timeout=15000)
except Exception:
print(" ⚡ 加载动画闪过过快或未出现强制安全缓冲1秒...")
page.wait_for_timeout(1000)
empty_desc = page.locator(
".ant-empty-description", has_text="暂无数据"
).first
data_rows = page.locator(".ant-table-tbody > tr.ant-table-row")
if empty_desc.is_visible():
print(" ⚠️ 当前表格显示【暂无数据】。")
if attempt == 0:
print(" -> 疑似前端 DOM 旧状态未刷新,触发二次兜底查询...")
page.wait_for_timeout(500)
else:
print(" -> 已二次确认为空数据环境。")
elif data_rows.count() > 0:
has_data = True
print(" ✅ 数据已成功加载。")
break
else:
pass
if not has_data:
print(">> ⚠️ 本次查询未产生任何卸车记录,提前结束流程。")
_close_tab(page, "卸车扫描记录")
return True
# ====================================================================
# 3. 直接发起全局导出
print(">> 正在发起全局数据导出请求...")
page.get_by_role("button", name="export 导出").click()
page.locator(
"span.ant-transfer-list-header-title:has-text('待选导出列')"
).wait_for(state="visible")
page.locator(".ant-transfer-list").first.locator(
".ant-transfer-list-header label"
).click()
page.get_by_label("导出").get_by_role("button", name="right").click()
page.get_by_role("button", name="export 导出数据").click()
export_times.append(datetime.now())
page.locator("text=任务添加成功!").wait_for(state="visible")
page.get_by_role("button", name="知道了").click()
print("✅ 卸车扫描记录导出任务已成功提交!")
# 4. 关闭标签页
_close_tab(page, "卸车扫描记录")
# 5. 前往数据导出页面去下载
print(">> 正在前往【数据导出】界面...")
page.locator("a[href='/dataExport']").click()
page.get_by_role("columnheader", name="任务标题").wait_for(state="visible")
page.wait_for_timeout(2000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 6. 轮询任务状态
print(">> 列表已加载,开始匹配并检查任务状态...")
poll_deadline = (
time.monotonic() + 300
) # 5 分钟上限:任务卡死/匹配不上时超时失败,交由上层重置重试
while True:
if time.monotonic() > poll_deadline:
raise RuntimeError(
"轮询导出任务超时5 分钟未全部完成/匹配),疑似任务卡死"
)
rows = page.locator(".ant-table-tbody > tr.ant-table-row")
row_count = rows.count()
pending_tasks = 0
current_ready_timestamps = []
# 单账号无并发:仅按提交时间容差(40s)认领本批任务,不再校验任务标题
# (站点可能调整标题名,写死标题会导致匹配失败、任务一直查不到)。
for i in range(row_count):
tds = rows.nth(i).locator("td")
if tds.count() < 9:
continue
submit_time_str = tds.nth(2).inner_text().strip()
status_str = tds.nth(6).inner_text().strip()
try:
row_time = datetime.strptime(submit_time_str, "%Y-%m-%d %H:%M:%S")
matched = any(
abs((row_time - et).total_seconds()) <= 40
for et in export_times
)
if matched:
if status_str != "执行完成":
pending_tasks += 1
if submit_time_str not in current_ready_timestamps:
print(
f" ⏳ 任务 [{submit_time_str}] 状态为【{status_str}】,数据生成中..."
)
else:
if submit_time_str not in current_ready_timestamps:
current_ready_timestamps.append(submit_time_str)
except Exception as e:
print(f" ⚠️ 解析时间时出错: {e}")
if pending_tasks > 0:
print(
f">> 共有 {pending_tasks} 个匹配任务还在处理中,等待 5 秒后刷新..."
)
page.wait_for_timeout(5000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
else:
target_task_timestamps = current_ready_timestamps
if len(target_task_timestamps) > 0:
print(">> 目标任务已就绪,开始下载...")
break
# 7. 下载逻辑
downloaded_files = []
for time_str in target_task_timestamps:
try:
target_row = page.locator(".ant-table-tbody > tr.ant-table-row").filter(
has=page.locator(f"td:nth-child(3):has-text('{time_str}')")
)
print(f" 开始下载任务 [{time_str}] ...")
with page.expect_download() as download_info:
target_row.locator("td").nth(8).locator(
"button", has_text=re.compile(r"\s*载")
).click()
download = download_info.value
safe_timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
custom_filename = f"顺心_temp_{safe_timestamp}.xlsx"
save_path = os.path.join(download_dir, custom_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" 已下载: downloads/{custom_filename}")
except Exception as e:
print(f" ❌ 下载任务 [{time_str}] 失败: {e}")
# 汇总校验:下载成功数必须等于目标任务数,否则判失败
if len(downloaded_files) < len(target_task_timestamps):
raise RuntimeError(
f"仅成功下载 {len(downloaded_files)}/{len(target_task_timestamps)} 个任务,数据不完整"
)
# 8. 合并数据
if downloaded_files:
print("\n>> 正在合并下载的数据...")
all_data_frames = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path, dtype=str)
if not df.empty:
all_data_frames.append(df)
except Exception as e:
pass
if all_data_frames:
combined_df = pd.concat(all_data_frames, ignore_index=True)
suffix = f"-{out_tag}" if out_tag else ""
final_output_path = os.path.join(
download_dir, f"顺心{suffix}-实到货物数据.xlsx"
)
combined_df.to_excel(final_output_path, index=False)
print(f"====================================================")
print(f" 合并完成,输出文件: {final_output_path}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print("✅ 临时文件已清理。")
# 9. 关闭标签页
_close_tab(page, "数据导出")
print("\n【顺心 - 实到货物数据下载】流程结束。")
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False

View File

@@ -0,0 +1,847 @@
# sites/yunda.py
import os
import re
import time
import yaml
from datetime import datetime, timedelta
import pandas as pd
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
"""
for attempt in range(1, max_attempts + 1):
try:
ret = flow()
if ret is False:
raise RuntimeError("流程返回失败状态")
if attempt > 1:
print(f">> 【{site_name}-{label}】第 {attempt} 次尝试成功 ✅")
return True
except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try:
reset()
except Exception as re:
print(f" ⚠️ 重置异常: {re}")
if attempt < max_attempts:
continue
print(
f"❌ 【{site_name}-{label}】已达最大尝试次数 {max_attempts},放弃(环境已清理)。"
)
return False
# 站点首页 URL异常兜底重置用也供 runtime 的 SITES_CONFIG 引用(单一来源)
HOME_URL = "https://ky-sso.yunda56.com"
def dismiss_audio_prompt(page):
"""关闭韵达登录后弹出的阻塞式提示弹窗(如音频设备未授权/未找到)。
不依赖具体文案(不同主机/音频状态下文案不同:未授权/未找到…),只要出现
.ivu-modal-confirm 就点其「确定」。主页加载后调用(初始 + 重试重载)。
"""
try:
modal = page.locator(".ivu-modal-confirm").first
modal.wait_for(state="visible", timeout=2000)
modal.locator(".ivu-modal-confirm-footer button.ivu-btn-primary").click()
print(" ✅ 【韵达】已关闭提示弹窗。")
return True
except Exception:
return False
def yunda_reset(page):
"""异常兜底:重置韵达到初始态(跳首页 URL丢弃当前页面状态登录态保留"""
page.goto(HOME_URL)
page.wait_for_timeout(1500)
dismiss_audio_prompt(page) # 主页重载后音频授权提示会复现,清理之
def _remove_if_exists(path):
"""删除文件(若存在):流程开头清理上次的最终文件,避免无数据/失败时残留旧数据。"""
try:
if os.path.exists(path):
os.remove(path)
except Exception:
pass
def _yunda_pick_laydate_new(ws_frame, page, date_ymd):
"""应到(新版 laydate .layui-laydate在已打开的面板上选中指定日期含跨月翻月
目标格子 td[lay-ymd='YYYY-M-D'](非补零)不在当前月视窗时,读 .laydate-set-ym 的
当前年月形如「2026年7月」按差值点 .laydate-prev-m / .laydate-next-m 翻到目标月,
再点格子;同月则直接点。调用前提:#startTime/#endTime 已点开,.layui-laydate:visible 就绪。
"""
cal = ws_frame.locator(".layui-laydate:visible").first
cell = cal.locator(f"td[lay-ymd='{date_ymd}']").first
if not cell.is_visible():
ty, tm = (int(x) for x in date_ymd.split("-")[:2])
print(f" 目标日期 {date_ymd} 不在当前月视窗,正在翻月导航 ...")
for _ in range(24):
nums = re.findall(r"\d+", cal.locator(".laydate-set-ym").first.inner_text())
if len(nums) >= 2:
cur_y, cur_m = int(nums[0]), int(nums[1])
if cur_y == ty and cur_m == tm:
break
cur = cur_y * 12 + (cur_m - 1)
btn = (
".laydate-prev-m"
if (ty * 12 + (tm - 1)) < cur
else ".laydate-next-m"
)
cal.locator(btn).first.click()
page.wait_for_timeout(300)
cell = cal.locator(f"td[lay-ymd='{date_ymd}']").first
cell.click()
def _yunda_pick_laydate_old(ws_frame, page, date_ymd):
"""实到(旧版 laydate #laydate_box在已打开的面板上选中指定日期含跨月翻月
目标格子 td[y][m][d](非补零)不在当前月视窗时,读 #laydate_y/#laydate_m 输入框值
形如「2026年」「07月」得当前年月按差值点 #laydate_MM 内 .laydate_chprev /
.laydate_chnext 翻到目标月,再点格子;同月则直接点。调用前提:#startDate/#endDate
已点开force=True#laydate_box:visible 就绪。
"""
box = ws_frame.locator("#laydate_box:visible").first
ty, tm, td = (int(x) for x in date_ymd.split("-")[:3])
cell = box.locator(f"td[y='{ty}'][m='{tm}'][d='{td}']").first
if not cell.is_visible():
print(f" 目标日期 {date_ymd} 不在当前月视窗,正在翻月导航 ...")
for _ in range(24):
yv = box.locator("#laydate_y").first.evaluate("e=>e.value")
mv = box.locator("#laydate_m").first.evaluate("e=>e.value")
cur_y = int(re.search(r"\d+", yv).group())
cur_m = int(re.search(r"\d+", mv).group())
if cur_y == ty and cur_m == tm:
break
cur = cur_y * 12 + (cur_m - 1)
btn = ".laydate_chprev" if (ty * 12 + (tm - 1)) < cur else ".laydate_chnext"
box.locator(f"#laydate_MM {btn}").first.click()
page.wait_for_timeout(300)
cell = box.locator(f"td[y='{ty}'][m='{tm}'][d='{td}']").first
cell.click()
def _resolve_export_frame(ws_frame):
"""定位韵达数据导出面板内嵌的 iframe。
韵达改版后导出面板换用 Element UI全选/向右转移/导出按钮)。实到流程的面板
iframe 名为 myFrame已验证应到流程历史上为 target1。这里短超时轮流探测
返回首个出现「全选」按钮的 frame都未命中则 dump 面板内所有 iframe 名便于排查。
"""
for name in ("myFrame", "target1"):
frame = ws_frame.frame_locator(f'iframe[name="{name}"]')
try:
frame.locator("button.el-button", has_text="全选").first.wait_for(
state="visible", timeout=5000
)
print(f" [导出iframe] 命中 iframe[name={name}]")
return frame
except Exception:
continue
try:
names = ws_frame.locator("iframe").evaluate_all(
"els => els.map(e => e.name || '(无name)')"
)
print(
f" [导出iframe] myFrame/target1 均未命中全选;面板 iframe 名: {names}"
)
except Exception as e:
print(f" [导出iframe] dump 失败: {e}")
return ws_frame.frame_locator('iframe[name="myFrame"]')
def yunda_login(page):
"""韵达自动登录:未登录则填充表单并提交,已登录则跳过。"""
print(">> 正在检查韵达登录状态...")
try:
# 定位“账号密码登录”切换按钮
switch_btn = page.locator("span", has_text="账号密码登录")
# 5 秒内若出现该按钮,说明当前未登录
if switch_btn.is_visible(timeout=5000):
print(" -> 检测到未登录界面,正在切换到【账号密码登录】...")
switch_btn.click()
page.wait_for_timeout(500)
# 凭证存 state.db由前端站点配置
username = state_store.get_setting("韵达", "username")
password = state_store.get_setting("韵达", "password")
print(f" -> 正在填充登录表单 (账号: {username})...")
page.locator("#username").fill(username)
page.locator("#password").fill(password)
page.wait_for_timeout(300)
print(" -> 正在点击【登录】按钮并提交表单...")
page.locator('button[type="submit"]', has_text="登录").click()
page.wait_for_timeout(1000)
else:
print(" -> 未发现登录按钮,判定为已登录,跳过。")
except Exception as e:
print(f" ⚠️ 登录检测出错(可能已在工作台内): {e}")
def yunda_smart_menu_click(page, menu_path):
"""韵达多级菜单导航:展开父级菜单并点击目标项(已展开则跳过,避免误折叠)。"""
print(f">> 导航韵达菜单: {' -> '.join(menu_path)}")
for item in menu_path:
title_locator = page.locator(
f"xpath=//div[contains(@class, 'el-submenu__title') and .//span[normalize-space(.)='{item}']]"
).first
parent_li = page.locator(
f"xpath=//div[contains(@class, 'el-submenu__title') and .//span[normalize-space(.)='{item}']]/.."
).first
leaf_locator = page.locator(
f"xpath=//li[contains(@class, 'el-menu-item') and .//span[normalize-space(.)='{item}']]"
).first
if title_locator.is_visible():
current_class = parent_li.get_attribute("class") or ""
is_opened = "is-opened" in current_class
if not is_opened:
print(f" -> 父菜单 [{item}] 处于收起状态,点击展开")
title_locator.click()
page.wait_for_timeout(500)
else:
print(f" -> 父菜单 [{item}] 已展开,跳过点击")
elif leaf_locator.is_visible():
print(f" -> 点击菜单项 [{item}]")
leaf_locator.click()
page.wait_for_timeout(1000)
def yunda_expected_download(page, force=False, date=None):
"""韵达:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"韵达",
"应到",
lambda: yunda_expected_download_impl(page, force=force, date=date),
lambda: yunda_reset(page),
page=page,
)
def yunda_expected_download_impl(page, force=False, date=None):
"""韵达:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【韵达 - 应到货物数据下载】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次的最终文件,避免本次无数据/失败时残留旧数据误导比对
_remove_if_exists(os.path.join(download_dir, "韵达-应到货物数据.xlsx"))
export_times = []
try:
# 1. 验证首页并导航菜单
page.locator(".el-menu-item", has_text="首页").wait_for(
state="visible", timeout=15000
)
print("✅ 韵达工作台首页已加载")
yunda_smart_menu_click(page, ["运营管理", "进站管理", "进站交接单查询"])
print(">> 正在定位【进站交接单查询】iframe...")
ws_frame = page.frame_locator("section iframe")
ws_frame.locator("#startTime").wait_for(state="attached", timeout=15000)
print("✅ 进站交接单查询页面就绪")
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("韵达")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_ymd = f"{target.year}-{target.month}-{target.day}"
start_date_ymd = target_ymd
today_ymd = target_ymd
src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 设置查询日期: [{target_ymd}]{src}")
# 设定起始时间
print(" >> 设置起始时间...")
page.wait_for_timeout(1000)
ws_frame.locator("#startTime").click(force=True)
calendar1 = ws_frame.locator(".layui-laydate:visible").first
calendar1.wait_for(state="visible", timeout=5000)
_yunda_pick_laydate_new(ws_frame, page, start_date_ymd)
calendar1.locator(".laydate-btns-confirm").click()
page.wait_for_timeout(400)
# 设定截止时间
print(" >> 设置截止时间...")
ws_frame.locator("#endTime").click(force=True)
calendar2 = ws_frame.locator(".layui-laydate:visible").first
calendar2.wait_for(state="visible", timeout=5000)
_yunda_pick_laydate_new(ws_frame, page, today_ymd)
calendar2.locator(".laydate-btns-confirm").click()
page.wait_for_timeout(500)
# 3. 等待数据加载完成
print(">> 正在执行查询...")
ws_frame.locator("a.btn-success", has_text="查询").click()
page.wait_for_timeout(800)
# 将 Loading 蒙层与数据判断限制在 #tab-1 内
loading_mask = ws_frame.locator(
"#tab-1 .fixed-table-loading", has_text="正在努力地加载数据中"
).first
if loading_mask.is_visible():
print(" ⏳ 检测到数据加载遮罩,等待加载完成...")
loading_mask.wait_for(state="hidden", timeout=30000)
page.wait_for_timeout(500)
sum_panel = ws_frame.locator("#sum").first
has_data = False
if sum_panel.is_visible():
sum_text = sum_panel.inner_text()
match_tickets = re.search(r"进站实际票数:(\d+)", sum_text)
if match_tickets and int(match_tickets.group(1)) > 0:
has_data = True
print(f" ✅ 统计面板已加载,实际票数: [{match_tickets.group(1)}]")
if not has_data:
if ws_frame.locator(
"#tab-1 .no-records-found", has_text="没有找到匹配的记录"
).first.is_visible():
print(" ⚠️ 确认为空数据,正在关闭当前标签页...")
page.locator(".tags-view-item", has_text="进站交接单查询").locator(
".el-icon-close"
).click()
return
# 4. 深度等待表格第一行数据行渲染就绪
ws_frame.locator("#exampleTable1 tbody tr[data-index='0']").wait_for(
state="visible", timeout=10000
)
main_rows = ws_frame.locator("#exampleTable1 tbody tr[data-index]")
row_count = main_rows.count()
print(f">> 当前视窗共捕获到活跃交接单记录: {row_count}")
# 【去重】加载本站已落库交接单号force=True 或查询失败时 existing=空集(不去重)
if force:
existing = set()
print(">> [去重] 强制重下,跳过去重。")
else:
try:
from inbound_verify import store
existing = store.get_existing_handover_nos("韵达")
except Exception as _e:
existing = set()
print(f">> [去重] 加载已落库交接单号失败,本次不去重: {_e}")
# 5. 逐行双击并提交导出
for i in range(row_count):
print(f" ⏳ 正在处理第 {i+1}/{row_count} 个交接单模块...")
current_row = ws_frame.locator("#exampleTable1 tbody tr[data-index]").nth(i)
raw_no = current_row.locator("td").nth(1).inner_text().strip()
# 【去重】已落库的交接单号不再提交导出任务
if raw_no in existing:
print(f" ⏭️ 交接单号 {raw_no} 已落库,跳过提交导出。")
continue
# 跳过已绑定的交接单
bind_status = current_row.locator("td").nth(2).inner_text().strip()
print(f" -> 交接单号: {raw_no} [绑定状态: {bind_status}]")
if bind_status == "已绑定":
print(" ⏭️ 该交接单已绑定,跳过。")
continue
current_row.dblclick()
ws_frame.locator("#docSum").wait_for(state="visible", timeout=15000)
page.wait_for_timeout(500)
# 导出弹窗重试:外层重新打开面板(最多 3 次)。
# 韵达站点已将导出面板从 jQuery(.allRight/#submitbutton) 改版为 Element UI
# 与实到流程同一导出组件iframe=myFrame
# 全选(button“全选”) → 向右转移(i.el-icon-d-arrow-right) → 导出(i.el-icon-download)
# → 正在导出中(.el-loading-mask) → 成功提示(.el-message-box 导出任务建立成功) → 确定
task_success = False
for major_attempt in range(3):
print(f" >> 正在打开数据导出面板 (尝试 {major_attempt + 1}/3)...")
ws_frame.locator('a.btn-info[onclick*="exportFile"]').click()
ws_frame.locator(".layui-layer-title", has_text="数据导出").wait_for(
state="visible", timeout=15000
)
export_frame = _resolve_export_frame(ws_frame)
try:
# 校验 Element UI 字段选择区是否加载完成(以「全选」按钮就绪为标志)
export_frame.locator(
"button.el-button", has_text="全选"
).first.wait_for(state="visible", timeout=8000)
except Exception:
print(" ⚠️ 导出面板字段区未加载,关闭面板后重试...")
ws_frame.locator(".layui-layer-close1").click()
page.wait_for_timeout(1000)
continue
print(" -> 全选字段并向右转移...")
export_frame.locator("button.el-button", has_text="全选").first.click()
page.wait_for_timeout(400)
export_frame.locator(
"button.el-button:has(i.el-icon-d-arrow-right)"
).first.click()
page.wait_for_timeout(500)
print(" -> 正在提交导出任务...")
export_frame.locator(
"button.el-button:has(i.el-icon-download)"
).first.click()
# 等待「正在导出中」遮罩出现并消失
loading = export_frame.locator(".el-loading-mask").first
try:
loading.wait_for(state="visible", timeout=5000)
loading.wait_for(state="hidden", timeout=60000)
except Exception:
pass
# 等待结果提示并判定
inner_success = False
try:
msg_box = export_frame.locator(".el-message-box.my-alert").first
msg_box.wait_for(state="visible", timeout=30000)
if msg_box.get_by_text("导出任务建立成功").is_visible():
print(" ✅ 导出任务已建立成功。")
inner_success = True
else:
print(" ⚠️ 导出结果提示非成功状态,将重试。")
msg_box.locator("button.el-button", has_text="确定").first.click()
page.wait_for_timeout(500)
except Exception:
print(" ⚠️ 未检测到导出结果提示,将重试。")
ws_frame.locator(".layui-layer-close1").click()
page.wait_for_timeout(500)
if inner_success:
task_success = True
break
if not task_success:
raise RuntimeError("多次重试后仍未能建立应到数据离线任务。")
ws_frame.locator("#myTab a", has_text="交接单信息").click()
page.wait_for_timeout(800)
export_times.append(datetime.now())
print(">> 任务提交完成,正在关闭【进站交接单查询】标签页...")
page.locator(".tags-view-item", has_text="进站交接单查询").locator(
".el-icon-close"
).click()
page.wait_for_timeout(500)
# 若所有记录都被跳过export_times 为空,直接结束
if not export_times:
print(">> ⚠️ 本次未产生任何离线下载任务(无数据或已全部跳过),结束。")
return
_yunda_poll_and_download_tasks(
page,
export_times,
download_dir,
"韵达-应到货物数据.xlsx",
)
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False
def yunda_actual_download(page, force=False, date=None):
"""韵达:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"韵达",
"实到",
lambda: yunda_actual_download_impl(page, date=date),
lambda: yunda_reset(page),
page=page,
)
def yunda_actual_download_impl(page, date=None):
"""韵达:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【韵达 - 实到货物数据下载】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次的最终文件,避免本次无数据/失败时残留旧数据误导比对
_remove_if_exists(os.path.join(download_dir, "韵达-实到货物数据.xlsx"))
export_times = []
try:
page.locator(".el-menu-item", has_text="首页").wait_for(
state="visible", timeout=15000
)
print("✅ 韵达工作台首页已加载")
yunda_smart_menu_click(page, ["报表管理", "扫描记录查询"])
print(">> 正在定位【扫描记录查询】iframe...")
ws_frame = page.frame_locator("section iframe")
ws_frame.locator(
".no-records-found", has_text="没有找到匹配的记录"
).first.wait_for(state="visible", timeout=15000)
print("✅ 扫描记录查询页面已初始化")
offset = state_store.get_offset("韵达", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
# 旧版 laydate 的日期格子 td[y][m][d] 用非补零整数值;单日范围起止同日
target_ymd = f"{target.year}-{target.month}-{target.day}"
src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(
f">> 设置实到查询日期: [{target.year}-{target.month}-{target.day}]{src}"
)
print(" >> 正在设定起始时间...")
ws_frame.locator("#startDate").click()
page.wait_for_timeout(400)
ws_frame.locator("#laydate_box:visible").first.wait_for(
state="visible", timeout=5000
)
_yunda_pick_laydate_old(ws_frame, page, target_ymd)
page.wait_for_timeout(400)
print(" >> 正在设定截止时间...")
ws_frame.locator("#endDate").click()
page.wait_for_timeout(400)
ws_frame.locator("#laydate_box:visible").first.wait_for(
state="visible", timeout=5000
)
_yunda_pick_laydate_old(ws_frame, page, target_ymd)
page.wait_for_timeout(500)
print(" >> 正在变更扫描类型为【到件】...")
ws_frame.locator("#scanRecordTyp").select_option(value="03")
page.wait_for_timeout(500)
print(">> 正在执行查询...")
ws_frame.locator('input[type="button"][value="查询"]').click()
page.wait_for_timeout(800)
loading_mask = ws_frame.locator(
".fixed-table-loading", has_text="正在努力地加载数据中"
).first
if loading_mask.is_visible():
print(" ⏳ 检测到数据加载遮罩,等待加载完成...")
loading_mask.wait_for(state="hidden", timeout=30000)
page.wait_for_timeout(500)
pg_info = ws_frame.locator(".pagination-info").first
has_records = False
if pg_info.is_visible():
info_text = pg_info.inner_text()
match_total = re.search(r"总共\s*(\d+)\s*条记录", info_text)
if match_total and int(match_total.group(1)) > 0:
has_records = True
print(f" ✅ 实到数据已加载,总记录数: [{match_total.group(1)}] 条。")
if not has_records:
if ws_frame.locator(
".no-records-found", has_text="没有找到匹配的记录"
).first.is_visible():
print(" ⚠️ 当前查询范围内为空数据,终止并关闭标签页。")
page.locator(".tags-view-item", has_text="扫描记录查询").locator(
".el-icon-close"
).click()
return
else:
print(" ⚠️ 未找到数据,也未出现空数据提示,结束。")
page.locator(".tags-view-item", has_text="扫描记录查询").locator(
".el-icon-close"
).click()
return
# 导出弹窗重试:外层重新打开面板(最多 3 次)。
# 韵达站点已将导出面板从 jQuery(.allRight/#submitbutton) 改版为 Element UI
# 全选(button“全选”) → 向右转移(i.el-icon-d-arrow-right) → 导出(i.el-icon-download)
# → 正在导出中(.el-loading-mask) → 成功提示(.el-message-box 导出任务建立成功) → 确定
print(">> 正在发起导出...")
task_success = False
for major_attempt in range(3):
print(f" >> 正在打开数据导出面板 (尝试 {major_attempt + 1}/3)...")
ws_frame.locator('input[type="button"][id="export"]').click()
ws_frame.locator(".layui-layer-title", has_text="数据导出").wait_for(
state="visible", timeout=15000
)
export_frame = ws_frame.frame_locator('iframe[name="myFrame"]')
try:
# 校验 Element UI 字段选择区是否加载完成(以「全选」按钮就绪为标志)
export_frame.locator(
"button.el-button", has_text="全选"
).first.wait_for(state="visible", timeout=8000)
except Exception:
print(" ⚠️ 导出面板字段区未加载,关闭面板后重试...")
ws_frame.locator(".layui-layer-close1").click()
page.wait_for_timeout(1000)
continue
print(" -> 全选字段并向右转移...")
export_frame.locator("button.el-button", has_text="全选").first.click()
page.wait_for_timeout(400)
export_frame.locator(
"button.el-button:has(i.el-icon-d-arrow-right)"
).first.click()
page.wait_for_timeout(500)
print(" -> 正在提交导出任务...")
export_frame.locator(
"button.el-button:has(i.el-icon-download)"
).first.click()
# 等待「正在导出中」遮罩出现并消失
loading = export_frame.locator(".el-loading-mask").first
try:
loading.wait_for(state="visible", timeout=5000)
loading.wait_for(state="hidden", timeout=60000)
except Exception:
pass
# 等待结果提示并判定
inner_success = False
try:
msg_box = export_frame.locator(".el-message-box.my-alert").first
msg_box.wait_for(state="visible", timeout=30000)
if msg_box.get_by_text("导出任务建立成功").is_visible():
print(" ✅ 导出任务已建立成功。")
inner_success = True
else:
print(" ⚠️ 导出结果提示非成功状态,将重试。")
msg_box.locator("button.el-button", has_text="确定").first.click()
page.wait_for_timeout(500)
except Exception:
print(" ⚠️ 未检测到导出结果提示,将重试。")
ws_frame.locator(".layui-layer-close1").click()
page.wait_for_timeout(500)
if inner_success:
task_success = True
break
if not task_success:
raise RuntimeError("多次重试后仍未能建立实到数据离线任务。")
export_times.append(datetime.now())
print(">> 任务提交完成,正在关闭【扫描记录查询】标签页...")
page.locator(".tags-view-item", has_text="扫描记录查询").locator(
".el-icon-close"
).click()
page.wait_for_timeout(500)
# 若未产生导出任务则结束
if not export_times:
print(">> ⚠️ 本次未产生离线下载任务,结束。")
return
# 6. 轮询并下载
_yunda_poll_and_download_tasks(
page,
export_times,
download_dir,
"韵达-实到货物数据.xlsx",
)
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False
def _yunda_poll_and_download_tasks(page, export_times, download_dir, final_filename):
"""韵达离线任务的轮询与下载"""
print("\n>> 正在前往【导出服务】界面...")
yunda_smart_menu_click(page, ["基础数据", "导出服务"])
export_ws_frame = page.frame_locator("section iframe")
export_ws_frame.get_by_role("cell", name="模块名称", exact=True).wait_for(
state="visible", timeout=15000
)
page.wait_for_timeout(1000)
print(">> 开始轮询离线任务队列,直到全部完成...")
total_expected = len(export_times)
poll_deadline = (
time.monotonic() + 300
) # 5 分钟上限:任务卡死/匹配不上时超时失败,交由上层重置重试
while True:
if time.monotonic() > poll_deadline:
raise RuntimeError(
"轮询导出任务超时5 分钟未全部完成/匹配),疑似任务卡死"
)
task_rows = export_ws_frame.locator(
".datagrid-view2 .datagrid-btable tbody tr.datagrid-row"
)
row_count = task_rows.count()
ready_indices = []
processing_indices = []
# 单账号无并发:仅按创建时间容差(40s)认领本批任务,不再校验模块名称
# (站点可能调整标题名,写死标题会导致匹配失败、任务一直查不到)。
for idx in range(row_count):
row = task_rows.nth(idx)
status_name = row.locator("td[field='fileStatus']").inner_text().strip()
create_time_str = (
row.locator("td[field='createdTime']").inner_text().strip()
)
try:
row_time = datetime.strptime(create_time_str, "%Y-%m-%d %H:%M:%S")
matched = any(
abs((row_time - et).total_seconds()) <= 40 for et in export_times
)
if matched:
if status_name == "导出完成":
ready_indices.append(idx)
else:
processing_indices.append(idx)
except Exception:
pass
total_found = len(ready_indices) + len(processing_indices)
print(
f" 📊 状态统计:期望 [{total_expected}],已入表 [{total_found}] (完成 [{len(ready_indices)}],生成中 [{len(processing_indices)}])"
)
if total_found < total_expected or len(processing_indices) > 0:
print(" ⏳ 队列未齐全,点击查询刷新...")
export_ws_frame.locator(
"#ydkyimport_basic_export_searchData1_ky_export_common"
).click()
page.wait_for_timeout(3000)
else:
print(">> 所有离线任务已就绪,开始依次下载...")
break
downloaded_files = []
for row_idx in ready_indices:
try:
target_row = export_ws_frame.locator(
".datagrid-view2 .datagrid-btable tbody tr.datagrid-row"
).nth(row_idx)
time_flag = (
target_row.locator("td[field='createdTime']").inner_text().strip()
)
print(f" 开始下载任务 [{time_flag}] ...")
with page.expect_download() as download_info:
target_row.locator("td[field='extreFile'] a").get_by_text(
"下载"
).first.click()
download = download_info.value
safe_timestamp = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
custom_filename = f"韵达_temp_{safe_timestamp}.xlsx"
save_path = os.path.join(download_dir, custom_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" 已下载: downloads/{custom_filename}")
page.wait_for_timeout(500)
except Exception as e:
print(f" ❌ 下载失败: {e}")
# 汇总校验:下载成功数必须等于本批提交的任务数,否则判失败
if len(downloaded_files) < total_expected:
raise RuntimeError(
f"仅成功下载 {len(downloaded_files)}/{total_expected} 个任务,数据不完整"
)
print(">> 【导出服务】下载完成,正在关闭标签页...")
try:
page.locator(".tags-view-item", has_text="导出服务").locator(
".el-icon-close"
).click()
print(" ✅ 【导出服务】标签页已关闭。")
except Exception:
pass
if downloaded_files:
print("\n>> 正在合并下载的数据...")
all_dfs = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path, dtype=str)
if not df.empty:
all_dfs.append(df)
except Exception:
pass
if all_dfs:
combined_df = pd.concat(all_dfs, ignore_index=True)
final_output = os.path.join(download_dir, final_filename)
combined_df.to_excel(final_output, index=False)
print(f"====================================================")
print(f" 合并完成。")
print(f" 📁 输出路径: {final_output}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print(" 临时文件已清理。")

760
inbound_verify/sites/zto.py Normal file
View File

@@ -0,0 +1,760 @@
# sites/zto.py
import os
import re
import time
import yaml
from datetime import datetime, timedelta
import pandas as pd
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
"""
for attempt in range(1, max_attempts + 1):
try:
ret = flow()
if ret is False:
raise RuntimeError("流程返回失败状态")
if attempt > 1:
print(f">> 【{site_name}-{label}】第 {attempt} 次尝试成功 ✅")
return True
except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try:
reset()
except Exception as re:
print(f" ⚠️ 重置异常: {re}")
if attempt < max_attempts:
continue
print(
f"❌ 【{site_name}-{label}】已达最大尝试次数 {max_attempts},放弃(环境已清理)。"
)
return False
# 站点首页 URL异常兜底重置用也供 runtime 的 SITES_CONFIG 引用(单一来源)
HOME_URL = "https://ws.zto56.com/"
def zto_reset(page):
"""异常兜底:重置中通到初始态(跳首页 URL丢弃当前页面状态登录态保留"""
page.goto(HOME_URL)
page.wait_for_timeout(1500)
def _remove_if_exists(path):
"""删除文件(若存在):流程开头清理上次的最终文件,避免无数据/失败时残留旧数据。"""
try:
if os.path.exists(path):
os.remove(path)
except Exception:
pass
def _wait_and_get_frame(page, text_indicator, timeout_ms=20000):
"""在主页面与所有 iframe 中查找包含指定文本的窗口"""
start_time = datetime.now()
while (datetime.now() - start_time).total_seconds() * 1000 < timeout_ms:
try:
if page.get_by_text(text_indicator).count() > 0:
return page
except Exception:
pass
for frame in page.frames:
try:
if frame.get_by_text(text_indicator).count() > 0:
return frame
except Exception:
pass
page.wait_for_timeout(300)
raise TimeoutError(f"超时:未找到包含 [{text_indicator}] 的窗口。")
def _dom_click(locator):
"""直接在元素上派发 mousedown+mouseup+click 事件,不走 Playwright 的坐标命中测试,
避免被遮挡元素(如日期控件的 hover 提示气泡)拦截。事件直接发到目标元素并冒泡,
兼容绑定 mousedown 或 click 的控件。"""
locator.evaluate(
"el => { const o = { bubbles: true, cancelable: true, view: window, button: 0 };"
" el.dispatchEvent(new MouseEvent('mousedown', o));"
" el.dispatchEvent(new MouseEvent('mouseup', o));"
" el.dispatchEvent(new MouseEvent('click', o)); }"
)
def _zto_compute_target_time(offset):
"""直接从 Python datetime 计算目标日期的毫秒级时间戳(本地时区零点)。
不再从 DOM 的 real-today 元素读取 time 属性,避免双月视图下 real-today
同时出现在 month1隐藏 ghost cell和 month2可见导致 .first 取到隐藏元素。
"""
target_date = datetime.now().date() - timedelta(days=offset)
target_dt = datetime(target_date.year, target_date.month, target_date.day)
return int(target_dt.timestamp() * 1000)
def _zto_find_visible_day(frame_locator, target_time):
"""在双月日期控件中查找可见的日期格子。
jQuery Date Range Picker 双月视图下,同一天可能出现在两个面板中:
- month1左面板的溢出 ghost celldisplay:none不可见
- month2右面板的正常 cell可见
同一日期在 DOM 中可能有毫秒级差异(零点 vs 23:59:59遍历匹配并返回
第一个 visible 的;无可见匹配返回 None。
"""
# 尝试两个时间变体:零点 和 23:59:59部分 checked/selected 格用后者)
for time_variant in (target_time, target_time + 86399000):
sel = f"td div.day[time='{time_variant}']"
cells = frame_locator.locator(sel)
count = cells.count()
for i in range(count):
if cells.nth(i).is_visible():
return cells.nth(i)
return None
def _zto_flip_to_target_month(frame_locator, page, target_time, max_flips=12):
"""中通日历(jQuery-Date-Range-Picker 双月视图)跨月导航:目标日期不在当前视窗时,
循环点 .prev 把目标月翻进视窗。用 _zto_find_visible_day 判可见(跳过隐藏 ghost cell
返回 True 若目标格子最终可见。"""
for _ in range(max_flips):
if _zto_find_visible_day(frame_locator, target_time) is not None:
return True
frame_locator.locator(".date-picker-wrapper .prev").first.evaluate(
"el => el.click()"
)
page.wait_for_timeout(450)
return _zto_find_visible_day(frame_locator, target_time) is not None
def zto_smart_menu_click(page, menu_path):
"""中通菜单导航"""
print(f">> 正在导航: {' -> '.join(menu_path)}")
for i in range(len(menu_path)):
current_menu = menu_path[i]
if i < len(menu_path) - 1:
next_menu = menu_path[i + 1]
next_locator = page.locator("span.menu-name", has_text=next_menu).first
if not next_locator.is_visible():
page.locator("span.menu-name", has_text=current_menu).first.click()
page.wait_for_timeout(800)
else:
page.locator("span.menu-name", has_text=current_menu).first.click()
page.wait_for_timeout(1000)
def zto_expected_download(page, force=False, date=None):
"""中通:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"应到",
lambda: zto_expected_download_impl(page, force=force, date=date),
lambda: zto_reset(page),
page=page,
)
def zto_expected_download_impl(page, force=False, date=None):
"""中通:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【中通 - 应到货物数据下载】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次的最终文件,避免本次无数据/失败时残留旧数据误导比对
_remove_if_exists(os.path.join(download_dir, "中通-应到货物数据.xlsx"))
export_times = []
try:
# 1. 菜单导航
zto_smart_menu_click(page, ["运营管理", "进站管理", "进站交接单查询"])
ewb_frame = page.frame_locator('iframe[src*="inEwbsListNoSearch"]')
print(">> 正在检测表格统计面板 (#inEwbCount)...")
ewb_frame.locator("#inEwbCount").wait_for(state="attached", timeout=15000)
# 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通")
if date:
# 指定日期:折算成相对今天的有效偏移,复用下方 target_time 计算与跨月翻月
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
ewb_frame.locator("#beginDate").click()
page.wait_for_timeout(500)
# 直接从 Python datetime 计算目标时间戳,不再依赖 DOM real-today双月视图
# 下 real-today 可能同时出现在 month1 隐藏 ghost cell 和 month2 可见 cell
# .first 会取到隐藏的那个导致 wait_for(visible) 超时)。
target_time = _zto_compute_target_time(offset)
target_cell = _zto_find_visible_day(ewb_frame, target_time)
if target_cell is None:
print(" 目标日期不在当前视窗,正在翻月导航 ...")
if not _zto_flip_to_target_month(ewb_frame, page, target_time):
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
target_cell = _zto_find_visible_day(ewb_frame, target_time)
if target_cell is None:
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
# 日期格子用 _dom_click 直接派发事件:.click() 会先 hover 格子,触发
# "范围长度"提示气泡(.date-range-length-tip)盖住格子导致点击被遮挡超时。
_dom_click(target_cell)
page.wait_for_timeout(300)
_dom_click(target_cell)
page.wait_for_timeout(500)
# 3. 触发查询并判断数据状态
print(">> 正在点击【查询】按钮并等待数据响应...")
old_count = ewb_frame.locator("#inEwbCount").inner_text().strip()
ewb_frame.locator("#searchbtn").click()
page.wait_for_timeout(1000)
loading_mask = ewb_frame.locator(".mini-mask-loading", has_text="加载中")
if loading_mask.is_visible():
print(" ⏳ 检测到数据加载遮罩层,等待系统渲染...")
loading_mask.wait_for(state="hidden", timeout=30000)
page.wait_for_timeout(1000)
# 轮询判定统计面板更新
wait_cycles = 0
while wait_cycles < 20:
current_count = ewb_frame.locator("#inEwbCount").inner_text().strip()
if current_count != old_count:
break
page.wait_for_timeout(500)
wait_cycles += 1
ticket_count_str = ewb_frame.locator("#inEwbCount").inner_text().strip()
print(f" ✅ 数据已加载,进站实际票数: [{ticket_count_str}]")
# 数据分流
if (
ticket_count_str == "0"
or not ticket_count_str.isdigit()
or int(ticket_count_str) == 0
):
print(" >> 票数为 0正在确认是否为空数据...")
empty_flag = ewb_frame.locator("#datagrid1").get_by_text(
"没有搜索到符合条件的数据记录"
)
if empty_flag.is_visible():
print(" ⚠️ 确认为空数据,正在关闭当前标签页...")
# 即使无数据也关闭已打开的标签页
try:
page.locator(".mini-tab", has_text="进站交接单查询").locator(
".mini-tab-close"
).click()
except Exception:
pass
return
else:
print(" ⚠️ 票数为 0 但未出现空记录提示,页面状态异常,判失败。")
try:
page.locator(".mini-tab", has_text="进站交接单查询").locator(
".mini-tab-close"
).click()
except Exception:
pass
raise RuntimeError("票数为 0 但未出现空记录提示,页面状态异常")
else:
print(" >> 票数校验通过,等待主表格渲染数据行...")
ewb_frame.locator(
"#datagrid1 .mini-grid-rows-view .mini-grid-row"
).first.wait_for(state="visible", timeout=15000)
# 4. 提取主表记录并循环双击
main_rows = ewb_frame.locator("#datagrid1 .mini-grid-rows-view .mini-grid-row")
count = main_rows.count()
print(f">> 共发现 {count} 个交接单需要导出。")
# 【去重】加载本站已落库交接单号force=True 或查询失败时 existing=空集(不去重)
if force:
existing = set()
print(">> [去重] 强制重下,跳过去重。")
else:
try:
from inbound_verify import store
existing = store.get_existing_handover_nos("中通")
except Exception as _e:
existing = set()
print(f">> [去重] 加载已落库交接单号失败,本次不去重: {_e}")
for i in range(count):
print(f" ⏳ 正在处理第 {i+1}/{count} 个交接单...")
row = ewb_frame.locator(
"#datagrid1 .mini-grid-rows-view .mini-grid-row"
).nth(i)
raw_text = row.locator("td").nth(3).inner_text()
match = re.search(r"\d{18}", raw_text)
handover_no = match.group(0) if match else raw_text.strip()
print(f" -> 当前交接单号:{handover_no}")
# 【去重】已落库的交接单号不再提交导出任务(不双击、不 append export_times
if handover_no in existing:
print(f" ⏭️ 交接单号 {handover_no} 已落库,跳过提交导出。")
continue
row.dblclick()
ewb_frame.locator("#datagrid2").get_by_text("运单号").wait_for(
state="visible"
)
ewb_frame.locator(
"#datagrid2 .mini-grid-row", has_text=handover_no
).first.wait_for(state="visible")
# 5. 执行导出流程
ewb_frame.locator("#exportExcel").click()
page.locator(".mini-panel-title", has_text="导出选择列").wait_for(
state="visible"
)
export_frame = page.frame_locator('iframe[src*="download"]')
export_frame.locator(".mini-button-text", has_text=">>").click()
page.wait_for_timeout(300)
export_frame.locator(".mini-button-text", has_text="确定").click()
print(" >> 正在查找【温馨提示】弹窗...")
ctx_alert = _wait_and_get_frame(page, "温馨提示")
ctx_alert.locator(
".mini-messagebox-buttons .mini-button-text", has_text="确定"
).click()
print(" ✅ 已确认【温馨提示】弹窗。")
print(" >> 正在等待服务器建立后台离线任务...")
# 提示「生成离线导出任务成功」出现在导出列 iframe(/comm/download) 中;
# 提交完成后该 iframe 会被站点销毁,此时 wait_for 会抛 "Frame was detached"
# ——这恰恰说明提示已随 iframe 消失、任务已建立,属正常,不视为失败。
ctx_tips = _wait_and_get_frame(
page, "生成离线导出任务成功", timeout_ms=10000
)
try:
ctx_tips.locator(".mini-tips-success").wait_for(
state="hidden", timeout=15000
)
except Exception as e:
if "detached" in str(e).lower():
print(" 提示框所在 iframe 已随提交关闭,任务已建立。")
else:
raise
print(" ✅ 成功提示框已消失。")
export_times.append(datetime.now())
print(" >> 切换回【交接单信息】标签页...")
ewb_frame.locator("#ewbsListNo").click()
page.wait_for_timeout(1000)
# ====================================================================
# 完成所有交接单导出提交后,关闭当前标签页
# ====================================================================
print(">> 【进站交接单查询】已完成,正在关闭标签页...")
try:
page.locator(".mini-tab", has_text="进站交接单查询").locator(
".mini-tab-close"
).click()
page.wait_for_timeout(500)
print(" ✅ 【进站交接单查询】标签页已关闭。")
except Exception as e:
print(f" ⚠️ 关闭【进站交接单查询】标签页时出错: {e}")
# 【去重兜底】全部已落库/无数据 → 无导出任务,标签页已关,直接结束不进轮询
if not export_times:
print(">> 本次无新交接单需导出(全部已落库或无数据),结束。")
return True
# 交由统一的轮询下载流程处理
_zto_poll_and_download_tasks(
page,
export_times,
download_dir,
"中通-应到货物数据.xlsx",
)
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False
def zto_actual_download(page, force=False, date=None):
"""中通:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"实到",
lambda: zto_actual_download_impl(page, date=date),
lambda: zto_reset(page),
page=page,
)
def zto_actual_download_impl(page, date=None):
"""中通:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【中通 - 实到货物数据下载】任务...")
download_dir = DOWNLOAD_DIR
if not os.path.exists(download_dir):
os.makedirs(download_dir)
# 清理上次的最终文件,避免本次无数据/失败时残留旧数据误导比对
_remove_if_exists(os.path.join(download_dir, "中通-实到货物数据.xlsx"))
export_times = []
try:
# 1. 菜单导航
zto_smart_menu_click(page, ["运营管理", "扫描操作与监控", "到件扫描监控"])
arr_frame = page.frame_locator('iframe[src*="ArriveScan"]')
print(">> 正在检测主页面 (#daterange)...")
arr_frame.locator("#daterange").wait_for(state="attached", timeout=15000)
# 2. 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通", "actual")
if date:
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
arr_frame.locator("#daterange").click()
page.wait_for_timeout(500)
# 直接从 Python datetime 计算目标时间戳,不再依赖 DOM real-today双月视图
# 下 real-today 可能同时出现在 month1 隐藏 ghost cell 和 month2 可见 cell
# .first 会取到隐藏的那个导致 wait_for(visible) 超时)。
target_time = _zto_compute_target_time(offset)
target_cell = _zto_find_visible_day(arr_frame, target_time)
if target_cell is None:
print(" 目标日期不在当前视窗,正在翻月导航 ...")
if not _zto_flip_to_target_month(arr_frame, page, target_time):
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
target_cell = _zto_find_visible_day(arr_frame, target_time)
if target_cell is None:
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
# 日期格子用 _dom_click 直接派发事件Playwright 的 .click() 会先 hover 格子,
# 触发"范围长度"提示气泡(.date-range-length-tip)盖住格子,导致点击被判遮挡而超时。
_dom_click(target_cell)
page.wait_for_timeout(300)
_dom_click(target_cell)
page.wait_for_timeout(500)
# 3. 设定单号类型
print(">> 正在设定单号类型为【子单】...")
arr_frame.locator('[id="bandEwbType$text"]').click()
page.wait_for_timeout(500)
arr_frame.locator(".mini-tree-nodeshow").filter(
has_text=re.compile(r"^子单$")
).locator(".mini-tree-checkbox").click()
page.wait_for_timeout(300)
# 4. 触发查询并判断数据状态
print(">> 正在点击【查询】按钮并等待数据响应...")
arr_frame.locator("#searchbtn").click()
page.wait_for_timeout(1000)
loading_mask = arr_frame.locator(".mini-mask-loading", has_text="加载中")
if loading_mask.is_visible():
print(" ⏳ 检测到数据加载遮罩层,等待系统渲染...")
loading_mask.wait_for(state="hidden", timeout=30000)
page.wait_for_timeout(1000)
empty_flag = arr_frame.locator("#datagrid1").get_by_text(
"没有搜索到符合条件的数据记录"
)
if empty_flag.is_visible():
print(" ⚠️ 当前查询范围内没有数据,终止并关闭标签页。")
try:
page.locator(".mini-tab", has_text="到件扫描监控").locator(
".mini-tab-close"
).click()
except Exception:
pass
return
arr_frame.locator("#page1").wait_for(state="visible", timeout=15000)
print(" ✅ 数据已加载。(底部分页控件已就绪)")
# 5. 执行导出流程
arr_frame.locator("#exportExcel").click()
page.locator(".mini-panel-title", has_text="导出选择列").wait_for(
state="visible"
)
export_frame = page.frame_locator('iframe[src*="download"]')
export_frame.locator(".mini-button-text", has_text=">>").click()
page.wait_for_timeout(300)
export_frame.locator(".mini-button-text", has_text="确定").click()
print(" >> 正在查找【温馨提示】弹窗...")
ctx_alert = _wait_and_get_frame(page, "温馨提示")
ctx_alert.locator(
".mini-messagebox-buttons .mini-button-text", has_text="确定"
).click()
print(" ✅ 已确认【温馨提示】弹窗。")
print(" >> 正在等待服务器建立后台离线任务...")
# 提示「生成离线导出任务成功」出现在导出列 iframe(/comm/download) 中;
# 提交完成后该 iframe 会被站点销毁,此时 wait_for 会抛 "Frame was detached"
# ——这恰恰说明提示已随 iframe 消失、任务已建立,属正常,不视为失败。
ctx_tips = _wait_and_get_frame(page, "生成离线导出任务成功", timeout_ms=10000)
try:
ctx_tips.locator(".mini-tips-success").wait_for(
state="hidden", timeout=15000
)
except Exception as e:
if "detached" in str(e).lower():
print(" 提示框所在 iframe 已随提交关闭,任务已建立。")
else:
raise
print(" ✅ 成功提示框已消失。")
export_times.append(datetime.now())
# ====================================================================
# 完成实到数据导出提交后,关闭当前标签页
# ====================================================================
print(">> 【到件扫描监控】已完成,正在关闭标签页...")
try:
page.locator(".mini-tab", has_text="到件扫描监控").locator(
".mini-tab-close"
).click()
page.wait_for_timeout(500)
print(" ✅ 【到件扫描监控】标签页已关闭。")
except Exception as e:
print(f" ⚠️ 关闭【到件扫描监控】标签页时出错: {e}")
# 交由统一的轮询下载流程处理
_zto_poll_and_download_tasks(
page,
export_times,
download_dir,
"中通-实到货物数据.xlsx",
)
return True
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
return False
def _zto_poll_and_download_tasks(page, export_times, download_dir, final_filename):
"""中通离线任务的轮询与下载"""
print("\n>> 正在前往【导出任务管理】界面...")
zto_smart_menu_click(page, ["系统配置", "导出任务管理"])
taskdone_frame = page.frame_locator('iframe[src*="taskdone"]')
taskdone_frame.locator("#taskdoneDatagrid").get_by_text("任务标题").wait_for(
state="visible"
)
page.wait_for_timeout(1000)
print(">> 列表已加载,开始匹配并检查任务状态...")
total_expected = len(export_times)
poll_deadline = (
time.monotonic() + 300
) # 5 分钟上限:任务卡死/匹配不上时超时失败,交由上层重置重试
last_total_found = -1
stall_rounds = 0 # 连续无进展轮数:刷新后 total_found 不增长则累计,超阈值快速失败
while True:
if time.monotonic() > poll_deadline:
raise RuntimeError(
"轮询导出任务超时5 分钟未全部完成/匹配),疑似任务卡死"
)
task_rows = taskdone_frame.locator(
"#taskdoneDatagrid .mini-grid-rows-view .mini-grid-row"
)
row_count = task_rows.count()
ready_timestamps = set()
processing_timestamps = set()
# 单账号无并发:仅按提交时间容差(40s)认领本批任务,不再校验任务标题
# (站点可能调整标题名,写死标题会导致匹配失败、任务一直查不到)。
for i in range(row_count):
tds = task_rows.nth(i).locator("td")
if tds.count() < 10:
continue
submit_time_str = tds.nth(4).inner_text().strip()
status_str = tds.nth(6).inner_text().strip()
try:
row_time = datetime.strptime(submit_time_str, "%Y-%m-%d %H:%M:%S")
matched = any(
abs((row_time - et).total_seconds()) <= 40 for et in export_times
)
if matched:
if status_str == "成功执行":
ready_timestamps.add(submit_time_str)
else:
processing_timestamps.add(submit_time_str)
except Exception:
pass
total_found = len(ready_timestamps) + len(processing_timestamps)
print(
f" 📊 状态统计:期望 [{total_expected}],已入表 [{total_found}] (就绪 [{len(ready_timestamps)}],处理中 [{len(processing_timestamps)}])"
)
if total_found < total_expected or len(processing_timestamps) > 0:
# 连续无进展即快速失败:避免页面被遮挡/任务卡住时干等到 5 分钟超时
if total_found == last_total_found:
stall_rounds += 1
else:
stall_rounds = 0
last_total_found = total_found
if stall_rounds >= 4:
raise RuntimeError(
f"连续 {stall_rounds} 轮刷新无进展(仍 {total_found}/{total_expected}"
"疑似页面被遮挡或任务异常,触发重试"
)
print(" ⏳ 任务尚未齐全或仍在生成,点击查询刷新...")
# 查询按钮加短超时;失败则菜单刷新,两者都失败直接报错触发重试
try:
taskdone_frame.locator(
".mini-button-text", has_text="查询"
).first.click(timeout=8000)
except Exception as e:
print(f" ⚠️ 查询按钮不可用({e}),改用菜单刷新...")
try:
page.locator(
"li.leaf span.menu-name", has_text="导出任务管理"
).click(timeout=8000)
except Exception as e2:
raise RuntimeError(f"查询与菜单刷新均失败,疑似页面被遮挡: {e2}")
page.wait_for_timeout(3000)
else:
target_task_timestamps = list(ready_timestamps)
print(">> 所有目标任务已生成,开始下载...")
break
# 8. 下载逻辑
downloaded_files = []
for time_str in target_task_timestamps:
try:
target_row = (
taskdone_frame.locator(
"#taskdoneDatagrid .mini-grid-rows-view .mini-grid-row"
)
.filter(
has=taskdone_frame.locator(
f"td:nth-child(5):has-text('{time_str}')"
)
)
.first
)
print(f" 开始下载任务 [{time_str}] ...")
checkbox = target_row.locator(".mini-grid-checkbox")
if checkbox.is_visible():
checkbox.click()
print(" -> 已勾选当前记录的 Checkbox")
page.wait_for_timeout(500)
with page.expect_download() as download_info:
target_row.locator("td").nth(10).locator(".ui-btn-download").click()
download = download_info.value
save_path = os.path.join(download_dir, download.suggested_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" 已下载: downloads/{download.suggested_filename}")
if checkbox.is_visible():
checkbox.click()
print(" -> 已取消勾选,继续下一条")
page.wait_for_timeout(500)
except Exception as e:
print(f" ❌ 下载任务 [{time_str}] 失败: {e}")
# 汇总校验:下载成功数必须等于本批提交的任务数,否则判失败
# (防"部分/全失败却被判成功"
if len(downloaded_files) < total_expected:
raise RuntimeError(
f"仅成功下载 {len(downloaded_files)}/{total_expected} 个任务,数据不完整"
)
# ====================================================================
# 所有目标文件下载完成后,关闭"导出任务管理"标签页
# ====================================================================
print(">> 【导出任务管理】下载完成,正在关闭标签页...")
try:
page.locator(".mini-tab", has_text="导出任务管理").locator(
".mini-tab-close"
).click()
page.wait_for_timeout(500)
print(" ✅ 【导出任务管理】标签页已关闭。")
except Exception as e:
print(f" ⚠️ 关闭【导出任务管理】标签页时出错: {e}")
# 9. 合并数据
if downloaded_files:
print("\n>> 正在合并下载的数据...")
all_data_frames = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path, dtype=str)
if not df.empty:
all_data_frames.append(df)
except Exception:
pass
if all_data_frames:
combined_df = pd.concat(all_data_frames, ignore_index=True)
final_output_path = os.path.join(download_dir, final_filename)
combined_df.to_excel(final_output_path, index=False)
print(f"====================================================")
print(f" 合并完成。")
print(f" 📁 输出路径: {final_output_path}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print(" 临时文件已清理。")

View File

@@ -0,0 +1,695 @@
# state_store.py
# 阶段0站点状态持久化SQLite。记录各站登录态 + 应到/实到数据文件生成时间,
# 供后台心跳刷新、菜单状态盘展示,以及未来 FastAPI / Web 前端读取。
# 重启不丢——程序重启后状态从本库恢复(登录态会随心跳重新探测校正)。
#
# 设计:纯 Pythonsqlite3 标准库,无新依赖),每次读写开短连接,主线程使用。
import os
import sqlite3
from datetime import datetime, timedelta
from inbound_verify.paths import STATE_DB_PATH
# 登录态枚举
LOGIN_UNKNOWN = "unknown" # 尚未探测过
LOGIN_IN = "logged_in"
LOGIN_OUT = "logged_out"
# 任务状态枚举task_history.status
TASK_PENDING = "pending" # 已入队,待执行
TASK_RUNNING = "running" # 正在执行
TASK_SUCCESS = "success" # 成功(有数据)
TASK_NO_DATA = "no_data" # 成功但本站本次无数据
TASK_FAILED = "failed" # 失败(重试耗尽 / 未登录 / 异常)
def _now():
"""本地时间的字符串(到秒),用于时间戳列。"""
return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
def init_db():
"""建库建表(幂等)。确保 state 目录存在。"""
os.makedirs(os.path.dirname(STATE_DB_PATH), exist_ok=True)
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute("""
CREATE TABLE IF NOT EXISTS site_status (
site TEXT PRIMARY KEY,
login_state TEXT,
login_checked_at TEXT,
expected_ready INTEGER,
expected_generated_at TEXT,
actual_ready INTEGER,
actual_generated_at TEXT,
undelivered_ready INTEGER,
undelivered_generated_at TEXT,
expected_business_date TEXT,
actual_business_date TEXT,
undelivered_business_date TEXT,
updated_at TEXT
)
""")
# 旧库迁移:补 undelivered 两列(新库已含;重复添加抛 OperationalError忽略
for _col, _typedef in [
("undelivered_ready", "INTEGER NOT NULL DEFAULT 0"),
("undelivered_generated_at", "TEXT NOT NULL DEFAULT ''"),
("expected_business_date", "TEXT NOT NULL DEFAULT ''"),
("actual_business_date", "TEXT NOT NULL DEFAULT ''"),
("undelivered_business_date", "TEXT NOT NULL DEFAULT ''"),
]:
try:
conn.execute(f"ALTER TABLE site_status ADD COLUMN {_col} {_typedef}")
except sqlite3.OperationalError:
pass
conn.execute("""
CREATE TABLE IF NOT EXISTS task_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site TEXT,
kind TEXT,
status TEXT,
started_at TEXT,
finished_at TEXT,
error TEXT,
trigger TEXT NOT NULL DEFAULT '',
target_date TEXT NOT NULL DEFAULT '',
force INTEGER NOT NULL DEFAULT 0
)
""")
# 旧库迁移:补触发方式/目标日期/强制重下三列(新库已含;重复添加抛 OperationalError忽略
for _col, _typedef in [
("trigger", "TEXT NOT NULL DEFAULT ''"),
("target_date", "TEXT NOT NULL DEFAULT ''"),
("force", "INTEGER NOT NULL DEFAULT 0"),
]:
try:
conn.execute(f"ALTER TABLE task_history ADD COLUMN {_col} {_typedef}")
except sqlite3.OperationalError:
pass
conn.execute("""
CREATE TABLE IF NOT EXISTS site_config (
site TEXT PRIMARY KEY,
expected_offset INTEGER NOT NULL DEFAULT 0,
actual_offset INTEGER NOT NULL DEFAULT 0,
schedule_enabled INTEGER NOT NULL DEFAULT 0,
schedule_time TEXT NOT NULL DEFAULT '',
updated_at TEXT
)
""")
# 旧库迁移:补 schedule 两列 + expected/actual 偏移(新库已含;重复添加抛错忽略)
for _col, _typedef in [
("schedule_enabled", "INTEGER NOT NULL DEFAULT 0"),
("schedule_time", "TEXT NOT NULL DEFAULT ''"),
("expected_offset", "INTEGER NOT NULL DEFAULT 0"),
("actual_offset", "INTEGER NOT NULL DEFAULT 0"),
]:
try:
conn.execute(f"ALTER TABLE site_config ADD COLUMN {_col} {_typedef}")
except sqlite3.OperationalError:
pass
# 旧库若有 date_offset 列,把值搬到 expected/actual一次性新库无此列则跳过
try:
conn.execute(
"UPDATE site_config SET expected_offset=date_offset, actual_offset=date_offset "
"WHERE expected_offset=0 AND date_offset IS NOT NULL AND date_offset>0"
)
except sqlite3.OperationalError:
pass
conn.execute("""
CREATE TABLE IF NOT EXISTS site_settings (
site TEXT,
key TEXT,
value TEXT,
PRIMARY KEY (site, key)
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS ingest_state (
site TEXT,
kind TEXT,
ok INTEGER,
ingested_at TEXT,
count INTEGER,
error TEXT,
PRIMARY KEY (site, kind)
)
""")
# 周期性抓取调度(每 site×kind 一行):取代旧 site_config.schedule_* 的每日单时点。
# enabled=总开关active_start/end=激活时段"HH:MM"(空=不限时段,避免半夜空跑);
# interval_minutes=激活时段内的抓取间隔。百世 kind 固定为 undelivered无应到/实到二分)。
conn.execute("""
CREATE TABLE IF NOT EXISTS fetch_schedule (
site TEXT NOT NULL,
kind TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 0,
active_start TEXT NOT NULL DEFAULT '',
active_end TEXT NOT NULL DEFAULT '',
interval_minutes INTEGER NOT NULL DEFAULT 30,
updated_at TEXT,
PRIMARY KEY (site, kind)
)
""")
conn.commit()
def _upsert(conn, site, **fields):
"""更新(或插入)单站:保留未传字段,刷新 updated_at。"""
row = conn.execute(
"SELECT login_state, login_checked_at, expected_ready, expected_generated_at, "
"expected_business_date, actual_ready, actual_generated_at, actual_business_date, "
"undelivered_ready, undelivered_generated_at, undelivered_business_date "
"FROM site_status WHERE site = ?",
(site,),
).fetchone()
cur = {
"login_state": LOGIN_UNKNOWN,
"login_checked_at": "",
"expected_ready": 0,
"expected_generated_at": "",
"expected_business_date": "",
"actual_ready": 0,
"actual_generated_at": "",
"actual_business_date": "",
"undelivered_ready": 0,
"undelivered_generated_at": "",
"undelivered_business_date": "",
}
if row:
(
cur["login_state"],
cur["login_checked_at"],
cur["expected_ready"],
cur["expected_generated_at"],
cur["expected_business_date"],
cur["actual_ready"],
cur["actual_generated_at"],
cur["actual_business_date"],
cur["undelivered_ready"],
cur["undelivered_generated_at"],
cur["undelivered_business_date"],
) = row
cur.update(fields)
cur["updated_at"] = _now()
values = (
site,
cur["login_state"],
cur["login_checked_at"],
cur["expected_ready"],
cur["expected_generated_at"],
cur["expected_business_date"],
cur["actual_ready"],
cur["actual_generated_at"],
cur["actual_business_date"],
cur["undelivered_ready"],
cur["undelivered_generated_at"],
cur["undelivered_business_date"],
cur["updated_at"],
)
if row:
conn.execute(
"UPDATE site_status SET login_state=?, login_checked_at=?, expected_ready=?, "
"expected_generated_at=?, expected_business_date=?, actual_ready=?, "
"actual_generated_at=?, actual_business_date=?, undelivered_ready=?, "
"undelivered_generated_at=?, undelivered_business_date=?, updated_at=? "
"WHERE site=?",
values[1:] + (site,),
)
else:
conn.execute(
"INSERT INTO site_status (site, login_state, login_checked_at, expected_ready, "
"expected_generated_at, expected_business_date, actual_ready, actual_generated_at, "
"actual_business_date, undelivered_ready, undelivered_generated_at, "
"undelivered_business_date, updated_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
values,
)
conn.commit()
def set_login_state(site, logged_in):
"""更新单站登录态。logged_in: bool。"""
state = LOGIN_IN if logged_in else LOGIN_OUT
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, login_state=state, login_checked_at=_now())
def reset_login_states(sites):
"""启动时把给定站点的登录态重置为 unknown避免显示上一会话的陈旧登录态
登录态是会话级的;数据态(文件就绪)会话无关、保留不动,心跳就绪后会重新探测。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
for site in sites:
_upsert(conn, site, login_state=LOGIN_UNKNOWN, login_checked_at="")
def set_data_state(site, kind, ready, generated_at, business_date=None):
"""更新单站数据态。kind: 'expected'/'actual'/'undelivered'ready: bool
generated_at: str。business_date: str 或 None——None 时保留原值(心跳不覆盖下载快照)。"""
fields = {
f"{kind}_ready": 1 if ready else 0,
f"{kind}_generated_at": generated_at or "",
}
if business_date is not None:
fields[f"{kind}_business_date"] = business_date or ""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, **fields)
def set_business_date(site, kind, business_date):
"""仅写业务日期快照(不碰 ready/generated_at
下载成功钩子用ready 语义已移交「入库成功」(见 reset_data_ready / _persist_to_db
下载阶段只记业务日期,供前端状态盘显示「是哪天的数据」。
"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, **{f"{kind}_business_date": business_date or ""})
def set_ready(site, kind, ready):
"""仅写就绪态(不碰 business_date/generated_at
供心跳从 ingest_state 派生 ready 用——ready 现为 DB 入库真相的派生视图,
非启动重置、不读 Excel。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, **{f"{kind}_ready": 1 if ready else 0})
def get_all_status():
"""返回 {site: {各字段}};库不存在则返回 {}"""
if not os.path.exists(STATE_DB_PATH):
return {}
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT site, login_state, login_checked_at, expected_ready, "
"expected_generated_at, expected_business_date, actual_ready, "
"actual_generated_at, actual_business_date, undelivered_ready, "
"undelivered_generated_at, undelivered_business_date, updated_at "
"FROM site_status"
).fetchall()
return {
r[0]: {
"login_state": r[1],
"login_checked_at": r[2],
"expected_ready": bool(r[3]),
"expected_generated_at": r[4],
"expected_business_date": r[5],
"actual_ready": bool(r[6]),
"actual_generated_at": r[7],
"actual_business_date": r[8],
"undelivered_ready": bool(r[9]),
"undelivered_generated_at": r[10],
"undelivered_business_date": r[11],
"updated_at": r[12],
}
for r in rows
}
def set_ingest_state(site, kind, ok, count=0, error=None):
"""记录一次入库结果UPSERT。ok: boolcount: 入库条数error: 失败原因或 None。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute(
"INSERT INTO ingest_state (site, kind, ok, ingested_at, count, error) "
"VALUES (?, ?, ?, ?, ?, ?) "
"ON CONFLICT(site, kind) DO UPDATE SET "
"ok=excluded.ok, ingested_at=excluded.ingested_at, "
"count=excluded.count, error=excluded.error",
(site, kind, 1 if ok else 0, _now(), int(count or 0), error or ""),
)
conn.commit()
def get_all_ingest_state():
"""返回 {site: {kind: {ok, ingested_at, count, error}}};库不存在返回 {}"""
if not os.path.exists(STATE_DB_PATH):
return {}
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT site, kind, ok, ingested_at, count, error FROM ingest_state"
).fetchall()
out = {}
for site, kind, ok, ingested_at, count, error in rows:
out.setdefault(site, {})[kind] = {
"ok": bool(ok),
"ingested_at": ingested_at or "",
"count": int(count or 0),
"error": error or "",
}
return out
# ============================ 下载日期偏移site_config============================
MAX_DATE_OFFSET = 30 # 0=今天,最大回溯 30 天
def get_offset(site, kind="expected"):
"""读取单站下载日期偏移kind: 'expected'/'actual'0=今天1=昨天…);未配置返回 0。"""
col = "expected_offset" if kind == "expected" else "actual_offset"
if not os.path.exists(STATE_DB_PATH):
return 0
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute(
f"SELECT {col} FROM site_config WHERE site=?", (site,)
).fetchone()
return int(row[0]) if row else 0
def set_offset(site, kind, offset):
"""设置单站下载日期偏移kind: 'expected'/'actual'),钳制到 [0, MAX_DATE_OFFSET]。"""
col = "expected_offset" if kind == "expected" else "actual_offset"
offset = max(0, min(MAX_DATE_OFFSET, int(offset)))
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute(
f"INSERT INTO site_config (site, {col}, updated_at) VALUES (?, ?, ?) "
f"ON CONFLICT(site) DO UPDATE SET {col}=excluded.{col}, "
f"updated_at=excluded.updated_at",
(site, offset, _now()),
)
conn.commit()
return offset
def resolve_target_date(site, kind, date=None):
"""计算一条任务的目标下载日期YYYY-MM-DD供任务日志展示 / 重试回放)。
有 date 用 date否则按站点偏移推算与 runtime._record_business_date 同源):
expected → 应到偏移actual → 实到偏移;百世 undelivered → 当天;
4 站 undelivered → 跟随应到偏移。__compare__ 无数据概念,返回 ''"""
if site == "__compare__":
return ""
if date:
return date
today = datetime.now().date()
if kind == "expected":
return (today - timedelta(days=get_offset(site, "expected"))).strftime(
"%Y-%m-%d"
)
if kind == "actual":
return (today - timedelta(days=get_offset(site, "actual"))).strftime("%Y-%m-%d")
if site == "百世":
return today.strftime("%Y-%m-%d")
return (today - timedelta(days=get_offset(site, "expected"))).strftime("%Y-%m-%d")
def set_schedule(site, enabled, time_str):
"""【DEPRECATED】旧"每日单时点定时"——已被 fetch_schedule 的周期+激活时段模式取代。
保留死代码以防外部残留调用;新代码请用 set_fetch_schedule。"""
enabled_int = 1 if enabled else 0
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute(
"INSERT INTO site_config (site, schedule_enabled, schedule_time, updated_at) "
"VALUES (?, ?, ?, ?) "
"ON CONFLICT(site) DO UPDATE SET "
"schedule_enabled=excluded.schedule_enabled, "
"schedule_time=excluded.schedule_time, updated_at=excluded.updated_at",
(site, enabled_int, time_str or "", _now()),
)
conn.commit()
return bool(enabled_int), (time_str or "")
# ============================ 周期性抓取调度fetch_schedule============================
# 取代旧 site_config.schedule_* 的"每日单时点":每 site×kind 一行,
# 在激活时段 [active_start, active_end) 内按 interval_minutes 周期抓取。
# 应到/实到各自独立配置;百世只有 undelivered站点直供未到明细无应到/实到二分)。
# 各站允许的周期抓取 kind单一来源server.py 复用)
SITE_FETCH_KINDS = {
"顺心": ("expected", "actual"),
"中通": ("expected", "actual"),
"韵达": ("expected", "actual"),
"安能": ("expected", "actual"),
"百世": ("undelivered",),
}
DEFAULT_FETCH_SCHEDULE = {
"enabled": False,
"active_start": "",
"active_end": "",
"interval_minutes": 30,
}
def allowed_kinds(site):
"""该站允许的周期抓取 kind 元组;未知站点返回空元组。"""
return SITE_FETCH_KINDS.get(site, ())
def _fetch_spec(row):
"""把 fetch_schedule 行转成 spec dict。"""
return {
"enabled": bool(row[0]),
"active_start": row[1] or "",
"active_end": row[2] or "",
"interval_minutes": int(row[3]),
}
def get_fetch_schedule(site, kind):
"""读单 (site,kind) 周期抓取配置;未配置返回 None。"""
if not os.path.exists(STATE_DB_PATH):
return None
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute(
"SELECT enabled, active_start, active_end, interval_minutes "
"FROM fetch_schedule WHERE site=? AND kind=?",
(site, kind),
).fetchone()
return _fetch_spec(row) if row else None
def set_fetch_schedule(site, kind, enabled, active_start, active_end, interval_minutes):
"""UPSERT 单 (site,kind) 周期抓取配置;返回写入后的 spec dict。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute(
"INSERT INTO fetch_schedule "
"(site, kind, enabled, active_start, active_end, interval_minutes, updated_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?) "
"ON CONFLICT(site, kind) DO UPDATE SET "
"enabled=excluded.enabled, active_start=excluded.active_start, "
"active_end=excluded.active_end, interval_minutes=excluded.interval_minutes, "
"updated_at=excluded.updated_at",
(
site,
kind,
1 if enabled else 0,
(active_start or ""),
(active_end or ""),
int(interval_minutes),
_now(),
),
)
conn.commit()
return {
"enabled": bool(enabled),
"active_start": active_start or "",
"active_end": active_end or "",
"interval_minutes": int(interval_minutes),
}
def get_all_fetch_schedules():
"""返回 {site: {kind: spec}};对每个站点的每个 allowed kind 都补齐(缺失用默认值)。
保证前端永远拿到完整 kind 集,不必前端补默认 spec。"""
out = {}
if not os.path.exists(STATE_DB_PATH):
for site, kinds in SITE_FETCH_KINDS.items():
out[site] = {k: dict(DEFAULT_FETCH_SCHEDULE) for k in kinds}
return out
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT site, kind, enabled, active_start, active_end, interval_minutes "
"FROM fetch_schedule"
).fetchall()
by_key = {(r[0], r[1]): _fetch_spec(r[2:]) for r in rows}
for site, kinds in SITE_FETCH_KINDS.items():
out[site] = {
k: by_key.get((site, k), dict(DEFAULT_FETCH_SCHEDULE)) for k in kinds
}
return out
def get_all_config():
"""返回 {site: {expected_offset, actual_offset, fetch_schedules}}。
站点集以 fetch_schedule 的 allowed kinds 为准覆盖全业务站点offsets 缺失默认 0。"""
schedules = get_all_fetch_schedules()
offsets = {}
if os.path.exists(STATE_DB_PATH):
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT site, expected_offset, actual_offset FROM site_config"
).fetchall()
offsets = {
r[0]: {"expected_offset": int(r[1]), "actual_offset": int(r[2])}
for r in rows
}
return {
site: {
"expected_offset": offsets.get(site, {}).get("expected_offset", 0),
"actual_offset": offsets.get(site, {}).get("actual_offset", 0),
"fetch_schedules": schedules.get(site, {}),
}
for site in schedules
}
# ============================ 站点键值配置site_settings============================
# 各站专属配置(百世密码、韵达账密、安能 exe 路径…),由前端配置弹窗设置。
def get_setting(site, key):
"""读取单站某个配置值;未设置返回 ''"""
if not os.path.exists(STATE_DB_PATH):
return ""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute(
"SELECT value FROM site_settings WHERE site=? AND key=?", (site, key)
).fetchone()
return row[0] if row else ""
def set_setting(site, key, value):
"""设置单站某个配置值upsert"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
conn.execute(
"INSERT INTO site_settings (site, key, value) VALUES (?, ?, ?) "
"ON CONFLICT(site, key) DO UPDATE SET value=excluded.value",
(site, key, value if value is not None else ""),
)
conn.commit()
def get_site_settings(site):
"""返回单站全部配置 {key: value}。"""
if not os.path.exists(STATE_DB_PATH):
return {}
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT key, value FROM site_settings WHERE site=?", (site,)
).fetchall()
return {r[0]: r[1] for r in rows}
# ============================ 任务历史 ============================
def create_task(site, kind, trigger="manual", target_date="", force=False):
"""新建一条 pending 任务(手动触发),返回其 id。trigger='manual'/'auto'
target_date 为该任务的目标下载日期YYYY-MM-DD可为 ''force 是否强制重下。"""
now = _now()
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
cur = conn.execute(
"INSERT INTO task_history "
"(site, kind, status, started_at, finished_at, error, trigger, target_date, force) "
"VALUES (?, ?, ?, ?, '', '', ?, ?, ?)",
(site, kind, TASK_PENDING, now, trigger, target_date, 1 if force else 0),
)
conn.commit()
return cur.lastrowid
def create_task_if_idle(site, kind, trigger="auto", target_date=""):
"""周期调度专用:若该 (site,kind) 已有 pending/running 任务则返回 None跳过本次周期
否则建一条 pending 任务返回其 id。单连接内 check-then-insert靠 SQLite 写锁把竞态压到忽略不计。
与 create_task 的区别:手动触发(POST /tasks)用 create_task用户点的必建周期 job 用本函数
——上一次还没跑完时跳过,避免同 (site,kind) 任务堆积。手动建的任务会让紧随其后的周期 fire
判到 inflight 而跳过天然互斥。trigger='auto'target_date 为目标下载日期YYYY-MM-DD"""
now = _now()
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute(
"SELECT 1 FROM task_history WHERE site=? AND kind=? "
"AND status IN (?, ?) LIMIT 1",
(site, kind, TASK_PENDING, TASK_RUNNING),
).fetchone()
if row:
return None
cur = conn.execute(
"INSERT INTO task_history "
"(site, kind, status, started_at, finished_at, error, trigger, target_date, force) "
"VALUES (?, ?, ?, ?, '', '', ?, ?, 0)",
(site, kind, TASK_PENDING, now, trigger, target_date),
)
conn.commit()
return cur.lastrowid
def update_task(task_id, status, error=None):
"""更新任务状态。终态(success/no_data/failed)写入 finished_at。"""
finished = _now() if status in (TASK_SUCCESS, TASK_NO_DATA, TASK_FAILED) else ""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
if finished:
conn.execute(
"UPDATE task_history SET status=?, finished_at=?, error=? WHERE id=?",
(status, finished, error or "", task_id),
)
else:
conn.execute(
"UPDATE task_history SET status=?, error=? WHERE id=?",
(status, error or "", task_id),
)
conn.commit()
def get_task(task_id):
"""返回单条任务 dict不存在返回 None。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute(
"SELECT id, site, kind, status, started_at, finished_at, error, "
"trigger, target_date, force "
"FROM task_history WHERE id=?",
(task_id,),
).fetchone()
if not row:
return None
return {
"id": row[0],
"site": row[1],
"kind": row[2],
"status": row[3],
"started_at": row[4],
"finished_at": row[5],
"error": row[6],
"trigger": row[7],
"target_date": row[8],
"force": bool(row[9]),
}
def list_tasks(limit=20):
"""返回最近 limit 条任务(按 id 倒序)。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute(
"SELECT id, site, kind, status, started_at, finished_at, error, "
"trigger, target_date, force "
"FROM task_history ORDER BY id DESC LIMIT ?",
(limit,),
).fetchall()
return [
{
"id": r[0],
"site": r[1],
"kind": r[2],
"status": r[3],
"started_at": r[4],
"finished_at": r[5],
"error": r[6],
"trigger": r[7],
"target_date": r[8],
"force": bool(r[9]),
}
for r in rows
]
def fail_stale_tasks(reason: str = "服务重启,上轮未完成任务,请手动重跑") -> int:
"""worker 启动时调用:把遗留的 pending/running 任务标记为 failed实现重启自愈。
返回被清理的任务数量。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
cur = conn.execute(
"UPDATE task_history SET status=?, finished_at=?, error=? "
"WHERE status IN (?, ?)",
(TASK_FAILED, _now(), reason, TASK_PENDING, TASK_RUNNING),
)
conn.commit()
return cur.rowcount

562
inbound_verify/store.py Normal file
View File

@@ -0,0 +1,562 @@
# -*- coding: utf-8 -*-
"""
store.py — 到货核销数据持久化PostgreSQL
职责:把 downloads/ 下各站点下载的应到 / 实到 / 未到 Excel 解析后,幂等写入 PostgreSQL。
与下载流程解耦:本模块只读 downloads/ 现有文件入库,不关心谁触发下载、下载了几次。
设计要点:
- 三张表expected_record运单级/ actual_record扫描件级
/ undelivered_record百世站点直供未到明细子单级
- 每行 = 统一核心列 + raw JSONB站点原始全列key=原列名,一字段不丢)
- 幂等:业务唯一键 UPSERT重复下载天然合并、零冗余
- 单号一律按文本读写dtype=str防长数字被科学计数 / 精度丢失
命令行:
python -m inbound_verify.store createdb 创建数据库(幂等)
python -m inbound_verify.store init 建表(幂等 CREATE TABLE IF NOT EXISTS
python -m inbound_verify.store ingest [site] 入库全站或单站(幂等 UPSERT
python -m inbound_verify.store ingest-one <site> <kind> 仅入库指定站/类(钩子同款路由)
python -m inbound_verify.store all createdb → init → 全站 ingest 一条龙
"""
import os
import sys
from datetime import date, datetime
import pandas as pd
import psycopg
import yaml
from psycopg.types.json import Jsonb
from inbound_verify.paths import BASE_DIR, CONFIG_PATH, DOWNLOAD_DIR
from inbound_verify.domain import (
BAISHI_FILE,
_site_cfg,
) # 站点 / 文件名配置(单一来源)
from inbound_verify import compare # _read_business_dates比对侧业务日期读取
SCHEMA_PATH = os.path.join(BASE_DIR, "schema.sql")
# 有应到 / 实到的 4 站(百世只有站点直供的未到明细,单独处理)
ALL_SITES = ["顺心", "中通", "韵达", "安能"]
# ============================== 配置 / 连接 ==============================
def _load_pg_config():
"""从 config.yaml 读 postgres 段;缺失项给默认。"""
if not os.path.exists(CONFIG_PATH):
raise FileNotFoundError(
f"未找到配置文件 {CONFIG_PATH}(请参考 config.example.yaml 创建 config.yaml"
)
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
pg = cfg.get("postgres") or {}
return {
"host": pg.get("host", "127.0.0.1"),
"port": int(pg.get("port", 5432)),
"user": pg.get("user", "postgres"),
"password": pg.get("password", ""),
"dbname": pg.get("dbname", "CQHXDB"),
"schema": pg.get("schema", "inbound_verify"),
"auto_ingest": bool(pg.get("auto_ingest", True)),
"connect_timeout_seconds": int(pg.get("connect_timeout_seconds", 5)),
}
def _connect(dbname):
"""用关键字参数连接(避开 conninfo 对密码特殊字符的解析)。
options 设 search_path 到专用 schema + 会话级 statement_timeout=30s
cpolar 隧道上防失控查询connect_timeout 守连接阶段)。"""
c = _load_pg_config()
return psycopg.connect(
host=c["host"],
port=c["port"],
dbname=dbname,
user=c["user"],
password=c["password"],
options=f"-c search_path={c['schema']} -c statement_timeout=30s",
connect_timeout=c["connect_timeout_seconds"],
)
def ingest_enabled():
"""是否启用下载后自动入库config.yaml postgres.auto_ingest默认 True
供 runtime 钩子判定开关,避免它伸手进 _load_pg_config。"""
return _load_pg_config()["auto_ingest"]
# ============================== 建库 / 建表 ==============================
def create_database():
"""连接维护库 postgres创建目标数据库幂等"""
c = _load_pg_config()
target = c["dbname"]
with _connect("postgres") as conn: # autocommitCREATE DATABASE 不能在事务里
conn.autocommit = True
with conn.cursor() as cur:
cur.execute("SELECT 1 FROM pg_database WHERE datname = %s", (target,))
if cur.fetchone():
print(f">> [db] 数据库 {target} 已存在,跳过创建")
return
cur.execute(f'CREATE DATABASE "{target}"')
print(f">> [db] 已创建数据库 {target}")
def init_schema():
"""在目标库执行 schema.sql幂等"""
with open(SCHEMA_PATH, "r", encoding="utf-8") as f:
sql = f.read()
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(sql)
conn.commit()
print(">> [db] 表结构已就绪")
# ============================== 解析辅助 ==============================
# 实到单号列 / 基号列映射(口径取自 domain与 STATIONS 对齐):
# piece = 实到表里「每件」的单号列(扫描单号 / 子单号 / 复合串)
# waybill = 与应到运单号对齐的干净列(中通无干净列,由 piece 复合串 v[:-8] 推导)
# scan_time = 扫描时间列(缺失则不填,原始值仍在 raw
# scan_site = 扫描网点列
ACTUAL_COLMAP = {
"中通": {
"piece": "运单号",
"waybill": None,
"scan_time": "扫描时间",
"scan_site": "扫描网点",
},
"顺心": {
"piece": "子单号",
"waybill": "运单号",
"scan_time": "操作时间",
"scan_site": "操作网点",
},
"韵达": {
"piece": "子单号",
"waybill": "主单号",
"scan_time": "扫描时间",
"scan_site": "扫描站点",
},
"安能": {
"piece": "扫描单号",
"waybill": "所属单号",
"scan_time": "扫描时间",
"scan_site": "扫描网点",
},
}
_TIME_FMTS = (
"%Y-%m-%d %H:%M:%S",
"%Y-%m-%d %H:%M",
"%Y/%m/%d %H:%M:%S",
"%Y/%m/%d %H:%M",
"%Y-%m-%d",
"%Y/%m/%d",
)
def _to_int(v):
"""尽力把单元格转 int件数空 / 非数返回 None。"""
s = str(v).strip().replace(",", "")
if s in ("", "-", "nan", "None"):
return None
try:
return int(float(s))
except (TypeError, ValueError):
return None
def _parse_time(v):
"""尽力解析多种时间格式为 datetime失败返回 None原始值在 raw 里)。"""
if v is None:
return None
if isinstance(v, datetime):
return v
s = str(v).strip()
if not s or s in ("nan", "NaT"):
return None
for fmt in _TIME_FMTS:
try:
return datetime.strptime(s, fmt)
except ValueError:
continue
try: # 兜底:交给 pandas 推断
return pd.to_datetime(s).to_pydatetime()
except Exception:
return None
def _parse_date(v):
if not v:
return None
try:
return datetime.strptime(str(v).strip(), "%Y-%m-%d").date()
except (ValueError, TypeError):
return None
def _raw_row(row):
"""把一行原始记录转成 JSONB 兼容 dictkey=原列名;丢空值,保留全部有值字段)。"""
out = {}
for k, v in row.items():
if v is None:
continue
if isinstance(v, float) and pd.isna(v):
continue
s = str(v).strip()
if s == "":
continue
if isinstance(v, (datetime, date)):
out[str(k)] = v.isoformat()
else:
out[str(k)] = v
return out
def _read_business_dates():
"""从状态库读各站本次业务日期(与报告口径一致;读不到返回空 dict"""
try:
return compare._read_business_dates(ALL_SITES + ["百世"]) or {}
except Exception as e:
print(f">> [warn] 读取业务日期失败(不影响入库): {e}")
return {}
# ============================== UPSERT SQL ==============================
_SQL_EXPECTED = """
INSERT INTO expected_record
(site, waybill_no, handover_no, handover_pieces, order_pieces, business_date, raw)
VALUES (%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, waybill_no) DO UPDATE SET
handover_no = EXCLUDED.handover_no,
handover_pieces = EXCLUDED.handover_pieces,
order_pieces = EXCLUDED.order_pieces,
business_date = COALESCE(EXCLUDED.business_date, expected_record.business_date),
raw = EXCLUDED.raw,
ingested_at = now()
"""
_SQL_ACTUAL = """
INSERT INTO actual_record
(site, waybill_no, piece_no, scan_time, scan_site, raw)
VALUES (%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, piece_no) DO UPDATE SET
waybill_no = COALESCE(EXCLUDED.waybill_no, actual_record.waybill_no),
scan_time = EXCLUDED.scan_time,
scan_site = EXCLUDED.scan_site,
raw = EXCLUDED.raw,
ingested_at = now()
"""
_SQL_UNDELIVERED = """
INSERT INTO undelivered_record
(site, waybill_no, piece_no, biz_type, last_scan, raw)
VALUES (%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, piece_no) DO UPDATE SET
waybill_no = COALESCE(EXCLUDED.waybill_no, undelivered_record.waybill_no),
biz_type = EXCLUDED.biz_type,
last_scan = EXCLUDED.last_scan,
raw = EXCLUDED.raw,
ingested_at = now()
"""
_SQL_BAISHI_DAILY_STATS = """
INSERT INTO baishi_daily_stats
(site, business_date, expected_pieces, arrived_pieces, undelivered_pieces, raw)
VALUES (%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, business_date) DO UPDATE SET
expected_pieces = COALESCE(EXCLUDED.expected_pieces, baishi_daily_stats.expected_pieces),
arrived_pieces = COALESCE(EXCLUDED.arrived_pieces, baishi_daily_stats.arrived_pieces),
undelivered_pieces = COALESCE(EXCLUDED.undelivered_pieces, baishi_daily_stats.undelivered_pieces),
raw = EXCLUDED.raw,
ingested_at = now()
"""
# ============================== 入库 ==============================
def _ingest_expected(cur, site, business_date):
"""入库单站应到(运单级,按 waybill_no 去重 keep-first 后 UPSERT"""
cfg = _site_cfg(site)
path = os.path.join(DOWNLOAD_DIR, cfg["exp"])
if not os.path.exists(path):
print(f" [跳过] {site} 应到:文件不存在 {cfg['exp']}")
return 0
df = pd.read_excel(path, dtype=str).fillna("")
df = df.drop_duplicates(subset=[cfg["exp_wb"]], keep="first")
biz = _parse_date(business_date)
rows = []
for r in df.to_dict("records"):
wb = str(r.get(cfg["exp_wb"], "")).strip()
if not wb:
continue
rows.append(
(
site,
wb,
str(r.get(cfg["exp_jd"], "")).strip() or None,
_to_int(r.get(cfg["exp_qty"])),
_to_int(r.get("录单件数")),
biz,
Jsonb(_raw_row(r)),
)
)
if rows:
cur.executemany(_SQL_EXPECTED, rows)
print(f" [应到] {site}{len(rows)} 条运单")
return len(rows)
def _ingest_actual(cur, site):
"""入库单站实到(扫描件级,按 piece_no UPSERT"""
cfg = _site_cfg(site)
path = os.path.join(DOWNLOAD_DIR, cfg["act"])
if not os.path.exists(path):
print(f" [跳过] {site} 实到:文件不存在 {cfg['act']}")
return 0
cm = ACTUAL_COLMAP[site]
df = pd.read_excel(path, dtype=str).fillna("")
if site == "韵达":
# 韵达业务清洗:保留「交接单号」为空的行(到/接件扫描),
# 抛弃「交接单号」不为空的行(派件/签收等,属重复数据)。
# 再按子单号去重一件多扫只留一条清洗后子单号已天然唯一drop 为保险)。
df = df[df["交接单号"].astype(str).str.strip() == ""]
df = df.drop_duplicates(subset=[cm["piece"]], keep="last")
rows = []
for r in df.to_dict("records"):
piece = str(r.get(cm["piece"], "")).strip()
if not piece:
continue
if site == "中通": # 复合串 H+运单号(12)+总数(4)+顺序(4):基号 = v[:-8]
waybill = piece[:-8] if (len(piece) > 8 and piece[-4:].isdigit()) else piece
else:
waybill = str(r.get(cm["waybill"], "")).strip() or None
rows.append(
(
site,
waybill,
piece,
_parse_time(r.get(cm["scan_time"])),
str(r.get(cm["scan_site"], "")).strip() or None,
Jsonb(_raw_row(r)),
)
)
if rows:
cur.executemany(_SQL_ACTUAL, rows)
print(f" [实到] {site}{len(rows)} 条扫描")
return len(rows)
def _ingest_undelivered_baishi(cur):
"""入库百世应到未到明细(子单级,按 (site, piece_no) UPSERT"""
path = os.path.join(DOWNLOAD_DIR, BAISHI_FILE)
if not os.path.exists(path):
print(f" [跳过] 百世 未到:文件不存在 {BAISHI_FILE}")
return 0
df = pd.read_excel(path, dtype=str).fillna("")
rows = []
for r in df.to_dict("records"):
rows.append(
(
"百世",
str(r.get("运单号", "")).strip() or None,
str(r.get("子单号", "")).strip() or None,
str(r.get("类型", "")).strip() or None,
str(r.get("最新扫描记录", "")).strip() or None,
Jsonb(_raw_row(r)),
)
)
if rows:
cur.executemany(_SQL_UNDELIVERED, rows)
print(f" [未到] 百世:{len(rows)}")
return len(rows)
def upsert_baishi_daily_stats(exp, arr, business_date=None):
"""直接落库百世当日应到/实到基数(应扫/已扫,站级日聚合)。
供 baishi 下载时抓到基数后直接调用(一步落库,不绕 state_store→store
business_date 默认今天百世固定当天。best-effort失败只告警不影响下载流程。"""
biz = business_date or date.today()
if exp is None and arr is None:
return
undel = (exp - arr) if (exp is not None and arr is not None) else None
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(
_SQL_BAISHI_DAILY_STATS,
(
"百世",
biz,
exp,
arr,
undel,
Jsonb({"expected": exp, "arrived": arr, "undelivered": undel}),
),
)
conn.commit()
print(f" [基数] 百世 {biz}: 应扫 {exp} / 已扫 {arr} / 未扫 {undel}")
except Exception as e:
print(f" [基数] 百世 {biz} 入库失败(不影响下载): {e}")
def ingest(site=None):
"""入库:指定 site 则单站(百世只入未到),否则全站。返回总条数。"""
dates = _read_business_dates()
total = 0
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
sites = ALL_SITES if site in (None, "百世") else [site]
for s in sites:
if s not in ALL_SITES:
print(f" [跳过] 不支持应到/实到的站点: {s}")
continue
total += _ingest_expected(cur, s, dates.get(s))
total += _ingest_actual(cur, s)
if site in (None, "百世"):
total += _ingest_undelivered_baishi(cur)
conn.commit()
print(f">> [ingest] 完成,共 {total}")
return total
def ingest_task(site, kind):
"""按 (site, kind) 入库本次刚下载的文件(幂等 UPSERT返回总条数。
与 ingest(site) 的区别:只入本次刷新的那一类,避免重读写另一类文件(同步钩子里减少阻塞)。
kind 路由:
expected/actual 各入其列;
undelivered 百世 入未到;
undelivered 4 站 _site_undelivered_handler 内部连带下了 expected+actual故入两者
__compare__ / 其它组合 返回 0。
"""
if site == "__compare__":
return 0
# 百世无应到/实到(只有站点直供的未到);非 undelivered 直接返回 0避免
# _ingest_expected/_ingest_actual 走到 _site_cfg(百世)=None 而 TypeError。
if site == "百世" and kind != "undelivered":
return 0
dates = _read_business_dates()
total = 0
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
if kind == "expected":
total += _ingest_expected(cur, site, dates.get(site))
elif kind == "actual":
total += _ingest_actual(cur, site)
elif kind == "undelivered":
if site == "百世":
total += _ingest_undelivered_baishi(cur)
else: # 顺心/中通/韵达/安能
total += _ingest_expected(cur, site, dates.get(site))
total += _ingest_actual(cur, site)
# 其它组合(如 百世/expected正常不经钩子触发防御性返回 0
conn.commit()
return total
def get_existing_handover_nos(site):
"""查该站点已落库的交接单号集合expected_record.handover_no
"提交导出任务前"去重:已落库的交接单号不再重复提交导出任务。
PG 不可用cpolar 抖动等)时返回空集 + 告警,调用方按"未确认存在"处理
继续提交导出UPSERT 兜底,绝不因去重查询失败而漏数据)。"""
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT handover_no FROM expected_record "
"WHERE site=%s AND handover_no IS NOT NULL AND handover_no <> ''",
(site,),
)
return {str(r[0]).strip() for r in cur.fetchall()}
except Exception as e:
print(f">> [去重] 查询已落库交接单号失败({site}),本次不去重: {e}")
return set()
# ============================== PG 数据存在性查询 ==============================
def has_data(site, kind, target_date):
"""查询 PG指定站点在 target_date 是否有业务数据。
target_date: str 'YYYY-MM-DD' 或 date 对象。
返回 (has_rows: bool, count: int)。
PG 不可达时返回 (False, 0),不抛异常——调用方按「未确认存在」处理。
kind 路由:
expected → expected_record (business_date)
actual → actual_record (scan_time::date)
undelivered → 百世: baishi_daily_stats4 站: 不单独查(由调用方 expected∧actual 派生)
"""
if site == "百世" and kind == "undelivered":
sql = (
"SELECT COUNT(*) FROM baishi_daily_stats"
" WHERE site = %s AND business_date = %s"
)
params = (site, target_date)
elif kind == "expected":
sql = (
"SELECT COUNT(*) FROM expected_record"
" WHERE site = %s AND business_date = %s"
)
params = (site, target_date)
elif kind == "actual":
sql = (
"SELECT COUNT(*) FROM actual_record"
" WHERE site = %s AND scan_time::date = %s"
)
params = (site, target_date)
else:
return (False, 0)
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(sql, params)
row = cur.fetchone()
cnt = int(row[0]) if row else 0
return (cnt > 0, cnt)
except Exception as e:
print(f">> [状态] PG 查询 {site}/{kind}/{target_date} 失败: {e}")
return (False, 0)
# ============================== 命令行 ==============================
def main():
cmd = sys.argv[1] if len(sys.argv) > 1 else "all"
site = sys.argv[2] if len(sys.argv) > 2 else None
if cmd == "createdb":
create_database()
elif cmd == "init":
init_schema()
elif cmd == "ingest":
ingest(site)
elif cmd == "all":
create_database()
init_schema()
ingest()
elif cmd == "ingest-one":
kind = sys.argv[3] if len(sys.argv) > 3 else None
if not site or kind not in ("expected", "actual", "undelivered"):
print(
"用法: python -m inbound_verify.store ingest-one <site> <expected|actual|undelivered>"
)
sys.exit(1)
total = ingest_task(site, kind)
print(f">> [ingest-one] {site}/{kind} 入库 {total}")
else:
print(__doc__)
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -1,482 +0,0 @@
import os
import re
import glob
from datetime import datetime
import pandas as pd
from playwright.sync_api import sync_playwright
def task_expected_goods_download(page):
"""模块一:应到货物数据下载"""
print("\n▶ 开始执行【应到货物数据下载】任务...")
# 初始化并创建下载目录
download_dir = os.path.join(os.getcwd(), "downloads")
if not os.path.exists(download_dir):
os.makedirs(download_dir)
print(f">> 已创建专属下载文件夹: {download_dir}")
export_times = []
target_task_timestamps = []
try:
# 1. 导航与页面加载判断
print(">> 正在进入【车辆点到】界面...")
page.locator("span.ant-pro-menu-item-title:has-text('派件管理')").click()
page.locator("div.ant-pro-menu-item:has-text('车辆点到')").click()
page.get_by_role("button", name="点到").wait_for(state="visible")
page.get_by_role("button", name="打印交接单").wait_for(state="visible")
page.get_by_role("button", name="强卸").wait_for(state="visible")
print("✅ 车辆点到界面加载完毕")
# 2. 筛选条件1天、状态=已发 -> 已到、查询
print(">> 正在设置筛选条件: 选择【1天】...")
page.get_by_role("radio", name="1天").check()
print(">> 正在展开【状态】下拉菜单...")
# 匹配包含 已发 或 已到 的下拉选项
page.locator(
".ant-select-selection-item", has_text=re.compile(r"已发|已到")
).click()
print(">> 正在选择状态为【已到】...")
page.locator(".ant-select-item-option", has_text="已到").click()
print(">> 正在点击【查询】按钮...")
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 3. 获取所有【运单列表】按钮并循环处理
waybill_btns = page.get_by_role("button", name="运单列表")
count = waybill_btns.count()
print(f">> 共发现 {count} 个班次需要导出。")
for i in range(count):
print(f" ⏳ 正在处理第 {i+1}/{count} 个班次...")
waybill_btns.nth(i).click()
page.locator("label[title='运单查询']").wait_for(state="visible")
# 4. 执行导出流程
page.get_by_role("button", name="export 导出").click()
page.locator(
"span.ant-transfer-list-header-title:has-text('待选导出列')"
).wait_for(state="visible")
page.locator(".ant-transfer-list").first.locator(
".ant-transfer-list-header label"
).click()
page.get_by_label("导出").get_by_role("button", name="right").click()
page.get_by_role("button", name="export 导出数据").click()
export_times.append(datetime.now())
page.locator("text=任务添加成功!").wait_for(state="visible")
page.get_by_role("button", name="知道了").click()
page.get_by_role("tab", name="车辆点到").click()
page.wait_for_timeout(500)
if count > 0:
print("✅ 所有班次的导出任务已成功提交!")
else:
print("⚠️ 未发现任何运单列表,直接跳转至下载环节。")
# 6. 前往数据导出页面去下载
print(">> 正在前往【数据导出】界面...")
page.locator("a[href='/dataExport']").click()
page.get_by_role("columnheader", name="任务标题").wait_for(state="visible")
page.wait_for_timeout(2000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 7. 轮询任务状态
print(">> 列表中已渲染,开始匹配并检查后端处理状态...")
while True:
rows = page.locator(".ant-table-tbody > tr.ant-table-row")
row_count = rows.count()
pending_tasks = 0
current_ready_timestamps = []
for i in range(row_count):
tds = rows.nth(i).locator("td")
if tds.count() < 9:
continue
submit_time_str = tds.nth(2).inner_text().strip()
title_str = tds.nth(5).inner_text().strip()
status_str = tds.nth(6).inner_text().strip()
if title_str == "发车管理运单列表":
try:
row_time = datetime.strptime(
submit_time_str, "%Y-%m-%d %H:%M:%S"
)
matched = any(
abs((row_time - et).total_seconds()) <= 60
for et in export_times
)
if matched:
if status_str != "执行完成":
pending_tasks += 1
if submit_time_str not in current_ready_timestamps:
print(
f" ⏳ 任务 [{submit_time_str}] 状态为【{status_str}】,数据生成中..."
)
else:
if submit_time_str not in current_ready_timestamps:
current_ready_timestamps.append(submit_time_str)
except Exception as e:
print(f" ⚠️ 解析时间时出错: {e}")
if pending_tasks > 0:
print(
f">> 共有 {pending_tasks} 个匹配任务还在处理中,等待 5 秒后刷新..."
)
page.wait_for_timeout(5000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
else:
target_task_timestamps = current_ready_timestamps
if len(target_task_timestamps) > 0:
print(">> ✅ 所有目标任务已就绪!开始并行下载...")
break
# 8. 下载逻辑
downloaded_files = []
for time_str in target_task_timestamps:
try:
target_row = page.locator(".ant-table-tbody > tr.ant-table-row").filter(
has=page.locator(f"td:nth-child(3):has-text('{time_str}')")
)
print(f" 🎯 触发下载 -> 任务 [{time_str}] ...")
with page.expect_download() as download_info:
target_row.locator("td").nth(8).locator(
"button", has_text=re.compile(r"\s*载")
).click()
download = download_info.value
save_path = os.path.join(download_dir, download.suggested_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" ⬇️ 文件已落盘: downloads/{download.suggested_filename}")
except Exception as e:
print(f" ❌ 下载任务 [{time_str}] 失败: {e}")
# 9. 合并数据
if downloaded_files:
print("\n>> 🧪 正在开始执行扁平数据高能合并流程...")
all_data_frames = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path)
if not df.empty:
all_data_frames.append(df)
except Exception as e:
pass
if all_data_frames:
combined_df = pd.concat(all_data_frames, ignore_index=True)
final_output_path = os.path.join(download_dir, "应到货物数据.xlsx")
combined_df.to_excel(final_output_path, index=False)
print(f"====================================================")
print(f" 🎉 恭喜!合并成功!最终输出路径: {final_output_path}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print("✅ 临时数据清理完毕。")
print("\n🎉 【应到货物数据下载】全流程测试完毕!")
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
def task_actual_goods_download(page):
"""模块二:实到货物数据下载"""
print("\n▶ 开始执行【实到货物数据下载】任务...")
# 初始化并创建下载目录
download_dir = os.path.join(os.getcwd(), "downloads")
if not os.path.exists(download_dir):
os.makedirs(download_dir)
export_times = []
target_task_timestamps = []
try:
# 1. 导航与页面加载
print(">> 正在进入【卸车扫描记录】界面...")
page.locator("span.ant-pro-menu-item-title:has-text('派件管理')").click()
page.locator("div.ant-pro-menu-item:has-text('卸车扫描记录')").click()
page.get_by_role("radio", name="1天").wait_for(state="visible")
# page.locator("button:has-text('查询')").wait_for(state="visible")
print("✅ 卸车扫描记录界面加载完毕")
# 2. 筛选条件1天、查询
print(">> 正在设置筛选条件: 选择【1天】...")
page.get_by_role("radio", name="1天").check()
# page.pause()
print(">> 正在点击【查询】按钮...")
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 3. 直接发起全局导出
print(">> 正在发起导出请求...")
page.get_by_role("button", name="export 导出").click()
page.locator(
"span.ant-transfer-list-header-title:has-text('待选导出列')"
).wait_for(state="visible")
page.locator(".ant-transfer-list").first.locator(
".ant-transfer-list-header label"
).click()
page.get_by_label("导出").get_by_role("button", name="right").click()
page.get_by_role("button", name="export 导出数据").click()
export_times.append(datetime.now())
page.locator("text=任务添加成功!").wait_for(state="visible")
page.get_by_role("button", name="知道了").click()
print("✅ 卸车扫描记录导出任务已成功提交!")
# 4. 前往数据导出页面去下载
print(">> 正在前往【数据导出】界面...")
page.locator("a[href='/dataExport']").click()
# page.pause()
page.get_by_role("columnheader", name="任务标题").wait_for(state="visible")
page.wait_for_timeout(2000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
# 5. 轮询任务状态
print(">> 列表中已渲染,开始匹配并检查后端处理状态...")
while True:
rows = page.locator(".ant-table-tbody > tr.ant-table-row")
row_count = rows.count()
pending_tasks = 0
current_ready_timestamps = []
for i in range(row_count):
tds = rows.nth(i).locator("td")
if tds.count() < 9:
continue
submit_time_str = tds.nth(2).inner_text().strip()
title_str = tds.nth(5).inner_text().strip()
status_str = tds.nth(6).inner_text().strip()
if title_str.startswith("卸车扫描记录"):
try:
row_time = datetime.strptime(
submit_time_str, "%Y-%m-%d %H:%M:%S"
)
matched = any(
abs((row_time - et).total_seconds()) <= 60
for et in export_times
)
if matched:
if status_str != "执行完成":
pending_tasks += 1
if submit_time_str not in current_ready_timestamps:
print(
f" ⏳ 任务 [{submit_time_str}] 状态为【{status_str}】,数据生成中..."
)
else:
if submit_time_str not in current_ready_timestamps:
current_ready_timestamps.append(submit_time_str)
except Exception as e:
print(f" ⚠️ 解析时间时出错: {e}")
if pending_tasks > 0:
print(
f">> 共有 {pending_tasks} 个匹配任务还在处理中,等待 5 秒后刷新..."
)
page.wait_for_timeout(5000)
page.get_by_role("button", name="search 查询").click()
page.wait_for_timeout(2000)
else:
target_task_timestamps = current_ready_timestamps
if len(target_task_timestamps) > 0:
print(">> ✅ 目标任务已就绪!开始下载...")
break
# 6. 下载逻辑
downloaded_files = []
for time_str in target_task_timestamps:
try:
target_row = page.locator(".ant-table-tbody > tr.ant-table-row").filter(
has=page.locator(f"td:nth-child(3):has-text('{time_str}')")
)
print(f" 🎯 触发下载 -> 任务 [{time_str}] ...")
with page.expect_download() as download_info:
target_row.locator("td").nth(8).locator(
"button", has_text=re.compile(r"\s*载")
).click()
download = download_info.value
save_path = os.path.join(download_dir, download.suggested_filename)
download.save_as(save_path)
downloaded_files.append(save_path)
print(f" ⬇️ 文件已落盘: downloads/{download.suggested_filename}")
except Exception as e:
print(f" ❌ 下载任务 [{time_str}] 失败: {e}")
# 7. 合并数据
if downloaded_files:
print("\n>> 🧪 正在开始执行数据归档整理...")
all_data_frames = []
for file_path in downloaded_files:
try:
df = pd.read_excel(file_path)
if not df.empty:
all_data_frames.append(df)
except Exception as e:
pass
if all_data_frames:
combined_df = pd.concat(all_data_frames, ignore_index=True)
final_output_path = os.path.join(download_dir, "实到货物数据.xlsx")
combined_df.to_excel(final_output_path, index=False)
print(f"====================================================")
print(f" 🎉 恭喜!处理成功!最终输出路径: {final_output_path}")
print(f"====================================================")
for file_path in downloaded_files:
os.remove(file_path)
print("✅ 临时数据清理完毕。")
print("\n🎉 【实到货物数据下载】全流程测试完毕!")
except Exception as e:
print(f"\n❌ 任务执行过程中发生异常: {e}")
def task_process_undelivered_data():
"""模块三:对下载的应到与实到数据进行高级清洗与比对,抽取出应到未到的异常运单"""
print("\n▶ 开始执行【应到未到数据处理】任务...")
download_dir = os.path.join(os.getcwd(), "downloads")
expected_path = os.path.join(download_dir, "应到货物数据.xlsx")
actual_path = os.path.join(download_dir, "实到货物数据.xlsx")
output_path = os.path.join(download_dir, "应到未到货物数据.xlsx")
# 基础校验:确保源文件全部存在
if not os.path.exists(expected_path):
print(f"❌ 错误:找不到【应到货物数据】主文档:{expected_path}")
print("💡 请先执行菜单选项 [1] 进行提取下载。")
return
if not os.path.exists(actual_path):
print(f"❌ 错误:找不到【实到货物数据】主文档:{actual_path}")
print("💡 请先执行菜单选项 [2] 进行提取下载。")
return
try:
print(">> 正在载入本地 Excel 文档...")
df_expected = pd.read_excel(expected_path)
df_actual = pd.read_excel(actual_path)
# 字段安全检查
if "运单号" not in df_expected.columns:
print(
"❌ 核心资产校验失败:应到货物数据中缺失【运单号】字段,请检查系统导出配置。"
)
return
if "运单号" not in df_actual.columns:
print(
"❌ 核心资产校验失败:实到货物数据中缺失【运单号】字段,请检查系统导出配置。"
)
return
print(">> 正在启动多维数据集比对引擎...")
# 核心比对清洗逻辑:在应到中剔除已经实到的运单号 (Left Anti-Join 机制)
df_undelivered = df_expected[~df_expected["运单号"].isin(df_actual["运单号"])]
# 按需求抽取指定的输出列:班次号、交接单号、运单号
target_columns = ["班次号", "交接单号", "运单号"]
# 确保列名完全匹配
available_columns = [
col for col in target_columns if col in df_undelivered.columns
]
df_output = df_undelivered[available_columns]
print(f">> 筛选完毕!共捕获到异常【应到未到】货物数据: {len(df_output)} 条。")
# 输出最终汇总报告
df_output.to_excel(output_path, index=False)
print(f"====================================================")
print(f" 🎉 异常比对流完成!独立数据已安全输出。")
print(f" 📁 成果归档成果路径: {output_path}")
print(f"====================================================")
except Exception as e:
print(f"❌ 数据处理引擎在执行连接和输出时发生致命异常: {e}")
def run_daemon_automation():
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
context = browser.new_context()
page = context.new_page()
target_url = "https://sxne.sxjdfreight.com"
print(f"正在打开登录页面: {target_url}")
page.goto(target_url)
print("====================================================")
print("【等待人工介入】请手动完成登录。")
print("====================================================")
try:
page.wait_for_selector('h1:has-text("盟商门户网")', timeout=120000)
print("🎉 登录成功!系统已接管浏览器。")
page.wait_for_timeout(1000)
# 跳过初始一些弹窗干扰
page.locator("a").nth(4).click()
page.wait_for_timeout(1000)
page.get_by_role("button", name="Close").click()
page.wait_for_timeout(1000)
page.get_by_role("button", name="不再询问").click()
page.wait_for_timeout(1000)
while True:
print("\n==============================")
print(" 物流数据自动提取系统 ")
print("==============================")
print("1. 执行【应到货物数据下载】")
print("2. 执行【实到货物数据下载】")
print("3. 执行【应到未到数据处理】 (★新)")
print("4. 退出程序")
print("==============================")
choice = input("请输入任务编号并回车: ")
if choice == "1":
task_expected_goods_download(page)
elif choice == "2":
task_actual_goods_download(page)
elif choice == "3":
task_process_undelivered_data()
elif choice == "4":
print("\n准备退出程序...")
break
else:
print("\n⚠️ 无效输入,请重新选一下。")
except Exception as e:
print(f"❌ 运行发生致命错误: {e}")
finally:
browser.close()
print("浏览器已安全关闭。")
if __name__ == "__main__":
run_daemon_automation()

28
pyproject.toml Normal file
View File

@@ -0,0 +1,28 @@
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "inbound-verify"
version = "0.1.0"
description = "物流到货数据自动下载与应到未到核对工具"
requires-python = ">=3.10"
dependencies = [
"pandas>=2.0.0",
"playwright>=1.40.0",
"openpyxl>=3.1.0",
"PyYAML>=6.0",
"websocket-client>=1.0.0",
"fastapi>=0.110.0",
"uvicorn>=0.27.0",
"apscheduler>=3.10.0",
"psycopg[binary]>=3.1",
]
[project.scripts]
inbound-verify = "inbound_verify.cli.router:main"
inbound-verify-server = "inbound_verify.cli.server:main"
inbound-verify-db = "inbound_verify.store:main"
[tool.setuptools.packages.find]
include = ["inbound_verify*"]

View File

@@ -1,3 +1,9 @@
pandas>=2.0.0
playwright>=1.40.0
openpyxl>=3.1.0
PyYAML>=6.0
websocket-client>=1.0.0
fastapi>=0.110.0
uvicorn>=0.27.0
apscheduler>=3.10.0
psycopg[binary]>=3.1

68
schema.sql Normal file
View File

@@ -0,0 +1,68 @@
-- =====================================================================
-- CQHXDB / 到货核销数据持久化 schema
-- 设计:统一核心表 + JSONB raw 兜底(保留站点原始全列,一字段不丢)
-- 幂等:业务唯一键 UPSERT重复下载天然合并、零冗余
-- 表expected_record运单级/ actual_record扫描件级
-- / undelivered_record百世站点直供未到明细子单级
-- =====================================================================
-- 隔离到专用 schema不污染 public表名无需再加前缀。
CREATE SCHEMA IF NOT EXISTS inbound_verify;
SET search_path TO inbound_verify;
-- 应到货物(运单级:一运单一行;按运单号去重 keep-first 后入库)
CREATE TABLE IF NOT EXISTS expected_record (
id BIGSERIAL PRIMARY KEY,
site TEXT NOT NULL, -- 站点:顺心 / 中通 / 韵达 / 安能
waybill_no TEXT NOT NULL, -- 运单基号(业务唯一键,去重键)
handover_no TEXT, -- 交接单号
handover_pieces INTEGER, -- 交接件数(应到件数口径)
order_pieces INTEGER, -- 录单件数
business_date DATE, -- 业务日期(属性,非唯一键;读不到则 NULL
raw JSONB NOT NULL, -- 站点原始全列key=原列名)
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, waybill_no)
);
CREATE INDEX IF NOT EXISTS idx_expected_site_date ON expected_record (site, business_date);
CREATE INDEX IF NOT EXISTS idx_expected_handover ON expected_record (site, handover_no);
-- 实到货物(扫描件级:一扫描一行;每扫描一件系统生成一个单号)
CREATE TABLE IF NOT EXISTS actual_record (
id BIGSERIAL PRIMARY KEY,
site TEXT NOT NULL,
waybill_no TEXT NOT NULL, -- 运单基号(由扫描单号推导)
piece_no TEXT NOT NULL, -- 扫描单号 / 子单号(业务唯一键)
scan_time TIMESTAMPTZ, -- 扫描时间(尽力解析,失败存 NULL原始值在 raw
scan_site TEXT, -- 扫描网点
raw JSONB NOT NULL,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, piece_no)
);
CREATE INDEX IF NOT EXISTS idx_actual_waybill ON actual_record (site, waybill_no);
CREATE INDEX IF NOT EXISTS idx_actual_scan_time ON actual_record (scan_time);
-- 应到未到(百世站点直供未到明细:子单级)
CREATE TABLE IF NOT EXISTS undelivered_record (
id BIGSERIAL PRIMARY KEY,
site TEXT NOT NULL, -- 百世
waybill_no TEXT, -- 运单号
piece_no TEXT, -- 子单号(业务唯一键)
biz_type TEXT, -- 类型
last_scan TEXT, -- 最新扫描记录
raw JSONB NOT NULL,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, piece_no)
);
-- 百世日聚合(应扫/已扫基数:站级日聚合,区别于运单级/件级/子单级表)
CREATE TABLE IF NOT EXISTS baishi_daily_stats (
id BIGSERIAL PRIMARY KEY,
site TEXT NOT NULL, -- 百世
business_date DATE NOT NULL, -- 业务日期(百世固定当天)
expected_pieces INTEGER, -- 应扫(应到基数)
arrived_pieces INTEGER, -- 已扫(实到基数)
undelivered_pieces INTEGER, -- 未扫(=应扫-已扫,任一缺失则 NULL
raw JSONB NOT NULL, -- 原始抓取值
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, business_date)
);