Compare commits

26 Commits

Author SHA1 Message Date
Misaka
36b204abe5 refactor(db_compare): 差缺统计改为应到驱动,以出库日为批次归属日
将未到统计从实到驱动(实到锚点反推批次)改为应到驱动(batch_out_date 归属日直接取应到批次),以吸收应到任务提前 1~2 天提交的扰动(韵达固定 +1、中通偶发 +1)。

- expected_record 新增 out_date / batch_out_date + 索引;入库解析出库时间并聚合批次归属日,支持历史回填

- 新增 compare_site_outdate 应到驱动入口,保留 compare_site_date 实到驱动作对照

- 未到任务 / POST /compare / 全站汇总切换到应到驱动

- 附现状梳理、设计、实现总结三篇文档
2026-08-03 22:52:28 +08:00
Misaka
bccf7cd396 fix(runtime): strip host-injected env vars when launching 安能 2026-08-02 14:14:34 +08:00
Misaka
c1bd53d832 feat(tasks): record trigger mode, target date and force in task history 2026-08-02 14:14:30 +08:00
Misaka
3c32720985 feat(sites): auto-screenshot on final download failure for debugging
所有站点 with_retry 在最后一次重试失败、reset 之前自动截图,
保存到 logs/screenshots/。网页站点走 Playwright page.screenshot(),
安能走 CDP Page.captureScreenshot。截图失败绝不阻塞任务流程。

- paths.py: 新增 SCREENSHOT_DIR (BASE_DIR/logs/screenshots/)
- runtime.py: 新增 capture_error_screenshot() 工具函数
- .gitignore: 新增 logs/ 忽略规则

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-02 11:18:21 +08:00
Misaka
53c71aeeac refactor(status): derive site ready/business_date from PG instead of ingest_state
_ready_flags 改从 PostgreSQL 直接查询(expected_record /
actual_record / baishi_daily_stats),target_date = today − offset。
消除因 ingested_at 日期比对导致的每日零点全站 ready 集体重置。

store.py: 新增 has_data(site, kind, target_date) 查 PG 数据存在性
runtime.py: _ready_flags 返回 (flags, dates) 同源元组,_apply_ready
  同步写入 business_date,修正 ready 与 business_date 不同源导致的
  前端日期标签漂移(如实到就绪却显示'前天')

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-02 09:22:54 +08:00
Misaka
837264f7b0 fix(zto): resolve real-today date picker failure on month-boundary days
The ZTO jQuery Date Range Picker always displays a dual-month view.
When today falls on the 1st (or early days) of a month, the same date
appears in both panels: a hidden ghost cell (month1, display:none) and
a visible cell (month2). Both carry the real-today CSS class, so .first
picks the hidden one, causing wait_for(visible) to timeout.

Replace DOM-based real-today time extraction with Python datetime
computation. Add _zto_find_visible_day to locate the actually visible
date cell (skipping hidden ghost cells, trying both midnight and
23:59:59 time variants). Fix _zto_flip_to_target_month to use the
same visibility-aware lookup.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:50:30 +08:00
Misaka
eaf56c1c9e refactor(status): derive readiness from ingest_state; drop Excel probe & /data
状态盘就绪态 {kind}_ready 改为从持久的 ingest_state 派生(DB 真相、重启不丢),
不再由心跳读 downloads/ Excel、也不在启动时重置:
- 心跳 _fresh_ingest/_ready_flags/_apply_ready:expected/actual_ready=该类今天入库成功;
  undelivered_ready=百世原生(今天入库) / 4 站派生(expected ∧ actual)。
- _persist_to_db 入库后调 _refresh_ready 立即派生(省 30s 心跳等待,与心跳同源);
  4 站 undelivered 连入 expected+actual,故 ingest_state 补记 expected/actual/undelivered 三行。
- _record_business_date 只写 business_date,不再碰 ready。
- 删 run_heartbeat 的 Excel 探测循环、DATA_FILENAMES、probe_data_file、遗物清理。
- state_store 增 set_business_date / set_ready(仅写单字段,不碰彼此)。
- 删 GET /data/{filename}(前端不再下载原始中转 Excel);/report 保留。

原则:Excel 只作「站点下载→入库」中转,状态/比对一律走 DB。
真机+单元验证:重启后心跳派生(百世今日入库→立即绿、重启不掉灰);韵达 expected-only
时 undelivered 不亮、补 actual 后亮(派生);百世原生;_ready_flags 7 例边界全过。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 23:47:23 +08:00
Misaka
dc7653c256 fix(db_compare): anchor comparison on actual_offset so yunda reads fresh data
比对以实到扫描日(actual.scan_time::date)为锚,锚点日期必须等于实到下载日 = actual_offset。
原 _site_undelivered_handler(runtime.py)与 _target_date_for(db_compare.py)误用
expected_offset 算锚点:韵达 exp=1 / act=0,锚点落到 today-1(昨天),读到历史数据。
两处改用 actual_offset 后韵达锚点 = today,反推出 expected 的昨天批次正确参与比对。
其余 3 站 exp == act == 0,锚点不变。

5 站真机验证:韵达 07-31 比对现读新鲜数据 107/104/3(修复前读昨天 116/115/1);
中通/安能/顺心 不变;全站汇总报表重新生成。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 22:58:12 +08:00
Misaka
3c7e9f2522 feat(db_compare): DB-based full summary report + wire 跑比对 to it
新增 build_full_report:4 站走 compare_site_date、百世走 baishi_daily_stats(基数) + undelivered_record(按 ingested_at 日期过滤),复用 compare.build_summary 渲染 KPI/柱状图/口径说明,产 output/应到未到数据.xlsx。跑比对入口(__compare__)从 compare.main() Excel 路径切换到 build_full_report。百世未到件用基数差(undelivered_pieces)与应到/已到自洽。

集成验证:前端跑比对 -> build_full_report -> /report 下载,各站未到件 顺心8/中通10/韵达2/安能0/百世8。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 21:56:39 +08:00
Misaka
8521c200ab feat(store): persist 百世 daily basis to PG (baishi_daily_stats)
百世应到/实到基数(应扫/已扫)原仅在 state_store(单值、无历史)。新增百世专用聚合表 baishi_daily_stats(site+business_date UPSERT),baishi 下载时抓到基数直接落库(store.upsert_baishi_daily_stats,一步,不绕 state_store→store)。state_store 双写保留以兼容旧 Excel 汇总(process_baishi),后续统一清理。

真机端到端验证通过:PG (2026-07-31, 194, 186, 8),state_store 一致。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 21:23:08 +08:00
Misaka
c6ad6a0ca2 fix(runtime): ingest before db_compare so first-run/force reads fresh data
DB 比对发生在入库之前,导致首次/force 时 PG 无当天数据,比对返回 None、不产出 Excel。在 _site_undelivered_handler 下载成功后、比对前,前置 _record_business_date + ingest_task,使比对能读到本次下载的数据。dispatch_task 后置 _persist_to_db 保持不变(对 undelivered 幂等重复一次,安全)。

四站真机验证通过(中通/韵达/安能 + 顺心):前置入库日志均出现在 db_compare 之前,force 首跑即产出 Excel。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 20:25:52 +08:00
Misaka_Company
541836fd1b feat(db_compare): add PostgreSQL-based comparison engine with SF handling
Replace Excel-based undelivered comparison with DB queries for all four
sites. The engine anchors on actual scan_time, reverse-lookups handover
batches, and compares expected vs actual waybill-by-waybill.

Shunxin SF waybills: use COUNT(*) instead of COUNT(DISTINCT piece_no)
since SF piece numbers are random and not derivable from the waybill.

Changes:
- db_compare.py: new module with compare_site_date(), compare_site_batch(),
  write_result_excel(), and POST /compare API endpoint
- runtime.py: switch _site_undelivered_handler from compare.write_site_file
  (Excel) to db_compare (DB); downloads succeed independently of comparison
- server.py: add POST /compare endpoint with date validation
- docs: implementation plan for Shunxin DB comparison

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 16:31:54 +08:00
Misaka_Company
95597fbb0c docs: add four-site comparison logic review report
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 14:48:24 +08:00
Misaka_Company
f66e6dd39e fix(yunda): invert handover number filter to keep empty rows
The previous filter kept rows with non-empty handover numbers
(派件/签收 scans), which were duplicate rows. The correct logic
is to keep rows with empty handover numbers (到/接件 scans).

- store.py: change != "" to == "" in ingest filter
- compare.py: add same filter before comparison (previously missing)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-31 14:44:50 +08:00
Misaka
09e05f8dfc feat(runtime): disable microphone by default to suppress permission popup
韵达等站点打开时会请求麦克风权限,触发浏览器系统级授权弹窗。在共享 context 上加
add_init_script,于每个页面/iframe 加载前覆盖 navigator.mediaDevices.getUserMedia
(及 webkit/moz 旧版) 为直接 reject(NotAllowedError: Permission disabled):站点调用时
立即被拒、不再弹出系统授权窗,且麦克风被真正挡住(非授权给它)。物流工作台无需音视频
采集,故对所有网页站点统一生效。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 20:58:46 +08:00
Misaka
047036f46b feat(yunda): cross-month calendar navigation for date-specific download
韵达两个流程的日期组件不同,分别适配跨月翻月:
- 应到(.layui-laydate 新版):读 .laydate-set-ym 当前年月,点 .laydate-prev-m/.laydate-next-m
  翻月,格子 td[lay-ymd='YYYY-M-D']。
- 实到(#laydate_box 旧版):读 #laydate_y/#laydate_m 输入框值,点 #laydate_MM 内
  .laydate_chprev/.laydate_chnext 翻月,格子 td[y][m][d]。
两处设日期改走 _yunda_pick_laydate_new / _yunda_pick_laydate_old,同月(offset 当月)
行为不变,仅跨月时翻月。年*12+月 比较天然支持跨年。

实到 #startDate/#endDate 保持普通 click(force=True 会在导航后 laydate 绑定完成前
抢先点击导致面板打不开);新增 #laydate_box:visible 就绪等待。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 20:26:45 +08:00
Misaka
21662d3944 feat(shunxin): cross-month calendar navigation for date-specific download
顺心 Ant Design 日期面板跨月导航:目标日期不在当前月视窗时,读面板头部
.ant-picker-year-btn/.ant-picker-month-btn 得当前年月,按差值点
.ant-picker-header-prev-btn/next-btn 翻到目标月再选格子。应到(车辆点到)与
实到(卸车扫描记录)两处设日期统一改走 _shunxin_pick_date。

同月(offset 当月)行为不变,仅在跨月时多走翻月;与中通 _zto_flip_to_target_month
思路对称,适配 Ant Design 面板。年*12+月 比较天然支持跨年。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 19:58:10 +08:00
Misaka_Company
4f0ef69739 fix(server): lower date backtrack limit from 90 to 31 days
Aneng only supports querying data up to 31 days back. The previous
90-day cap allowed dates that would silently fail at download time,
so restrict the POST /tasks `date` validation to a 31-day window.
2026-07-29 16:55:14 +08:00
Misaka_Company
5e2ef72afd feat(server): add date field to POST /tasks with legality validation
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:11:12 +08:00
Misaka_Company
fcb75643d0 feat(runtime): propagate date through dispatch chain and business-date snapshot
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:10:23 +08:00
Misaka_Company
cda764a305 feat(anneng): support date arg (date takes precedence over offset)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:09:30 +08:00
Misaka_Company
e51aab2e0b feat(shunxin): support date arg, propagate to both accounts
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:08:51 +08:00
Misaka_Company
bacccc43ab feat(yunda): support date arg (date takes precedence over offset)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:08:06 +08:00
Misaka_Company
e240d92ad9 feat(zto): support date arg via effective-offset (reuses cross-month nav)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:07:27 +08:00
Misaka_Company
30e9203fca feat(baishi): accept date kwarg (ignored) for unified dispatch signature
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:06:37 +08:00
Misaka_Company
93e48118bf docs: add implementation plan for date-specific download API
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 13:03:49 +08:00
20 changed files with 3544 additions and 256 deletions

1
.gitignore vendored
View File

@@ -42,6 +42,7 @@ desktop.ini
downloads/ downloads/
output/ output/
state/ state/
logs/
*.xlsx *.xlsx
*.xls *.xls
*.log *.log

View File

@@ -0,0 +1,682 @@
# 指定日期下载接口开发者Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:**`POST /tasks` 加可选 `date`YYYY-MM-DD支持顺心/中通/韵达/安能指定一个过去日期下载应到/实到数据;不传 date 时行为不变(走 offset
**Architecture:** 复用现有 `force` 透传链路加一个 `date` 字段:`TaskRequest.date``task_spec["date"]``dispatch_task``handler(ctx, force, date)` → 各站 `impl(page, force, date)`。各站把"算 target 日期"的来源从 `today - offset` 改为"有 date 用 date否则 today - offset"。中通把 date 折算成 effective offset 以复用已验证的跨月翻页。
**Tech Stack:** Python 3.10、FastAPI、Playwright、SQLitestate_store、PostgreSQLstore可选入库
## Global Constraints
- **环境**:所有 Python 用 `D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe`项目虚拟环境无需激活。PYTHONPATH 含 `InboundVerify` 根。
- **格式化/自检(每改一个 .py 必做)**`python -m black <file>`target py310+ `python -m py_compile <file>`。Black 若提示 "Python 3.10 cannot parse code formatted for 3.15",加 `--target-version py310`"left unchanged" 即合格。
- **无 pytest**:本项目无单元测试框架(见 `InboundVerify/CLAUDE.md`)。每个任务的"验证"= Black + py_compile端到端API 校验、各站实测)集中在 Task 8需重启服务加载新代码
- **submodule 工作流**:改动在 `InboundVerify` submodule`dev` 分支)。每个 Task 末尾在 submodule 内 `git add <file> && git commit`。**push 到 origin/dev + 父仓库 bump** 统一在 Task 8项目约定不自动 push等用户确认但 plan 内 commit 步骤照写)。
- **顺序安全**Task 1-5站点只给 impl 加 `date=None` 形参 + date 逻辑,**date 默认 None 时走原 offset 路径,向后兼容**Task 6runtime才把 date 从 task_spec 透传进 implTask 7server才允许 date 入队。任一 Task 完成后系统均可正常运行。
- **合法性边界**(来自 spec`date` 传了才校验——格式 `YYYY-MM-DD``今天-90 ≤ date ≤ 今天`、百世不支持 date。非法返回 HTTP 400。
---
## File Structure
| 文件 | 责任 | 本计划改动 |
| --- | --- | --- |
| `inbound_verify/sites/baishi.py` | 百世下载(固定当天) | 入口加 `date=None` 形参(忽略) |
| `inbound_verify/sites/zto.py` | 中通下载(日历格子,跨月) | 入口+impl 加 `date`date→effective offset 复用跨月 |
| `inbound_verify/sites/yunda.py` | 韵达下载(日期字符串) | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/sites/shunxin.py` | 顺心下载(双账号,日期字符串) | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/sites/anneng.py` | 安能下载CDP日期字符串 | expected/actual 入口+impl 加 `date`date→target |
| `inbound_verify/runtime.py` | 任务派发/心跳共享核心 | handler/dispatch 透传 date`_record_business_date` 用 date |
| `inbound_verify/cli/server.py` | FastAPI 服务 | `TaskRequest.date` + 合法性校验 + task_spec 透传 |
---
## Task 1: baishi.py — 入口加 date=None 形参(兼容)
**Files:**
- Modify: `inbound_verify/sites/baishi.py:140``baishi_download_undelivered_data`
**Interfaces:**
- Produces: `baishi_download_undelivered_data(page, force=False, date=None)` —— 后续 Task 6 的 `_web_handler` 会以 `download_func(pg, force=force, date=date)` 调用它,必须接受 `date` kwarg百世忽略
- [ ] **Step 1: 改签名**
`def baishi_download_undelivered_data(page, force=False):` 改为:
```python
def baishi_download_undelivered_data(page, force=False, date=None):
"""百世应到未到数据下载固定当天date 形参仅为对齐统一透传签名,忽略)。"""
```
(函数体不动;`date` 不使用。)
- [ ] **Step 2: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/sites/baishi.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
Expected: black "left unchanged" 或 reformat 后通过OK。
- [ ] **Step 3: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/baishi.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(baishi): accept date kwarg (ignored) for unified dispatch signature" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 2: zto.py — date 折算成 effective offset复用跨月
**Files:**
- Modify: `inbound_verify/sites/zto.py``zto_expected_download``zto_expected_download_impl``zto_actual_download``zto_actual_download_impl`
**Interfaces:**
- Produces: `zto_expected_download(page, force=False, date=None)` / `zto_actual_download(page, force=False, date=None)`impl 同签名。Task 6 的 `_web_handler``download_func(pg, force=force, date=date)` 调用。
- [ ] **Step 1: expected 入口加 date 并透传**
`def zto_expected_download(page, force=False):` 及其 `with_retry` 改为:
```python
def zto_expected_download(page, force=False, date=None):
"""中通:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"应到",
lambda: zto_expected_download_impl(page, force=force, date=date),
lambda: zto_reset(page),
)
```
- [ ] **Step 2: expected impl 加 datedate→effective offset**
`def zto_expected_download_impl(page, force=False):` 改签名加 `date=None`。其内"读取服务端日期偏移"段(`offset = state_store.get_offset("中通")` 与随后的 `print(...偏移...)`)改为:
```python
# 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通")
if date:
# 指定日期:折算成相对今天的有效偏移,复用下方 target_time 计算与跨月翻月
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
```
(其后的 `target_time = today_time - offset * 86400000` 与跨月翻月逻辑**不动**——date 经折算后走同一条路径。)
- [ ] **Step 3: actual 入口加 date 并透传**
`def zto_actual_download(page, force=False):` 及其 `with_retry` 改为:
```python
def zto_actual_download(page, force=False, date=None):
"""中通:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry(
"中通",
"实到",
lambda: zto_actual_download_impl(page, date=date),
lambda: zto_reset(page),
)
```
> 注:`zto_actual_download_impl` 现签名 `(page)`(无 forceactual 不去重Step 4 给它加 `date`。
- [ ] **Step 4: actual impl 加 datedate→effective offset**
`def zto_actual_download_impl(page):` 改为 `def zto_actual_download_impl(page, date=None):`。其内"# 2. 读取服务端日期偏移"段(`offset = state_store.get_offset("中通", "actual")` 与随后的 `print`)改为:
```python
# 2. 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通", "actual")
if date:
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
```
(其后 `target_time = today_time - offset * 86400000` 与跨月翻月不动。)
- [ ] **Step 5: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/sites/zto.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/zto.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(zto): support date arg via effective-offset (reuses cross-month nav)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 3: yunda.py — date→targetexpected + actual
**Files:**
- Modify: `inbound_verify/sites/yunda.py``yunda_expected_download(_impl)``yunda_actual_download(_impl)`
**Interfaces:**
- Produces: `yunda_expected_download(page, force=False, date=None)` / `yunda_actual_download(page, force=False, date=None)`impl 同加 `date=None`
- [ ] **Step 1: expected 入口透传 date**
```python
def yunda_expected_download(page, force=False, date=None):
...
return with_retry(
"韵达",
"应到",
lambda: yunda_expected_download_impl(page, force=force, date=date),
lambda: yunda_reset(page),
)
```
- [ ] **Step 2: expected impl 加 date + date→target**
`def yunda_expected_download_impl(page, force=False):``def yunda_expected_download_impl(page, force=False, date=None):`。其内日期段(`offset = state_store.get_offset("韵达")` 起几行)改为:
```python
offset = state_store.get_offset("韵达")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_ymd = f"{target.year}-{target.month}-{target.day}"
start_date_ymd = target_ymd
today_ymd = target_ymd
print(f">> 设置查询日期: [{target_ymd}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
- [ ] **Step 3: actual 入口透传 date**
```python
def yunda_actual_download(page, force=False, date=None):
...
return with_retry(
"韵达",
"实到",
lambda: yunda_actual_download_impl(page, date=date),
lambda: yunda_reset(page),
)
```
- [ ] **Step 4: actual impl 加 date + date→target**
`def yunda_actual_download_impl(page):``def yunda_actual_download_impl(page, date=None):`。其内日期段改为:
```python
offset = state_store.get_offset("韵达", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
start_date = target # 单日范围:起止同日
today = target # 让下方"截止时间"选择器也指向 target
print(
f">> 设置实到查询日期: [{target.year}-{target.month}-{target.day}]"
f"{'指定 ' + date if date else f'偏移 {offset}0=今天'}"
)
```
- [ ] **Step 5: Black + py_compile**(同 Task 2 命令,文件换 yunda.py
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/yunda.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(yunda): support date arg (date takes precedence over offset)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 4: shunxin.py — date→target双账号透传
**Files:**
- Modify: `inbound_verify/sites/shunxin.py``shunxin_expected_download(_impl)``shunxin_actual_download(_impl)`
**Interfaces:**
- Produces: `shunxin_expected_download(pages, foreground=True, force=False, date=None)` / `shunxin_actual_download(pages, foreground=True, force=False, date=None)`。Task 6 的 `_web_handler``download_func(pg, foreground=ctx.foreground, force=force, date=date)` 调用pg 是 page 列表)。
- [ ] **Step 1: expected 入口加 date 并向 impl 透传**
`def shunxin_expected_download(pages, foreground=True, force=False):` → 加 `date=None`。在函数内调用 `shunxin_expected_download_impl(page, out_tag=..., force=force)` 的位置,加上 `date=date`
```python
def shunxin_expected_download(pages, foreground=True, force=False, date=None):
...
# 对每个账号调用 impl 时透传 date
... shunxin_expected_download_impl(page, out_tag=归属地, force=force, date=date) ...
```
> 执行者:用 Grep 定位 `shunxin_expected_download_impl(page,` 的调用处(在 `shunxin_expected_download` 函数体内,对每个账号调用一次),给每处加 `date=date`。签名行只加 `date=None`其余函数体归属地读取、去重校验、merge不动。
- [ ] **Step 2: expected impl 加 date + date→target**
`def shunxin_expected_download_impl(page, out_tag="", force=False):` → 加 `date=None`。其内日期段(`offset = state_store.get_offset("顺心")` 起几行)改为:
```python
offset = state_store.get_offset("顺心")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}...")
```
- [ ] **Step 3: actual 入口加 date 并透传**
`def shunxin_actual_download(pages, foreground=True, force=False):` → 加 `date=None`;调用 `shunxin_actual_download_impl(page, out_tag=...)` 处加 `date=date`
- [ ] **Step 4: actual impl 加 date + date→target**
`def shunxin_actual_download_impl(page, out_tag=""):``def shunxin_actual_download_impl(page, out_tag="", date=None):`。其内日期段(`offset = state_store.get_offset("顺心", "actual")` 起几行)改为:
```python
offset = state_store.get_offset("顺心", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str
today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}...")
```
- [ ] **Step 5: Black + py_compile**(文件 shunxin.py
- [ ] **Step 6: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/shunxin.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(shunxin): support date arg, propagate to both accounts" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 5: anneng.py — date→targetCDPexpected + actual
**Files:**
- Modify: `inbound_verify/sites/anneng.py``anneng_expected_download(_impl)``anneng_actual_download(_impl)`
**Interfaces:**
- Produces: `anneng_expected_download(force=False, date=None)` / `anneng_actual_download(force=False, date=None)`。Task 6 的 TASK_HANDLERS 安能项以 `lambda ctx, force=False, date=None: anneng.anneng_expected_download(force=force, date=date)` 调用。
- [ ] **Step 1: expected 入口 + impl 加 datedate→target**
```python
def anneng_expected_download(force=False, date=None):
return with_retry(
"安能", "应到", lambda: anneng_expected_download_impl(force=force, date=date), anneng_reset
)
def anneng_expected_download_impl(force=False, date=None):
```
expected impl 内日期段(`offset = state_store.get_offset("安能")` 起几行)改为:
```python
offset = state_store.get_offset("安能")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = f"{target.year}-{target.month:02d}-{target.day:02d}"
start_str = target_str
today_str = target_str
print(f">> 查询日期: [{target_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
- [ ] **Step 2: actual 入口 + impl 加 datedate→target**
```python
def anneng_actual_download(force=False, date=None):
return with_retry("安能", "实到", lambda: anneng_actual_download_impl(date=date), anneng_reset)
def anneng_actual_download_impl(date=None):
```
actual impl 内日期段(`offset = state_store.get_offset("安能", "actual")` 起几行)改为:
```python
offset = state_store.get_offset("安能", "actual")
today = datetime.now()
if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
start_str = f"{target.year}\{target.month:02d}/{target.day:02d} 00:00:00"
end_str = f"{target.year}\{target.month:02d}/{target.day:02d} 23:59:59"
print(f">> 扫描日期: [{start_str}{end_str}]{'指定 ' + date if date else f'偏移 {offset}0=今天'}")
```
> actual 的 `start_str/end_str` 沿用现有 `{year}\{month}/{day}` 格式(含反斜杠,站点如此),只把 target 来源改成 date。
- [ ] **Step 3: Black + py_compile**(文件 anneng.py
- [ ] **Step 4: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/sites/anneng.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(anneng): support date arg (date takes precedence over offset)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 6: runtime.py — 全链路透传 date + _record_business_date 用 date
**Files:**
- Modify: `inbound_verify/runtime.py``_web_handler``_site_undelivered_handler``TASK_HANDLERS` 安能项、`dispatch_task``_record_business_date`
**Interfaces:**
- Consumes: Task 1-5 产出的各站 `*(..., date=None)` 签名。
- Produces: `dispatch_task``task_spec["date"]` 读 date 透传给 handler 与 `_record_business_date`handler 签名 `(ctx, force=False, date=None)`
- [ ] **Step 1: _web_handler 透传 date**
```python
def _web_handler(site, download_func):
def handler(ctx, force=False, date=None):
pg = ctx.pages_map[site]
if isinstance(pg, list):
# 顺心双账号:置顶与否交给 shunxin_download 在逐账号循环里按 foreground 决定
return download_func(pg, foreground=ctx.foreground, force=force, date=date)
if ctx.foreground:
pg.bring_to_front()
return download_func(pg, force=force, date=date)
return handler
```
- [ ] **Step 2: _site_undelivered_handler 透传 date连下 expected+actual 共用同一 date**
```python
def _site_undelivered_handler(site):
def handler(ctx, force=False, date=None):
exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force, date) is not False
act_ok = (
(TASK_HANDLERS[(site, "actual")](ctx, force, date) is not False)
if exp_ok
else False
)
if exp_ok and act_ok:
return compare.write_site_file(site)
stale = os.path.join(DOWNLOAD_DIR, SITE_UNDELIVERED_FILE.format(name=site))
if os.path.exists(stale):
os.remove(stale)
return False
return handler
```
- [ ] **Step 3: TASK_HANDLERS 安能项透传 date**
```python
("安能", "expected"): lambda ctx, force=False, date=None: anneng.anneng_expected_download(
force=force, date=date
),
("安能", "actual"): lambda ctx, force=False, date=None: anneng.anneng_actual_download(
force=force, date=date
),
```
`__compare__` 项与百世/网页项不动——百世经 `_web_handler` 已透传 datebaishi 忽略。)
- [ ] **Step 4: dispatch_task 透传 date**
`dispatch_task` 内,把 `ret = handler(ctx, bool(task_spec.get("force", False)))` 改为:
```python
ret = handler(
ctx, bool(task_spec.get("force", False)), task_spec.get("date")
)
if ret is False:
return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)")
_record_business_date(site, kind, task_spec.get("date"))
_persist_to_db(site, kind)
return (state_store.TASK_SUCCESS, None)
```
- [ ] **Step 5: _record_business_date 接受 date**
```python
def _record_business_date(site, kind, date=None):
"""下载成功后,把本次数据的业务日期快照写进状态库(供前端/报告显示「是哪天的数据」)。
有 date 用 date否则 = 下载当天 该数据对应的日期偏移。__compare__ 无数据概念,跳过。"""
if site == "__compare__":
return
today = datetime.now().date()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
def _write(k, biz_or_off):
# biz_or_off: int=offsettodayoffsetstr=已确定的业务日期date
biz = (
(today - timedelta(days=biz_or_off)).strftime("%Y-%m-%d")
if isinstance(biz_or_off, int)
else biz_or_off
)
try:
state_store.set_data_state(
site, k, ready=True, generated_at=now, business_date=biz
)
except Exception as e:
print(f">> [状态] 写业务日期失败 {site}/{k}: {e}")
def off(kind_key):
return state_store.get_offset(site, kind_key)
if kind == "expected":
_write("expected", date if date else off("expected"))
elif kind == "actual":
_write("actual", date if date else off("actual"))
elif site == "百世":
_write("undelivered", 0)
else: # 4 站 undelivered连带补写 expected/actual/undelivered 三列
_write("expected", date if date else off("expected"))
_write("actual", date if date else off("actual"))
_write("undelivered", date if date else off("expected"))
```
- [ ] **Step 6: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/runtime.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 7: 冒烟date=None 行为不变)**
服务仍跑旧代码,但 runtime 模块可独立 import 校验:
```bash
PYTHONPATH="D:/projects/LogisticsHubIPA/InboundVerify" "$PY" -c "import inbound_verify.runtime as r; import inspect; print('handler', inspect.signature(r._web_handler('中通', lambda *a, **k: None))); print('rec', inspect.signature(r._record_business_date))"
```
Expected: handler 含 `(ctx, force=False, date=None)`rec 含 `(site, kind, date=None)`
- [ ] **Step 8: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/runtime.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(runtime): propagate date through dispatch chain and business-date snapshot" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 7: server.py — TaskRequest.date + 合法性校验 + task_spec
**Files:**
- Modify: `inbound_verify/cli/server.py` — 顶部 import、`TaskRequest``create_task`
**Interfaces:**
- Produces: `POST /tasks` 接受 `{site, kind, force, date}`;合法 date 入队为 `task_spec["date"]`YYYY-MM-DD非法返回 400。
- [ ] **Step 1: 顶部 import 加 timedelta**
`from datetime import datetime``from datetime import datetime, timedelta`
- [ ] **Step 2: TaskRequest 加 date**
```python
class TaskRequest(BaseModel):
site: str
kind: str
force: bool = False
date: Optional[str] = None # YYYY-MM-DD指定则下载该日数据否则走站点 offset
```
- [ ] **Step 3: create_task 加合法性校验 + task_spec 透传 date**
```python
@app.post("/tasks")
def create_task(req: TaskRequest):
"""提交任务 {site, kind, force, date?} → 入队,返回 task_id。"""
if not worker_state["ready"]:
raise HTTPException(
status_code=409, detail="后端尚未就绪,请等待各站点登录完成后再操作"
)
if (req.site, req.kind) not in TASK_HANDLERS:
raise HTTPException(status_code=400, detail=f"无效任务: {req.site}/{req.kind}")
# 指定日期合法性校验(仅在传了 date 时)
if req.date:
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(
status_code=400, detail=f"date 不可为未来日期: {req.date}"
)
if target_date < today - timedelta(days=90):
raise HTTPException(
status_code=400, detail=f"date 超出 90 天回溯上限: {req.date}"
)
if req.site == "百世":
raise HTTPException(
status_code=400, detail="百世固定下载当天,不支持指定日期"
)
task_id = state_store.create_task(req.site, req.kind)
spec = {"site": req.site, "kind": req.kind, "force": req.force}
if req.date:
spec["date"] = req.date
task_queue.put((task_id, spec))
return {"task_id": task_id}
```
- [ ] **Step 4: Black + py_compile**
```bash
PY="D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe"
F="D:/projects/LogisticsHubIPA/InboundVerify/inbound_verify/cli/server.py"
"$PY" -m black "$F" && "$PY" -m py_compile "$F" && echo OK
```
- [ ] **Step 5: Commit**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" add inbound_verify/cli/server.py
git -C "D:/projects/LogisticsHubIPA/InboundVerify" commit -m "feat(server): add date field to POST /tasks with legality validation" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
```
---
## Task 8: 端到端验证 + 收尾push / 父 bump
**Files:** 无代码改动;验证 + 提交推送。
- [ ] **Step 1: 重启服务加载全部新代码**
停掉旧服务进程重启中通调试模式config.yaml 已是中通):
```bash
PYTHONPATH="D:/projects/LogisticsHubIPA/InboundVerify" "D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe" -m inbound_verify.cli.server
```
(后台运行;等中通登录就绪。)
- [ ] **Step 2: 合法性校验端到端API 400/202**
用 Python urllib避免 curl 中文编码问题)逐一验证,期望:
| 请求 | 期望 |
| --- | --- |
| `{site:"中通", kind:"expected", date:"2026-06-14"}` | 202 + task_id |
| `date:"2099-01-01"`(未来) | 400 "不可为未来日期" |
| `date:"2020-01-01"`(超 90 天) | 400 "超出 90 天回溯上限" |
| `date:"2026/06/14"`(格式错) | 400 "格式非法" |
| `{site:"百世", kind:"undelivered", date:"2026-06-14"}` | 400 "百世…不支持指定日期" |
| `{site:"中通", kind:"expected"}`(不传 date | 202走 offset行为不变 |
```bash
"D:/projects/LogisticsHubIPA/InboundVerify/.venv/Scripts/python.exe" - <<'PYEOF'
import json, urllib.request, urllib.error
def post(body):
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request("http://127.0.0.1:8000/tasks", data=data,
headers={"Content-Type":"application/json"}, method="POST")
try:
print(body, "->", urllib.request.urlopen(req, timeout=10).read().decode())
except urllib.error.HTTPError as e:
print(body, "->", e.code, e.read().decode())
post({"site":"中通","kind":"expected","date":"2099-01-01"})
post({"site":"中通","kind":"expected","date":"2020-01-01"})
post({"site":"中通","kind":"expected","date":"2026/06/14"})
post({"site":"百世","kind":"undelivered","date":"2026-06-14"})
PYEOF
```
- [ ] **Step 3: 中通跨月 date 实测(已知 OK**
触发 `{site:"中通", kind:"expected", date:"2026-06-14"}`,观察 worker 日志:出现 `指定日期 2026-06-14折算偏移 …)` + `偏移日期跨月,正在向前翻月导航` + 查询/下载成功。任务 `success`
- [ ] **Step 4: 顺心/韵达/安能 date 实测**
对顺心/韵达/安能各触发一个 expected `date`(取一个近 1 周内的过去日期,确保站点有数据且控件能接受)。观察日志:`指定日期 …` + 正常查询下载。**若某站日期控件不接受字符串而需日历翻月**记录现象按中通同法_zto_flip 模式)追加改动(可能产生新 Task
- [ ] **Step 5: 业务日期快照校验**
下载后 `curl -s http://127.0.0.1:8000/status`,确认对应站 `expected_business_date` == 指定 date。
- [ ] **Step 6: push submodule dev + 父仓库 bump**
```bash
git -C "D:/projects/LogisticsHubIPA/InboundVerify" push origin dev
git -C "D:/projects/LogisticsHubIPA" add InboundVerify
git -C "D:/projects/LogisticsHubIPA" commit -m "chore: bump InboundVerify submodule (date-specific download API)" -m "Co-Authored-By: Claude <noreply@anthropic.com>"
git -C "D:/projects/LogisticsHubIPA" push origin master
```
---
## Self-Reviewplan 作者自检)
1. **Spec 覆盖**接口契约→Task 7透传链路→Task 6中通 effective offset→Task 2韵达/顺心/安能 date→target→Task 3/4/5百世兼容→Task 1业务日期快照→Task 6 Step 5周期调度不动→无需 taskspec 明示合法性校验→Task 7验证→Task 8。✅ 全覆盖。
2. **占位符**:无 TBD/TODO每步含完整代码或精确命令。✅
3. **类型/签名一致**`date=None` 贯穿 server→runtime→sitesactual implyunda/shunxin/anneng原本无 force本计划只加 `date`,与 runtime 调用 `download_func(pg, force=, date=)` / 安能 `lambda(ctx,force,date)` 一致shunxin 双账号 `(pages, foreground, force, date)``_web_handler` 的 list 分支一致。✅
4. **顺序安全**Task 1-5 向后兼容date=None 走 offsetTask 6 启用透传但 server 未传 date仍 NoneTask 7 启用 date。✅

View File

@@ -0,0 +1,332 @@
# 四站点差缺对比逻辑审查报告
> 审查日期2026-07-31
> 审查范围:顺心、中通、韵达、安能 四个站点的应到 vs 实到差缺对比逻辑
> 排除:百世(站点直供未到明细,不参与四站比对)
---
## 一、比对算法总览(四站共用)
`compare.py:process()` 对四个站点执行**完全相同**的算法步骤。站点间的差异仅由 `domain.py:STATIONS` 配置注入——列名映射 + 实到单号解析器。
```
步骤1: 读应到Excel → 按运单号去重keep-first → 构建 {运单号 → (交接单号, 交接件数=n)}
步骤2: 读实到Excel → 站点专用解析器 → 构建 {运单基号 → {已到单号集合}}
步骤3: 逐运单比对
arrived_cnt >= n → 足额到货,跳过
arrived_cnt == 0 → 完全未到
0 < arrived < n → 部分未到
步骤4: 产出未到明细(交接单号 | 运单号 | 总件数 | 已到单号1 | 已到单号2 | ...
```
### 核心口径
| 指标 | 口径 |
|------|------|
| 应到件数 | **交接件数**(非录单件数);按运单号去重 keep-first |
| 实到件数 | 单号去重计数(每扫描一件=一个单号) |
| 未到件数 | max(0, 应到件数 实到件数) |
| 未到率 | 未到件数 ÷ 应到件数 |
### 未到明细输出约定
- 仅列出**短少运单**(实到 < 应到)
- 列出该运单**实际已到的单号**已到单号1, 已到单号2, ...
- **不编造缺件子单号**——实到扫描顺序号乱序,无法反推缺了哪个顺序号
### 统计指标
| 指标 | 含义 |
|------|------|
| 运单数 | 应到运单去重数 |
| 应到件 | Σ 交接件数 |
| 已到件 | Σ 实到单号去重数 |
| 未到件 | max(0, 应到件 已到件) |
| 涉及运单 | 存在短少的运单数 |
| 完全未到 | 整单零到货运单数 |
| 部分未到 | 部分缺件运单数 |
---
## 二、四站配置对照
`domain.py:STATIONS` — 所有差异集中于此配置表,比对核心代码不感知站点差异。
| 维度 | 中通 | 顺心 | 韵达 | 安能 |
|------|------|------|------|------|
| 应到文件 | `中通-应到货物数据.xlsx` | `顺心-应到货物数据.xlsx` | `韵达-应到货物数据.xlsx` | `安能-应到货物数据.xlsx` |
| 实到文件 | `中通-实到货物数据.xlsx` | `顺心-实到货物数据.xlsx` | `韵达-实到货物数据.xlsx` | `安能-实到货物数据.xlsx` |
| 应到-运单号列 | `运单号` | `运单号` | `运单号` | `运单号` |
| 应到-件数列 | `交接件数` | `交接件数` | `交接件数` | `交接件数` |
| 应到-交接单号列 | `交接单号` | `交接单号` | `交接单号` | `交接单号` |
| 实到-基号列 | —(从复合串推导) | `运单号` | **`主单号`** | **`所属单号`** |
| 实到-单号列 | `运单号`(复合串) | `子单号` | `子单号` | `扫描单号` |
| 解析器 | `arrived_pieces_zhongtong` | `arrived_pieces_by_cols` | `arrived_pieces_by_cols` | `arrived_pieces_by_cols` |
---
## 三、逐站点详细分析
### 3.1 中通ZTO
#### 业务逻辑
实到货物数据中的「运单号」为复合串,由三部分构成:
```
┌──────────┬────────────┬──────────┐
│ 运单号 │ 录单件数 │ 顺序号 │
│ (12位) │ (4位) │ (4位) │
└──────────┴────────────┴──────────┘
总长 20 位
示例: 330953527953 0001 0001
├─ 运单号 ─┤├录单┤├顺序┤
```
- **运单号(12位)**: 与应到货物数据中的运单号对齐
- **录单件数(4位)**: 该运单在系统中的录单总件数0占位
- **顺序号(4位)**: 0占位`0001`, `0002`, `0003`, `0004`
对比逻辑:
1. 从应到数据取运单号 + 交接件数(**非录单件数**
2. 从实到数据取复合串掐尾8位得运单基号完整串为子运单号
3. 按运单基号分组,子运单号去重得实到件数
4. 实到件数 < 交接件数 → 差缺
> **重要**: 录单件数仅作参考。举例:某运单录单件数=4、交接件数=2实到最多出现2条数据。如果只出现了1条我们只知道差缺了但**无法判断具体差缺了哪一件**(顺序号乱序)。
#### 代码实现
`domain.py:17-26` — 实到解析器:
```python
def arrived_pieces_zhongtong(df):
res = defaultdict(set)
for v in df["运单号"]:
v = str(v).strip()
if len(v) > 8 and v[-4:].isdigit():
res[v[:-8]].add(v) # 基号=前12位, 已到单号=完整20位复合串
return res
```
`domain.py:48-56` — 站点配置:
```python
{
"name": "中通",
"exp_qty": "交接件数", # 应到件数口径:交接件数(非录单件数)
"exp_wb": "运单号",
"exp_jd": "交接单号",
"arrived_pieces": arrived_pieces_zhongtong,
"columns": ["交接单号", "运单号", "总件数"],
}
```
#### 对齐情况:✅ 对齐
代码实现与业务逻辑一致。`v[:-8]` 掐尾8位得12位运单基号保留完整复合串作为已到单号——不解析、不推断录单件数和顺序号的具体含义。
---
### 3.2 安能Anneng
#### 业务逻辑
与中通相同的差缺对比逻辑。
安能实到数据同样为复合串,结构:`运单号(12位) + 录单件数(4位) + 顺序号(4位)`20位
与中通的关键区别:安能实到表有**独立的「所属单号」列**干净运单基号无需像中通那样从复合串掐尾8位推导基号。
#### 代码实现
`domain.py:76-86`
```python
{
"name": "安能",
"arrived_pieces": arrived_pieces_by_cols("所属单号", "扫描单号"),
...
}
```
安能使用 `arrived_pieces_by_cols` 而非 `arrived_pieces_zhongtong`——直接从「所属单号」列读基号、从「扫描单号」列读完整单号,效果等价。
| 差异点 | 中通 | 安能 |
|--------|------|------|
| 实到基号来源 | 从复合串解析(`v[:-8]` | 直接读「所属单号」列 |
| 实到单号来源 | 复合串本身(「运单号」列) | 「扫描单号」列 |
| 解析器 | `arrived_pieces_zhongtong` | `arrived_pieces_by_cols` |
| 最终产出 | `{基号 → {完整单号集合}}` | 相同 |
#### 数据库验证
```
piece_no=61003282264500140014 → waybill_no=610032822645 (12位), total=0014, seq=0014
```
#### 对齐情况:✅ 对齐
---
### 3.3 顺心Shunxin
#### 业务逻辑
顺心站点需区分两类运单:
**A. 非SF开头运单占 97%:**
实到「子单号」结构为两部分:
```
┌──────────┬──────────┐
│ 运单号 │ 顺序号 │
│ (不定长) │ (3位) │
└──────────┴──────────┘
示例: S71623721115 001
├─ 运单号 ──┤├顺序┤
注意:顺心子单号无录单件数部分(仅两部分)
```
对比时从实到取「子单号」列,按「运单号」分组,子单号去重得实到件数。
**B. SF开头运单占 3%:**
SF订单的「子单号」为**随机号码**(非由运单号衍生),不能用于差缺推导。
对比逻辑:
1. 在实到数据中按「运单号」字段查找,统计出现次数
2. 出现次数 < 交接件数 → 差缺
3. 将找到的子单号(虽随机但可以列出来)填入「已到单号」列
SF订单的差缺判定**只基于交接件数与实到运单号出现次数的比较**,不依赖子单号的结构解析。
#### 代码实现
`domain.py:57-65`
```python
{
"name": "顺心",
"arrived_pieces": arrived_pieces_by_cols("运单号", "子单号"),
}
```
**SF 与非 SF 没有任何区分处理。** 所有运单走同一条路径。
#### 数据库验证
**非SF正常:**
```
子单号=S71623721115001 → 运单号=S71623721115 + 后缀=001 ✅
子单号=S71934073996002 → 运单号=S71934073996 + 后缀=002 ✅
```
**SF异常:**
```
运单号=SF1225002296515 的两条实到记录:
子单号=SF2025318183224 (随机SF号码)
子单号=SF1225002296515 (与运单号相同)
```
数据中有 10 个SF运单存在多条实到记录。
#### 对齐情况:⚠️ 部分对齐SF特殊逻辑缺失
| 检查项 | 代码现状 | 业务要求 |
|--------|----------|----------|
| 非SF处理 | ✅ `arrived_pieces_by_cols("运单号", "子单号")` | 一致 |
| 非SF子单号结构 | ✅ 运单号 + 顺序号(两部分) | 一致 |
| SF处理 | ❌ 与非SF完全一致使用子单号去重 | **不能**使用子单号,只按运单号行数计数 |
| 功能影响 | 子单号虽随机但值唯一,按目前逻辑也能正确去重计数 | 但语义不正确——SF子单号不由运单号衍生 |
---
### 3.4 韵达Yunda
#### 业务逻辑
**去重规则:** 韵达实到数据存在重复行(同一子单号出现两次)。去重依据为「交接单号」字段:
- **保留**交接单号为**空**的行
- **丢弃**交接单号**非空**的行
**子单号结构:** 两部分——单号 + 顺序号(无录单件数部分)。
```
┌──────────┬──────────┐
│ 主单号 │ 顺序号 │
│ (不定长) │ (4位) │
└──────────┴──────────┘
示例: 713326603 0003
├─主单号─┤├顺序┤
```
**对比方式:** 与中通/安能同——按「主单号」分组,「子单号」去重得实到件数,与交接件数比对。
#### 代码实现
`store.py:316-320`(入库过滤):
```python
if site == "韵达":
# 韵达业务清洗:抛弃「交接单号」为空的行(派件/签收等其他扫描无交接单号),
# 再按子单号去重一件多扫只留一条清洗后子单号已天然唯一drop 为保险)。
df = df[df["交接单号"].astype(str).str.strip() != ""] # ← 保留非空
df = df.drop_duplicates(subset=[cm["piece"]], keep="last")
```
`domain.py:67-75`(比对配置):
```python
{
"name": "韵达",
"exp_wb": "运单号",
"arrived_pieces": arrived_pieces_by_cols("主单号", "子单号"),
}
```
#### 对齐情况:❌ 交接单号过滤逻辑完全相反
| 检查项 | 代码现状 | 业务要求 |
|--------|----------|----------|
| 交接单号过滤 | 保留 `!= ""`**非空** | 保留 `== ""`**空** |
| 子单号结构 | ✅ `7133266030003` = wb`713326603` + seq`0003` | 一致 |
| 实到解析 | ✅ `arrived_pieces_by_cols("主单号", "子单号")` | 一致 |
| compare.py 过滤 | ❌ **无过滤**,所有行参与比对 | 需要过滤 |
**影响分析:**
1. `store.py` 过滤反了——入库时留下了错误的数据集
2. `compare.py` 完全没有交接单号过滤——如果原始 Excel 中同时存在空和非空行,比对阶段会全部读入导致重复计数
3. 当前数据库中韵达 3483 条记录全部为非空交接单号——说明当前 Excel 数据中空交接单号行偏少或不存在,但这不改变逻辑错误
---
## 四、差异汇总
| # | 站点 | 问题 | 严重程度 | 影响范围 |
|---|------|------|----------|----------|
| 1 | **韵达** | 交接单号过滤反了:`!= ""` 应改为 `== ""` | ❌ 严重 | `store.py:319` + `compare.py` 需新增过滤 |
| 2 | **顺心** | SF运单无特殊处理与非SF混用子单号 | ⚠️ 中等 | `domain.py` 需新增SF判断分支 |
| 3 | **中通** | 录单件数0占位描述与实际数据完全一致 | ✅ 无影响 | 代码不依赖此区分 |
---
## 五、代码位置索引
| 逻辑 | 文件 | 行号 |
|------|------|------|
| 单站比对 `process()` | `compare.py` | 61-143 |
| 站点配置 `STATIONS` | `domain.py` | 46-87 |
| 中通实到解析器 | `domain.py` | 17-26 |
| 通用实到解析器 | `domain.py` | 29-42 |
| 单站未到文件写入 | `compare.py` | 258-273 |
| 全量汇总报告 | `compare.py` | 293-324 |
| 未到触发编排 | `runtime.py` | 478-497 |
| 韵达入库过滤(需修) | `store.py` | 316-320 |
| 顺心实到配置(需修) | `domain.py` | 57-65 |

View File

@@ -0,0 +1,264 @@
# 顺心 DB 差缺对比 — 实施计划
> 日期2026-07-31
> 目标:将顺心站点差缺对比从 Excel 读取改为 PostgreSQL 查询,并修正 SF 运单特殊处理逻辑
---
## 一、背景
### 当前状态Excel 方式)
```
compare.py:process("顺心")
├── 读 downloads/顺心-应到货物数据.xlsx
├── 读 downloads/顺心-实到货物数据.xlsx
├── arrived_pieces_by_cols("运单号", "子单号") ← SF/non-SF 无区分
└── 产出 {站}-未到数据.xlsx + 统计 dict
```
### 需要解决的两个问题
1. **从 Excel 切换到 DB**:数据已持久化到 PostgreSQL比对应直接从 DB 查询
2. **顺心 SF 运单特殊处理**SF 运单的子单号为随机号码,不能用于去重计数,应使用行计数
---
## 二、数据结构
### PostgreSQL 表
**expected_record**(关键列):
| 列 | 类型 | 说明 |
|----|------|------|
| site | TEXT | 站点 |
| waybill_no | TEXT | 运单号唯一键之一SF 以 "SF" 开头) |
| handover_no | TEXT | 交接单号(批次标识) |
| handover_pieces | INTEGER | 交接件数(应到口径) |
| order_pieces | INTEGER | 录单件数(参考) |
| business_date | DATE | 下载目标日期 |
**actual_record**(关键列):
| 列 | 类型 | 说明 |
|----|------|------|
| site | TEXT | 站点 |
| waybill_no | TEXT | 运单基号(关联 expected_record |
| piece_no | TEXT | 扫描单号non-SF运单号+顺序号SF随机号码 |
| scan_time | TIMESTAMPTZ | 扫描时间(可靠,当天数据=当天扫描) |
### SF 数据特征(已验证)
- 顺心 actual_record 中 SF 运单148 条
- `piece_no == waybill_no`86 条58%
- `piece_no != waybill_no`62 条42%)← 随机 SF 号码
- SF 运单 expected99 条,分布在 31 个交接批次中
---
## 三、算法设计
### 核心思路:以实到为锚,通过交接单号反推批次
```
输入: site="顺心", date="2026-07-25"
Step 1 — 取实到锚点
SELECT DISTINCT waybill_no FROM actual_record
WHERE site='顺心' AND scan_time::date = '2026-07-25'
Step 2 — 反推交接批次
SELECT DISTINCT handover_no FROM expected_record
WHERE site='顺心'
AND waybill_no IN (Step 1 的运单集合)
Step 3 — 展开批次全量应到
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site='顺心'
AND handover_no IN (Step 2 的交接单号集合)
Step 4 — 取批次全量实到
SELECT waybill_no, piece_no FROM actual_record
WHERE site='顺心'
AND waybill_no IN (Step 3 的运单集合)
Step 5 — 逐运单比对
for each waybill in Step 3:
if waybill_no LIKE 'SF%':
arrived_cnt = COUNT(*) ← 行计数,不去重
else:
arrived_cnt = COUNT(DISTINCT piece_no) ← 子单号去重
if arrived_cnt < handover_pieces → 差缺
```
### SF vs non-SF 处理差异
| | non-SF | SF |
|------|--------|-----|
| piece_no 含义 | 运单号 + 顺序号(可推导) | 随机 SF 号码(无推导意义) |
| 实到计数方式 | `COUNT(DISTINCT piece_no)` | `COUNT(*)`(行计数) |
| 已到单号列表 | 列出去重后的子单号 | 列出所有 piece_no含重复 |
### 统计指标
| 指标 | 公式 |
|------|------|
| 运单数 | Step 3 去重运单数 |
| 应到件 | Σ handover_pieces |
| 已到件 | Σ arrived_cnt |
| 未到件 | max(0, 应到件 已到件) |
| 涉及运单 | arrived_cnt < handover_pieces 的运单数 |
| 完全未到 | arrived_cnt = 0 的运单数 |
| 部分未到 | 0 < arrived_cnt < handover_pieces 的运单数 |
| 未到率 | 未到件 ÷ 应到件 |
### 边界情况覆盖
| 情况 | 覆盖方式 |
|------|----------|
| 同日多批次 | Step 2 查出全部涉及的 handover_no |
| 跨天到达(延迟) | Step 4 不限 scan_time历史扫描全计入 |
| 溢到(实到 > 应到) | arrived_cnt >= n 跳过,不进差缺表 |
| 完全沉默批次 | 一件未扫 = 实到无锚点,该批次不会被触发——在首次有扫描那天被纳入 |
| SF 子单号重复 | 用 COUNT(*) 而非 COUNT(DISTINCT),不会漏计 |
---
## 四、模块设计
### 新增文件
**`inbound_verify/db_compare.py`** — DB 比对引擎(纯 PostgreSQL + Python
```python
# 核心函数签名
def compare_site_date(site: str, date: str) -> CompareResult | None:
"""对指定站点和日期执行 DB 差缺比对。
返回 CompareResultstats + undelivered_rows
当天无实到数据时返回 None。
"""
def compare_site_batch(site: str, handover_no: str) -> CompareResult | None:
"""按指定交接单号执行全批次比对(不依赖实到锚点)。"""
```
**数据类型**
```python
@dataclass
class CompareResult:
stats: dict # 统计指标
rows: list[dict] # 差缺明细行
batches: list[str] # 涉及的交接批次
@dataclass
class UndeliveredRow:
handover_no: str # 交接单号
waybill_no: str # 运单号
total_pieces: int # 总件数(=交接件数)
arrived_pieces: int # 已到件数
arrived_list: list[str] # 已到单号列表
is_sf: bool # 是否 SF 运单
```
### 修改文件
**`inbound_verify/cli/server.py`** — 新增 API 端点
```python
@app.post("/compare")
def run_compare(req: CompareRequest):
"""DB 比对:{site, date} → 返回差缺结果"""
@app.get("/compare/{site}/{date}")
def get_compare(site: str, date: str):
"""查询某站点某日的差缺结果(缓存)"""
```
### 现有文件保持不动
- `compare.py` — 保留不动Excel 比对继续可用
- `domain.py` — 可能需要新增 DB 版站点配置(或复用现有)
- `runtime.py` — 暂不改动,`_site_undelivered_handler` 仍走 Excel 路径
---
## 五、实施步骤
### Phase 1 — `db_compare.py` 核心引擎
- [ ] 新建 `inbound_verify/db_compare.py`
- [ ] 实现 `compare_site_date("顺心", date)`
- [ ] SF/non-SF 分支处理
- [ ] 返回 `CompareResult`
- [ ] 终端手动验证(直接调函数,打印结果)
### Phase 2 — API 端点
- [ ]`server.py` 新增 `POST /compare`
- [ ] `CompareRequest { site, date }`
- [ ] 调用 `db_compare.compare_site_date()`
- [ ] 返回 JSONstats + undelivered rows
- [ ] HTTP 验证curl 调 `/compare` 对比不同日期结果
### Phase 3 — Excel 输出(可选)
- [ ] `db_compare` 生成 Excel 报告(复用现有 `compare.py` 的 openpyxl 样式)
- [ ] 输出到 `output/顺心-{date}-未到数据.xlsx`
- [ ] 或者只输出 JSON前端自行渲染
### Phase 4 — 替换 undelivered 任务流
- [ ] `runtime.py` 新增 `_db_undelivered_handler`
- [ ] 下载完成后不再调 Excel 比对,改调 DB 比对
- [ ] 逐步替换 `TASK_HANDLERS` 中的顺心 undelivered handler
### Phase 5 — 扩展到中通/韵达/安能
- [ ] 各站适配(主要是 piece_no 去重方式差异)
- [ ] 中通:`COUNT(DISTINCT piece_no)`,无 SF 问题
- [ ] 韵达:同上
- [ ] 安能:同上
---
## 六、测试策略
### 手工验证Phase 1
```python
# 终端直接调
from inbound_verify.db_compare import compare_site_date
result = compare_site_date("顺心", "2026-07-25")
print(result.stats)
# 对比基于 Excel 版的 compare.process("顺心") 结果
```
### API 验证Phase 2
```bash
curl -X POST http://127.0.0.1:8000/compare \
-H "Content-Type: application/json" \
-d '{"site":"顺心","date":"2026-07-25"}'
```
### 回归验证
- 新 DB 比对结果 vs 旧 Excel 比对结果(同一份数据)
- SF 运单的 arrived_cnt 对比DB 版COUNT(*)vs Excel 版COUNT DISTINCT piece_no
- 确认 SF 运单不再被漏计
---
## 七、风险与注意事项
| 风险 | 缓解 |
|------|------|
| DB 连接超时cpolar 隧道) | 加 connect_timeout + try/except 降级 |
| 全表扫描性能 | 依赖 (site, waybill_no) 和 (site, scan_time) 索引 |
| SF 运单数据量小(~1% | 测试覆盖可能不足——需找有 SF 差缺的日期验证 |
| `scan_time` 时区 | 统一用 `::date` cast确认与服务器时区一致 |

View File

@@ -0,0 +1,232 @@
# 未到(差缺)统计逻辑现状梳理
> 梳理日期2026-08-02
> 分支:`refactor/undelivered-by-expected`
> 范围:顺心、中通、韵达、安能四站的"应到 / 实到 / 差缺"统计链路(百世为站点直供未到,单独处理)
> 目的:固定当前真实逻辑,为"改为基于应到数据统计差缺"的重构提供基线
---
## 一、系统总览与数据流
系统分三层:
1. **下载层**`inbound_verify/sites/*.py`):用 Playwright / CDP 登录各物流站点按目标日期导出「应到货物数据」「实到货物数据」Excel 到 `downloads/`
2. **持久化层**`store.py` + PostgreSQL把 Excel 幂等 UPSERT 进 `expected_record`(应到,运单级)/ `actual_record`(实到,扫描件级)/ `undelivered_record`(百世直供未到,子单级)/ `baishi_daily_stats`(百世日聚合基数)。
3. **统计层**`db_compare.py`):直接查 PostgreSQL以**实到扫描日期为锚点**反推交接批次,展开批次全量应到后逐运单比对,产出差缺统计与明细,落盘 `output/` 下 Excel供前端下载。
```mermaid
flowchart LR
subgraph 下载层["下载层 sites/*.py"]
A1[定时任务<br/>fetch_schedule] --> D[Playwright/CDP 登录站点]
A2[手动任务<br/>POST /tasks] --> D
D --> E1[应到货物数据.xlsx]
D --> E2[实到货物数据.xlsx]
end
subgraph 持久化层["持久化层 store.py + PG"]
E1 --> F[ingest_task<br/>expected_record]
E2 --> G[ingest_task<br/>actual_record]
end
subgraph 统计层["统计层 db_compare.py"]
F --> H[compare_site_date<br/>实到锚点→反推批次→展开应到]
G --> H
H --> I[output/*-未到数据.xlsx]
H --> J[output/应到未到数据.xlsx<br/>全站汇总]
end
I --> K[前端下载]
J --> K
```
---
## 二、数据模型
### 2.1 PostgreSQL 业务表schema.sql
| 表 | 粒度 | 业务唯一键 | 关键列 |
|----|------|-----------|--------|
| `expected_record` | 运单级,一运单一行 | `(site, waybill_no)` | `handover_no``handover_pieces`(交接件数=应到口径)、`order_pieces`(录单件数)、`business_date`(属性,非唯一键)、`raw` JSONB |
| `actual_record` | 扫描件级,一扫描一行 | `(site, piece_no)` | `waybill_no`(运单基号)、`piece_no`(扫描/子单号)、`scan_time``scan_site` |
| `undelivered_record` | 百世直供未到明细,子单级 | `(site, piece_no)` | `biz_type``last_scan` |
| `baishi_daily_stats` | 百世站级日聚合 | `(site, business_date)` | `expected_pieces`(应扫)、`arrived_pieces`(已扫)、`undelivered_pieces`(未扫) |
> 索引:`expected_record` 有 `(site, business_date)` 与 `(site, handover_no)` 索引;`actual_record` 有 `(site, waybill_no)` 与 `(scan_time)` 索引——差缺比对的主查询路径均命中。
### 2.2 SQLite 状态库state.db
- `site_config`:各站 `expected_offset` / `actual_offset`0=今天,最大回溯 30 天)。
- `fetch_schedule`:周期抓取开关、激活时段、间隔(分钟)。
- `task_history`:任务记录(`trigger``target_date``force`)。
- `ingest_state`:最近一次入库健康状态(`ok` / `count` / `error`)。
- `site_status`:登录态 + 各 kind 的 `*_ready``*_business_date`(由 PG 派生,见 §4.4)。
---
## 三、当前差缺统计算法(核心)
### 3.1 一句话概括
> **以目标日期的实到扫描为锚点 → 反推这些运单所属的交接批次 → 展开批次全量应到 → 逐运单比对差缺。**
即:**先有实到,再找应到**。这是本次重构要推翻的核心假设。
### 3.2 算法步骤(`db_compare.compare_site_date`
```mermaid
flowchart TD
S[compare_site_date site, target_date] --> S1
S1["Step1 实到锚点<br/>SELECT DISTINCT waybill_no<br/>FROM actual_record<br/>WHERE site=? AND scan_time::date = target_date"]
S1 -->|当天无实到| X[返回 None<br/>“当天无实到数据,无法比对”]
S1 -->|有实到运单| S2
S2["Step2 反推交接批次<br/>SELECT DISTINCT handover_no<br/>FROM expected_record<br/>WHERE waybill_no = ANY(锚点运单)"]
S2 --> S3
S3["Step3 展开批次全量应到<br/>SELECT waybill_no, handover_no, handover_pieces<br/>FROM expected_record<br/>WHERE handover_no = ANY(批次)"]
S3 -->|无应到| X2[返回 None]
S3 --> S4
S4["Step4 取批次全量实到<br/>SELECT waybill_no, piece_no<br/>FROM actual_record<br/>WHERE waybill_no = ANY(展开的全部运单)"]
S4 --> S5
S5["Step5 逐运单比对 _do_compare"]
S5 --> R[CompareResult<br/>stats + 差缺明细 rows]
```
### 3.3 逐运单比对口径(`_do_compare`
对展开出的每一条应到运单:
| 判断 | 结论 |
|------|------|
| `handover_pieces <= 0` | 跳过,不计入应到 |
| 实到件数 `>=` 应到件数(`handover_pieces` | **足额到货**(含溢到),不进差缺 |
| 实到件数 `== 0` | **完全未到**`full_miss++` |
| `0 <` 实到件数 `<` 应到件数 | **部分未到**`part_miss++` |
实到件数口径SF / 非 SF 分支):
- **非 SF中通/韵达/安能)**`COUNT(DISTINCT piece_no)`,子单号去重。
- **顺心 SF 运单**`waybill_no``SF` 开头):`COUNT(*)` 行计数不去重SF 子单号为随机号码,不能去重计数)。
### 3.4 统计指标定义(`CompareStats`
| 指标 | 定义 |
|------|------|
| `waybill_count` | 应到运单数(`handover_pieces>0` 的展开运单) |
| `expected_pieces` | Σ `handover_pieces`(交接件数,非录单件数) |
| `arrived_pieces` | Σ 各运单实到件数(按上节口径) |
| `undelivered_pieces` | `max(0, expected_pieces arrived_pieces)` |
| `undelivered_wb` | `full_miss + part_miss`(差缺运单数) |
| `full_miss` / `part_miss` | 完全未到 / 部分未到运单数 |
| `sf_wb_count` / `sf_undelivered` | 顺心 SF 运单总数 / 其中差缺数 |
### 3.5 差缺明细(`UndeliveredRow`
仅含短少运单:`交接单号 | 运单号 | 总件数(交接件数) | 已到单号1 | 已到单号2 | ...`。已到单号按实到记录顺序列出,**不编造缺件子单号**(扫描顺序号乱序,无法反推缺了哪个)。
### 3.6 另一个入口:按批次比对(`compare_site_batch`
已知交接单号时可直接按 `handover_no` 精确比对,**不依赖实到锚点**。展开该批次全量应到 → 取全量实到 → 走同一 `_do_compare`。此入口当前未接入任务链路,主要用于调试/复核。
---
## 四、任务触发与执行链路
### 4.1 任务种类
| kind | 含义 | 四站行为 | 百世行为 |
|------|------|---------|---------|
| `expected` | 应到下载 | 下载应到 Excel → 入库 | 不支持 |
| `actual` | 实到下载 | 下载实到 Excel → 入库 | 不支持 |
| `undelivered` | 未到(差缺) | 应到+实到 → 入库 → DB 比对 → 写单站未到 Excel | 站点直供未到 → 入库 |
| `compare``__compare__` | 全站跑比对 | 4 站 DB 比对 + 百世 PG → 全站汇总 Excel | 同上 |
### 4.2 触发方式
- **定时**APScheduler `IntervalTrigger``fetch_schedule` 配置周期投递,激活时段内才投;任务空闲才投(`create_task_if_idle`)。
- **手动**`POST /tasks {site, kind, force?, date?}`,前端「获取未到数据」主按钮触发各站 primary kind前端「跑比对」按钮触发 `__compare__/compare`
任务执行统一走 `dispatch_task``runtime.py`):登录态校验 → 调 handler → 成功后写业务日期(`_record_business_date`)→ 入库(`_persist_to_db`)。
### 4.3 目标日期target_date怎么定
`state_store.resolve_target_date(site, kind, date)`
- 显式传 `date` → 用之(前端重试按钮会回放原 `target_date`)。
- 未传 → `today offset``expected``expected_offset``actual``actual_offset`、四站 `undelivered` 跟随 `expected_offset`、百世恒当天。
### 4.4 就绪态与业务日期ready / business_date
- `_ready_flags`:直接查 PG`expected` / `actual` 看对应 `today offset` 日期是否有数据;四站 `undelivered = expected ∧ actual`;百世看 `baishi_daily_stats` 当天。
- `_record_business_date`:任务成功后在 SQLite 快照本次数据日期,供前端状态盘与报表「数据日期」列展示。
- **注意**`expected_business_date` / `actual_business_date` 是**按偏移派生的目标日期**,不是下载文件内的真实业务日期——`expected_record.business_date` 入库时取自状态库的 `expected_business_date``store._read_business_dates`),可能与交接单实际生成日不同(详见 §6 疑点)。
---
## 五、站点差异与配置
### 5.1 比对配置(`db_compare.SITE_COMPARE_CONFIG`
| 站点 | has_sf | 备注 |
|------|--------|------|
| 顺心 | `True` | SF 运单走行计数 |
| 中通 / 韵达 / 安能 | `False` | 子单号去重计数 |
### 5.2 入库列映射(`store.ACTUAL_COLMAP` 与 `domain.STATIONS`
| 站点 | 应到运单号列 | 实到运单基号列 | 实到单件列 | 扫描时间列 |
|------|-------------|---------------|-----------|-----------|
| 顺心 | 运单号 | 运单号 | 子单号 | 操作时间 |
| 中通 | 运单号 | 由子单号复合串 `v[:-8]` 推导 | 运单号(复合串) | 扫描时间 |
| 韵达 | 运单号 | 主单号 | 子单号 | 扫描时间 |
| 安能 | 运单号 | 所属单号 | 扫描单号 | 扫描时间 |
### 5.3 入库清洗特例
- **韵达实到**:只保留「交接单号为空」的行(到/接件扫描),丢弃非空行(派件/签收等重复数据);再按子单号去重 keep-last。
- **中通实到**`piece_no` 为复合串 `H+运单号(12)+总数(4)+顺序(4)`,基号 = `v[:-8]`,每串计一件。
---
## 六、当前逻辑的特征与疑点(重构输入)
### 6.1 特征
1. **锚点=实到扫描日**:某天实到为空 → 比对直接返回 None不产出任何差缺"先有实到才有结论")。
2. **批次是反推出来的**:只要当天有 1 个运单扫到,其所属整个交接批次都会被展开,把该批次的历史未到也一起统计进来(跨日旧账混入当天报表)。
3. **应到=批次全量**`expected_pieces` 是"被命中批次的全部应到",不是"当天的应到交接单"`business_date` 标签也可能滞后/漂移。
4. **溢到不抵消**`undelivered_pieces = max(0, Σ应到 Σ实到)` 是全局差;单运单溢到(实到>应到)只会让该运单不进差缺,不会抵消他单未到。
### 6.2 疑点(来自 08-02 韵达实测)
- 08-02 实到 130 条/58 运单全部命中应到且件数一致,但差缺 9 单里有 7 单是 07-21 批次的历史未到——因为当天有 1 个运单(`295468511`)属于该批次被扫到,整批被展开。
- `task_history` 无记录却有 18:28 入库 130 条实到:入库链路与常规任务链路不一致(`ingest_state` 也未刷新),说明存在绕过任务系统的直入路径,需在重构时统一入口。
---
## 七、重构目标对照(待细化)
| 维度 | 现状(实到驱动) | 重构方向(应到驱动) |
|------|----------------|--------------------|
| 统计起点 | 目标日期实到扫描运单 | 目标日期/批次的应到数据(`expected_record` |
| 批次来源 | 实到锚点反推 | 应到自身携带的 `handover_no` / `business_date` |
| 无数据表现 | 实到空 → 不产出 | 应到空 → 明确"无应到";应到有、实到空 → 全部记差缺 |
| 历史批次混入 | 会1 单命中即整批展开) | 应到锚定,天然按应到口径隔离 |
| 报表口径 | 批次全量 | 按应到日期/批次统计 |
> 重构后仍需保持:应到=交接件数、实到=子单号去重SF 行计数)、未到=`max(0, 应到−实到)`、差缺明细含已到单号。
---
## 八、涉及文件清单
| 文件 | 职责 |
|------|------|
| `inbound_verify/db_compare.py` | DB 差缺比对引擎 + 全站汇总(重构主战场) |
| `inbound_verify/runtime.py` | 任务派发、未到 handler、入库钩子、就绪派生 |
| `inbound_verify/store.py` | Excel → PG 入库expected/actual/百世) |
| `inbound_verify/state_store.py` | SQLite 状态/配置/任务/偏移/目标日期 |
| `inbound_verify/domain.py` | 站点/文件/列映射单一配置源 |
| `inbound_verify/schema.sql` | PG 表结构 |
| `inbound_verify/cli/server.py` | FastAPI 端点(/tasks /compare /status /config /report |
| `dashboard/app/page.tsx` | 前端触发任务、状态盘、下载报表 |

View File

@@ -0,0 +1,179 @@
# 应到驱动差缺统计重构 · 设计方案
> 日期2026-08-03
> 分支:`refactor/undelivered-by-expected`
> 状态:待审核
> 目标:差缺统计从"实到驱动(反推应到)"改为"应到驱动(以应到为统计起点)",并用「出库日」作为批次归属日,吸收"提前提交"扰动
---
## 一、背景与问题
### 1.1 业务诉求
- 业务部门日常看的是**应到货物数据**:要"应到了哪些、实到了哪些、差缺是什么"。
- 当前实现是**实到驱动**:以目标日实到扫描为锚点 → 反推交接批次 → 展开批次全量应到 → 逐单比对。
- 方向与业务诉求相反,需重构为**应到驱动**。
### 1.2 核心扰动:应到任务可能被提前提交
- 应到数据理论当天提交(韵达固定提前 1 天),但实际可能提前 1~2 天。
- 若完全按"下载日"统计应到,会把"提前提交、货次日才到"的批次计入当天,产生假差缺。
### 1.3 数据实证7 月全量)
`downloads/archive/` 7 月应到文件 × PG 实到数据交叉验证:
| 站点 | 下载日=出库大头日 | 出库大头日=下载日+1 |
|------|-----------------|-------------------|
| 顺心 | 83/83100% | 0 |
| 中通 | 28/3093% | 2`071402``071902` |
| 韵达 | 0 | 29/29100%,固定提前) |
| 安能 | 31/31100% | 0 |
**关键结论**
1. **出库大头日 ≈ 实到峰值日**154/158 一致97.5%)——"出库日"基本等于"这批货实际到的那天"。
2. "提前提交"的批次(中通 `071402` 出库 7/15、`071902` 出库 7/20在**出库时间上如实体现**了真实归属日。
3. 四站应到文件的「出库时间」字段非空率 100%,且已完整保留在 PG `expected_record.raw` JSONB 中(四站 100% 有值)。
---
## 二、核心口径
### 2.1 批次归属日 = 出库大头日
一个交接批次内,取运单「出库时间」的**日期众数(大头日)**作为该批次归属日:
```
批次归属日 = mode(运单.出库时间::date)
```
- 顺心/安能:归属日 = 下载日(无扰动)
- 中通:偶发提前批次自动归属次日(`071402` → 7/15
- 韵达:所有批次归属日 = 下载日 + 1与实到对齐不再依赖 `expected_offset=1`
### 2.2 统计 D 日差缺 = 取所有「出库日 = D」的应到批次
无论批次在 D / D-1 / D-2 哪天下载(`business_date` 为何),只要**出库日 = D** 即纳入 D 日统计:
```
目标批次 = expected_record WHERE site=? AND 出库日 = D
```
这样:
- 提前提交的批次(下载于 D-1/D-2、出库于 D会被**自然归入 D 日**,不再遗漏也不提前计入;
- 不再需要"实到为 0 → 抛弃/标记留存"的状态机;
- 不再需要为韵达单独配置 `expected_offset`
---
## 三、统计流程compare_site_date 重构后)
```mermaid
flowchart TD
S[查询 D 日差缺] --> S1
S1["Step1 取应到批次<br/>expected_record<br/>WHERE site=? AND 出库日 = D"]
S1 -->|无应到| X[返回:明确当日无应到]
S1 -->|有批次| S2
S2["Step2 展开批次全量应到<br/>waybill_no, handover_no, handover_pieces"]
S2 --> S3
S3["Step3 查这些运单的全量实到<br/>actual_record WHERE waybill_no = ANY(应到)"]
S3 --> S4
S4["Step4 逐运单比对_do_compare<br/>应到=交接件数 实到=子单号去重/SF行计数"]
S4 --> R[CompareResult<br/>stats + 差缺明细]
```
### 3.1 与现实现的差异
| 环节 | 现状 | 重构后 |
|------|------|--------|
| 应到来源 | 实到锚点反推批次 | 按出库日直接取应到批次 |
| 无实到表现 | 返回 None不产出 | 应到空 → 明确"无应到";应到有实到空 → 记差缺 |
| 历史批次混入 | 1 单命中即整批展开 | 按出库日隔离,天然干净 |
| 提前提交 | 无感知(靠实到锚定) | 出库日归属,自动吸收 |
### 3.2 保留的能力
- `compare_site_batch`(按交接单号精确比对)保留,供复核。
- 实到驱动入口 `compare_site_date` 旧逻辑保留为对照模式(或通过配置切换),便于回溯验证差异。
- 统计口径不变:应到=交接件数、实到=子单号去重(顺心 SF 行计数)、未到=`max(0,应到−实到)`、明细含已到单号。
---
## 四、数据层改造
### 4.1 新增列:`out_date`
`expected_record` 新增 `out_date DATE`(出库日,批次归属日的持久化依据):
```sql
ALTER TABLE expected_record ADD COLUMN IF NOT EXISTS out_date DATE;
CREATE INDEX IF NOT EXISTS idx_expected_out_date ON expected_record (site, out_date);
```
- 入库时(`store._ingest_expected`):从 raw 的「出库时间」解析出日期写入 `out_date`
- 历史数据回填:一次性 UPDATE`raw->>'出库时间'` 提取日期。
- 出库时间缺失/解析失败 → `out_date` 置 NULL统计时回退 `business_date`(下载日),保证不丢数据。
### 4.2 `business_date` 语义保持不变
- `business_date` 继续表示"下载目标日快照"(兼容就绪态派生 / 报表数据日期列 / 现有 API
- 差缺统计改用 `out_date`,两者解耦,避免连锁改动。
### 4.3 出库时间字段来源(已验证)
| 站点 | 字段 | 覆盖率 |
|------|------|--------|
| 顺心 | `出库时间` | raw 100% |
| 中通 | `出库时间` | raw 100% |
| 韵达 | `出库时间` | raw 100% |
| 安能 | `出库时间` | raw 100% |
---
## 五、边界与特殊处理
| 场景 | 处理 |
|------|------|
| 批次内出库日跨多天 | 取**大头日(众数)**;众数并列时取较早日期 |
| 出库时间缺失/解析失败 | `out_date` 置 NULL回退 `business_date` |
| 应到有、实到空 | 全部计入差缺(不再因"无实到"而返回 None |
| 实到有、应到无(孤儿) | 保持现状,报表/明细可另行提示,不混入应到统计 |
| 韵达 `expected_offset` | 保留配置但重构后不再参与归属日计算(由 `out_date` 取代) |
| 异常小批次 | 规模很小1~4 单)按常规逻辑走;如出现系统性偏差再单独讨论 |
---
## 六、涉及改动清单
| 文件 | 改动 |
|------|------|
| `schema.sql` | `expected_record``out_date` 列 + 索引 |
| `store.py` | `_ingest_expected``out_date`新增历史回填逻辑CLI |
| `db_compare.py` | `compare_site_date` 改为按 `out_date` 取应到;新增"无应到"返回语义;保留批次入口与实到驱动对照 |
| `runtime.py` | `_site_undelivered_handler` 锚点日期逻辑随新口径调整 |
| `cli/server.py` | `/compare` 响应补充 `out_date` 语义说明;行为兼容 |
| `state_store.py` | 视需要暴露 `out_date` 相关查询 |
| 前端 `dashboard` | 报表说明文案(批次归属=出库日);无结构变更预期 |
---
## 七、验证计划
1. **单元验证**`out_date` 回填后,抽样核对与归档 Excel 出库日一致。
2. **回溯对照**:用 7 月归档应到 + PG 实到,分别跑"旧实到驱动"与"新应到驱动",对比差缺差异,重点:
- 韵达 7 月各日(应到归属日整体 +1 是否对齐实到)
- 中通 7/14、7/19`071402`/`071902` 是否归入次日)
- 顺心/安能(应无差异)
3. **报表烟测**:跑一次 `__compare__/compare` 全站汇总,人工核对韵达 08-02 数据。
---
## 八、决策记录(已确认)
- ✅ 出库时间字段业务含义 = **货物实际发出时间**(按此处理)。
- ✅ 批次内出库日并列众数取法 = **取较早日期**
- ✅ "无应到"呈现 = 报表中**直接写**(如实呈现,无需特殊文案)。
-**保留实到驱动入口**,长期作为对照(不删除)。

View File

@@ -0,0 +1,120 @@
# 应到驱动差缺统计重构 · 实现总结(备忘录)
> 完成日期2026-08-03
> 分支:`refactor/undelivered-by-expected`
> 关联文档:`docs/2026-08-02-未到差缺统计逻辑现状梳理.md`、`docs/2026-08-03-应到驱动差缺统计重构-design.md`
> 状态:已完成并验证,待提交
---
## 一、背景
原差缺统计为**实到驱动**:以目标日实到扫描为锚点 → 反推交接批次 → 展开批次全量应到 → 逐单比对。
业务部门日常以**应到数据**为依据,且应到任务可能被**提前提交 1~2 天**(韵达固定提前 1 天),导致:
- 完全按应到统计会出现假差缺;
- 实到驱动会把历史批次混入当天报表1 单命中即整批展开)。
重构目标:改为**应到驱动**,并用「出库日」作为批次归属日,自动吸收提前提交。
## 二、核心口径
- **批次归属日 = 批次内运单「出库时间」日期众数,并列取较早**(字段:`expected_record.batch_out_date`)。
- **统计 D 日差缺 = 取所有 `batch_out_date = D` 的应到批次**,展开全量应到 → 查全量实到 → 逐运单比对。
- 无论批次在 D / D-1 / D-2 哪天下载(`business_date` 为何),只要出库日 = D 即纳入 D 日统计。
## 三、数据实证7 月全量)
| 站点 | 下载日=出库大头日 | 出库大头日=下载日+1 |
|------|-----------------|-------------------|
| 顺心 | 83/83100% | 0 |
| 中通 | 28/3093% | 2`071402``071902` |
| 韵达 | 0 | 29/29100%,固定提前) |
| 安能 | 31/31100% | 0 |
关键结论:
- **出库大头日 ≈ 实到峰值日**154/158 一致97.5%)——"出库日"基本等于"这批货实际到的那天"。
- "提前提交"的批次(中通 `071402` 出库 7/15、`071902` 出库 7/20在出库时间上如实体现真实归属日。
- 四站应到文件的「出库时间」字段非空率 100%,且已完整保留在 PG `expected_record.raw` JSONB 中。
## 四、代码改动
### 4.1 数据层
- `schema.sql``expected_record` 新增 `out_date DATE`(运单出库日)、`batch_out_date DATE`(批次归属日)+ `idx_expected_out_date` / `idx_expected_batch_out_date` 索引。
- `store.py`
- `_ingest_expected`:解析「出库时间」写 `out_date`;按交接单号聚合出库日众数(并列取较早)写 `batch_out_date`
- 新增 `_batch_out_date_map()` 辅助函数。
- 新增 `backfill_out_date()` + CLI 子命令 `backfill-out-date`,历史数据一次性回填。
### 4.2 比对层
- `db_compare.py`
- 新增 `compare_site_outdate(site, target_date)`:应到驱动入口,`WHERE batch_out_date = target_date` 取批次 → 展开 → 比对。
- 保留 `compare_site_date()`(实到驱动)作对照,不删除。
- `_target_date_for()` 改为默认取今天(不再依赖 actual_offset
- `build_full_report()` 改用 `compare_site_outdate`
- `runtime.py``_site_undelivered_handler` 切到应到驱动,锚点日期默认今天。
- `cli/server.py``POST /compare` 切到应到驱动。
## 五、实施与验证
### 5.1 数据迁移
```
python -m inbound_verify.store init # 建表/补列(幂等)
python -m inbound_verify.store backfill-out-date # 历史回填
```
回填结果:
- `out_date`:顺心 3290 / 中通 5419 / 韵达 1640 / 安能 3459 条,共 13808 条。
- `batch_out_date`:顺心 93 / 中通 35 / 韵达 32 / 安能 35 个批次。
- 抽样核对 PG `out_date` vs 归档 Excel 出库日一致率 96~100%。
### 5.2 批次归属验证
| 批次 | 下载日 | batch_out_date | 预期 |
|------|--------|----------------|------|
| 中通 `...071401` | 7/14 | 7/14 | 正常 |
| 中通 `...071402` | 7/14 | **7/15** | 提前提交归位 |
| 中通 `...071901` | 7/19 | 7/19 | 正常 |
| 中通 `...071902` | 7/19 | **7/20** | 提前提交归位 |
| 韵达 `...07312001` | 7/31 | 8/1 | 固定 +1 |
| 顺心/安能 | — | = 下载日 | 无扰动 |
### 5.3 回溯对照7/02~7/31
新应到驱动 vs 旧实到驱动,差异方向符合设计:
- 旧驱动混入历史批次(如中通 7/12 旧 236 件 vs 新 11 件;顺心 7/13 旧 59 vs 新 1
- 新驱动只统计出库日=当天批次,数字更聚焦。
- 个别日期新驱动未到偏大(如韵达 7/03、安能 7/29属"当天出库、次日扫描"的真实差缺口径。
### 5.4 接口联调(真实后端)
| 用例 | 结果 |
|------|------|
| `POST /compare` 韵达 2026-08-02 | 批次 1 个(`...08012001`),差缺 2 件/2 单(`988350756``988415586`),历史批次不再混入 |
| `POST /compare` 中通 2026-07-15 | 提前提交批次 `...071402` 正确归位到 7/15 |
| `__compare__/compare` 全站汇总 2026-08-01 | 顺心 17 件 / 中通 23 件 / 韵达 0 件 / 安能 0 件,合计 40 件,报表正常生成 |
## 六、待确认 / 遗留事项
- `out_date` / `batch_out_date` 依赖站点「出库时间」字段语义(当前按"货物实际发出时间"处理,已与业务确认)。
- 批次内出库日并列众数取较早(已确认)。
- "无应到"时报表直接写(如实呈现,无特殊文案,已确认)。
- 实到驱动入口保留作对照(已确认)。
- `docs/2026-08-02-未到统计重构讨论纪要与下一步.md` 中记录的 18:28 直入入库等链路疑点,本重构未处理,留待后续。
## 七、附:涉及文件
| 文件 | 说明 |
|------|------|
| `schema.sql` | 表结构:新增 `out_date` / `batch_out_date` |
| `inbound_verify/store.py` | 入库解析 + 历史回填 |
| `inbound_verify/db_compare.py` | 应到驱动比对入口(保留实到驱动对照) |
| `inbound_verify/runtime.py` | 未到任务切到应到驱动 |
| `inbound_verify/cli/server.py` | `/compare` API 切到应到驱动 |

View File

@@ -17,7 +17,7 @@ import queue
import threading import threading
import time import time
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from datetime import datetime from datetime import datetime, timedelta
from typing import Dict, Optional from typing import Dict, Optional
import uvicorn import uvicorn
@@ -27,7 +27,7 @@ from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse from fastapi.responses import FileResponse
from pydantic import BaseModel from pydantic import BaseModel
from inbound_verify.paths import DOWNLOAD_DIR, OUTPUT_DIR from inbound_verify.paths import OUTPUT_DIR
from inbound_verify import state_store from inbound_verify import state_store
from inbound_verify.runtime import ( from inbound_verify.runtime import (
HEARTBEAT_INTERVAL, HEARTBEAT_INTERVAL,
@@ -36,6 +36,7 @@ from inbound_verify.runtime import (
launch_and_prepare, launch_and_prepare,
run_heartbeat, run_heartbeat,
) )
from inbound_verify import db_compare
# 全部站点;百世固定下载当天,不可配置偏移 # 全部站点;百世固定下载当天,不可配置偏移
ALL_SITES = ["顺心", "百世", "中通", "韵达", "安能"] ALL_SITES = ["顺心", "百世", "中通", "韵达", "安能"]
@@ -127,7 +128,10 @@ def _enqueue_fetch(site, kind):
if not _in_active_window(cfg["active_start"], cfg["active_end"]): if not _in_active_window(cfg["active_start"], cfg["active_end"]):
return # 不在激活时段,跳过本次 fire return # 不在激活时段,跳过本次 fire
try: try:
tid = state_store.create_task_if_idle(site, kind) target_date = state_store.resolve_target_date(site, kind)
tid = state_store.create_task_if_idle(
site, kind, trigger="auto", target_date=target_date
)
if tid is None: if tid is None:
return # 上一次同类任务还没跑完,跳过避免堆积 return # 上一次同类任务还没跑完,跳过避免堆积
task_queue.put((tid, {"site": site, "kind": kind})) task_queue.put((tid, {"site": site, "kind": kind}))
@@ -190,11 +194,12 @@ class TaskRequest(BaseModel):
force: bool = ( force: bool = (
False # 强制重下:忽略已落库去重,重新提交所有班次/交接单的导出任务(默认关) False # 强制重下:忽略已落库去重,重新提交所有班次/交接单的导出任务(默认关)
) )
date: Optional[str] = None # YYYY-MM-DD指定则下载该日数据否则走站点 offset
@app.post("/tasks") @app.post("/tasks")
def create_task(req: TaskRequest): def create_task(req: TaskRequest):
"""提交任务 {site, kind} → 入队,返回 task_id。""" """提交任务 {site, kind, force, date?} → 入队,返回 task_id。"""
# 【P0】后端未就绪时直接拒绝避免任务在 worker 启动前入队卡死 # 【P0】后端未就绪时直接拒绝避免任务在 worker 启动前入队卡死
if not worker_state["ready"]: if not worker_state["ready"]:
raise HTTPException( raise HTTPException(
@@ -202,8 +207,38 @@ def create_task(req: TaskRequest):
) )
if (req.site, req.kind) not in TASK_HANDLERS: if (req.site, req.kind) not in TASK_HANDLERS:
raise HTTPException(status_code=400, detail=f"无效任务: {req.site}/{req.kind}") raise HTTPException(status_code=400, detail=f"无效任务: {req.site}/{req.kind}")
task_id = state_store.create_task(req.site, req.kind) # 指定日期合法性校验(仅在传了 date 时)
task_queue.put((task_id, {"site": req.site, "kind": req.kind, "force": req.force})) if req.date:
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(
status_code=400, detail=f"date 不可为未来日期: {req.date}"
)
if target_date < today - timedelta(days=31):
raise HTTPException(
status_code=400, detail=f"date 超出 31 天回溯上限: {req.date}"
)
if req.site == "百世":
raise HTTPException(
status_code=400, detail="百世固定下载当天,不支持指定日期"
)
task_id = state_store.create_task(
req.site,
req.kind,
trigger="manual",
target_date=state_store.resolve_target_date(req.site, req.kind, req.date),
force=req.force,
)
spec = {"site": req.site, "kind": req.kind, "force": req.force}
if req.date:
spec["date"] = req.date
task_queue.put((task_id, spec))
return {"task_id": task_id} return {"task_id": task_id}
@@ -220,6 +255,71 @@ def list_tasks(limit: int = 20):
return state_store.list_tasks(limit) return state_store.list_tasks(limit)
# ── DB 比对(基于 PostgreSQL不依赖 Excel 文件)──
class CompareRequest(BaseModel):
site: str
date: str # YYYY-MM-DD
@app.post("/compare")
def run_compare(req: CompareRequest):
"""DB 差缺比对应到驱动以批次归属日batch_out_date为锚展开全量比对。
返回统计指标 + 差缺明细。
"""
# 合法性校验
if req.site not in db_compare.SITE_COMPARE_CONFIG:
raise HTTPException(
status_code=400,
detail=f"不支持的站点: {req.site}(支持: {list(db_compare.SITE_COMPARE_CONFIG.keys())}",
)
try:
target_date = datetime.strptime(req.date, "%Y-%m-%d").date()
except ValueError:
raise HTTPException(
status_code=400, detail=f"date 格式非法,需 YYYY-MM-DD: {req.date}"
)
today = datetime.now().date()
if target_date > today:
raise HTTPException(status_code=400, detail=f"date 不可为未来日期: {req.date}")
result = db_compare.compare_site_outdate(req.site, req.date)
if result is None:
raise HTTPException(
status_code=404,
detail=f"{req.site} {req.date}: 当日无应到批次,无法比对",
)
return {
"site": result.site,
"date": result.date,
"batches": result.batches,
"stats": {
"waybill_count": result.stats.waybill_count,
"sf_wb_count": result.stats.sf_wb_count,
"expected_pieces": result.stats.expected_pieces,
"arrived_pieces": result.stats.arrived_pieces,
"undelivered_pieces": result.stats.undelivered_pieces,
"undelivered_wb": result.stats.undelivered_wb,
"full_miss": result.stats.full_miss,
"part_miss": result.stats.part_miss,
"sf_undelivered": result.stats.sf_undelivered,
},
"rows": [
{
"handover_no": r.handover_no,
"waybill_no": r.waybill_no,
"total_pieces": r.total_pieces,
"arrived_pieces": r.arrived_pieces,
"arrived_list": r.arrived_list,
"is_sf": r.is_sf,
}
for r in result.rows
],
}
@app.get("/status") @app.get("/status")
def get_status(): def get_status():
"""各站登录态 + 数据态 + 入库态(前端状态盘用),另含 worker 就绪状态。""" """各站登录态 + 数据态 + 入库态(前端状态盘用),另含 worker 就绪状态。"""
@@ -326,20 +426,6 @@ def download_report():
return FileResponse(path, filename=REPORT_FILE) return FileResponse(path, filename=REPORT_FILE)
@app.get("/data/{filename}")
def download_data(filename: str):
"""下载 downloads/ 下的数据文件(防路径穿越)。"""
if not filename or "/" in filename or "\\" in filename or ".." in filename:
raise HTTPException(status_code=400, detail="非法文件名")
path = os.path.join(DOWNLOAD_DIR, filename)
# 双重校验:解析后绝对路径仍在 DOWNLOAD_DIR 内
if not os.path.abspath(path).startswith(os.path.abspath(DOWNLOAD_DIR) + os.sep):
raise HTTPException(status_code=400, detail="非法路径")
if not os.path.isfile(path):
raise HTTPException(status_code=404, detail="文件不存在")
return FileResponse(path, filename=filename)
def main(): def main():
"""服务模式入口。传字符串导入路径(规范写法;不开 reload/workers 时进程内 import行为等价""" """服务模式入口。传字符串导入路径(规范写法;不开 reload/workers 时进程内 import行为等价"""
uvicorn.run("inbound_verify.cli.server:app", host="0.0.0.0", port=8000) uvicorn.run("inbound_verify.cli.server:app", host="0.0.0.0", port=8000)

View File

@@ -75,6 +75,12 @@ def process(name):
df_exp = pd.read_excel(exp_path, dtype=str).fillna("") df_exp = pd.read_excel(exp_path, dtype=str).fillna("")
df_act = pd.read_excel(act_path, dtype=str).fillna("") df_act = pd.read_excel(act_path, dtype=str).fillna("")
if name == "韵达":
# 韵达实到数据有重复行(同子单号出现两次),保留交接单号为空的(到/接件扫描),
# 丢弃交接单号不为空的(派件/签收等),再按子单号去重。
df_act = df_act[df_act["交接单号"].astype(str).str.strip() == ""]
df_act = df_act.drop_duplicates(subset=["子单号"], keep="last")
# 同一运单可能有多条交接记录,按运单号去重、保留首条 # 同一运单可能有多条交接记录,按运单号去重、保留首条
dup = int(df_exp[cfg["exp_wb"]].duplicated().sum()) dup = int(df_exp[cfg["exp_wb"]].duplicated().sum())
df_exp = df_exp.drop_duplicates(subset=[cfg["exp_wb"]], keep="first") df_exp = df_exp.drop_duplicates(subset=[cfg["exp_wb"]], keep="first")

View File

@@ -0,0 +1,733 @@
# -*- coding: utf-8 -*-
"""
db_compare.py — 基于 PostgreSQL 的应到未到差缺比对引擎。
与 compare.pyExcel 版)并行:本模块直接从 DB 查询数据进行比对,
不依赖 downloads/ 下的 Excel 文件。
核心思路:以实到扫描日期为锚点 → 反推交接批次 → 展开批次全量比对。
每个站点只需提供配置waybill 列名 / piece 列名 / 是否有 SF 特殊处理),
核心比对逻辑完全通用。
顺心站点 SF 运单特殊处理SF 运单的子单号piece_no为随机号码不能用
COUNT(DISTINCT piece_no) 去重计数,改为 COUNT(*) 行计数。
用法:
from inbound_verify.db_compare import compare_site_date, SITE_COMPARE_CONFIG
result = compare_site_date("顺心", "2026-07-25")
if result:
print(result.stats)
for row in result.rows:
print(row)
"""
import os
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta
import psycopg
import yaml
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from inbound_verify.paths import CONFIG_PATH, OUTPUT_DIR, DOWNLOAD_DIR
from inbound_verify.domain import _site_cfg, ALL_REPORT_SITES, BAISHI_COLUMNS
# ============================== 结果类型 ==============================
@dataclass
class CompareStats:
"""单站点/单批次比对统计。"""
waybill_count: int = 0 # 应到运单数
expected_pieces: int = 0 # 应到件数
arrived_pieces: int = 0 # 实到件数
undelivered_pieces: int = 0 # 未到件数
undelivered_wb: int = 0 # 差缺运单数
full_miss: int = 0 # 完全未到
part_miss: int = 0 # 部分未到
sf_wb_count: int = 0 # SF 运单数
sf_undelivered: int = 0 # SF 差缺数
@dataclass
class UndeliveredRow:
"""单条差缺明细。"""
handover_no: str = "" # 交接单号
waybill_no: str = "" # 运单号
total_pieces: int = 0 # 总件数(交接件数)
arrived_pieces: int = 0 # 已到件数
arrived_list: list = field(default_factory=list) # 已到单号列表
is_sf: bool = False # 是否 SF 运单
@dataclass
class CompareResult:
"""一次比对的完整结果。"""
site: str = ""
date: str = ""
batches: list = field(default_factory=list) # 涉及的交接批次
stats: CompareStats = field(default_factory=CompareStats)
rows: list = field(default_factory=list) # UndeliveredRow 列表
# ============================== 站点比对配置 ==============================
@dataclass
class SiteCompareConfig:
"""DB 比对的站点参数。"""
name: str # 站点名
has_sf: bool = False # 是否需要区分 SF 运单
# 四站点 DB 比对配置(百世不参与 4 站比对)
SITE_COMPARE_CONFIG: dict[str, SiteCompareConfig] = {
"顺心": SiteCompareConfig(name="顺心", has_sf=True),
"中通": SiteCompareConfig(name="中通", has_sf=False),
"韵达": SiteCompareConfig(name="韵达", has_sf=False),
"安能": SiteCompareConfig(name="安能", has_sf=False),
}
# ============================== DB 连接 ==============================
def _load_pg_config():
"""从 config.yaml 读 postgres 段。与 store.py 共用同一配置源。"""
if not os.path.exists(CONFIG_PATH):
raise FileNotFoundError(
f"未找到配置文件 {CONFIG_PATH}(请参考 config.example.yaml 创建 config.yaml"
)
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
pg = cfg.get("postgres") or {}
return {
"host": pg.get("host", "127.0.0.1"),
"port": int(pg.get("port", 5432)),
"user": pg.get("user", "postgres"),
"password": pg.get("password", ""),
"dbname": pg.get("dbname", "CQHXDB"),
"schema": pg.get("schema", "inbound_verify"),
"connect_timeout_seconds": int(pg.get("connect_timeout_seconds", 5)),
}
def _connect():
c = _load_pg_config()
return psycopg.connect(
host=c["host"],
port=c["port"],
dbname=c["dbname"],
user=c["user"],
password=c["password"],
options=f"-c search_path={c['schema']} -c statement_timeout=30s",
connect_timeout=c["connect_timeout_seconds"],
)
# ============================== 核心比对逻辑 ==============================
def compare_site_date(site: str, target_date: str) -> CompareResult | None:
"""对指定站点和日期执行 DB 差缺比对。
算法:
1. 取 scan_time::date = target_date 的实到运单(锚点)
2. 反推这些运单所属的交接批次handover_no
3. 展开批次全量应到运单
4. 查询批次全量实到扫描
5. 逐运单比对差缺SF/non-SF 分支处理)
Args:
site: 站点名("顺心"/"中通"/"韵达"/"安能"
target_date: 日期 "YYYY-MM-DD"
Returns:
CompareResult 或 None当天无实到数据时返回 None
"""
cfg = SITE_COMPARE_CONFIG.get(site)
if cfg is None:
print(f"[db_compare] 不支持的站点: {site}")
return None
try:
conn = _connect()
cur = conn.cursor()
# ── Step 1: 取实到锚点 ──
cur.execute(
"""
SELECT DISTINCT waybill_no FROM actual_record
WHERE site = %s AND scan_time::date = %s
""",
(site, target_date),
)
anchor_wbs = [r[0] for r in cur.fetchall()]
if not anchor_wbs:
print(f"[db_compare] {site} {target_date}: 当天无实到数据")
conn.close()
return None
# ── Step 2: 反推交接批次 ──
cur.execute(
"""
SELECT DISTINCT e.handover_no FROM expected_record e
WHERE e.site = %s AND e.waybill_no = ANY(%s)
""",
(site, anchor_wbs),
)
batches = [r[0] for r in cur.fetchall()]
# ── Step 3: 展开批次全量应到 ──
cur.execute(
"""
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site = %s AND handover_no = ANY(%s)
ORDER BY handover_no, waybill_no
""",
(site, batches),
)
exp_rows = cur.fetchall() # [(waybill_no, handover_no, handover_pieces), ...]
if not exp_rows:
conn.close()
return None
all_wbs = [r[0] for r in exp_rows]
# ── Step 4: 取批次全量实到 ──
cur.execute(
"""
SELECT waybill_no, piece_no FROM actual_record
WHERE site = %s AND waybill_no = ANY(%s)
ORDER BY waybill_no, piece_no
""",
(site, all_wbs),
)
act_rows = cur.fetchall() # [(waybill_no, piece_no), ...]
conn.close()
# ── Step 5: 逐运单比对 ──
return _do_compare(site, target_date, batches, exp_rows, act_rows, cfg)
except Exception as e:
print(f"[db_compare] {site} {target_date} 比对异常: {e}")
return None
def compare_site_outdate(site: str, target_date: str) -> CompareResult | None:
"""应到驱动差缺比对以「批次归属日batch_out_date」为准取应到。
与 compare_site_date实到驱动区别
1. 应到来源 = expected_record WHERE batch_out_date = target_date
2. 不再依赖实到锚点反推;应到空时返回 None明确"当日无应到"
3. 提前提交的批次按其出库日归属,自动归入正确日期
Args:
site: 站点名("顺心"/"中通"/"韵达"/"安能"
target_date: 目标业务日期 "YYYY-MM-DD"
Returns:
CompareResult 或 None当日无应到批次
"""
cfg = SITE_COMPARE_CONFIG.get(site)
if cfg is None:
print(f"[db_compare] 不支持的站点: {site}")
return None
try:
conn = _connect()
cur = conn.cursor()
# ── Step 1: 取目标日应到批次(按批次归属日)──
cur.execute(
"""
SELECT DISTINCT handover_no FROM expected_record
WHERE site = %s AND batch_out_date = %s
ORDER BY handover_no
""",
(site, target_date),
)
batches = [r[0] for r in cur.fetchall()]
if not batches:
print(f"[db_compare] {site} {target_date}: 当日无应到批次")
conn.close()
return None
# ── Step 2: 展开批次全量应到 ──
cur.execute(
"""
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site = %s AND handover_no = ANY(%s)
ORDER BY handover_no, waybill_no
""",
(site, batches),
)
exp_rows = cur.fetchall()
if not exp_rows:
conn.close()
return None
all_wbs = [r[0] for r in exp_rows]
# ── Step 3: 取批次全量实到 ──
cur.execute(
"""
SELECT waybill_no, piece_no FROM actual_record
WHERE site = %s AND waybill_no = ANY(%s)
ORDER BY waybill_no, piece_no
""",
(site, all_wbs),
)
act_rows = cur.fetchall()
conn.close()
# ── Step 4: 逐运单比对 ──
return _do_compare(site, target_date, batches, exp_rows, act_rows, cfg)
except Exception as e:
print(f"[db_compare] {site} {target_date} 应到驱动比对异常: {e}")
return None
def compare_site_batch(site: str, handover_no: str) -> CompareResult | None:
"""按指定交接单号执行全批次比对(不依赖实到锚点)。
用于已知交接单号后精确比对某一批次。
"""
cfg = SITE_COMPARE_CONFIG.get(site)
if cfg is None:
print(f"[db_compare] 不支持的站点: {site}")
return None
try:
conn = _connect()
cur = conn.cursor()
cur.execute(
"""
SELECT waybill_no, handover_no, handover_pieces
FROM expected_record
WHERE site = %s AND handover_no = %s
ORDER BY waybill_no
""",
(site, handover_no),
)
exp_rows = cur.fetchall()
if not exp_rows:
conn.close()
return None
all_wbs = [r[0] for r in exp_rows]
cur.execute(
"""
SELECT waybill_no, piece_no FROM actual_record
WHERE site = %s AND waybill_no = ANY(%s)
ORDER BY waybill_no, piece_no
""",
(site, all_wbs),
)
act_rows = cur.fetchall()
conn.close()
return _do_compare(
site,
f"batch:{handover_no}",
[handover_no],
exp_rows,
act_rows,
cfg,
)
except Exception as e:
print(f"[db_compare] {site} batch:{handover_no} 比对异常: {e}")
return None
# ============================== 比对核心 ==============================
def _do_compare(
site: str,
label: str,
batches: list[str],
exp_rows: list[tuple], # [(waybill_no, handover_no, handover_pieces), ...]
act_rows: list[tuple], # [(waybill_no, piece_no), ...]
cfg: SiteCompareConfig,
) -> CompareResult:
"""执行逐运单比对,产出统计 + 差缺明细。
与 compare.py:process() 口径一致:
- 应到件数 = handover_pieces交接件数
- 实到件数 = SF ? COUNT(*) : COUNT(DISTINCT piece_no)
- arrived_cnt >= handover_pieces → 足额到货,跳过
"""
# 构建实到索引: waybill_no → [piece_no, ...](保留所有行,不去重)
act_by_wb: dict[str, list[str]] = {}
for wb, piece in act_rows:
act_by_wb.setdefault(wb, []).append(piece)
stats = CompareStats()
rows: list[UndeliveredRow] = []
max_arrived = 0
for wb, handover_no, handover_pcs in exp_rows:
handover_pcs = handover_pcs or 0
if handover_pcs <= 0:
continue
stats.waybill_count += 1
stats.expected_pieces += handover_pcs
is_sf = cfg.has_sf and wb.startswith("SF")
if is_sf:
stats.sf_wb_count += 1
all_pieces = act_by_wb.get(wb, [])
if is_sf:
# SF: 行计数不去重piece_no 是随机号码)
arrived_cnt = len(all_pieces)
arrived_list = list(all_pieces)
else:
# non-SF: 子单号去重
unique_pieces = list(dict.fromkeys(all_pieces)) # 保序去重
arrived_cnt = len(unique_pieces)
arrived_list = unique_pieces
stats.arrived_pieces += arrived_cnt
if arrived_cnt >= handover_pcs:
continue # 足额或溢到,不进差缺表
if arrived_cnt == 0:
stats.full_miss += 1
else:
stats.part_miss += 1
if is_sf:
stats.sf_undelivered += 1
max_arrived = max(max_arrived, arrived_cnt)
rows.append(
UndeliveredRow(
handover_no=handover_no,
waybill_no=wb,
total_pieces=handover_pcs,
arrived_pieces=arrived_cnt,
arrived_list=arrived_list,
is_sf=is_sf,
)
)
stats.undelivered_pieces = max(0, stats.expected_pieces - stats.arrived_pieces)
stats.undelivered_wb = stats.full_miss + stats.part_miss
result = CompareResult(
site=site,
date=label,
batches=batches,
stats=stats,
rows=rows,
)
# 打印摘要
print(
f"[db_compare] {site} {label}: "
f"batches={len(batches)}, "
f"wb={stats.waybill_count}(SF:{stats.sf_wb_count}), "
f"exp={stats.expected_pieces}, arr={stats.arrived_pieces}, "
f"miss={stats.undelivered_pieces}, "
f"miss_wb={stats.undelivered_wb}(full={stats.full_miss}, part={stats.part_miss})"
)
if stats.sf_undelivered:
print(f" SF 差缺: {stats.sf_undelivered} 个运单")
return result
# ============================== Excel 输出 ==============================
# 样式常量(与 compare.py 对齐)
_FONT = "微软雅黑"
_BLUE = "305496"
_HEADER_FILL = PatternFill("solid", fgColor=_BLUE)
_HEADER_FONT = Font(name=_FONT, bold=True, color="FFFFFF", size=11)
_BODY_FONT = Font(name=_FONT, size=10)
_THIN = Side(style="thin", color="D9D9D9")
_BORDER = Border(left=_THIN, right=_THIN, top=_THIN, bottom=_THIN)
def write_result_excel(result: CompareResult, output_path: str | None = None) -> str:
"""将比对结果写入 Excel 文件。
Args:
result: compare_site_date 或 compare_site_batch 的返回值
output_path: 输出路径,为 None 时自动生成:
output/{站}-{日期}-未到数据.xlsx
Returns:
实际写入的文件路径
"""
if output_path is None:
os.makedirs(OUTPUT_DIR, exist_ok=True)
date_tag = result.date.replace(":", "-").replace("batch:", "batch-")
output_path = os.path.join(
OUTPUT_DIR, f"{result.site}-{date_tag}-未到数据.xlsx"
)
wb = Workbook()
ws = wb.active
ws.title = result.site
_write_sheet(ws, result)
wb.save(output_path)
print(f"[db_compare] Excel 已输出: {output_path}")
return output_path
def _write_sheet(ws, result: CompareResult):
"""写单个站点的差缺明细 sheet。"""
s = result.stats
rows = result.rows
# 动态列: 交接单号 | 运单号 | 总件数 | 已到单号1 | 已到单号2 | ...
max_arrived = max((len(r.arrived_list) for r in rows), default=0)
columns = ["交接单号", "运单号", "总件数"] + [
f"已到单号{i + 1}" for i in range(max_arrived)
]
ws.sheet_view.showGridLines = False
# 表头
ws.append(columns)
for c in range(1, len(columns) + 1):
cell = ws.cell(row=1, column=c)
cell.fill = _HEADER_FILL
cell.font = _HEADER_FONT
cell.alignment = Alignment(horizontal="center", vertical="center")
cell.border = _BORDER
# 数据行
for row in rows:
values = {
"交接单号": row.handover_no,
"运单号": row.waybill_no,
"总件数": row.total_pieces,
}
for i, piece in enumerate(row.arrived_list):
values[f"已到单号{i + 1}"] = piece
ws.append([values.get(c, "") for c in columns])
# 格式
for r in range(2, ws.max_row + 1):
for c, col in enumerate(columns, start=1):
cell = ws.cell(row=r, column=c)
cell.font = _BODY_FONT
cell.border = _BORDER
if col == "总件数":
cell.number_format = "#,##0"
cell.alignment = Alignment(horizontal="right", vertical="center")
else:
cell.number_format = "@"
# 列宽
for c, col in enumerate(columns, start=1):
body_lens = [
len(str(ws.cell(row=r, column=c).value or ""))
for r in range(2, ws.max_row + 1)
]
width = min(max([len(str(col))] + body_lens) + 4, 36)
ws.column_dimensions[ws.cell(row=1, column=c).column_letter].width = max(
width, 12
)
ws.freeze_panes = "A2"
# ============================== 全站汇总报表DB 版)=============================
def _stats_to_dict(s: CompareStats) -> dict:
"""CompareStats -> build_summary 要的中文键 stats dict。"""
return {
"运单数": s.waybill_count,
"应到件": s.expected_pieces,
"已到件": s.arrived_pieces,
"未到件": s.undelivered_pieces,
"完全未到": s.full_miss,
"部分未到": s.part_miss,
}
def _target_date_for(site: str) -> str:
"""4 站比对锚点(应到驱动):批次归属日默认取今天。
各站统一以出库日batch_out_date为准不再依赖站点偏移配置。"""
return date.today().strftime("%Y-%m-%d")
def _baishi_from_pg(cur, target: str):
"""查百世当日基数baishi_daily_stats+ 当天未到明细undelivered_record 按 ingested_at 过滤)。
返回 (stats_dict_or_None, rows_or_None);基数与明细均无 → (None, None)。
undelivered_record 是 UPSERT 累积表;按 ingested_at::date = target 取当天入库的未到快照
= 当天下载的当前未到,站点已剔除已到),避免累积偏大。
"""
cur.execute(
"SELECT expected_pieces, arrived_pieces, undelivered_pieces "
"FROM baishi_daily_stats WHERE site = %s AND business_date = %s",
("百世", target),
)
basis = cur.fetchone()
cur.execute(
"SELECT waybill_no, piece_no, biz_type, last_scan FROM undelivered_record "
"WHERE site = %s AND ingested_at::date = %s",
("百世", target),
)
detail = cur.fetchall()
if basis is None and not detail:
return (None, None)
exp = basis[0] if basis else None
arr = basis[1] if basis else None
# 未到件优先取基数差baishi_daily_stats.undelivered_pieces与应到/已到同源自洽);
# 基数缺失时退回明细行数。
undel = basis[2] if (basis and basis[2] is not None) else len(detail)
wb_count = len({r[0] for r in detail if r[0]}) # 运单号去重
rows = [
{
"类型": r[2] or "",
"子单号": r[1] or "",
"运单号": r[0] or "",
"最新扫描记录": r[3] or "",
}
for r in detail
]
stats = {
"运单数": wb_count,
"应到件": exp,
"已到件": arr,
"未到件": undel,
"完全未到": None,
"部分未到": None,
}
return (stats, rows)
def build_full_report(date=None) -> str:
"""DB 版全站汇总报表4 站走 DB 比对、百世走 PG复用 compare.build_summary 渲染。
产出 output/应到未到数据.xlsx/report 下载。date=None 时各站取今天为批次归属锚点(应到驱动)。
返回输出路径。"""
from inbound_verify import compare # 复用 build_summary / write_station / OUTFILE
print("[db_compare] 开始生成全站汇总报表 ...")
wb = Workbook()
wb.remove(wb.active)
summary_ws = wb.create_sheet("汇总报表")
results = [] # [(name, stats_dict_or_None)],顺序 ALL_REPORT_SITES
site_targets = {} # name -> target_date汇总表"数据日期"列)
conn = _connect()
cur = conn.cursor()
try:
for name in ALL_REPORT_SITES:
if name == "百世":
target = date or datetime.now().strftime("%Y-%m-%d")
site_targets[name] = target
stats, rows = _baishi_from_pg(cur, target)
results.append((name, stats))
if rows is not None:
compare.write_station(wb.create_sheet(name), BAISHI_COLUMNS, rows)
continue
if name not in SITE_COMPARE_CONFIG:
results.append((name, None))
continue
target = date or _target_date_for(name)
site_targets[name] = target
result = compare_site_outdate(name, target)
if result is not None:
results.append((name, _stats_to_dict(result.stats)))
_write_sheet(wb.create_sheet(name), result)
else:
results.append((name, None))
finally:
conn.close()
compare.build_summary(
summary_ws,
results,
datetime.now().strftime("%Y-%m-%d %H:%M"),
dates=site_targets,
)
os.makedirs(OUTPUT_DIR, exist_ok=True)
wb.save(compare.OUTFILE)
print(f"[db_compare] 全站汇总已输出: {compare.OUTFILE}")
for name, s in results:
print(f" {name}:未到 {s['未到件']}" if s else f" {name}:无数据,跳过")
return compare.OUTFILE
# ============================== 终端验证入口 ==============================
def main():
"""命令行验证入口:
python -m inbound_verify.db_compare 顺心 2026-07-25
"""
import sys
site = sys.argv[1] if len(sys.argv) > 1 else "顺心"
target_date = sys.argv[2] if len(sys.argv) > 2 else "2026-07-25"
result = compare_site_date(site, target_date)
if result is None:
print(f"{site} {target_date}: 无结果")
return
print(f"\n=== {result.site} {result.date} 差缺明细 ===")
print(f"涉及批次: {result.batches}")
print(f"应到运单: {result.stats.waybill_count} (SF: {result.stats.sf_wb_count})")
print(f"应到件数: {result.stats.expected_pieces}")
print(f"实到件数: {result.stats.arrived_pieces}")
print(f"未到件数: {result.stats.undelivered_pieces}")
print(
f"差缺运单: {result.stats.undelivered_wb} (完全未到: {result.stats.full_miss}, 部分未到: {result.stats.part_miss})"
)
if result.stats.sf_undelivered:
print(f"SF 差缺: {result.stats.sf_undelivered}")
if result.rows:
print(f"\n--- 差缺明细 (共 {len(result.rows)} 条) ---")
for row in result.rows[:20]:
sf = "[SF]" if row.is_sf else ""
arrived_preview = row.arrived_list[:5]
print(
f" {sf} {row.waybill_no}: "
f"应到{row.total_pieces}件, 实到{row.arrived_pieces}"
f" {f'已到: {arrived_preview}' if arrived_preview else ''}"
)
if len(result.rows) > 20:
print(f" ... 还有 {len(result.rows) - 20}")
# 输出 Excel
path = write_result_excel(result)
print(f"\n结果文件: {path}")
if __name__ == "__main__":
main()

View File

@@ -18,3 +18,6 @@ CONFIG_PATH = os.path.join(BASE_DIR, "config.yaml")
# 状态存储SQLite阶段0心跳 / 登录态 / 数据态持久化,重启不丢) # 状态存储SQLite阶段0心跳 / 登录态 / 数据态持久化,重启不丢)
STATE_DB_PATH = os.path.join(BASE_DIR, "state", "state.db") STATE_DB_PATH = os.path.join(BASE_DIR, "state", "state.db")
# 错误截图目录(下载流程失败时自动截取,供问题排查)
SCREENSHOT_DIR = os.path.join(BASE_DIR, "logs", "screenshots")

View File

@@ -16,13 +16,12 @@ import socket
import subprocess import subprocess
import time import time
import urllib.request import urllib.request
from datetime import datetime, timedelta from datetime import date, datetime, timedelta
import yaml import yaml
from playwright.sync_api import sync_playwright from playwright.sync_api import sync_playwright
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH from inbound_verify.paths import CONFIG_PATH, SCREENSHOT_DIR
from inbound_verify.domain import SITE_UNDELIVERED_FILE
from inbound_verify import state_store from inbound_verify import state_store
from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng from inbound_verify.sites import shunxin, baishi, zto, yunda, anneng
@@ -50,31 +49,6 @@ APP_SITES = {"安能"}
# 心跳间隔(秒) # 心跳间隔(秒)
HEARTBEAT_INTERVAL = 30 HEARTBEAT_INTERVAL = 30
# 各站最终数据文件名(探测"数据是否已跑出来");百世为单流程
DATA_FILENAMES = {
"顺心": {
"expected": "顺心-应到货物数据.xlsx",
"actual": "顺心-实到货物数据.xlsx",
"undelivered": "顺心-未到数据.xlsx",
},
"中通": {
"expected": "中通-应到货物数据.xlsx",
"actual": "中通-实到货物数据.xlsx",
"undelivered": "中通-未到数据.xlsx",
},
"韵达": {
"expected": "韵达-应到货物数据.xlsx",
"actual": "韵达-实到货物数据.xlsx",
"undelivered": "韵达-未到数据.xlsx",
},
"安能": {
"expected": "安能-应到货物数据.xlsx",
"actual": "安能-实到货物数据.xlsx",
"undelivered": "安能-未到数据.xlsx",
},
"百世": {"expected": "", "actual": "", "undelivered": "百世-应到未到货物数据.xlsx"},
}
# ============================ 安能启动CDP============================ # ============================ 安能启动CDP============================
@@ -102,13 +76,37 @@ def _wait_cdp_up(port, timeout=60.0):
return False return False
# 【环境兼容】宿主 shellCodex/VS Code 插件、WorkBuddy 等)会向子进程注入一批与业务
# 无关的变量,实测会让安能应用登录后反复弹出“获取试用网点接口报错”:
# - HTTP(S)_PROXY=http://127.0.0.1:8800QuickQ 加速器代理):安能的 wnp.ane56.com
# 接口请求被塞进第三方代理后返回 400/用户未登录;
# - VSCODE_* / CODEX_* / EFC_*VS Code 扩展宿主注入IPC、PID、NLS、ESM 等);
# - NODE_TLS_REJECT_UNAUTHORIZED / DEBUG / RUST_LOG 等宿主调试变量。
# 另ELECTRON_RUN_AS_NODE=1 会把安能当作纯 Node 运行(拒绝 Chromium 参数、启动即
# 退出 rc=9NODE_OPTIONS 同样会干扰。拉起前全部摘掉,尽量还原终端手动启动环境。
_ANNENG_STRIP_PREFIXES = ("VSCODE_", "CODEX_", "EFC_")
_ANNENG_STRIP_EXACT = {
"NODE_OPTIONS",
"ELECTRON_RUN_AS_NODE",
"HTTP_PROXY",
"HTTPS_PROXY",
"ALL_PROXY",
"NO_PROXY",
"NODE_TLS_REJECT_UNAUTHORIZED",
"NODEFAULTCURRENTDIRECTORYINEXEPATH",
"DEBUG",
"RUST_LOG",
"APPLICATION_INSIGHTS_NO_STATSBEAT",
}
def launch_anneng(app_path): def launch_anneng(app_path):
"""以调试模式启动安能 Electron 应用(自动选取空闲端口),返回子进程对象。""" """以调试模式启动安能 Electron 应用(自动选取空闲端口),返回子进程对象。"""
# 【环境兼容】WorkBuddy 等 shell 会注入 NODE_OPTIONS含 --use-system-ca anneng_env = {
# Electron 内置 Node 拒绝该 flag 导致安能启动即退出rc=9 key: value
# 拉起前从子进程环境里摘掉 NODE_OPTIONS。 for key, value in os.environ.items()
anneng_env = os.environ.copy() if key not in _ANNENG_STRIP_EXACT and not key.startswith(_ANNENG_STRIP_PREFIXES)
anneng_env.pop("NODE_OPTIONS", None) }
port = _find_free_port() port = _find_free_port()
print(f">> 以调试模式启动【安能】应用(端口 {port}{app_path}") print(f">> 以调试模式启动【安能】应用(端口 {port}{app_path}")
proc = subprocess.Popen( proc = subprocess.Popen(
@@ -150,19 +148,6 @@ def probe_site_login(site_name, pages_map):
return False return False
def probe_data_file(site_name, kind):
"""探测单站应到/实到数据文件是否存在且为今天。返回 (is_today, mtime_str)。"""
fname = DATA_FILENAMES.get(site_name, {}).get(kind, "")
if not fname:
return (False, "")
path = os.path.join(DOWNLOAD_DIR, fname)
if not os.path.exists(path):
return (False, "")
dt = datetime.fromtimestamp(os.path.getmtime(path))
is_today = dt.date() == datetime.now().date()
return (is_today, dt.strftime("%Y-%m-%d %H:%M:%S"))
# ============================ 运行上下文 ============================ # ============================ 运行上下文 ============================
@@ -301,6 +286,15 @@ def launch_and_prepare(debug_mode=False, debug_target="", foreground=True):
_launch_args = ["--remote-debugging-port=9223"] if debug_mode else [] _launch_args = ["--remote-debugging-port=9223"] if debug_mode else []
browser = pw.chromium.launch(headless=False, args=_launch_args) browser = pw.chromium.launch(headless=False, args=_launch_args)
context = browser.new_context(viewport={"width": 1920, "height": 1080}) context = browser.new_context(viewport={"width": 1920, "height": 1080})
# 默认禁用麦克风/摄像头:在每个页面/iframe 加载前覆盖 getUserMedia 为“直接拒绝”,
# 这样站点(如韵达登录/工作台会请求麦克风)调用时立即 NotAllowedErrorChromium 不再
# 弹出系统授权窗,且麦克风被真正挡住(不是授权给它)。物流工作台无需音视频采集。
context.add_init_script(
"(()=>{const d=()=>Promise.reject(new DOMException('Permission disabled','NotAllowedError'));"
"if(navigator.mediaDevices)navigator.mediaDevices.getUserMedia=d;"
"for(const k of ['getUserMedia','webkitGetUserMedia','mozGetUserMedia']){"
"if(typeof navigator[k]==='function')navigator[k]=function(){return d();};}})();"
)
pages_map = {} pages_map = {}
print("\n====================================================") print("\n====================================================")
@@ -454,41 +448,85 @@ def _web_handler(site, download_func):
焦点;交互模式置顶便于调试。顺心是 page 列表,置顶标志透传给 shunxin_download。 焦点;交互模式置顶便于调试。顺心是 page 列表,置顶标志透传给 shunxin_download。
""" """
def handler(ctx, force=False): def handler(ctx, force=False, date=None):
pg = ctx.pages_map[site] pg = ctx.pages_map[site]
if isinstance(pg, list): if isinstance(pg, list):
# 顺心双账号:置顶与否交给 shunxin_download 在逐账号循环里按 foreground 决定 # 顺心双账号:置顶与否交给 shunxin_download 在逐账号循环里按 foreground 决定
return download_func(pg, foreground=ctx.foreground, force=force) return download_func(pg, foreground=ctx.foreground, force=force, date=date)
if ctx.foreground: if ctx.foreground:
pg.bring_to_front() pg.bring_to_front()
return download_func(pg, force=force) return download_func(pg, force=force, date=date)
return handler return handler
def _site_undelivered_handler(site): def _site_undelivered_handler(site):
"""4 站未到:下应到+实到 → 比对写 downloads/<站>-未到数据.xlsx。 """4 站未到:下应到+实到 → DB 比对 → 写 output/<站>-<日期>-未到数据.xlsx。
任一下载失败 → 清掉旧未到文件、返回 False前端不展示陈旧未到""" 应到全量去重(已落库则跳过导出),因此比对不依赖 Excel 文件,走数据库查询。
下载成功则返回 True比对失败不影响任务判定数据已入库"""
def handler(ctx, force=False): def handler(ctx, force=False, date=None):
# 各站下载入口约定返回 True/False顺心历史返回 None视为成功与 dispatch 一致) exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force, date) is not False
exp_ok = TASK_HANDLERS[(site, "expected")](ctx, force) is not False
act_ok = ( act_ok = (
(TASK_HANDLERS[(site, "actual")](ctx, force) is not False) (TASK_HANDLERS[(site, "actual")](ctx, force, date) is not False)
if exp_ok if exp_ok
else False else False
) )
if exp_ok and act_ok: if not exp_ok or not act_ok:
return compare.write_site_file(site) return False
stale = os.path.join(DOWNLOAD_DIR, SITE_UNDELIVERED_FILE.format(name=site))
if os.path.exists(stale): # ── 先入库再比对(修复时序:比对须读到本次下载的数据,
os.remove(stale) # 否则首次/force 时 PG 无当天数据,比对返回 None、不产出 Excel──
return False try:
_record_business_date(site, "undelivered", date)
except Exception:
pass
try:
from inbound_verify import store # 懒导入,避免成环
if store.ingest_enabled():
store.ingest_task(
site, "undelivered"
) # 4 站 = ingest expected + actual
print(f">> [入库] {site} 前置入库完成")
except Exception as e:
print(f">> [入库] {site} 前置入库失败(不影响比对尝试): {e}")
# ── DB 比对(应到驱动:以批次归属日 batch_out_date 为锚)──
try:
from inbound_verify import db_compare # 懒导入,避免成环
if date:
target_date = date
else:
target_date = datetime.now().date().strftime("%Y-%m-%d")
result = db_compare.compare_site_outdate(site, target_date)
if result is not None:
db_compare.write_result_excel(result)
else:
print(f">> [未到] {site} {target_date}: 当日无应到批次,跳过比对")
except Exception as e:
print(f">> [未到] {site} DB 比对异常(不影响下载结果): {e}")
return True # 下载成功即返回 True比对失败不影响任务判定
return handler return handler
# 「跑比对」= 纯离线比对(用 downloads/ 现有文件生成全站汇总;下载交由各站定时/手动)。 # 「跑比对」= DB 版全站汇总报表(替代旧 compare.main Excel 路径;下载交由各站定时/手动)。
def _run_db_full_report(date=None):
"""生成 DB 版全站汇总报表output/应到未到数据.xlsx
懒导入 db_comparebest-effort失败只告警返回 True与旧 lambda 契约一致)。"""
try:
from inbound_verify import db_compare
db_compare.build_full_report(date)
except Exception as e:
print(f">> [跑比对] DB 汇总报表生成失败: {e}")
return True
TASK_HANDLERS = { TASK_HANDLERS = {
@@ -504,51 +542,135 @@ TASK_HANDLERS = {
("韵达", "expected"): _web_handler("韵达", yunda.yunda_expected_download), ("韵达", "expected"): _web_handler("韵达", yunda.yunda_expected_download),
("韵达", "actual"): _web_handler("韵达", yunda.yunda_actual_download), ("韵达", "actual"): _web_handler("韵达", yunda.yunda_actual_download),
("韵达", "undelivered"): _site_undelivered_handler("韵达"), ("韵达", "undelivered"): _site_undelivered_handler("韵达"),
("安能", "expected"): lambda ctx, force=False: anneng.anneng_expected_download( (
force=force "安能",
"expected",
): lambda ctx, force=False, date=None: anneng.anneng_expected_download(
force=force, date=date
), ),
("安能", "actual"): lambda ctx, force=False: anneng.anneng_actual_download( (
force=force "安能",
"actual",
): lambda ctx, force=False, date=None: anneng.anneng_actual_download(
force=force, date=date
), ),
("安能", "undelivered"): _site_undelivered_handler("安能"), ("安能", "undelivered"): _site_undelivered_handler("安能"),
("__compare__", "compare"): lambda ctx, force=False: (compare.main() or True), ("__compare__", "compare"): lambda ctx, force=False, date=None: _run_db_full_report(
date
),
} }
def _record_business_date(site, kind): def _record_business_date(site, kind, date=None):
"""下载成功后,把本次数据的业务日期快照写进状态库(供前端/报告显示「是哪天的数据」)。 """下载成功后,把本次数据的业务日期快照写进状态库(供前端/报告显示「是哪天的数据」)。
业务日期 = 下载当天 该数据对应的日期偏移。__compare__ 无数据概念,跳过。 有 date 用 date否则 = 下载当天 该数据对应的日期偏移。__compare__ 无数据概念,跳过。
kind → 写入: kind → 写入:
expected/actual各写自己一列(偏移各取其列) expected/actual各写自己一列。
undelivered百世直供 0)写 undelivered4 站未到由 _site_undelivered_handler undelivered百世直供当天)写 undelivered4 站未到由 _site_undelivered_handler
内部连带下了 expected+actual不经 dispatch无业务日期写入故此处一并补写 内部连带下了 expected+actual不经 dispatch无业务日期写入故此处一并补写
expected/actual/undelivered 三列——actual 用 actual 偏移、未到跟随 expected 偏移。 expected/actual/undelivered 三列——actual 用 actual 偏移、未到跟随 expected 偏移。
顺带置 ready=True让前端不必等心跳即可反映下载成功写入失败仅告警、不影响任务判定""" 只写业务日期ready 语义已移交「入库成功」_persist_to_db 置位),此处不再碰 ready"""
if site == "__compare__": if site == "__compare__":
return return
today = datetime.now().date() today = datetime.now().date()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
def _write(k, off): def _write(k, biz_or_off):
biz = (today - timedelta(days=off)).strftime("%Y-%m-%d") # biz_or_off: int=偏移todayoffstr=已确定业务日期date
biz = (
(today - timedelta(days=biz_or_off)).strftime("%Y-%m-%d")
if isinstance(biz_or_off, int)
else biz_or_off
)
try: try:
state_store.set_data_state( state_store.set_business_date(site, k, biz)
site, k, ready=True, generated_at=now, business_date=biz
)
except Exception as e: except Exception as e:
print(f">> [状态] 写业务日期失败 {site}/{k}: {e}") print(f">> [状态] 写业务日期失败 {site}/{k}: {e}")
def off(kind_key):
return state_store.get_offset(site, kind_key)
if kind == "expected": if kind == "expected":
_write("expected", state_store.get_offset(site, "expected")) _write("expected", date if date else off("expected"))
elif kind == "actual": elif kind == "actual":
_write("actual", state_store.get_offset(site, "actual")) _write("actual", date if date else off("actual"))
elif site == "百世": elif site == "百世":
_write("undelivered", 0) _write("undelivered", 0)
else: # 4 站 undelivered连带补写 expected/actual/undelivered 三列 else: # 4 站 undelivered连带补写 expected/actual/undelivered 三列
_write("expected", state_store.get_offset(site, "expected")) _write("expected", date if date else off("expected"))
_write("actual", state_store.get_offset(site, "actual")) _write("actual", date if date else off("actual"))
_write("undelivered", state_store.get_offset(site, "expected")) _write("undelivered", date if date else off("expected"))
def _ready_flags(site):
"""从 PG 业务表派生单站三就绪态ready = DB 数据真相)。
expected/actual = PG 中存在对应 target_datetoday offset的数据
百世 undelivered = baishi_daily_stats 中存在 target_date 的数据;
4 站 undelivered = expected_ready ∧ actual_ready派生
PG 不可达时返回全 False降级安全不阻塞心跳
返回 (flags: {kind: bool}, dates: {kind: target_date_str})。
dates 与 flags 同源——ready=True 时 business_date 即该 target_date
彻底消除 ready 与 business_date 不同源导致的日期标签漂移。"""
from inbound_verify import store # 懒导入:避免模块级循环
today = date.today()
today_str = today.isoformat()
if site == "百世":
has_und, _ = store.has_data(site, "undelivered", today_str)
return (
{"expected": False, "actual": False, "undelivered": has_und},
{"undelivered": today_str},
)
exp_off = state_store.get_offset(site, "expected")
act_off = state_store.get_offset(site, "actual")
exp_date = (today - timedelta(days=exp_off)).isoformat()
act_date = (today - timedelta(days=act_off)).isoformat()
has_exp, _ = store.has_data(site, "expected", exp_date)
has_act, _ = store.has_data(site, "actual", act_date)
return (
{"expected": has_exp, "actual": has_act, "undelivered": has_exp and has_act},
{"expected": exp_date, "actual": act_date, "undelivered": exp_date},
)
def _apply_ready(site, flags, dates=None):
"""写入单站就绪态 + 业务日期同源ready 与 business_date 均据 PG + offset 派生)。
ready=True 时同步写入 target_date 作为 business_date消除不同源导致的日期标签漂移。
失败仅告警。"""
for k, rdy in flags.items():
try:
state_store.set_ready(site, k, rdy)
if rdy and dates and dates.get(k):
state_store.set_business_date(site, k, dates[k])
except Exception as e:
print(f">> [状态] 置就绪态失败 {site}/{k}: {e}")
def capture_error_screenshot(page, site, kind, attempt, error):
"""流程失败时截取当前页面,保存到 logs/screenshots/。
page: Playwright Page 对象(安能传 None 走 CDP 分支,调用方自行处理)。
截图失败绝不外抛——只打告警,不干扰任务重试/清场流程。"""
try:
os.makedirs(SCREENSHOT_DIR, exist_ok=True)
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
err_short = (error or "unknown")[:40].replace("/", "_").replace("\\", "_")
fname = f"{site}_{kind}_{ts}_attempt{attempt}_{err_short}.png"
path = os.path.join(SCREENSHOT_DIR, fname)
page.screenshot(path=path, full_page=False)
print(f"📸 【{site}-{kind}】错误截图已保存: {path}")
except Exception as se:
print(f"📸 【{site}-{kind}】截图失败(不影响任务): {se}")
def _refresh_ready(site):
"""入库后立即据 PG 派生并写入该站就绪态(省 30s 心跳等待,与心跳同源)。"""
flags, dates = _ready_flags(site)
_apply_ready(site, flags, dates)
def _persist_to_db(site, kind): def _persist_to_db(site, kind):
@@ -570,7 +692,16 @@ def _persist_to_db(site, kind):
print(">> [入库] 已关闭 (auto_ingest=false),跳过") print(">> [入库] 已关闭 (auto_ingest=false),跳过")
return return
count = store.ingest_task(site, kind) count = store.ingest_task(site, kind)
state_store.set_ingest_state(site, kind, ok=True, count=count) # 4 站 undelivered 连带入了 expected+actual按实际入库的类补记 ingest_state
# 否则心跳派生 readyexpected ∧ actual → undelivered会读到陈旧值。
logged = (
["expected", "actual", "undelivered"]
if kind == "undelivered" and site != "百世"
else [kind]
)
for k in logged:
state_store.set_ingest_state(site, k, ok=True, count=count)
_refresh_ready(site) # 入库成功 → 立即据 ingest_state 派生就绪态(与心跳同源)
print(f">> [入库] {site}/{kind} 成功,{count}") print(f">> [入库] {site}/{kind} 成功,{count}")
except Exception as e: except Exception as e:
print(f">> [warn] 入库失败 {site}/{kind}: {e}") print(f">> [warn] 入库失败 {site}/{kind}: {e}")
@@ -600,10 +731,10 @@ def dispatch_task(ctx, task_spec):
if handler is None: if handler is None:
return (state_store.TASK_FAILED, f"未知任务: {site}/{kind}") return (state_store.TASK_FAILED, f"未知任务: {site}/{kind}")
try: try:
ret = handler(ctx, bool(task_spec.get("force", False))) ret = handler(ctx, bool(task_spec.get("force", False)), task_spec.get("date"))
if ret is False: if ret is False:
return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)") return (state_store.TASK_FAILED, "任务执行失败(重试耗尽)")
_record_business_date(site, kind) _record_business_date(site, kind, task_spec.get("date"))
_persist_to_db(site, kind) _persist_to_db(site, kind)
return (state_store.TASK_SUCCESS, None) return (state_store.TASK_SUCCESS, None)
except Exception as e: except Exception as e:
@@ -614,8 +745,11 @@ def dispatch_task(ctx, task_spec):
def run_heartbeat(ctx): def run_heartbeat(ctx):
"""一轮心跳:探测各站登录态 + 数据文件,写状态库;登录态变化时提示。 """一轮心跳:探测各站登录态 + 据 PG 业务表派生数据就绪态;登录态变化时提示。
ready 直接查询 PG 业务表expected_record / actual_record / baishi_daily_stats
以「目标业务日期是否有数据」为唯一依据,彻底消除 ingest_state 日期比对带来的每日零点重置。
_refresh_ready 在入库瞬间即据 PG 派生(省 30s 等待),心跳同源复核。
只在 Playwright 所属线程调用。 只在 Playwright 所属线程调用。
""" """
prev = state_store.get_all_status() prev = state_store.get_all_status()
@@ -626,6 +760,5 @@ def run_heartbeat(ctx):
now_login = state_store.LOGIN_IN if logged_in else state_store.LOGIN_OUT now_login = state_store.LOGIN_IN if logged_in else state_store.LOGIN_OUT
if prev_login and prev_login not in (now_login, state_store.LOGIN_UNKNOWN): if prev_login and prev_login not in (now_login, state_store.LOGIN_UNKNOWN):
print(f"\n ⚠️【{site_name}】登录态变化: {prev_login}{now_login}") print(f"\n ⚠️【{site_name}】登录态变化: {prev_login}{now_login}")
for kind in ("expected", "actual", "undelivered"): flags, dates = _ready_flags(site_name)
ready, gen_at = probe_data_file(site_name, kind) _apply_ready(site_name, flags, dates)
state_store.set_data_state(site_name, kind, ready, gen_at)

View File

@@ -43,10 +43,37 @@ from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3): def _capture_error_screenshot(site, kind, attempt, error):
"""安能 CDP 错误截图best-effort失败仅告警绝不外抛"""
try:
import base64, os
from datetime import datetime
from inbound_verify.paths import SCREENSHOT_DIR
os.makedirs(SCREENSHOT_DIR, exist_ok=True)
pages = list_pages()
if not pages:
return
cdp = CDP(pages[0]["webSocketDebuggerUrl"])
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
err_short = (error or "unknown")[:40].replace("/", "_").replace("\\", "_")
fname = f"{site}_{kind}_{ts}_attempt{attempt}_{err_short}.png"
path = os.path.join(SCREENSHOT_DIR, fname)
result = cdp.call("Page.captureScreenshot", format="png")
with open(path, "wb") as f:
f.write(base64.b64decode(result["data"]))
cdp.close()
print(f"📸 【{site}-{kind}】错误截图已保存: {path}")
except Exception as se:
print(f"📸 【{site}-{kind}】截图失败(不影响任务): {se}")
def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。 """异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。 每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
(安能通过 CDP 截图page 参数忽略;保留为统一签名兼容。)
flow 为零参可调用;返回 False 视为失败,其余视为成功。 flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。 返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
""" """
@@ -60,6 +87,11 @@ def with_retry(site_name, label, flow, reset, max_attempts=3):
return True return True
except Exception as e: except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}") print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts:
try:
_capture_error_screenshot(site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...") print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try: try:
reset() reset()
@@ -855,15 +887,18 @@ def _load_query_days():
return max(1, days) return max(1, days)
def anneng_expected_download(force=False): def anneng_expected_download(force=False, date=None):
"""安能:应到货物数据下载(内部含异常兜底重试,路由层无感)。""" """安能:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry( return with_retry(
"安能", "应到", lambda: anneng_expected_download_impl(force=force), anneng_reset "安能",
"应到",
lambda: anneng_expected_download_impl(force=force, date=date),
anneng_reset,
) )
def anneng_expected_download_impl(force=False): def anneng_expected_download_impl(force=False, date=None):
"""安能:应到货物数据(运单信息)下载,完整流程(单次执行,无重试;供自动化测试用)。""" """安能:应到货物数据(运单信息)下载,完整流程(单次执行,无重试;供自动化测试用)。"""
print("\n▶ 开始执行【安能 - 应到货物数据下载】任务 ...") print("\n▶ 开始执行【安能 - 应到货物数据下载】任务 ...")
download_dir = DOWNLOAD_DIR download_dir = DOWNLOAD_DIR
@@ -874,11 +909,15 @@ def anneng_expected_download_impl(force=False):
offset = state_store.get_offset("安能") offset = state_store.get_offset("安能")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = f"{target.year}-{target.month:02d}-{target.day:02d}" target_str = f"{target.year}-{target.month:02d}-{target.day:02d}"
start_str = target_str start_str = target_str
today_str = target_str today_str = target_str
print(f">> 查询日期: [{target_str}]偏移 {offset}0=今天") src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 查询日期: [{target_str}]{src}")
main_cdp = find_main_page_cdp() main_cdp = find_main_page_cdp()
export_times = [] export_times = []
@@ -1258,13 +1297,15 @@ def _save_actual(rows, download_dir):
print("====================================================") print("====================================================")
def anneng_actual_download(force=False): def anneng_actual_download(force=False, date=None):
"""安能:实到数据下载(内部含异常兜底重试,路由层无感)。""" """安能:实到数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry("安能", "实到", anneng_actual_download_impl, anneng_reset) return with_retry(
"安能", "实到", lambda: anneng_actual_download_impl(date=date), anneng_reset
)
def anneng_actual_download_impl(): def anneng_actual_download_impl(date=None):
"""安能:实到数据(网点到件扫描,子单)下载,完整流程(单次执行,无重试;供自动化测试用)。""" """安能:实到数据(网点到件扫描,子单)下载,完整流程(单次执行,无重试;供自动化测试用)。"""
print("\n▶ 开始执行【安能 - 实到货物数据下载】任务 ...") print("\n▶ 开始执行【安能 - 实到货物数据下载】任务 ...")
download_dir = DOWNLOAD_DIR download_dir = DOWNLOAD_DIR
@@ -1275,10 +1316,14 @@ def anneng_actual_download_impl():
offset = state_store.get_offset("安能", "actual") offset = state_store.get_offset("安能", "actual")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
start_str = f"{target.year}/{target.month:02d}/{target.day:02d} 00:00:00" start_str = f"{target.year}/{target.month:02d}/{target.day:02d} 00:00:00"
end_str = f"{target.year}/{target.month:02d}/{target.day:02d} 23:59:59" end_str = f"{target.year}/{target.month:02d}/{target.day:02d} 23:59:59"
print(f">> 扫描日期: [{start_str}{end_str}]偏移 {offset}0=今天") src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 扫描日期: [{start_str}{end_str}]{src}")
main_cdp = find_main_page_cdp() main_cdp = find_main_page_cdp()
try: try:

View File

@@ -8,10 +8,11 @@ from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3): def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。 """异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。 每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。 flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。 返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
""" """
@@ -25,6 +26,13 @@ def with_retry(site_name, label, flow, reset, max_attempts=3):
return True return True
except Exception as e: except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}") print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...") print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try: try:
reset() reset()
@@ -137,14 +145,17 @@ def _close_tab(page, tab_name):
print(f" ⚠️ 关闭标签页【{tab_name}】时出错: {e}") print(f" ⚠️ 关闭标签页【{tab_name}】时出错: {e}")
def baishi_download_undelivered_data(page, force=False): def baishi_download_undelivered_data(page, force=False, date=None):
"""百世:一键提取应到未到(当日未扫)数据(内部含异常兜底重试,路由层无感)。""" """百世:一键提取应到未到(当日未扫)数据(内部含异常兜底重试,路由层无感)。
date 形参仅为对齐统一透传签名(百世固定下载当天),忽略。"""
return with_retry( return with_retry(
"百世", "百世",
"应到未到", "应到未到",
lambda: baishi_download_undelivered_data_impl(page), lambda: baishi_download_undelivered_data_impl(page),
lambda: baishi_reset(page), lambda: baishi_reset(page),
page=page,
) )
@@ -198,6 +209,11 @@ def baishi_download_undelivered_data_impl(page):
if _exp_n > 0: if _exp_n > 0:
state_store.set_setting("百世", "scan_expected_pieces", str(_exp_n)) state_store.set_setting("百世", "scan_expected_pieces", str(_exp_n))
state_store.set_setting("百世", "scan_arrived_pieces", str(_arr_n)) state_store.set_setting("百世", "scan_arrived_pieces", str(_arr_n))
from inbound_verify import (
store,
) # 直接落库 PG一步不绕 state_store→store
store.upsert_baishi_daily_stats(_exp_n, _arr_n)
print(f" 已记录百世应到/实到基数:应扫 {_exp_n} / 已扫 {_arr_n}") print(f" 已记录百世应到/实到基数:应扫 {_exp_n} / 已扫 {_arr_n}")
except Exception as _e: except Exception as _e:
# 抓取失败绝不影响未到明细下载主流程 # 抓取失败绝不影响未到明细下载主流程

View File

@@ -11,10 +11,11 @@ from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3): def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。 """异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。 每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。 flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。 返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
""" """
@@ -28,6 +29,13 @@ def with_retry(site_name, label, flow, reset, max_attempts=3):
return True return True
except Exception as e: except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}") print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...") print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try: try:
reset() reset()
@@ -88,6 +96,64 @@ def _remove_if_exists(path):
pass pass
def _shunxin_navigate_picker_to_target(page, date_str, max_flips=24):
"""顺心 Ant Design 单月面板跨月导航(车辆点到 / 卸车扫描记录 共用同一组件)。
读面板头部 .ant-picker-year-btn / .ant-picker-month-btn 得当前显示的年月,按与目标
年月的差值点 .ant-picker-header-prev-btn上月/ .ant-picker-header-next-btn下月
翻到目标月视窗。返回 True 表示当前视窗已是目标月(目标格子随后可见可点)。
调用前提:开始/结束时间输入已点开,.ant-picker-dropdown:visible 已就绪。
"""
try:
ty, tm = (int(x) for x in date_str.split("-")[:2])
except Exception:
return True # 解析不出就不翻,交由后续 cell.click 自行成败
drop = page.locator(".ant-picker-dropdown:visible")
for _ in range(max_flips):
try:
cur_y = int(
re.search(
r"\d+", drop.locator(".ant-picker-year-btn").first.inner_text()
).group()
)
cur_m = int(
re.search(
r"\d+", drop.locator(".ant-picker-month-btn").first.inner_text()
).group()
)
except Exception:
return False
cur = cur_y * 12 + (cur_m - 1)
tgt = ty * 12 + (tm - 1)
if cur == tgt:
return True
btn_sel = (
".ant-picker-header-prev-btn"
if tgt < cur
else ".ant-picker-header-next-btn"
)
drop.locator(btn_sel).first.click()
page.wait_for_timeout(300)
return False
def _shunxin_pick_date(page, date_str):
"""在已打开的顺心 Ant Design 日期浮层上选中指定日期格子(含跨月翻月)。
目标格子不在当前月视窗(跨月)时,先调 _shunxin_navigate_picker_to_target 翻到目标月,
再点格子;同月则直接点。与中通 _zto_flip_to_target_month 思路对称,适配 Ant Design 面板。
"""
cell = page.locator(f".ant-picker-dropdown:visible td[title='{date_str}']").first
if not cell.is_visible():
print(f" 目标日期 {date_str} 不在当前月视窗,正在翻月导航 ...")
if not _shunxin_navigate_picker_to_target(page, date_str):
raise RuntimeError(f"翻月后仍无法定位目标日期格子 {date_str}")
cell = page.locator(
f".ant-picker-dropdown:visible td[title='{date_str}']"
).first
cell.click()
def shunxin_belonging(page): def shunxin_belonging(page):
"""读取顺心当前账号的归属网点名(仅在首页可见,须在导航离开首页前调用)。 """读取顺心当前账号的归属网点名(仅在首页可见,须在导航离开首页前调用)。
@@ -149,7 +215,7 @@ def shunxin_merge_final(kind, tags):
pass pass
def shunxin_expected_download(pages, foreground=True, force=False): def shunxin_expected_download(pages, foreground=True, force=False, date=None):
"""顺心:应到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。 """顺心:应到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。
pages 为该站点的 page 列表(双账号在同一窗口的各一个标签页)。 pages 为该站点的 page 列表(双账号在同一窗口的各一个标签页)。
@@ -175,10 +241,11 @@ def shunxin_expected_download(pages, foreground=True, force=False):
ok = with_retry( ok = with_retry(
f"顺心-{tag}", f"顺心-{tag}",
"应到", "应到",
lambda p=pg, t=tag, f=force: shunxin_expected_download_impl( lambda p=pg, t=tag, f=force, d=date: shunxin_expected_download_impl(
p, out_tag=t, force=f p, out_tag=t, force=f, date=d
), ),
lambda p=pg: shunxin_reset(p), lambda p=pg: shunxin_reset(p),
page=pg,
) )
if not ok: if not ok:
return False # 某账号重试耗尽 → 整体失败,不融合(避免部分数据) return False # 某账号重试耗尽 → 整体失败,不融合(避免部分数据)
@@ -187,7 +254,7 @@ def shunxin_expected_download(pages, foreground=True, force=False):
return True return True
def shunxin_expected_download_impl(page, out_tag="", force=False): def shunxin_expected_download_impl(page, out_tag="", force=False, date=None):
"""顺心:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。 """顺心:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。
out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-应到货物数据.xlsx」 out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-应到货物数据.xlsx」
@@ -224,27 +291,27 @@ def shunxin_expected_download_impl(page, out_tag="", force=False):
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日 # 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("顺心") offset = state_store.get_offset("顺心")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d") target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str start_date_str = target_str
today_str = target_str today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]偏移 {offset}0=今天...") src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 正在设置查询日期: [{target_str}]{src}...")
# 分两步精准呼出和点击时间控件 # 分两步精准呼出和点击时间控件
print(" >> 设置起始时间...") print(" >> 设置起始时间...")
page.get_by_placeholder("开始时间").click() page.get_by_placeholder("开始时间").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
page.locator( _shunxin_pick_date(page, start_date_str)
f".ant-picker-dropdown:visible td[title='{start_date_str}']"
).first.click()
page.wait_for_timeout(300) page.wait_for_timeout(300)
print(" >> 设置截止时间...") print(" >> 设置截止时间...")
page.get_by_placeholder("结束时间").click() page.get_by_placeholder("结束时间").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
page.locator( _shunxin_pick_date(page, today_str)
f".ant-picker-dropdown:visible td[title='{today_str}']"
).first.click()
page.wait_for_timeout(300) page.wait_for_timeout(300)
# 确认日期 # 确认日期
@@ -516,7 +583,7 @@ def shunxin_expected_download_impl(page, out_tag="", force=False):
return False return False
def shunxin_actual_download(pages, foreground=True, force=False): def shunxin_actual_download(pages, foreground=True, force=False, date=None):
"""顺心:实到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。 """顺心:实到货物数据下载(双账号/双归属地,内部含异常兜底重试与数据融合)。
与 shunxin_expected_download 同构:读归属地 → 去重校验 → 顺序各账号下载 → 与 shunxin_expected_download 同构:读归属地 → 去重校验 → 顺序各账号下载 →
@@ -540,8 +607,11 @@ def shunxin_actual_download(pages, foreground=True, force=False):
ok = with_retry( ok = with_retry(
f"顺心-{tag}", f"顺心-{tag}",
"实到", "实到",
lambda p=pg, t=tag: shunxin_actual_download_impl(p, out_tag=t), lambda p=pg, t=tag, d=date: shunxin_actual_download_impl(
p, out_tag=t, date=d
),
lambda p=pg: shunxin_reset(p), lambda p=pg: shunxin_reset(p),
page=pg,
) )
if not ok: if not ok:
return False return False
@@ -550,7 +620,7 @@ def shunxin_actual_download(pages, foreground=True, force=False):
return True return True
def shunxin_actual_download_impl(page, out_tag=""): def shunxin_actual_download_impl(page, out_tag="", date=None):
"""顺心:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。 """顺心:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。
out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-实到货物数据.xlsx」 out_tag 为归属地标签时,合并产物命名为「顺心-{out_tag}-实到货物数据.xlsx」
@@ -583,27 +653,27 @@ def shunxin_actual_download_impl(page, out_tag=""):
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日 # 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("顺心", "actual") offset = state_store.get_offset("顺心", "actual")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_str = target.strftime("%Y-%m-%d") target_str = target.strftime("%Y-%m-%d")
start_date_str = target_str start_date_str = target_str
today_str = target_str today_str = target_str
print(f">> 正在设置查询日期: [{target_str}]偏移 {offset}0=今天...") src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 正在设置查询日期: [{target_str}]{src}...")
# 分两步精准呼出和点击时间控件 # 分两步精准呼出和点击时间控件
print(" >> 设置起始时间...") print(" >> 设置起始时间...")
page.get_by_placeholder("开始时间").click() page.get_by_placeholder("开始时间").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
page.locator( _shunxin_pick_date(page, start_date_str)
f".ant-picker-dropdown:visible td[title='{start_date_str}']"
).first.click()
page.wait_for_timeout(300) page.wait_for_timeout(300)
print(" >> 设置截止时间...") print(" >> 设置截止时间...")
page.get_by_placeholder("结束时间").click() page.get_by_placeholder("结束时间").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
page.locator( _shunxin_pick_date(page, today_str)
f".ant-picker-dropdown:visible td[title='{today_str}']"
).first.click()
page.wait_for_timeout(300) page.wait_for_timeout(300)
# 确认日期 # 确认日期

View File

@@ -11,10 +11,11 @@ from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3): def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。 """异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。 每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。 flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。 返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
""" """
@@ -28,6 +29,13 @@ def with_retry(site_name, label, flow, reset, max_attempts=3):
return True return True
except Exception as e: except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}") print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...") print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try: try:
reset() reset()
@@ -77,6 +85,64 @@ def _remove_if_exists(path):
pass pass
def _yunda_pick_laydate_new(ws_frame, page, date_ymd):
"""应到(新版 laydate .layui-laydate在已打开的面板上选中指定日期含跨月翻月
目标格子 td[lay-ymd='YYYY-M-D'](非补零)不在当前月视窗时,读 .laydate-set-ym 的
当前年月形如「2026年7月」按差值点 .laydate-prev-m / .laydate-next-m 翻到目标月,
再点格子;同月则直接点。调用前提:#startTime/#endTime 已点开,.layui-laydate:visible 就绪。
"""
cal = ws_frame.locator(".layui-laydate:visible").first
cell = cal.locator(f"td[lay-ymd='{date_ymd}']").first
if not cell.is_visible():
ty, tm = (int(x) for x in date_ymd.split("-")[:2])
print(f" 目标日期 {date_ymd} 不在当前月视窗,正在翻月导航 ...")
for _ in range(24):
nums = re.findall(r"\d+", cal.locator(".laydate-set-ym").first.inner_text())
if len(nums) >= 2:
cur_y, cur_m = int(nums[0]), int(nums[1])
if cur_y == ty and cur_m == tm:
break
cur = cur_y * 12 + (cur_m - 1)
btn = (
".laydate-prev-m"
if (ty * 12 + (tm - 1)) < cur
else ".laydate-next-m"
)
cal.locator(btn).first.click()
page.wait_for_timeout(300)
cell = cal.locator(f"td[lay-ymd='{date_ymd}']").first
cell.click()
def _yunda_pick_laydate_old(ws_frame, page, date_ymd):
"""实到(旧版 laydate #laydate_box在已打开的面板上选中指定日期含跨月翻月
目标格子 td[y][m][d](非补零)不在当前月视窗时,读 #laydate_y/#laydate_m 输入框值
形如「2026年」「07月」得当前年月按差值点 #laydate_MM 内 .laydate_chprev /
.laydate_chnext 翻到目标月,再点格子;同月则直接点。调用前提:#startDate/#endDate
已点开force=True#laydate_box:visible 就绪。
"""
box = ws_frame.locator("#laydate_box:visible").first
ty, tm, td = (int(x) for x in date_ymd.split("-")[:3])
cell = box.locator(f"td[y='{ty}'][m='{tm}'][d='{td}']").first
if not cell.is_visible():
print(f" 目标日期 {date_ymd} 不在当前月视窗,正在翻月导航 ...")
for _ in range(24):
yv = box.locator("#laydate_y").first.evaluate("e=>e.value")
mv = box.locator("#laydate_m").first.evaluate("e=>e.value")
cur_y = int(re.search(r"\d+", yv).group())
cur_m = int(re.search(r"\d+", mv).group())
if cur_y == ty and cur_m == tm:
break
cur = cur_y * 12 + (cur_m - 1)
btn = ".laydate_chprev" if (ty * 12 + (tm - 1)) < cur else ".laydate_chnext"
box.locator(f"#laydate_MM {btn}").first.click()
page.wait_for_timeout(300)
cell = box.locator(f"td[y='{ty}'][m='{tm}'][d='{td}']").first
cell.click()
def _resolve_export_frame(ws_frame): def _resolve_export_frame(ws_frame):
"""定位韵达数据导出面板内嵌的 iframe。 """定位韵达数据导出面板内嵌的 iframe。
@@ -169,18 +235,19 @@ def yunda_smart_menu_click(page, menu_path):
page.wait_for_timeout(1000) page.wait_for_timeout(1000)
def yunda_expected_download(page, force=False): def yunda_expected_download(page, force=False, date=None):
"""韵达:应到货物数据下载(内部含异常兜底重试,路由层无感)。""" """韵达:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry( return with_retry(
"韵达", "韵达",
"应到", "应到",
lambda: yunda_expected_download_impl(page, force=force), lambda: yunda_expected_download_impl(page, force=force, date=date),
lambda: yunda_reset(page), lambda: yunda_reset(page),
page=page,
) )
def yunda_expected_download_impl(page, force=False): def yunda_expected_download_impl(page, force=False, date=None):
"""韵达:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。""" """韵达:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【韵达 - 应到货物数据下载】任务...") print("\n▶ 开始执行【韵达 - 应到货物数据下载】任务...")
@@ -211,11 +278,15 @@ def yunda_expected_download_impl(page, force=False):
# 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日 # 2. 读取服务端日期偏移0=今天1=昨天…),单日范围:起止同日
offset = state_store.get_offset("韵达") offset = state_store.get_offset("韵达")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
target = datetime.strptime(date, "%Y-%m-%d")
else:
target = today - timedelta(days=offset)
target_ymd = f"{target.year}-{target.month}-{target.day}" target_ymd = f"{target.year}-{target.month}-{target.day}"
start_date_ymd = target_ymd start_date_ymd = target_ymd
today_ymd = target_ymd today_ymd = target_ymd
print(f">> 设置查询日期: [{target_ymd}]偏移 {offset}0=今天") src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print(f">> 设置查询日期: [{target_ymd}]{src}")
# 设定起始时间 # 设定起始时间
print(" >> 设置起始时间...") print(" >> 设置起始时间...")
@@ -224,7 +295,7 @@ def yunda_expected_download_impl(page, force=False):
calendar1 = ws_frame.locator(".layui-laydate:visible").first calendar1 = ws_frame.locator(".layui-laydate:visible").first
calendar1.wait_for(state="visible", timeout=5000) calendar1.wait_for(state="visible", timeout=5000)
calendar1.locator(f"td[lay-ymd='{start_date_ymd}']").click() _yunda_pick_laydate_new(ws_frame, page, start_date_ymd)
calendar1.locator(".laydate-btns-confirm").click() calendar1.locator(".laydate-btns-confirm").click()
page.wait_for_timeout(400) page.wait_for_timeout(400)
@@ -234,7 +305,7 @@ def yunda_expected_download_impl(page, force=False):
calendar2 = ws_frame.locator(".layui-laydate:visible").first calendar2 = ws_frame.locator(".layui-laydate:visible").first
calendar2.wait_for(state="visible", timeout=5000) calendar2.wait_for(state="visible", timeout=5000)
calendar2.locator(f"td[lay-ymd='{today_ymd}']").click() _yunda_pick_laydate_new(ws_frame, page, today_ymd)
calendar2.locator(".laydate-btns-confirm").click() calendar2.locator(".laydate-btns-confirm").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
@@ -422,18 +493,19 @@ def yunda_expected_download_impl(page, force=False):
return False return False
def yunda_actual_download(page, force=False): def yunda_actual_download(page, force=False, date=None):
"""韵达:实到货物数据下载(内部含异常兜底重试,路由层无感)。""" """韵达:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry( return with_retry(
"韵达", "韵达",
"实到", "实到",
lambda: yunda_actual_download_impl(page), lambda: yunda_actual_download_impl(page, date=date),
lambda: yunda_reset(page), lambda: yunda_reset(page),
page=page,
) )
def yunda_actual_download_impl(page): def yunda_actual_download_impl(page, date=None):
"""韵达:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。""" """韵达:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【韵达 - 实到货物数据下载】任务...") print("\n▶ 开始执行【韵达 - 实到货物数据下载】任务...")
@@ -464,30 +536,33 @@ def yunda_actual_download_impl(page):
offset = state_store.get_offset("韵达", "actual") offset = state_store.get_offset("韵达", "actual")
today = datetime.now() today = datetime.now()
target = today - timedelta(days=offset) if date:
start_date = target # 单日范围:起止同日 target = datetime.strptime(date, "%Y-%m-%d")
today = target # 让下方"截止时间"选择器也指向 target else:
target = today - timedelta(days=offset)
# 旧版 laydate 的日期格子 td[y][m][d] 用非补零整数值;单日范围起止同日
target_ymd = f"{target.year}-{target.month}-{target.day}"
src = f"指定 {date}" if date else f"偏移 {offset}0=今天"
print( print(
f">> 设置实到查询日期: [{target.year}-{target.month}-{target.day}]" f">> 设置实到查询日期: [{target.year}-{target.month}-{target.day}]{src}"
f"(偏移 {offset}0=今天)"
) )
print(" >> 正在设定起始时间...") print(" >> 正在设定起始时间...")
ws_frame.locator("#startDate").click() ws_frame.locator("#startDate").click()
page.wait_for_timeout(400) page.wait_for_timeout(400)
box1 = ws_frame.locator("#laydate_box:visible").first ws_frame.locator("#laydate_box:visible").first.wait_for(
box1.locator( state="visible", timeout=5000
f"td[y='{start_date.year}'][m='{start_date.month}'][d='{start_date.day}']" )
).click() _yunda_pick_laydate_old(ws_frame, page, target_ymd)
page.wait_for_timeout(400) page.wait_for_timeout(400)
print(" >> 正在设定截止时间...") print(" >> 正在设定截止时间...")
ws_frame.locator("#endDate").click() ws_frame.locator("#endDate").click()
page.wait_for_timeout(400) page.wait_for_timeout(400)
box2 = ws_frame.locator("#laydate_box:visible").first ws_frame.locator("#laydate_box:visible").first.wait_for(
box2.locator( state="visible", timeout=5000
f"td[y='{today.year}'][m='{today.month}'][d='{today.day}']" )
).click() _yunda_pick_laydate_old(ws_frame, page, target_ymd)
page.wait_for_timeout(500) page.wait_for_timeout(500)
print(" >> 正在变更扫描类型为【到件】...") print(" >> 正在变更扫描类型为【到件】...")

View File

@@ -4,17 +4,18 @@ import os
import re import re
import time import time
import yaml import yaml
from datetime import datetime from datetime import datetime, timedelta
import pandas as pd import pandas as pd
from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH from inbound_verify.paths import DOWNLOAD_DIR, CONFIG_PATH
from inbound_verify import state_store from inbound_verify import state_store
def with_retry(site_name, label, flow, reset, max_attempts=3): def with_retry(site_name, label, flow, reset, max_attempts=3, page=None):
"""异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。 """异常兜底flow 失败 → 重置回初始态 → 重试,最多 max_attempts 次(含首次)。
每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。 每次失败都重置(含最终放弃那一次):既是重试前的清场,也保证最终放弃时环境干净。
最后一次失败时(重置前)自动截图保存到 logs/screenshots/,供问题排查。
flow 为零参可调用;返回 False 视为失败,其余视为成功。 flow 为零参可调用;返回 False 视为失败,其余视为成功。
返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。 返回 True=最终成功False=重试耗尽放弃(供调度层判断任务成败)。
""" """
@@ -28,6 +29,13 @@ def with_retry(site_name, label, flow, reset, max_attempts=3):
return True return True
except Exception as e: except Exception as e:
print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}") print(f"⚠️ 【{site_name}-{label}】第 {attempt}/{max_attempts} 次失败: {e}")
if attempt == max_attempts and page is not None:
try:
from inbound_verify.runtime import capture_error_screenshot
capture_error_screenshot(page, site_name, label, attempt, str(e))
except Exception:
pass
print(f" → 重置【{site_name}】到初始态,清理环境 ...") print(f" → 重置【{site_name}】到初始态,清理环境 ...")
try: try:
reset() reset()
@@ -93,20 +101,50 @@ def _dom_click(locator):
) )
def _zto_compute_target_time(offset):
"""直接从 Python datetime 计算目标日期的毫秒级时间戳(本地时区零点)。
不再从 DOM 的 real-today 元素读取 time 属性,避免双月视图下 real-today
同时出现在 month1隐藏 ghost cell和 month2可见导致 .first 取到隐藏元素。
"""
target_date = datetime.now().date() - timedelta(days=offset)
target_dt = datetime(target_date.year, target_date.month, target_date.day)
return int(target_dt.timestamp() * 1000)
def _zto_find_visible_day(frame_locator, target_time):
"""在双月日期控件中查找可见的日期格子。
jQuery Date Range Picker 双月视图下,同一天可能出现在两个面板中:
- month1左面板的溢出 ghost celldisplay:none不可见
- month2右面板的正常 cell可见
同一日期在 DOM 中可能有毫秒级差异(零点 vs 23:59:59遍历匹配并返回
第一个 visible 的;无可见匹配返回 None。
"""
# 尝试两个时间变体:零点 和 23:59:59部分 checked/selected 格用后者)
for time_variant in (target_time, target_time + 86399000):
sel = f"td div.day[time='{time_variant}']"
cells = frame_locator.locator(sel)
count = cells.count()
for i in range(count):
if cells.nth(i).is_visible():
return cells.nth(i)
return None
def _zto_flip_to_target_month(frame_locator, page, target_time, max_flips=12): def _zto_flip_to_target_month(frame_locator, page, target_time, max_flips=12):
"""中通日历(jQuery-Date-Range-Picker 双月视图)跨月导航:目标日期不在当前视窗时, """中通日历(jQuery-Date-Range-Picker 双月视图)跨月导航:目标日期不在当前视窗时,
循环点 .prev 把目标月翻进 month1 视窗。offset 恒指向过去,故只往前翻;步长 1 月/次 循环点 .prev 把目标月翻进视窗。用 _zto_find_visible_day 判可见(跳过隐藏 ghost cell
用 JS 派发点击(.evaluate("el=>el.click()"))避开 .date-range-length-tip 等 hover 遮挡。
返回 True 若目标格子最终可见。""" 返回 True 若目标格子最终可见。"""
sel = f"td div.day[time='{target_time}']"
for _ in range(max_flips): for _ in range(max_flips):
if frame_locator.locator(sel).first.is_visible(): if _zto_find_visible_day(frame_locator, target_time) is not None:
return True return True
frame_locator.locator(".date-picker-wrapper .prev").first.evaluate( frame_locator.locator(".date-picker-wrapper .prev").first.evaluate(
"el => el.click()" "el => el.click()"
) )
page.wait_for_timeout(450) page.wait_for_timeout(450)
return frame_locator.locator(sel).first.is_visible() return _zto_find_visible_day(frame_locator, target_time) is not None
def zto_smart_menu_click(page, menu_path): def zto_smart_menu_click(page, menu_path):
@@ -125,18 +163,19 @@ def zto_smart_menu_click(page, menu_path):
page.wait_for_timeout(1000) page.wait_for_timeout(1000)
def zto_expected_download(page, force=False): def zto_expected_download(page, force=False, date=None):
"""中通:应到货物数据下载(内部含异常兜底重试,路由层无感)。""" """中通:应到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry( return with_retry(
"中通", "中通",
"应到", "应到",
lambda: zto_expected_download_impl(page, force=force), lambda: zto_expected_download_impl(page, force=force, date=date),
lambda: zto_reset(page), lambda: zto_reset(page),
page=page,
) )
def zto_expected_download_impl(page, force=False): def zto_expected_download_impl(page, force=False, date=None):
"""中通:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。""" """中通:应到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【中通 - 应到货物数据下载】任务...") print("\n▶ 开始执行【中通 - 应到货物数据下载】任务...")
@@ -160,37 +199,35 @@ def zto_expected_download_impl(page, force=False):
# 读取服务端日期偏移0=今天1=昨天…),单日:起止同日 # 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通") offset = state_store.get_offset("中通")
if date:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...") # 指定日期:折算成相对今天的有效偏移,复用下方 target_time 计算与跨月翻月
target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
ewb_frame.locator("#beginDate").click() ewb_frame.locator("#beginDate").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
today_cell = ewb_frame.locator("td div.day.real-today").first # 直接从 Python datetime 计算目标时间戳,不再依赖 DOM real-today双月视图
today_cell.wait_for(state="visible") # 下 real-today 可能同时出现在 month1 隐藏 ghost cell 和 month2 可见 cell
# .first 会取到隐藏的那个导致 wait_for(visible) 超时)。
target_time = _zto_compute_target_time(offset)
target_cell = _zto_find_visible_day(ewb_frame, target_time)
today_time_str = today_cell.get_attribute("time") if target_cell is None:
if today_time_str: print(" 目标日期不在当前视窗,正在翻月导航 ...")
today_time = int(today_time_str) if not _zto_flip_to_target_month(ewb_frame, page, target_time):
target_time = today_time - offset * 86400000 raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
target_cell = ewb_frame.locator(f"td div.day[time='{target_time}']").first target_cell = _zto_find_visible_day(ewb_frame, target_time)
if target_cell is None:
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
# 偏移日期跨月时目标格子不在当前双月视窗 → 向前翻月把它带进视窗,不再降级为当天 # 日期格子用 _dom_click 直接派发事件:.click() 会先 hover 格子,触发
if not target_cell.is_visible(): # "范围长度"提示气泡(.date-range-length-tip)盖住格子导致点击被遮挡超时。
print(" 偏移日期跨月,正在向前翻月导航到目标 ...") _dom_click(target_cell)
if not _zto_flip_to_target_month(ewb_frame, page, target_time): page.wait_for_timeout(300)
raise RuntimeError( _dom_click(target_cell)
f"翻月后仍无法定位目标日期格子(time={target_time})"
)
# 日期格子用 _dom_click 直接派发事件(同实到):.click() 会先 hover 格子,触发
# “范围长度”提示气泡(.date-range-length-tip)盖住格子导致点击被遮挡超时,
# 跨月选中非今日格子时尤为明显。
_dom_click(target_cell)
page.wait_for_timeout(300)
_dom_click(target_cell)
else:
today_cell.click()
page.wait_for_timeout(300)
today_cell.click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
@@ -373,18 +410,19 @@ def zto_expected_download_impl(page, force=False):
return False return False
def zto_actual_download(page, force=False): def zto_actual_download(page, force=False, date=None):
"""中通:实到货物数据下载(内部含异常兜底重试,路由层无感)。""" """中通:实到货物数据下载(内部含异常兜底重试,路由层无感)。"""
return with_retry( return with_retry(
"中通", "中通",
"实到", "实到",
lambda: zto_actual_download_impl(page), lambda: zto_actual_download_impl(page, date=date),
lambda: zto_reset(page), lambda: zto_reset(page),
page=page,
) )
def zto_actual_download_impl(page): def zto_actual_download_impl(page, date=None):
"""中通:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。""" """中通:实到货物数据下载(单次执行,无重试;供自动化测试探测原始结果用)。"""
print("\n▶ 开始执行【中通 - 实到货物数据下载】任务...") print("\n▶ 开始执行【中通 - 实到货物数据下载】任务...")
@@ -408,36 +446,34 @@ def zto_actual_download_impl(page):
# 2. 读取服务端日期偏移0=今天1=昨天…),单日:起止同日 # 2. 读取服务端日期偏移0=今天1=昨天…),单日:起止同日
offset = state_store.get_offset("中通", "actual") offset = state_store.get_offset("中通", "actual")
if date:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...") target_date = datetime.strptime(date, "%Y-%m-%d").date()
offset = (datetime.now().date() - target_date).days
print(f">> 正在设定查询日期: 指定日期 {date}(折算偏移 {offset}...")
else:
print(f">> 正在设定查询日期: 偏移 {offset}0=今天)...")
arr_frame.locator("#daterange").click() arr_frame.locator("#daterange").click()
page.wait_for_timeout(500) page.wait_for_timeout(500)
today_cell = arr_frame.locator("td div.day.real-today").first # 直接从 Python datetime 计算目标时间戳,不再依赖 DOM real-today双月视图
today_cell.wait_for(state="visible") # 下 real-today 可能同时出现在 month1 隐藏 ghost cell 和 month2 可见 cell
# .first 会取到隐藏的那个导致 wait_for(visible) 超时)。
target_time = _zto_compute_target_time(offset)
target_cell = _zto_find_visible_day(arr_frame, target_time)
today_time_str = today_cell.get_attribute("time") if target_cell is None:
if today_time_str: print(" 目标日期不在当前视窗,正在翻月导航 ...")
today_time = int(today_time_str) if not _zto_flip_to_target_month(arr_frame, page, target_time):
target_time = today_time - offset * 86400000 raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
target_cell = arr_frame.locator(f"td div.day[time='{target_time}']").first target_cell = _zto_find_visible_day(arr_frame, target_time)
if target_cell is None:
raise RuntimeError(f"翻月后仍无法定位目标日期格子(time={target_time})")
# 日期格子用 _dom_click 直接派发事件Playwright 的 .click() 会先 hover 格子, # 日期格子用 _dom_click 直接派发事件Playwright 的 .click() 会先 hover 格子,
# 触发范围长度提示气泡(.date-range-length-tip)盖住格子,导致点击被判遮挡而超时。 # 触发"范围长度"提示气泡(.date-range-length-tip)盖住格子,导致点击被判遮挡而超时。
# 偏移日期跨月时目标格子不在当前双月视窗 → 向前翻月把它带进视窗,不再降级为当天。 _dom_click(target_cell)
if not target_cell.is_visible(): page.wait_for_timeout(300)
print(" 偏移日期跨月,正在向前翻月导航到目标 ...") _dom_click(target_cell)
if not _zto_flip_to_target_month(arr_frame, page, target_time):
raise RuntimeError(
f"翻月后仍无法定位目标日期格子(time={target_time})"
)
_dom_click(target_cell)
page.wait_for_timeout(300)
_dom_click(target_cell)
else:
_dom_click(today_cell)
page.wait_for_timeout(300)
_dom_click(today_cell)
page.wait_for_timeout(500) page.wait_for_timeout(500)
@@ -686,7 +722,7 @@ def _zto_poll_and_download_tasks(page, export_times, download_dir, final_filenam
) )
# ==================================================================== # ====================================================================
# 所有目标文件下载完成后,关闭导出任务管理标签页 # 所有目标文件下载完成后,关闭"导出任务管理"标签页
# ==================================================================== # ====================================================================
print(">> 【导出任务管理】下载完成,正在关闭标签页...") print(">> 【导出任务管理】下载完成,正在关闭标签页...")
try: try:

View File

@@ -7,7 +7,7 @@
import os import os
import sqlite3 import sqlite3
from datetime import datetime from datetime import datetime, timedelta
from inbound_verify.paths import STATE_DB_PATH from inbound_verify.paths import STATE_DB_PATH
@@ -70,9 +70,22 @@ def init_db():
status TEXT, status TEXT,
started_at TEXT, started_at TEXT,
finished_at TEXT, finished_at TEXT,
error TEXT error TEXT,
trigger TEXT NOT NULL DEFAULT '',
target_date TEXT NOT NULL DEFAULT '',
force INTEGER NOT NULL DEFAULT 0
) )
""") """)
# 旧库迁移:补触发方式/目标日期/强制重下三列(新库已含;重复添加抛 OperationalError忽略
for _col, _typedef in [
("trigger", "TEXT NOT NULL DEFAULT ''"),
("target_date", "TEXT NOT NULL DEFAULT ''"),
("force", "INTEGER NOT NULL DEFAULT 0"),
]:
try:
conn.execute(f"ALTER TABLE task_history ADD COLUMN {_col} {_typedef}")
except sqlite3.OperationalError:
pass
conn.execute(""" conn.execute("""
CREATE TABLE IF NOT EXISTS site_config ( CREATE TABLE IF NOT EXISTS site_config (
site TEXT PRIMARY KEY, site TEXT PRIMARY KEY,
@@ -242,6 +255,25 @@ def set_data_state(site, kind, ready, generated_at, business_date=None):
_upsert(conn, site, **fields) _upsert(conn, site, **fields)
def set_business_date(site, kind, business_date):
"""仅写业务日期快照(不碰 ready/generated_at
下载成功钩子用ready 语义已移交「入库成功」(见 reset_data_ready / _persist_to_db
下载阶段只记业务日期,供前端状态盘显示「是哪天的数据」。
"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, **{f"{kind}_business_date": business_date or ""})
def set_ready(site, kind, ready):
"""仅写就绪态(不碰 business_date/generated_at
供心跳从 ingest_state 派生 ready 用——ready 现为 DB 入库真相的派生视图,
非启动重置、不读 Excel。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
_upsert(conn, site, **{f"{kind}_ready": 1 if ready else 0})
def get_all_status(): def get_all_status():
"""返回 {site: {各字段}};库不存在则返回 {}""" """返回 {site: {各字段}};库不存在则返回 {}"""
if not os.path.exists(STATE_DB_PATH): if not os.path.exists(STATE_DB_PATH):
@@ -338,6 +370,27 @@ def set_offset(site, kind, offset):
return offset return offset
def resolve_target_date(site, kind, date=None):
"""计算一条任务的目标下载日期YYYY-MM-DD供任务日志展示 / 重试回放)。
有 date 用 date否则按站点偏移推算与 runtime._record_business_date 同源):
expected → 应到偏移actual → 实到偏移;百世 undelivered → 当天;
4 站 undelivered → 跟随应到偏移。__compare__ 无数据概念,返回 ''"""
if site == "__compare__":
return ""
if date:
return date
today = datetime.now().date()
if kind == "expected":
return (today - timedelta(days=get_offset(site, "expected"))).strftime(
"%Y-%m-%d"
)
if kind == "actual":
return (today - timedelta(days=get_offset(site, "actual"))).strftime("%Y-%m-%d")
if site == "百世":
return today.strftime("%Y-%m-%d")
return (today - timedelta(days=get_offset(site, "expected"))).strftime("%Y-%m-%d")
def set_schedule(site, enabled, time_str): def set_schedule(site, enabled, time_str):
"""【DEPRECATED】旧"每日单时点定时"——已被 fetch_schedule 的周期+激活时段模式取代。 """【DEPRECATED】旧"每日单时点定时"——已被 fetch_schedule 的周期+激活时段模式取代。
保留死代码以防外部残留调用;新代码请用 set_fetch_schedule。""" 保留死代码以防外部残留调用;新代码请用 set_fetch_schedule。"""
@@ -520,26 +573,28 @@ def get_site_settings(site):
# ============================ 任务历史 ============================ # ============================ 任务历史 ============================
def create_task(site, kind): def create_task(site, kind, trigger="manual", target_date="", force=False):
"""新建一条 pending 任务,返回其 id。""" """新建一条 pending 任务(手动触发),返回其 id。trigger='manual'/'auto'
target_date 为该任务的目标下载日期YYYY-MM-DD可为 ''force 是否强制重下。"""
now = _now() now = _now()
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn: with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
cur = conn.execute( cur = conn.execute(
"INSERT INTO task_history (site, kind, status, started_at, finished_at, error) " "INSERT INTO task_history "
"VALUES (?, ?, ?, ?, '', '')", "(site, kind, status, started_at, finished_at, error, trigger, target_date, force) "
(site, kind, TASK_PENDING, now), "VALUES (?, ?, ?, ?, '', '', ?, ?, ?)",
(site, kind, TASK_PENDING, now, trigger, target_date, 1 if force else 0),
) )
conn.commit() conn.commit()
return cur.lastrowid return cur.lastrowid
def create_task_if_idle(site, kind): def create_task_if_idle(site, kind, trigger="auto", target_date=""):
"""周期调度专用:若该 (site,kind) 已有 pending/running 任务则返回 None跳过本次周期 """周期调度专用:若该 (site,kind) 已有 pending/running 任务则返回 None跳过本次周期
否则建一条 pending 任务返回其 id。单连接内 check-then-insert靠 SQLite 写锁把竞态压到忽略不计。 否则建一条 pending 任务返回其 id。单连接内 check-then-insert靠 SQLite 写锁把竞态压到忽略不计。
与 create_task 的区别:手动触发(POST /tasks)用 create_task用户点的必建周期 job 用本函数 与 create_task 的区别:手动触发(POST /tasks)用 create_task用户点的必建周期 job 用本函数
——上一次还没跑完时跳过,避免同 (site,kind) 任务堆积。手动建的任务会让紧随其后的周期 fire ——上一次还没跑完时跳过,避免同 (site,kind) 任务堆积。手动建的任务会让紧随其后的周期 fire
判到 inflight 而跳过,天然互斥。""" 判到 inflight 而跳过,天然互斥。trigger='auto'target_date 为目标下载日期YYYY-MM-DD"""
now = _now() now = _now()
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn: with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute( row = conn.execute(
@@ -550,9 +605,10 @@ def create_task_if_idle(site, kind):
if row: if row:
return None return None
cur = conn.execute( cur = conn.execute(
"INSERT INTO task_history (site, kind, status, started_at, finished_at, error) " "INSERT INTO task_history "
"VALUES (?, ?, ?, ?, '', '')", "(site, kind, status, started_at, finished_at, error, trigger, target_date, force) "
(site, kind, TASK_PENDING, now), "VALUES (?, ?, ?, ?, '', '', ?, ?, 0)",
(site, kind, TASK_PENDING, now, trigger, target_date),
) )
conn.commit() conn.commit()
return cur.lastrowid return cur.lastrowid
@@ -579,7 +635,8 @@ def get_task(task_id):
"""返回单条任务 dict不存在返回 None。""" """返回单条任务 dict不存在返回 None。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn: with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
row = conn.execute( row = conn.execute(
"SELECT id, site, kind, status, started_at, finished_at, error " "SELECT id, site, kind, status, started_at, finished_at, error, "
"trigger, target_date, force "
"FROM task_history WHERE id=?", "FROM task_history WHERE id=?",
(task_id,), (task_id,),
).fetchone() ).fetchone()
@@ -593,6 +650,9 @@ def get_task(task_id):
"started_at": row[4], "started_at": row[4],
"finished_at": row[5], "finished_at": row[5],
"error": row[6], "error": row[6],
"trigger": row[7],
"target_date": row[8],
"force": bool(row[9]),
} }
@@ -600,7 +660,8 @@ def list_tasks(limit=20):
"""返回最近 limit 条任务(按 id 倒序)。""" """返回最近 limit 条任务(按 id 倒序)。"""
with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn: with sqlite3.connect(STATE_DB_PATH, timeout=3.0) as conn:
rows = conn.execute( rows = conn.execute(
"SELECT id, site, kind, status, started_at, finished_at, error " "SELECT id, site, kind, status, started_at, finished_at, error, "
"trigger, target_date, force "
"FROM task_history ORDER BY id DESC LIMIT ?", "FROM task_history ORDER BY id DESC LIMIT ?",
(limit,), (limit,),
).fetchall() ).fetchall()
@@ -613,6 +674,9 @@ def list_tasks(limit=20):
"started_at": r[4], "started_at": r[4],
"finished_at": r[5], "finished_at": r[5],
"error": r[6], "error": r[6],
"trigger": r[7],
"target_date": r[8],
"force": bool(r[9]),
} }
for r in rows for r in rows
] ]

View File

@@ -173,6 +173,33 @@ def _to_int(v):
return None return None
def _batch_out_date_map(df, cfg, out_col="出库时间"):
"""按交接单号分组,计算批次归属日:出库日期众数,并列取较早日期。
返回 {handover_no: date};无出库时间/无交接单号的行不参与。"""
from collections import Counter
jd_col = cfg.get("exp_jd", "交接单号")
if jd_col not in df.columns or out_col not in df.columns:
return {}
counter: dict[str, Counter] = {}
for _, r in df.iterrows():
hn = str(r.get(jd_col, "")).strip()
if not hn or hn == "nan":
continue
d = _parse_time(r.get(out_col))
if d is None:
continue
counter.setdefault(hn, Counter())[d.date()] += 1
out = {}
for hn, cnt in counter.items():
if not cnt:
continue
max_n = max(cnt.values())
earliest = min(d for d, n in cnt.items() if n == max_n)
out[hn] = earliest
return out
def _parse_time(v): def _parse_time(v):
"""尽力解析多种时间格式为 datetime失败返回 None原始值在 raw 里)。""" """尽力解析多种时间格式为 datetime失败返回 None原始值在 raw 里)。"""
if v is None: if v is None:
@@ -233,13 +260,16 @@ def _read_business_dates():
_SQL_EXPECTED = """ _SQL_EXPECTED = """
INSERT INTO expected_record INSERT INTO expected_record
(site, waybill_no, handover_no, handover_pieces, order_pieces, business_date, raw) (site, waybill_no, handover_no, handover_pieces, order_pieces,
VALUES (%s,%s,%s,%s,%s,%s,%s) business_date, out_date, batch_out_date, raw)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, waybill_no) DO UPDATE SET ON CONFLICT (site, waybill_no) DO UPDATE SET
handover_no = EXCLUDED.handover_no, handover_no = EXCLUDED.handover_no,
handover_pieces = EXCLUDED.handover_pieces, handover_pieces = EXCLUDED.handover_pieces,
order_pieces = EXCLUDED.order_pieces, order_pieces = EXCLUDED.order_pieces,
business_date = COALESCE(EXCLUDED.business_date, expected_record.business_date), business_date = COALESCE(EXCLUDED.business_date, expected_record.business_date),
out_date = COALESCE(EXCLUDED.out_date, expected_record.out_date),
batch_out_date = COALESCE(EXCLUDED.batch_out_date, expected_record.batch_out_date),
raw = EXCLUDED.raw, raw = EXCLUDED.raw,
ingested_at = now() ingested_at = now()
""" """
@@ -268,6 +298,18 @@ _SQL_UNDELIVERED = """
ingested_at = now() ingested_at = now()
""" """
_SQL_BAISHI_DAILY_STATS = """
INSERT INTO baishi_daily_stats
(site, business_date, expected_pieces, arrived_pieces, undelivered_pieces, raw)
VALUES (%s,%s,%s,%s,%s,%s)
ON CONFLICT (site, business_date) DO UPDATE SET
expected_pieces = COALESCE(EXCLUDED.expected_pieces, baishi_daily_stats.expected_pieces),
arrived_pieces = COALESCE(EXCLUDED.arrived_pieces, baishi_daily_stats.arrived_pieces),
undelivered_pieces = COALESCE(EXCLUDED.undelivered_pieces, baishi_daily_stats.undelivered_pieces),
raw = EXCLUDED.raw,
ingested_at = now()
"""
# ============================== 入库 ============================== # ============================== 入库 ==============================
@@ -282,19 +324,31 @@ def _ingest_expected(cur, site, business_date):
df = pd.read_excel(path, dtype=str).fillna("") df = pd.read_excel(path, dtype=str).fillna("")
df = df.drop_duplicates(subset=[cfg["exp_wb"]], keep="first") df = df.drop_duplicates(subset=[cfg["exp_wb"]], keep="first")
biz = _parse_date(business_date) biz = _parse_date(business_date)
out_col = "出库时间"
has_out_col = out_col in df.columns
# 批次归属日:同交接单号出库日众数,并列取较早
batch_out_date = _batch_out_date_map(df, cfg, out_col)
rows = [] rows = []
for r in df.to_dict("records"): for r in df.to_dict("records"):
wb = str(r.get(cfg["exp_wb"], "")).strip() wb = str(r.get(cfg["exp_wb"], "")).strip()
if not wb: if not wb:
continue continue
out_date = None
if has_out_col:
out_dt = _parse_time(r.get(out_col))
if out_dt is not None:
out_date = out_dt.date()
hn = str(r.get(cfg["exp_jd"], "")).strip() or None
rows.append( rows.append(
( (
site, site,
wb, wb,
str(r.get(cfg["exp_jd"], "")).strip() or None, hn,
_to_int(r.get(cfg["exp_qty"])), _to_int(r.get(cfg["exp_qty"])),
_to_int(r.get("录单件数")), _to_int(r.get("录单件数")),
biz, biz,
out_date,
batch_out_date.get(hn),
Jsonb(_raw_row(r)), Jsonb(_raw_row(r)),
) )
) )
@@ -314,9 +368,10 @@ def _ingest_actual(cur, site):
cm = ACTUAL_COLMAP[site] cm = ACTUAL_COLMAP[site]
df = pd.read_excel(path, dtype=str).fillna("") df = pd.read_excel(path, dtype=str).fillna("")
if site == "韵达": if site == "韵达":
# 韵达业务清洗:抛弃「交接单号」为空的行(派件/签收等其他扫描无交接单号 # 韵达业务清洗:保留「交接单号」为空的行(到/接件扫描
# 抛弃「交接单号」不为空的行(派件/签收等,属重复数据)。
# 再按子单号去重一件多扫只留一条清洗后子单号已天然唯一drop 为保险)。 # 再按子单号去重一件多扫只留一条清洗后子单号已天然唯一drop 为保险)。
df = df[df["交接单号"].astype(str).str.strip() != ""] df = df[df["交接单号"].astype(str).str.strip() == ""]
df = df.drop_duplicates(subset=[cm["piece"]], keep="last") df = df.drop_duplicates(subset=[cm["piece"]], keep="last")
rows = [] rows = []
for r in df.to_dict("records"): for r in df.to_dict("records"):
@@ -368,6 +423,34 @@ def _ingest_undelivered_baishi(cur):
return len(rows) return len(rows)
def upsert_baishi_daily_stats(exp, arr, business_date=None):
"""直接落库百世当日应到/实到基数(应扫/已扫,站级日聚合)。
供 baishi 下载时抓到基数后直接调用(一步落库,不绕 state_store→store
business_date 默认今天百世固定当天。best-effort失败只告警不影响下载流程。"""
biz = business_date or date.today()
if exp is None and arr is None:
return
undel = (exp - arr) if (exp is not None and arr is not None) else None
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(
_SQL_BAISHI_DAILY_STATS,
(
"百世",
biz,
exp,
arr,
undel,
Jsonb({"expected": exp, "arrived": arr, "undelivered": undel}),
),
)
conn.commit()
print(f" [基数] 百世 {biz}: 应扫 {exp} / 已扫 {arr} / 未扫 {undel}")
except Exception as e:
print(f" [基数] 百世 {biz} 入库失败(不影响下载): {e}")
def ingest(site=None): def ingest(site=None):
"""入库:指定 site 则单站(百世只入未到),否则全站。返回总条数。""" """入库:指定 site 则单站(百世只入未到),否则全站。返回总条数。"""
dates = _read_business_dates() dates = _read_business_dates()
@@ -422,6 +505,69 @@ def ingest_task(site, kind):
return total return total
def backfill_out_date(site=None):
"""历史数据回填:从 raw->>'出库时间' 解析出库日,写入 out_date
再按交接单号聚合出库日众数,回填 batch_out_date。
site 为空时处理全部站点。返回回填 out_date 条数。"""
sites = [site] if site else ALL_SITES
total = 0
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
for s in sites:
# ── Step 1: 回填 out_date仅 NULL 行)──
cur.execute(
"SELECT id, raw FROM expected_record "
"WHERE site=%s AND out_date IS NULL",
(s,),
)
rows = cur.fetchall()
updates = []
for rid, raw in rows:
if not isinstance(raw, dict):
continue
out_dt = _parse_time(raw.get("出库时间"))
if out_dt is None:
continue
updates.append((out_dt.date(), rid))
if updates:
cur.executemany(
"UPDATE expected_record SET out_date=%s WHERE id=%s",
updates,
)
total += len(updates)
print(f" [回填] {s}{len(updates)}/{len(rows)}")
# ── Step 2: 回填 batch_out_date仅 NULL 行)──
cur.execute(
"SELECT id, handover_no, out_date FROM expected_record "
"WHERE site=%s AND batch_out_date IS NULL",
(s,),
)
rows = cur.fetchall()
if rows:
from collections import Counter
cnt: dict[str, Counter] = {}
for _, hn, od in rows:
if not hn or od is None:
continue
cnt.setdefault(hn, Counter())[od] += 1
batch_map = {}
for hn, c in cnt.items():
max_n = max(c.values())
batch_map[hn] = min(d for d, n in c.items() if n == max_n)
if batch_map:
cur.executemany(
"UPDATE expected_record SET batch_out_date=%s "
"WHERE site=%s AND handover_no=%s",
[(d, s, hn) for hn, d in batch_map.items()],
)
print(f" [回填] {s} batch_out_date{len(batch_map)} 个批次")
conn.commit()
print(f">> [回填] out_date 完成,共 {total}")
return total
def get_existing_handover_nos(site): def get_existing_handover_nos(site):
"""查该站点已落库的交接单号集合expected_record.handover_no """查该站点已落库的交接单号集合expected_record.handover_no
"提交导出任务前"去重:已落库的交接单号不再重复提交导出任务。 "提交导出任务前"去重:已落库的交接单号不再重复提交导出任务。
@@ -441,6 +587,52 @@ def get_existing_handover_nos(site):
return set() return set()
# ============================== PG 数据存在性查询 ==============================
def has_data(site, kind, target_date):
"""查询 PG指定站点在 target_date 是否有业务数据。
target_date: str 'YYYY-MM-DD' 或 date 对象。
返回 (has_rows: bool, count: int)。
PG 不可达时返回 (False, 0),不抛异常——调用方按「未确认存在」处理。
kind 路由:
expected → expected_record (business_date)
actual → actual_record (scan_time::date)
undelivered → 百世: baishi_daily_stats4 站: 不单独查(由调用方 expected∧actual 派生)
"""
if site == "百世" and kind == "undelivered":
sql = (
"SELECT COUNT(*) FROM baishi_daily_stats"
" WHERE site = %s AND business_date = %s"
)
params = (site, target_date)
elif kind == "expected":
sql = (
"SELECT COUNT(*) FROM expected_record"
" WHERE site = %s AND business_date = %s"
)
params = (site, target_date)
elif kind == "actual":
sql = (
"SELECT COUNT(*) FROM actual_record"
" WHERE site = %s AND scan_time::date = %s"
)
params = (site, target_date)
else:
return (False, 0)
try:
with _connect(_load_pg_config()["dbname"]) as conn:
with conn.cursor() as cur:
cur.execute(sql, params)
row = cur.fetchone()
cnt = int(row[0]) if row else 0
return (cnt > 0, cnt)
except Exception as e:
print(f">> [状态] PG 查询 {site}/{kind}/{target_date} 失败: {e}")
return (False, 0)
# ============================== 命令行 ============================== # ============================== 命令行 ==============================
@@ -466,6 +658,8 @@ def main():
sys.exit(1) sys.exit(1)
total = ingest_task(site, kind) total = ingest_task(site, kind)
print(f">> [ingest-one] {site}/{kind} 入库 {total}") print(f">> [ingest-one] {site}/{kind} 入库 {total}")
elif cmd == "backfill-out-date":
backfill_out_date(site)
else: else:
print(__doc__) print(__doc__)
sys.exit(1) sys.exit(1)

View File

@@ -19,12 +19,16 @@ CREATE TABLE IF NOT EXISTS expected_record (
handover_pieces INTEGER, -- 交接件数(应到件数口径) handover_pieces INTEGER, -- 交接件数(应到件数口径)
order_pieces INTEGER, -- 录单件数 order_pieces INTEGER, -- 录单件数
business_date DATE, -- 业务日期(属性,非唯一键;读不到则 NULL business_date DATE, -- 业务日期(属性,非唯一键;读不到则 NULL
out_date DATE, -- 出库日(批次归属日口径;从 raw.出库时间 解析)
batch_out_date DATE, -- 批次归属日(同交接单号出库日众数,并列取较早)
raw JSONB NOT NULL, -- 站点原始全列key=原列名) raw JSONB NOT NULL, -- 站点原始全列key=原列名)
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, waybill_no) UNIQUE (site, waybill_no)
); );
CREATE INDEX IF NOT EXISTS idx_expected_site_date ON expected_record (site, business_date); CREATE INDEX IF NOT EXISTS idx_expected_site_date ON expected_record (site, business_date);
CREATE INDEX IF NOT EXISTS idx_expected_handover ON expected_record (site, handover_no); CREATE INDEX IF NOT EXISTS idx_expected_handover ON expected_record (site, handover_no);
CREATE INDEX IF NOT EXISTS idx_expected_out_date ON expected_record (site, out_date);
CREATE INDEX IF NOT EXISTS idx_expected_batch_out_date ON expected_record (site, batch_out_date);
-- 实到货物(扫描件级:一扫描一行;每扫描一件系统生成一个单号) -- 实到货物(扫描件级:一扫描一行;每扫描一件系统生成一个单号)
CREATE TABLE IF NOT EXISTS actual_record ( CREATE TABLE IF NOT EXISTS actual_record (
@@ -53,3 +57,16 @@ CREATE TABLE IF NOT EXISTS undelivered_record (
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, piece_no) UNIQUE (site, piece_no)
); );
-- 百世日聚合(应扫/已扫基数:站级日聚合,区别于运单级/件级/子单级表)
CREATE TABLE IF NOT EXISTS baishi_daily_stats (
id BIGSERIAL PRIMARY KEY,
site TEXT NOT NULL, -- 百世
business_date DATE NOT NULL, -- 业务日期(百世固定当天)
expected_pieces INTEGER, -- 应扫(应到基数)
arrived_pieces INTEGER, -- 已扫(实到基数)
undelivered_pieces INTEGER, -- 未扫(=应扫-已扫,任一缺失则 NULL
raw JSONB NOT NULL, -- 原始抓取值
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (site, business_date)
);