Replace Excel-based undelivered comparison with DB queries for all four
sites. The engine anchors on actual scan_time, reverse-lookups handover
batches, and compares expected vs actual waybill-by-waybill.
Shunxin SF waybills: use COUNT(*) instead of COUNT(DISTINCT piece_no)
since SF piece numbers are random and not derivable from the waybill.
Changes:
- db_compare.py: new module with compare_site_date(), compare_site_batch(),
write_result_excel(), and POST /compare API endpoint
- runtime.py: switch _site_undelivered_handler from compare.write_site_file
(Excel) to db_compare (DB); downloads succeed independently of comparison
- server.py: add POST /compare endpoint with date validation
- docs: implementation plan for Shunxin DB comparison
Co-Authored-By: Claude <noreply@anthropic.com>
The previous filter kept rows with non-empty handover numbers
(派件/签收 scans), which were duplicate rows. The correct logic
is to keep rows with empty handover numbers (到/接件 scans).
- store.py: change != "" to == "" in ingest filter
- compare.py: add same filter before comparison (previously missing)
Co-Authored-By: Claude <noreply@anthropic.com>
Aneng only supports querying data up to 31 days back. The previous
90-day cap allowed dates that would silently fail at download time,
so restrict the POST /tasks `date` validation to a 31-day window.
When the offset target date falls outside the current two-month view, flip .prev months (1 month/step) until the target day cell enters month1 view, instead of silently falling back to today. JS-dispatched clicks avoid the .date-range-length-tip hover interception.
Also switch the expected (#beginDate) day-cell click to _dom_click (matching actual #daterange): the range-length tooltip intercepts the second click on non-today cells during single-day range selection, causing timeouts.
Verified end-to-end: offset=45 (-> 2026-06-14, cross-month) expected task succeeded and downloaded 6-14 data without fallback.
Co-Authored-By: Claude <noreply@anthropic.com>
websocket.create_connection had no timeout, so a hung Runtime.evaluate
(Electron business tab not replying) blocked ws.recv() indefinitely — the
export-poll's 300s deadline and with_retry could never fire (observed as a
~22min hang on an 安能 expected download). A 15s socket timeout lets recv
raise WebSocketTimeoutException so wait_until/with_retry can fail and retry.
Co-Authored-By: Claude <noreply@anthropic.com>
Move the store.ingest_enabled() gate inside the main try/except. Previously
it sat outside as a standalone block: ingest_enabled() -> _load_pg_config()
raises FileNotFoundError when config.yaml is absent/malformed, which escaped
_persist_to_db, was caught by dispatch_task's outer except, and flipped a
successful download to FAILED -- violating the hook's never-raise invariant.
Now config errors print a [warn], record ok=False, and return silently.
Co-Authored-By: Claude <noreply@anthropic.com>
Third launch failure this session (after two 顺心 load-timeouts and a 中通 ERR_NAME_NOT_RESOLVED) — each a transient network blip that killed the whole worker because launch_and_prepare opened sites with no retry. Wrap each site open+goto in _open_page: up to 3 attempts (2s apart) on domcontentloaded, so transient DNS/timeout is absorbed; only persistent failure (all 3) aborts. All 5 sites + 安能 now launch clean.
Co-Authored-By: Claude <noreply@anthropic.com>
launch_and_prepare opened each site with Playwright's default wait_until='load', which waits for every resource (ads/trackers/images). sxne.sxjdfreight.com (顺心) reliably takes >30s to fire load, so its goto timed out and — launch not being retried — killed the whole worker, leaving only one page open. Switch the two launch gotos to wait_until='domcontentloaded' (return as soon as the DOM is ready; the readiness polling still gates login). Behavior for already-fast sites unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
git mv expected_undelivered.py -> compare.py; update the 3 importers (store/runtime/router) to import compare. All public names (main, write_site_file, _read_business_dates) unchanged. Also add the Tier 2 plan doc.
Co-Authored-By: Claude <noreply@anthropic.com>
Move ALL_REPORT_SITES / SITE_UNDELIVERED_FILE / BAISHI_FILE / BAISHI_COLUMNS / arrived_pieces_* / STATIONS / _site_cfg out of expected_undelivered into a new leaf module inbound_verify/domain.py. store.py and runtime.py now read site/file config from domain directly instead of through the compare engine (store keeps eu only for _read_business_dates). Behavior identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Remove the duplicate BASE/DOWNLOADS/OUTPUT self-anchor (the one that caused the Tier 1 hotfix bug when the file moved into the package). DOWNLOADS/OUTPUT now come from paths.py; OUTFILE derives from OUTPUT_DIR. Behavior identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Tier 1 regression: expected_undelivered.py carries its own BASE=dirname(__file__) anchor (a duplicate of paths.py). The package move dropped the file one level deeper, so BASE resolved to inbound_verify/ and DOWNLOADS/OUTPUT pointed at non-existent inbound_verify/downloads|output — while site downloads write to the project-root dirs via paths.py. process()/write_site_file() thus skipped the compare with '[跳过] downloads 下缺少 ...', returned None/False, and every undelivered task for the 4 web/app sites reported failed (重试耗尽) despite the files downloading fine. Fix: anchor BASE two levels up (mirrors paths.py). Verified: write_site_file returns True; all 4 undelivered tasks now success with 未到 files generated.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace pre-rename references (main_router, db_store, server.py) in code comments and docstrings with their new locations (cli/router, store, cli/server, runtime). Comment-only; no behavior change.
Co-Authored-By: Claude <noreply@anthropic.com>
Relocate 12 root .py modules into inbound_verify/ (sites/, cli/ subpackages). Rewrite all internal imports to package-qualified; drop the site_ prefix on the 5 site modules and their 28 call sites. Fix paths.py BASE_DIR to anchor at the project root. Add main() entry wrappers (cli/router, cli/server, store). No behavior change.
Co-Authored-By: Claude <noreply@anthropic.com>
Empty package markers (inbound_verify/, sites/, cli/) plus pyproject.toml with console_scripts entry points and dependency floors.
Co-Authored-By: Claude <noreply@anthropic.com>
