AllForCompany/BLD_sync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: remove Excel migration pipeline; keep Access-only sync

Browse Source 
Drop the Excel (.xlsm) production-execution-card pipeline now that the
project only synchronizes Access databases to SQL Server.

- Delete excel_sync_to_sql.py, migration.py, config/field_mappings.py
- Remove Excel-only config symbols (EXCEL_CONFIGS, MIGRATION_TASKS,
  EXECUTION_CARD_FIELDS, CONTRACT_DATA_*, CACHE_DIR, TEMP_DIR,
  EXCEL_SYNC_* settings) from the config package
- Drop now-unused deps from requirements.txt: pandas, sqlalchemy, openpyxl
- Update .env.example, CLAUDE.md, and the uptime_kuma_utils docstring
- Fix the tube-bending workshop Access table mapping in SYNC_MAPPING
  (source table renamed; old name no longer exists)

The three Excel-sourced tables in warehouseOutbound (executionCardData,
contractData, customerProductType) and their data are left untouched.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
2026-06-17 13:09:36 +08:00
parent f9b2d3b4da
commit f6580b4994
11 changed files with 231 additions and 944 deletions
Download Patch File Download Diff File Expand all files Collapse all files

125
CLAUDE.md
View File

@@ -6,7 +6,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
This is a data synchronization system (BLD_sync) that extracts data from:
- **Microsoft Access databases** (.accdb) located on network shares
- **Excel files** (.xlsm) containing production execution cards and contract data
And synchronizes it to:
- **Microsoft SQL Server** (CompanyDB) with multiple schemas organized by data type
@@ -21,15 +20,11 @@ The system supports both full initialization sync and incremental sync driven by
┌─────────────────────────────────────────────────────────────────────┐
│ Network File Sources │
│ Access DBs (\\192.168.110.114\生产进度表\) │
│ Excel Files (\\192.168.110.113\生产执行卡\) │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Sync Scripts │
│ • etl_manager.py - Main ETL orchestrator (primary) │
│ • sync_excel_to_sql.py - Excel to SQL sync (legacy/alternative) │
│ • migration.py - Legacy customer product type migration │
│ • run_incremental_sync.py - Change log driven incremental sync │
│ • init_full_sync.py - Full table truncation/reload │
└─────────────────────────────────────────────────────────────────────┘
@@ -43,36 +38,18 @@ The system supports both full initialization sync and incremental sync driven by
└─────────────────────────────────────────────────────────────────────┘
```
> Note: The system previously also synced Excel `.xlsm` production-execution-card files into
> the `warehouseOutbound` schema (`executionCardData`, `contractData`, `customerProductType`).
> That Excel pipeline has been removed; the existing tables and their data remain in CompanyDB
> but are no longer updated.
### Key Components
**etl_manager.py** - Primary ETL orchestrator
- `DataSynchronizer` class manages Excel to SQL sync
- Supports incremental sync via file modification time comparison
- Caches Excel files locally in `excel_cache/` directory
- Uses SQLAlchemy with `fast_executemany=True` for bulk operations
- Generates `contractData` table via MERGE statement from `executionCardData`
**config.py** - Central configuration
- `SYNC_MAPPING`: Nested dict mapping Access files → tables → SQL targets
- `EXCEL_CONFIGS`: List of Excel file configs with sheet names and field mappings
- `TABLE_SCHEMA`: Column type definitions for data cleaning
- `NTFY_CONFIG`: Push notification settings
- `LOG_TABLE_CONFIG`: Change log table schema for incremental sync
**db_utils.py** - Database connection utilities
- `get_sql_conn()`: SQL Server connection via pyodbc
- `get_access_conn()`: Access database connection
- `fmt_table()`: Safe table name formatting `[schema].[table]`
- `generate_insert_sql()`: Dynamic INSERT statement generation
**ntfy_utils.py** - Push notifications
- Sends alerts to ntfy server on errors/completion
- Uses Bearer token authentication
**run_incremental_sync.py** - Log-driven incremental sync
- Polls `TableChangeLog` table for unsynced records (`Synced=0`)
- Queries by `TableAddress` to match configured file paths
- Processes deletions and insertions in batches
- Queries by `TableAddress` to match configured Access file paths
- Connects to the source Access DB, reads latest rows by primary key, then DELETE + INSERT
- Per-batch verification (delete/insert checks by primary key) before marking `Synced=1`
- Long-running service with configurable polling interval
**init_full_sync.py** - Full table reload
@@ -80,6 +57,33 @@ The system supports both full initialization sync and incremental sync driven by
- Handles IDENTITY_INSERT ON/OFF for tables with identity columns
- Progress logging with row counts and throughput metrics
**config/** - Configuration package
- `database.py`: SQL Server / Access driver settings (read from env via `python-dotenv`)
- `file_sources.py`: `SYNC_MAPPING` — nested dict mapping Access files → tables → SQL targets
- `app_settings.py`: `LOG_TABLE_CONFIG`, `NTFY_CONFIG`, `UPTIME_KUMA_CONFIG`, `POLL_INTERVAL`, `BATCH_SIZE`
- Symbols are re-exported from `config/__init__.py` (e.g. `from config import SYNC_MAPPING`)
**db_utils.py** - Database connection utilities
- `get_sql_conn()`: SQL Server connection via pyodbc
- `get_access_conn()`: Access database connection
- `fmt_table()`: Safe table name formatting `[schema].[table]`
- `generate_insert_sql()`: Dynamic INSERT statement generation
- `create_table_from_access()`: Auto-create SQL Server table from Access `cursor.description`
**ntfy_utils.py** - Push notifications
- Sends alerts to ntfy server on errors/completion
- Uses Bearer token authentication
**uptime_kuma_utils.py** - Heartbeat monitoring
- `UptimeKumaMonitor` pushes heartbeats to Uptime Kuma; used by the incremental sync service
**log_utils.py** - Unified logging
- `LoggerManager` creates a timestamped log file under `log/` and archives old logs by year-month
**vbareplace.py** / **vba.txt** - Access VBA helper tool
- Standalone drag-and-drop tool that injects the change-log VBA macro into Access forms and
refreshes linked tables (packaged via PyInstaller; unrelated to the Python sync scripts)
## Common Development Tasks
### Install Dependencies
@@ -94,60 +98,35 @@ pip install -r requirements.txt
python init_full_sync.py
```
### Run Excel to SQL Sync (Standard)
```bash
python etl_manager.py
```
Force sync all files (ignore modification times):
```bash
python etl_manager.py --force
```
### Run Incremental Sync Service (Change Log Driven)
```bash
python run_incremental_sync.py
```
### Run Legacy Migration Script
```bash
python migration.py
```
Edit `FORCE_UPDATE = True` in `migration.py` to force full refresh.
## Configuration Management
**Main Config File**: `config.py`
- Contains all database credentials, file paths, and mapping configurations
- Modify `SYNC_MAPPING` to add new Access files/tables
- Modify `EXCEL_CONFIGS` for new Excel sources
**Alternative Config**: `update_config.py`
- Used by `sync_excel_to_sql.py`
- Similar structure but different variable names
- Contains `EXECUTION_CARD_FIELDS`, `CONTRACT_DATA_MAPPING`
Configuration lives in the **`config/` package** (re-exported via `config/__init__.py`):
- Database credentials and driver strings come from environment variables (see `.env.example`),
loaded by `python-dotenv`.
- Modify `SYNC_MAPPING` in `config/file_sources.py` to add new Access files/tables.
## Important Implementation Details
### SQL Server Connection
- Uses ODBC Driver 18 for SQL Server
- Requires `TrustServerCertificate=yes` due to self-signed cert
- SQLAlchemy URL: `mssql+pyodbc:///?odbc_connect=...`
- Always enable `fast_executemany=True` for bulk operations
- Direct pyodbc connections (no SQLAlchemy); `fast_executemany=True` for bulk inserts
### Access Database Connection
- Driver: `{Microsoft Access Driver (*.mdb, *.accdb)}`
- Direct file path connection via pyodbc
### Data Cleaning
- Integer fields: `pd.to_numeric().round().astype('Int64')`
- String fields: Truncate to max length, replace empty with None
- Date fields: Convert to `date()` objects, None for NaT
- Duplicate removal: Based on primary key (usually `ID` or `总排号`)
### Type Mapping / Auto Table Creation
- When a SQL Server target table does not exist, it is created from the Access `cursor.description`.
- Access types map to SQL Server types: `int`→`INT` (PK → `IDENTITY(1,1) PRIMARY KEY`),
`float`→`FLOAT`, `bool`→`BIT`, `datetime`→`DATETIME`, `Decimal`→`DECIMAL(p,s)`,
`str`→`NVARCHAR(size)` (size > 4000 → `NVARCHAR(MAX)`).
### Identity Column Handling
Tables with identity columns require:
@@ -157,7 +136,7 @@ SET IDENTITY_INSERT [schema].[table] ON
SET IDENTITY_INSERT [schema].[table] OFF
```
See `has_identity_column()` in `init_full_sync.py` for detection logic.
See `has_identity_column()` in `init_full_sync.py` / `run_incremental_sync.py` for detection logic.
### File Path Matching in Incremental Sync
The change log table stores paths in VBA format:
@@ -169,7 +148,6 @@ The code constructs multiple match patterns for robust matching.
## Database Schema Organization
SQL Server schemas by function:
- `warehouseOutbound` - Execution card data, contract data, customer product types
- `productionContractData` - Contract data by year (25年/26年压力表/温度计)
- `productWarehousing` - Finished product inspection/warehousing records
- `workshopOne/Two/Three` - Workshop production records
@@ -181,16 +159,21 @@ SQL Server schemas by function:
- `solderingData` - Soldering operation records
- `TIGWelding` - TIG welding records
- `executionCardIssuanceRecord` - Execution card issuance records
- `tubeBending` - Tube bending records
- `warehouseOutbound` - (Legacy) Excel-sourced tables: `executionCardData`, `contractData`,
`customerProductType` — no longer updated
## Notifications
The system uses [ntfy](https://ntfy.sh/) for push notifications:
- Configured in `NTFY_CONFIG` within `config.py`
- Configured in `NTFY_CONFIG` within `config/app_settings.py`
- Sends on: errors, critical failures, task completion
- Authenticated via Bearer token
## Logging
- File logs: `log/` directory with timestamp rotation
- File logs: `log/` directory, one timestamped file per run (`<prefix>_<YYYYMMDD>_<HHMMSS>.log`)
- Console output: With emoji prefixes for status (✅ ❌ ⚠️ 🔄)
- Incremental sync: Uses `TimedRotatingFileHandler` for daily log files
- Old logs are archived into `log/Archive/YYYY-MM/` automatically
(see `LoggerManager.archive_old_logs` in `log_utils.py`; `archive_existing_logs.py` is a
one-time helper for historical logs)