BLD_sync/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a data synchronization system (BLD_sync) that extracts data from:
- **Microsoft Access databases** (.accdb) located on network shares
- **Excel files** (.xlsm) containing production execution cards and contract data

And synchronizes it to:
- **Microsoft SQL Server** (CompanyDB) with multiple schemas organized by data type

The system supports both full initialization sync and incremental sync driven by a change log table.

## Architecture

### Data Flow

```
┌─────────────────────────────────────────────────────────────────────┐
│                         Network File Sources                        │
│  Access DBs (\\192.168.110.114\生产进度表\)                         │
│  Excel Files (\\192.168.110.113\生产执行卡\)                        │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Sync Scripts                               │
│  • etl_manager.py         - Main ETL orchestrator (primary)         │
│  • sync_excel_to_sql.py   - Excel to SQL sync (legacy/alternative)  │
│  • migration.py           - Legacy customer product type migration   │
│  • run_incremental_sync.py - Change log driven incremental sync     │
│  • init_full_sync.py      - Full table truncation/reload           │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      SQL Server (CompanyDB)                         │
│  Schemas: productionContractData, productWarehousing,               │
│           workshopOne/Two/Three, machining, contractPlanning,       │
│           inspectionRecords, partsWarehouse, etc.                   │
└─────────────────────────────────────────────────────────────────────┘
```

### Key Components

**etl_manager.py** - Primary ETL orchestrator
- `DataSynchronizer` class manages Excel to SQL sync
- Supports incremental sync via file modification time comparison
- Caches Excel files locally in `excel_cache/` directory
- Uses SQLAlchemy with `fast_executemany=True` for bulk operations
- Generates `contractData` table via MERGE statement from `executionCardData`

**config.py** - Central configuration
- `SYNC_MAPPING`: Nested dict mapping Access files → tables → SQL targets
- `EXCEL_CONFIGS`: List of Excel file configs with sheet names and field mappings
- `TABLE_SCHEMA`: Column type definitions for data cleaning
- `NTFY_CONFIG`: Push notification settings
- `LOG_TABLE_CONFIG`: Change log table schema for incremental sync

**db_utils.py** - Database connection utilities
- `get_sql_conn()`: SQL Server connection via pyodbc
- `get_access_conn()`: Access database connection
- `fmt_table()`: Safe table name formatting `[schema].[table]`
- `generate_insert_sql()`: Dynamic INSERT statement generation

**ntfy_utils.py** - Push notifications
- Sends alerts to ntfy server on errors/completion
- Uses Bearer token authentication

**run_incremental_sync.py** - Log-driven incremental sync
- Polls `TableChangeLog` table for unsynced records (`Synced=0`)
- Queries by `TableAddress` to match configured file paths
- Processes deletions and insertions in batches
- Long-running service with configurable polling interval

**init_full_sync.py** - Full table reload
- Truncates target tables and reloads all data from Access
- Handles IDENTITY_INSERT ON/OFF for tables with identity columns
- Progress logging with row counts and throughput metrics

## Common Development Tasks

### Install Dependencies

```bash
pip install -r requirements.txt
```

### Run Full Initial Sync (Reload All Data)

```bash
python init_full_sync.py
```

### Run Excel to SQL Sync (Standard)

```bash
python etl_manager.py
```

Force sync all files (ignore modification times):
```bash
python etl_manager.py --force
```

### Run Incremental Sync Service (Change Log Driven)

```bash
python run_incremental_sync.py
```

### Run Legacy Migration Script

```bash
python migration.py
```

Edit `FORCE_UPDATE = True` in `migration.py` to force full refresh.

## Configuration Management

**Main Config File**: `config.py`
- Contains all database credentials, file paths, and mapping configurations
- Modify `SYNC_MAPPING` to add new Access files/tables
- Modify `EXCEL_CONFIGS` for new Excel sources

**Alternative Config**: `update_config.py`
- Used by `sync_excel_to_sql.py`
- Similar structure but different variable names
- Contains `EXECUTION_CARD_FIELDS`, `CONTRACT_DATA_MAPPING`

## Important Implementation Details

### SQL Server Connection
- Uses ODBC Driver 18 for SQL Server
- Requires `TrustServerCertificate=yes` due to self-signed cert
- SQLAlchemy URL: `mssql+pyodbc:///?odbc_connect=...`
- Always enable `fast_executemany=True` for bulk operations

### Access Database Connection
- Driver: `{Microsoft Access Driver (*.mdb, *.accdb)}`
- Direct file path connection via pyodbc

### Data Cleaning
- Integer fields: `pd.to_numeric().round().astype('Int64')`
- String fields: Truncate to max length, replace empty with None
- Date fields: Convert to `date()` objects, None for NaT
- Duplicate removal: Based on primary key (usually `ID` or `总排号`)

### Identity Column Handling
Tables with identity columns require:
```sql
SET IDENTITY_INSERT [schema].[table] ON
-- perform inserts
SET IDENTITY_INSERT [schema].[table] OFF
```

See `has_identity_column()` in `init_full_sync.py` for detection logic.

### File Path Matching in Incremental Sync
The change log table stores paths in VBA format:
- Network: `;DATABASE=\\server\share\file.accdb`
- Local: `LOCAL:\path\to\file.accdb` or `LOCAL=\path\to\file.accdb`

The code constructs multiple match patterns for robust matching.

## Database Schema Organization

SQL Server schemas by function:
- `warehouseOutbound` - Execution card data, contract data, customer product types
- `productionContractData` - Contract data by year (25年/26年压力表/温度计)
- `productWarehousing` - Finished product inspection/warehousing records
- `workshopOne/Two/Three` - Workshop production records
- `machining` - Machining process records (膜片焊接, 喷涂寄出, etc.)
- `contractPlanning` - Work order assignment records
- `inspectionRecords` - Quality inspection records
- `partsWarehouse` - Component inventory records
- `thermometerRecord` - Thermometer calibration/testing records
- `solderingData` - Soldering operation records
- `TIGWelding` - TIG welding records
- `executionCardIssuanceRecord` - Execution card issuance records

## Notifications

The system uses [ntfy](https://ntfy.sh/) for push notifications:
- Configured in `NTFY_CONFIG` within `config.py`
- Sends on: errors, critical failures, task completion
- Authenticated via Bearer token

## Logging

- File logs: `log/` directory with timestamp rotation
- Console output: With emoji prefixes for status (✅ ❌ ⚠️ 🔄)
- Incremental sync: Uses `TimedRotatingFileHandler` for daily log files