BLD_sync/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is a data synchronization system (BLD_sync) that extracts data from:

• Microsoft Access databases (.accdb) located on network shares
• Excel files (.xlsm) containing production execution cards and contract data

And synchronizes it to:

• Microsoft SQL Server (CompanyDB) with multiple schemas organized by data type

The system supports both full initialization sync and incremental sync driven by a change log table.

Architecture

Data Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         Network File Sources                        │
│  Access DBs (\\192.168.110.114\生产进度表\)                         │
│  Excel Files (\\192.168.110.113\生产执行卡\)                        │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          Sync Scripts                               │
│  • etl_manager.py         - Main ETL orchestrator (primary)         │
│  • sync_excel_to_sql.py   - Excel to SQL sync (legacy/alternative)  │
│  • migration.py           - Legacy customer product type migration   │
│  • run_incremental_sync.py - Change log driven incremental sync     │
│  • init_full_sync.py      - Full table truncation/reload           │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      SQL Server (CompanyDB)                         │
│  Schemas: productionContractData, productWarehousing,               │
│           workshopOne/Two/Three, machining, contractPlanning,       │
│           inspectionRecords, partsWarehouse, etc.                   │
└─────────────────────────────────────────────────────────────────────┘

Key Components

etl_manager.py - Primary ETL orchestrator

• DataSynchronizer class manages Excel to SQL sync
• Supports incremental sync via file modification time comparison
• Caches Excel files locally in excel_cache/ directory
• Uses SQLAlchemy with fast_executemany=True for bulk operations
• Generates contractData table via MERGE statement from executionCardData

config.py - Central configuration

• SYNC_MAPPING: Nested dict mapping Access files → tables → SQL targets
• EXCEL_CONFIGS: List of Excel file configs with sheet names and field mappings
• TABLE_SCHEMA: Column type definitions for data cleaning
• NTFY_CONFIG: Push notification settings
• LOG_TABLE_CONFIG: Change log table schema for incremental sync

db_utils.py - Database connection utilities

• get_sql_conn(): SQL Server connection via pyodbc
• get_access_conn(): Access database connection
• fmt_table(): Safe table name formatting [schema].[table]
• generate_insert_sql(): Dynamic INSERT statement generation

ntfy_utils.py - Push notifications

• Sends alerts to ntfy server on errors/completion
• Uses Bearer token authentication

run_incremental_sync.py - Log-driven incremental sync

• Polls TableChangeLog table for unsynced records (Synced=0)
• Queries by TableAddress to match configured file paths
• Processes deletions and insertions in batches
• Long-running service with configurable polling interval

init_full_sync.py - Full table reload

• Truncates target tables and reloads all data from Access
• Handles IDENTITY_INSERT ON/OFF for tables with identity columns
• Progress logging with row counts and throughput metrics

Common Development Tasks

Install Dependencies

pip install -r requirements.txt

Run Full Initial Sync (Reload All Data)

python init_full_sync.py

Run Excel to SQL Sync (Standard)

python etl_manager.py

Force sync all files (ignore modification times):

python etl_manager.py --force

Run Incremental Sync Service (Change Log Driven)

python run_incremental_sync.py

Run Legacy Migration Script

python migration.py

Edit FORCE_UPDATE = True in migration.py to force full refresh.

Configuration Management

Main Config File: config.py

• Contains all database credentials, file paths, and mapping configurations
• Modify SYNC_MAPPING to add new Access files/tables
• Modify EXCEL_CONFIGS for new Excel sources

Alternative Config: update_config.py

• Used by sync_excel_to_sql.py
• Similar structure but different variable names
• Contains EXECUTION_CARD_FIELDS, CONTRACT_DATA_MAPPING

Important Implementation Details

SQL Server Connection

• Uses ODBC Driver 18 for SQL Server
• Requires TrustServerCertificate=yes due to self-signed cert
• SQLAlchemy URL: mssql+pyodbc:///?odbc_connect=...
• Always enable fast_executemany=True for bulk operations

Access Database Connection

• Driver: {Microsoft Access Driver (*.mdb, *.accdb)}
• Direct file path connection via pyodbc

Data Cleaning

• Integer fields: pd.to_numeric().round().astype('Int64')
• String fields: Truncate to max length, replace empty with None
• Date fields: Convert to date() objects, None for NaT
• Duplicate removal: Based on primary key (usually ID or 总排号)

Identity Column Handling

Tables with identity columns require:

SET IDENTITY_INSERT [schema].[table] ON
-- perform inserts
SET IDENTITY_INSERT [schema].[table] OFF

See has_identity_column() in init_full_sync.py for detection logic.

File Path Matching in Incremental Sync

The change log table stores paths in VBA format:

• Network: ;DATABASE=\\server\share\file.accdb
• Local: LOCAL:\path\to\file.accdb or LOCAL=\path\to\file.accdb

The code constructs multiple match patterns for robust matching.

Database Schema Organization

SQL Server schemas by function:

• warehouseOutbound - Execution card data, contract data, customer product types
• productionContractData - Contract data by year (25年/26年压力表/温度计)
• productWarehousing - Finished product inspection/warehousing records
• workshopOne/Two/Three - Workshop production records
• machining - Machining process records (膜片焊接, 喷涂寄出, etc.)
• contractPlanning - Work order assignment records
• inspectionRecords - Quality inspection records
• partsWarehouse - Component inventory records
• thermometerRecord - Thermometer calibration/testing records
• solderingData - Soldering operation records
• TIGWelding - TIG welding records
• executionCardIssuanceRecord - Execution card issuance records

Notifications

The system uses ntfy for push notifications:

• Configured in NTFY_CONFIG within config.py
• Sends on: errors, critical failures, task completion
• Authenticated via Bearer token

Logging

• File logs: log/ directory with timestamp rotation
• Console output: With emoji prefixes for status (✅ ❌ ⚠️ 🔄)
• Incremental sync: Uses TimedRotatingFileHandler for daily log files