Add configuration and synchronization logic for Excel to SQL Server data processing
This commit is contained in:
Misaka_Company
196
CLAUDE.md
Normal file
196
CLAUDE.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
This is a data synchronization system (BLD_sync) that extracts data from:
|
||||
- **Microsoft Access databases** (.accdb) located on network shares
|
||||
- **Excel files** (.xlsm) containing production execution cards and contract data
|
||||
|
||||
And synchronizes it to:
|
||||
- **Microsoft SQL Server** (CompanyDB) with multiple schemas organized by data type
|
||||
|
||||
The system supports both full initialization sync and incremental sync driven by a change log table.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Network File Sources │
|
||||
│ Access DBs (\\192.168.110.114\生产进度表\) │
|
||||
│ Excel Files (\\192.168.110.113\生产执行卡\) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Sync Scripts │
|
||||
│ • etl_manager.py - Main ETL orchestrator (primary) │
|
||||
│ • sync_excel_to_sql.py - Excel to SQL sync (legacy/alternative) │
|
||||
│ • migration.py - Legacy customer product type migration │
|
||||
│ • run_incremental_sync.py - Change log driven incremental sync │
|
||||
│ • init_full_sync.py - Full table truncation/reload │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ SQL Server (CompanyDB) │
|
||||
│ Schemas: productionContractData, productWarehousing, │
|
||||
│ workshopOne/Two/Three, machining, contractPlanning, │
|
||||
│ inspectionRecords, partsWarehouse, etc. │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
**etl_manager.py** - Primary ETL orchestrator
|
||||
- `DataSynchronizer` class manages Excel to SQL sync
|
||||
- Supports incremental sync via file modification time comparison
|
||||
- Caches Excel files locally in `excel_cache/` directory
|
||||
- Uses SQLAlchemy with `fast_executemany=True` for bulk operations
|
||||
- Generates `contractData` table via MERGE statement from `executionCardData`
|
||||
|
||||
**config.py** - Central configuration
|
||||
- `SYNC_MAPPING`: Nested dict mapping Access files → tables → SQL targets
|
||||
- `EXCEL_CONFIGS`: List of Excel file configs with sheet names and field mappings
|
||||
- `TABLE_SCHEMA`: Column type definitions for data cleaning
|
||||
- `NTFY_CONFIG`: Push notification settings
|
||||
- `LOG_TABLE_CONFIG`: Change log table schema for incremental sync
|
||||
|
||||
**db_utils.py** - Database connection utilities
|
||||
- `get_sql_conn()`: SQL Server connection via pyodbc
|
||||
- `get_access_conn()`: Access database connection
|
||||
- `fmt_table()`: Safe table name formatting `[schema].[table]`
|
||||
- `generate_insert_sql()`: Dynamic INSERT statement generation
|
||||
|
||||
**ntfy_utils.py** - Push notifications
|
||||
- Sends alerts to ntfy server on errors/completion
|
||||
- Uses Bearer token authentication
|
||||
|
||||
**run_incremental_sync.py** - Log-driven incremental sync
|
||||
- Polls `TableChangeLog` table for unsynced records (`Synced=0`)
|
||||
- Queries by `TableAddress` to match configured file paths
|
||||
- Processes deletions and insertions in batches
|
||||
- Long-running service with configurable polling interval
|
||||
|
||||
**init_full_sync.py** - Full table reload
|
||||
- Truncates target tables and reloads all data from Access
|
||||
- Handles IDENTITY_INSERT ON/OFF for tables with identity columns
|
||||
- Progress logging with row counts and throughput metrics
|
||||
|
||||
## Common Development Tasks
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### Run Full Initial Sync (Reload All Data)
|
||||
|
||||
```bash
|
||||
python init_full_sync.py
|
||||
```
|
||||
|
||||
### Run Excel to SQL Sync (Standard)
|
||||
|
||||
```bash
|
||||
python etl_manager.py
|
||||
```
|
||||
|
||||
Force sync all files (ignore modification times):
|
||||
```bash
|
||||
python etl_manager.py --force
|
||||
```
|
||||
|
||||
### Run Incremental Sync Service (Change Log Driven)
|
||||
|
||||
```bash
|
||||
python run_incremental_sync.py
|
||||
```
|
||||
|
||||
### Run Legacy Migration Script
|
||||
|
||||
```bash
|
||||
python migration.py
|
||||
```
|
||||
|
||||
Edit `FORCE_UPDATE = True` in `migration.py` to force full refresh.
|
||||
|
||||
## Configuration Management
|
||||
|
||||
**Main Config File**: `config.py`
|
||||
- Contains all database credentials, file paths, and mapping configurations
|
||||
- Modify `SYNC_MAPPING` to add new Access files/tables
|
||||
- Modify `EXCEL_CONFIGS` for new Excel sources
|
||||
|
||||
**Alternative Config**: `update_config.py`
|
||||
- Used by `sync_excel_to_sql.py`
|
||||
- Similar structure but different variable names
|
||||
- Contains `EXECUTION_CARD_FIELDS`, `CONTRACT_DATA_MAPPING`
|
||||
|
||||
## Important Implementation Details
|
||||
|
||||
### SQL Server Connection
|
||||
- Uses ODBC Driver 18 for SQL Server
|
||||
- Requires `TrustServerCertificate=yes` due to self-signed cert
|
||||
- SQLAlchemy URL: `mssql+pyodbc:///?odbc_connect=...`
|
||||
- Always enable `fast_executemany=True` for bulk operations
|
||||
|
||||
### Access Database Connection
|
||||
- Driver: `{Microsoft Access Driver (*.mdb, *.accdb)}`
|
||||
- Direct file path connection via pyodbc
|
||||
|
||||
### Data Cleaning
|
||||
- Integer fields: `pd.to_numeric().round().astype('Int64')`
|
||||
- String fields: Truncate to max length, replace empty with None
|
||||
- Date fields: Convert to `date()` objects, None for NaT
|
||||
- Duplicate removal: Based on primary key (usually `ID` or `总排号`)
|
||||
|
||||
### Identity Column Handling
|
||||
Tables with identity columns require:
|
||||
```sql
|
||||
SET IDENTITY_INSERT [schema].[table] ON
|
||||
-- perform inserts
|
||||
SET IDENTITY_INSERT [schema].[table] OFF
|
||||
```
|
||||
|
||||
See `has_identity_column()` in `init_full_sync.py` for detection logic.
|
||||
|
||||
### File Path Matching in Incremental Sync
|
||||
The change log table stores paths in VBA format:
|
||||
- Network: `;DATABASE=\\server\share\file.accdb`
|
||||
- Local: `LOCAL:\path\to\file.accdb` or `LOCAL=\path\to\file.accdb`
|
||||
|
||||
The code constructs multiple match patterns for robust matching.
|
||||
|
||||
## Database Schema Organization
|
||||
|
||||
SQL Server schemas by function:
|
||||
- `warehouseOutbound` - Execution card data, contract data, customer product types
|
||||
- `productionContractData` - Contract data by year (25年/26年压力表/温度计)
|
||||
- `productWarehousing` - Finished product inspection/warehousing records
|
||||
- `workshopOne/Two/Three` - Workshop production records
|
||||
- `machining` - Machining process records (膜片焊接, 喷涂寄出, etc.)
|
||||
- `contractPlanning` - Work order assignment records
|
||||
- `inspectionRecords` - Quality inspection records
|
||||
- `partsWarehouse` - Component inventory records
|
||||
- `thermometerRecord` - Thermometer calibration/testing records
|
||||
- `solderingData` - Soldering operation records
|
||||
- `TIGWelding` - TIG welding records
|
||||
- `executionCardIssuanceRecord` - Execution card issuance records
|
||||
|
||||
## Notifications
|
||||
|
||||
The system uses [ntfy](https://ntfy.sh/) for push notifications:
|
||||
- Configured in `NTFY_CONFIG` within `config.py`
|
||||
- Sends on: errors, critical failures, task completion
|
||||
- Authenticated via Bearer token
|
||||
|
||||
## Logging
|
||||
|
||||
- File logs: `log/` directory with timestamp rotation
|
||||
- Console output: With emoji prefixes for status (✅ ❌ ⚠️ 🔄)
|
||||
- Incremental sync: Uses `TimedRotatingFileHandler` for daily log files
|
||||
319
etl_manager.py
Normal file
319
etl_manager.py
Normal file
@@ -0,0 +1,319 @@
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
import logging
|
||||
import argparse
|
||||
import datetime
|
||||
import urllib.parse
|
||||
import warnings
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
from sqlalchemy.types import NVARCHAR, Integer, Date
|
||||
|
||||
# 导入配置
|
||||
import update_config as config
|
||||
|
||||
# ================= 抑制 openpyxl 的数据验证警告 =================
|
||||
warnings.filterwarnings('ignore', category=UserWarning, module='openpyxl')
|
||||
|
||||
# ================= 日志配置 =================
|
||||
# 配置控制台输出使用 UTF-8 编码，确保中文正确显示
|
||||
console_handler = logging.StreamHandler(sys.stdout)
|
||||
console_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
console_handler,
|
||||
logging.FileHandler("sync_log.txt", encoding='utf-8')
|
||||
]
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class DataSynchronizer:
|
||||
def __init__(self, force_sync=False):
|
||||
self.force_sync = force_sync
|
||||
self.engine = self._get_db_connection()
|
||||
self.cache_dir = config.CACHE_DIR
|
||||
|
||||
if not os.path.exists(self.cache_dir):
|
||||
os.makedirs(self.cache_dir)
|
||||
|
||||
def _get_db_connection(self):
|
||||
connection_string = (
|
||||
f"DRIVER={{{config.DB_CONFIG['driver']}}};"
|
||||
f"SERVER={config.DB_CONFIG['server']};"
|
||||
f"DATABASE={config.DB_CONFIG['database']};"
|
||||
f"UID={config.DB_CONFIG['username']};"
|
||||
f"PWD={config.DB_CONFIG['password']};"
|
||||
f"TrustServerCertificate={config.DB_CONFIG.get('TrustServerCertificate', 'no')};"
|
||||
)
|
||||
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
|
||||
return create_engine(connection_url, fast_executemany=True)
|
||||
|
||||
def _should_process_file(self, remote_path, local_path):
|
||||
if self.force_sync:
|
||||
return True, "强制同步"
|
||||
|
||||
if not os.path.exists(local_path):
|
||||
return True, "缓存不存在"
|
||||
|
||||
try:
|
||||
remote_mtime = os.path.getmtime(remote_path)
|
||||
local_mtime = os.path.getmtime(local_path)
|
||||
if remote_mtime > local_mtime + 1:
|
||||
return True, f"源文件更新"
|
||||
except OSError as e:
|
||||
logger.error(f"无法访问源文件: {remote_path}, Error: {e}")
|
||||
return False, "源文件无法访问"
|
||||
|
||||
return False, "文件未变更"
|
||||
|
||||
def _clean_dataframe(self, df, contract_year):
|
||||
"""主表数据清洗与验证"""
|
||||
# 1. 设置合同年份
|
||||
df['合同年份'] = contract_year
|
||||
|
||||
# 2. 移除总排号为空的行
|
||||
if '总排号' in df.columns:
|
||||
df = df.dropna(subset=['总排号'])
|
||||
df = df[df['总排号'].astype(str).str.strip() != '']
|
||||
else:
|
||||
logger.error("数据源中找不到映射后的[总排号]列，跳过此 sheet")
|
||||
return None, None
|
||||
|
||||
# ★ 新增：去除重复的总排号（保留第一条）
|
||||
if '总排号' in df.columns:
|
||||
df['总排号'] = df['总排号'].astype(str).str.strip()
|
||||
duplicates = df[df.duplicated(subset=['总排号'], keep='first')]
|
||||
if not duplicates.empty:
|
||||
logger.warning(f"发现 {len(duplicates)} 条重复的总排号，已自动去重。重复的总排号: {duplicates['总排号'].tolist()[:10]}")
|
||||
df = df.drop_duplicates(subset=['总排号'], keep='first')
|
||||
|
||||
# 3. 补全列
|
||||
for col in config.TABLE_SCHEMA.keys():
|
||||
if col not in df.columns:
|
||||
df[col] = None
|
||||
|
||||
# 用于存储每一列的 SQL 类型
|
||||
dtype_dict = {}
|
||||
|
||||
# 4. 字段清洗
|
||||
for col, rules in config.TABLE_SCHEMA.items():
|
||||
if col not in df.columns:
|
||||
continue
|
||||
|
||||
if rules['type'] == 'int':
|
||||
# ★ 修改：先转换为数值，然后四舍五入到整数
|
||||
df[col] = pd.to_numeric(df[col], errors='coerce')
|
||||
# 将浮点数四舍五入为整数（处理如 123.5 这样的值）
|
||||
df[col] = df[col].round(0)
|
||||
# 转换为可空整数类型
|
||||
df[col] = df[col].astype('Int64')
|
||||
# 将 NaN 替换为 None
|
||||
df[col] = df[col].replace({pd.NA: None})
|
||||
dtype_dict[col] = Integer()
|
||||
|
||||
elif rules['type'] == 'date':
|
||||
df[col] = pd.to_datetime(df[col], errors='coerce')
|
||||
df[col] = df[col].apply(lambda x: x.date() if pd.notnull(x) else None)
|
||||
dtype_dict[col] = Date()
|
||||
|
||||
elif rules['type'] == 'str':
|
||||
# 先转换为字符串
|
||||
df[col] = df[col].fillna('').astype(str)
|
||||
# 替换各种空值表示
|
||||
df[col] = df[col].replace({'nan': '', 'None': '', '<NA>': ''})
|
||||
|
||||
# 强制截断
|
||||
max_len = rules.get('max_len', 255)
|
||||
df[col] = df[col].str.slice(0, max_len)
|
||||
# 将空字符串转为 None
|
||||
df[col] = df[col].replace('', None)
|
||||
|
||||
dtype_dict[col] = NVARCHAR(max_len)
|
||||
|
||||
final_cols = list(config.TABLE_SCHEMA.keys())
|
||||
|
||||
return df[final_cols], dtype_dict
|
||||
|
||||
def _sync_to_db(self, df, dtype_dict):
|
||||
"""同步主表数据 - 使用更稳健的方法"""
|
||||
if df is None or df.empty:
|
||||
return
|
||||
|
||||
target_table = "[warehouseOutbound].[executionCardData]"
|
||||
|
||||
with self.engine.connect() as conn:
|
||||
existing_ids = pd.read_sql(f"SELECT [总排号] FROM {target_table}", conn)
|
||||
|
||||
existing_id_set = set(existing_ids['总排号'].astype(str))
|
||||
df['总排号'] = df['总排号'].astype(str).str.strip()
|
||||
|
||||
df_update = df[df['总排号'].isin(existing_id_set)].copy()
|
||||
df_insert = df[~df['总排号'].isin(existing_id_set)].copy()
|
||||
|
||||
logger.info(f"分析结果: 需插入 {len(df_insert)} 条, 需更新 {len(df_update)} 条")
|
||||
|
||||
# 1. 插入新数据
|
||||
if not df_insert.empty:
|
||||
logger.info("正在执行批量插入...")
|
||||
df_insert.to_sql('executionCardData', self.engine, schema='warehouseOutbound',
|
||||
if_exists='append', index=False, chunksize=config.BATCH_SIZE,
|
||||
dtype=dtype_dict)
|
||||
logger.info("批量插入完成。")
|
||||
|
||||
# 2. 更新现有数据 - 改用逐条或小批量 UPDATE
|
||||
if not df_update.empty:
|
||||
logger.info("正在执行批量更新...")
|
||||
|
||||
cols = [c for c in df.columns if c != '总排号']
|
||||
set_clause = ", ".join([f"[{c}] = :{c}" for c in cols])
|
||||
update_sql = f"""
|
||||
UPDATE [warehouseOutbound].[executionCardData]
|
||||
SET {set_clause}
|
||||
WHERE [总排号] = :总排号
|
||||
"""
|
||||
|
||||
with self.engine.begin() as conn:
|
||||
batch_size = 1000
|
||||
total_rows = len(df_update)
|
||||
update_count = 0
|
||||
|
||||
for i in range(0, total_rows, batch_size):
|
||||
batch = df_update.iloc[i:i+batch_size]
|
||||
records = batch.to_dict('records')
|
||||
|
||||
result = conn.execute(text(update_sql), records)
|
||||
update_count += result.rowcount
|
||||
|
||||
if (i + batch_size) % 5000 == 0:
|
||||
logger.info(f"已更新 {i + batch_size}/{total_rows} 条记录...")
|
||||
|
||||
# 修复 SQL Server executemany 返回负数 rowcount 的问题
|
||||
affected_rows = abs(update_count) if update_count < 0 else total_rows
|
||||
logger.info(f"批量更新完成，共影响 {affected_rows} 行。")
|
||||
|
||||
def process_excel_files(self):
|
||||
for cfg in config.EXCEL_CONFIGS:
|
||||
remote_path = cfg['file_path']
|
||||
filename = os.path.basename(remote_path)
|
||||
local_path = os.path.join(self.cache_dir, filename)
|
||||
|
||||
should_sync, reason = self._should_process_file(remote_path, local_path)
|
||||
|
||||
if should_sync:
|
||||
logger.info(f"开始处理文件: {filename} ({reason})")
|
||||
try:
|
||||
# 复制文件到本地缓存（只复制一次）
|
||||
if os.path.exists(remote_path):
|
||||
shutil.copy2(remote_path, local_path)
|
||||
|
||||
# 遍历该文件的所有指定 sheet
|
||||
for sheet_name in cfg['sheet_names']:
|
||||
logger.info(f" → 处理工作表: {sheet_name} (合同年份: {cfg['contract_year']})")
|
||||
try:
|
||||
df = pd.read_excel(local_path, sheet_name=sheet_name, header=0, engine='openpyxl')
|
||||
df.columns = [str(c).strip() for c in df.columns]
|
||||
df.rename(columns=cfg['field_mapping'], inplace=True)
|
||||
|
||||
cleaned_df, dtype_mapping = self._clean_dataframe(df, cfg['contract_year'])
|
||||
|
||||
if cleaned_df is not None:
|
||||
self._sync_to_db(cleaned_df, dtype_mapping)
|
||||
logger.info(f" 工作表 {sheet_name} 同步成功。")
|
||||
else:
|
||||
logger.warning(f" 工作表 {sheet_name} 清洗失败，跳过。")
|
||||
except Exception as e:
|
||||
logger.error(f" 处理工作表 {sheet_name} 时发生错误: {str(e)}", exc_info=True)
|
||||
|
||||
logger.info(f"文件 {filename} 所有工作表处理完成。")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"处理文件 {filename} 时发生错误: {str(e)}", exc_info=True)
|
||||
else:
|
||||
logger.info(f"跳过文件: {filename} ({reason})")
|
||||
|
||||
def generate_contract_data(self):
|
||||
logger.info("开始生成/更新 contractData 表...")
|
||||
|
||||
merge_sql = """
|
||||
WITH SourceData AS (
|
||||
SELECT
|
||||
CAST(ISNULL([合同年份], '') AS NVARCHAR(10)) AS [合同年份],
|
||||
CAST(ISNULL([车间号], '') AS NVARCHAR(20)) AS [车间号],
|
||||
CAST(ISNULL([工令号], '') AS NVARCHAR(200)) AS [工令号],
|
||||
CAST([订单号] AS NVARCHAR(150)) AS [订单号],
|
||||
CAST([客户名称] AS NVARCHAR(200)) AS [客户名称],
|
||||
CAST([产品名称] AS NVARCHAR(200)) AS [产品型号],
|
||||
CAST([量程] AS NVARCHAR(150)) AS [量程],
|
||||
TRY_CAST([数量] AS INT) AS [数量],
|
||||
CAST(NULL AS INT) AS [单价],
|
||||
TRY_CAST([序号] AS INT) AS [ID],
|
||||
CAST([位号] AS NVARCHAR(500)) AS [位号],
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY [合同年份], [车间号], [工令号]
|
||||
ORDER BY [总排号] DESC
|
||||
) as rn
|
||||
FROM [warehouseOutbound].[executionCardData]
|
||||
WHERE
|
||||
[车间号] IS NOT NULL AND [车间号] <> ''
|
||||
AND [工令号] IS NOT NULL AND [工令号] <> ''
|
||||
)
|
||||
|
||||
MERGE INTO [warehouseOutbound].[contractData] AS Target
|
||||
USING (SELECT * FROM SourceData WHERE rn = 1) AS Source
|
||||
ON (
|
||||
Target.[合同年份] = Source.[合同年份]
|
||||
AND Target.[车间号] = Source.[车间号]
|
||||
AND Target.[工令号] = Source.[工令号]
|
||||
)
|
||||
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET
|
||||
Target.[订单号] = Source.[订单号],
|
||||
Target.[客户名称] = Source.[客户名称],
|
||||
Target.[产品型号] = Source.[产品型号],
|
||||
Target.[量程] = Source.[量程],
|
||||
Target.[数量] = Source.[数量],
|
||||
Target.[ID] = Source.[ID],
|
||||
Target.[位号] = Source.[位号]
|
||||
|
||||
WHEN NOT MATCHED BY TARGET THEN
|
||||
INSERT (
|
||||
[合同年份], [车间号], [工令号],
|
||||
[订单号], [客户名称], [产品型号],
|
||||
[量程], [数量], [单价], [ID], [位号]
|
||||
)
|
||||
VALUES (
|
||||
Source.[合同年份], Source.[车间号], Source.[工令号],
|
||||
Source.[订单号], Source.[客户名称], Source.[产品型号],
|
||||
Source.[量程], Source.[数量], Source.[单价], Source.[ID], Source.[位号]
|
||||
)
|
||||
;
|
||||
"""
|
||||
|
||||
try:
|
||||
with self.engine.begin() as conn:
|
||||
result = conn.execute(text(merge_sql))
|
||||
logger.info(f"ContractData 表同步完成 (SQL Server 内部处理)。rowcount: {result.rowcount}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"生成 ContractData 失败: {e}", exc_info=True)
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Excel数据同步至SQL Server")
|
||||
parser.add_argument('--force', action='store_true', help='强制同步所有文件')
|
||||
args = parser.parse_args()
|
||||
|
||||
syncer = DataSynchronizer(force_sync=args.force)
|
||||
logger.info("================= 任务开始 =================")
|
||||
syncer.process_excel_files()
|
||||
syncer.generate_contract_data()
|
||||
logger.info("================= 任务结束 =================")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +1,4 @@
|
||||
pyodbc>=5.0.0
|
||||
pandas>=1.5.0
|
||||
sqlalchemy>=2.0.0
|
||||
openpyxl>=3.0.0
|
||||
154
update_config.py
Normal file
154
update_config.py
Normal file
@@ -0,0 +1,154 @@
|
||||
import os
|
||||
|
||||
# ================= 数据库配置 =================
|
||||
DB_CONFIG = {
|
||||
"server": "192.168.110.114",
|
||||
"database": "CompanyDB",
|
||||
"username": "peng",
|
||||
"password": "Cqbld123456.",
|
||||
"driver": "ODBC Driver 18 for SQL Server",
|
||||
"TrustServerCertificate": "yes"
|
||||
}
|
||||
|
||||
# ================= 全局配置 =================
|
||||
CACHE_DIR = os.path.join(os.getcwd(), "temp")
|
||||
BATCH_SIZE = 5000
|
||||
|
||||
# ================= 字段清洗规则 =================
|
||||
# 基于 NVARCHAR (按字符数计算长度)
|
||||
TABLE_SCHEMA = {
|
||||
"合同年份": {"type": "str", "max_len": 10},
|
||||
"总排号": {"type": "str", "max_len": 50},
|
||||
"序号": {"type": "int"},
|
||||
"订单号": {"type": "str", "max_len": 150},
|
||||
"车间号": {"type": "str", "max_len": 20},
|
||||
"销售内部号": {"type": "str", "max_len": 50},
|
||||
"经办人": {"type": "str", "max_len": 20},
|
||||
"签订日期": {"type": "date"},
|
||||
"交货日期": {"type": "date"},
|
||||
"客户名称": {"type": "str", "max_len": 200},
|
||||
"产品名称": {"type": "str", "max_len": 200},
|
||||
"客户型号": {"type": "str", "max_len": 200},
|
||||
"选型型号": {"type": "str", "max_len": 200},
|
||||
"量程": {"type": "str", "max_len": 150},
|
||||
"数量": {"type": "int"},
|
||||
"备注2": {"type": "str", "max_len": 500},
|
||||
"备注1": {"type": "str", "max_len": 500},
|
||||
"位号": {"type": "str", "max_len": 500},
|
||||
"技术参数": {"type": "str", "max_len": 1000},
|
||||
"车间": {"type": "str", "max_len": 200},
|
||||
"工令号": {"type": "str", "max_len": 200},
|
||||
"接单日期": {"type": "date"},
|
||||
"新参数": {"type": "str", "max_len": 1000},
|
||||
"基本型号": {"type": "str", "max_len": 200},
|
||||
"公称外径": {"type": "str", "max_len": 200},
|
||||
"安装代码": {"type": "str", "max_len": 200},
|
||||
"设计形式": {"type": "str", "max_len": 200},
|
||||
"技术安装代码": {"type": "str", "max_len": 200},
|
||||
"隔膜类型": {"type": "str", "max_len": 200},
|
||||
"标准": {"type": "str", "max_len": 100},
|
||||
"隔膜大小": {"type": "str", "max_len": 200},
|
||||
"隔膜材质": {"type": "str", "max_len": 200},
|
||||
"膜片": {"type": "str", "max_len": 200},
|
||||
"膜片材质": {"type": "str", "max_len": 200},
|
||||
"CRM明细ID号": {"type": "str", "max_len": 200},
|
||||
"成品物料编码": {"type": "str", "max_len": 200},
|
||||
"工单号": {"type": "str", "max_len": 200},
|
||||
"盘号": {"type": "str", "max_len": 200},
|
||||
"编码": {"type": "str", "max_len": 200},
|
||||
"型批名称": {"type": "str", "max_len": 200},
|
||||
"型批型号": {"type": "str", "max_len": 200},
|
||||
"开票名称": {"type": "str", "max_len": 200},
|
||||
"开票型号": {"type": "str", "max_len": 200},
|
||||
"上数据时间": {"type": "date"},
|
||||
"生产代码1": {"type": "str", "max_len": 50},
|
||||
"生产代码2": {"type": "str", "max_len": 50}
|
||||
}
|
||||
|
||||
CONTRACT_MAPPING = {
|
||||
"合同年份": "合同年份",
|
||||
"车间号": "车间号",
|
||||
"工令号": "工令号",
|
||||
"订单号": "订单号",
|
||||
"客户名称": "客户名称",
|
||||
"产品型号": "选型型号",
|
||||
"量程": "量程",
|
||||
"数量": "数量",
|
||||
"单价": None,
|
||||
"ID": None,
|
||||
"位号": "位号"
|
||||
}
|
||||
|
||||
EXCEL_CONFIGS = [
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\往年生产执行卡\生产执行卡2022.xlsm",
|
||||
"sheet_names": ["Sheet1"],
|
||||
"contract_year": "2022",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\往年生产执行卡\生产执行卡2023（1-5月）.xlsm",
|
||||
"sheet_names": ["Sheet1"],
|
||||
"contract_year": "2023",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\往年生产执行卡\生产执行卡2023(6月-.xlsm",
|
||||
"sheet_names": ["Sheet1"],
|
||||
"contract_year": "2023",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\往年生产执行卡\生产执行卡2024 6月.xlsm",
|
||||
"sheet_names": ["重庆数据","北京数据"],
|
||||
"contract_year": "2024",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\往年生产执行卡\生产执行卡2024.xlsm",
|
||||
"sheet_names": ["重庆数据","北京数据"],
|
||||
"contract_year": "2024",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\生产执行卡2025年.xlsm",
|
||||
"sheet_names": ["重庆数据","北京数据"],
|
||||
"contract_year": "2025",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
},
|
||||
{
|
||||
"file_path": r"\\192.168.110.113\生产执行卡\生产执行卡2026年.xlsm",
|
||||
"sheet_names": ["重庆数据","北京数据"],
|
||||
"contract_year": "2026",
|
||||
"field_mapping": {
|
||||
"产品型号": "选型型号",
|
||||
"备注": "备注1",
|
||||
"下单日期": "接单日期"
|
||||
}
|
||||
}
|
||||
]
|
||||
Reference in New Issue
Block a user