feat: Excel to SQL Server migration tool for production execution cards
Migrates pressure gauge (PG) and thermometer (TM) data from .xlsm files into SQL Server executionCard schema, one table per year (PG_2022-2026, TM_2022-2026). Supports single/multi-table migration, column remapping, type inference, and dry-run mode via config.yaml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
490
docs/plans/2026-06-01-excel-to-sqlserver-migration.md
Normal file
490
docs/plans/2026-06-01-excel-to-sqlserver-migration.md
Normal file
@@ -0,0 +1,490 @@
|
||||
# Excel to SQL Server Migration Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** 将 Excel 中压力表(PG)和温度计(TM)的生产执行卡数据迁移到 SQL Server 的 `executionCard` Schema 中，每年一张表。
|
||||
|
||||
**Architecture:** 单一 Python 脚本 + YAML 配置文件。使用 openpyxl 读取 Excel + pyodbc 写入 SQL Server。所有数据源映射、SQL Server 连接信息、类型推断规则等均由 `config.yaml` 管理，脚本仅负责执行逻辑。对每个目标表：读取所有源工作表 → 合并列名取并集 → 按总排号去重 → 动态建表 → 批量插入。
|
||||
|
||||
**Tech Stack:** Python 3, openpyxl, pyodbc, PyYAML, SQL Server (ODBC Driver 18)
|
||||
|
||||
---
|
||||
|
||||
## SQL Server 目标结构
|
||||
|
||||
- **Schema:** `executionCard`
|
||||
- **表命名:** `executionCard.PG_YYYY` / `executionCard.TM_YYYY`
|
||||
- **主键:** `总排号 NVARCHAR(50) PRIMARY KEY`
|
||||
- **自增列:** `id INT IDENTITY(1,1)` （非主键，仅作辅助序号）
|
||||
|
||||
## 数据源映射
|
||||
|
||||
### 压力表 (PG)
|
||||
|
||||
| 目标表 | 文件 | 工作表 | 去重策略 |
|
||||
|--------|------|--------|---------|
|
||||
| PG_2022 | 生产执行卡2022.xlsm | Sheet1 | 直接导入 |
|
||||
| PG_2023 | 生产执行卡2023（1-5月）.xlsm | Sheet1 | 先导入1-5月，再追加6月后数据，按总排号去重(保留后者) |
|
||||
| PG_2023 | 生产执行卡2023(6月-.xlsm | Sheet1 | 同上 |
|
||||
| PG_2024 | 生产执行卡2024 6月.xlsm | 重庆数据 + 北京数据 | 6月版为超集，直接用；重庆北京合并取列并集 |
|
||||
| PG_2025 | 生产执行卡2025年.xlsm | 重庆数据 + 北京数据 | 合并取列并集 |
|
||||
| PG_2026 | 生产执行卡2026年.xlsm | 重庆数据 + 北京数据 | 合并取列并集 |
|
||||
|
||||
### 温度计 (TM)
|
||||
|
||||
| 目标表 | 文件 | 工作表 |
|
||||
|--------|------|--------|
|
||||
| TM_2022 | 生产执行卡(2022合同数据).xlsm | 数据 |
|
||||
| TM_2023 | 生产执行卡(2023合同数据).xlsm | 数据 |
|
||||
| TM_2024 | 生产执行卡(新版1).xlsm | 数据 |
|
||||
| TM_2025 | 生产执行卡2025.xlsm | 数据 |
|
||||
| TM_2026 | 生产执行卡2026.xlsm | 数据 |
|
||||
|
||||
## 列名处理规则
|
||||
|
||||
1. 保留中文列名，SQL Server 中用方括号包裹（如 `[总排号]`）
|
||||
2. 空列名 → 替换为 `col_N`（N 为列索引）
|
||||
3. 同名列但不同位置（如 PG_2025 重庆的"订单号"在第3列 vs 2022 的"订单号"在第5列）→ 统一按列名对齐
|
||||
4. 重复列名（如 TM 中出现了两次"数量"、"操作者"）→ 第二次出现的加后缀 `_2`
|
||||
|
||||
## 数据类型推断规则
|
||||
|
||||
按列名关键字自动推断，其余默认 `NVARCHAR(500)`：
|
||||
|
||||
| 关键字匹配 | SQL 类型 | 说明 |
|
||||
|------------|----------|------|
|
||||
| 日期、Date | DATETIME | 签订日期、交货日期、下单日期等 |
|
||||
| 数量 | INT | 数值型 |
|
||||
| 单价 | DECIMAL(18,2) | 金额型 |
|
||||
| 备注、技术参数、说明、转换数据 | NVARCHAR(MAX) | 长文本 |
|
||||
| 总排号 | NVARCHAR(50) NOT NULL PRIMARY KEY | 主键 |
|
||||
| 其余 | NVARCHAR(500) | 默认文本 |
|
||||
|
||||
## 去重规则
|
||||
|
||||
- **同一表多来源时**：按 `总排号` 去重，保留**后读取**的记录（后导入覆盖先导入）
|
||||
- **PG_2023**：先读1-5月文件，再读6月文件，重叠部分保留6月版本
|
||||
- **PG_2024**：仅使用6月版文件（已是原版超集）
|
||||
- **重庆+北京合并**：总排号前缀不同（C vs B），天然不重叠，直接合并
|
||||
|
||||
## 批量插入
|
||||
|
||||
- 每批 500 行
|
||||
- 使用 parameterized insert（防注入 + 正确处理特殊字符）
|
||||
- 插入时跳过总排号为空的行
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: 项目初始化 — 安装依赖、创建目录结构
|
||||
|
||||
**Files:**
|
||||
- Create: `requirements.txt`
|
||||
- Modify: `.venv` (install deps)
|
||||
|
||||
**Step 1: 创建 requirements.txt**
|
||||
|
||||
```txt
|
||||
openpyxl>=3.1.0
|
||||
pyodbc>=5.0.0
|
||||
pyyaml>=6.0
|
||||
```
|
||||
|
||||
**Step 2: 安装依赖**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/pip install -r requirements.txt
|
||||
```
|
||||
|
||||
**Step 3: 验证安装**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python -c "import openpyxl, pyodbc, yaml; print('OK')"
|
||||
```
|
||||
|
||||
Expected: `OK`
|
||||
|
||||
---
|
||||
|
||||
### Task 2: 创建 YAML 配置文件
|
||||
|
||||
**Files:**
|
||||
- Create: `config.yaml`
|
||||
|
||||
**Step 1: 编写 config.yaml**
|
||||
|
||||
所有配置集中在此文件，包括 SQL Server 连接、Schema、数据源映射、类型推断规则：
|
||||
|
||||
```yaml
|
||||
# Excel to SQL Server Migration Configuration
|
||||
|
||||
# ================= SQL Server 连接 =================
|
||||
sql_server:
|
||||
driver: "ODBC Driver 18 for SQL Server"
|
||||
server: "192.168.110.114"
|
||||
database: "CompanyDB"
|
||||
username: "peng"
|
||||
password: "Cqbld123456."
|
||||
TrustServerCertificate: "yes"
|
||||
|
||||
# ================= 目标 Schema =================
|
||||
schema: "executionCard"
|
||||
|
||||
# ================= 批量插入参数 =================
|
||||
batch_size: 500
|
||||
|
||||
# ================= 主键列名（用于去重） =================
|
||||
primary_key: "总排号"
|
||||
|
||||
# ================= 数据类型推断规则 =================
|
||||
type_rules:
|
||||
# 列名包含关键字 → SQL 类型
|
||||
datetime:
|
||||
- "日期"
|
||||
- "Date"
|
||||
int:
|
||||
- name: "数量"
|
||||
exact: true
|
||||
decimal:
|
||||
- name: "单价"
|
||||
exact: true
|
||||
long_text:
|
||||
- "备注"
|
||||
- "技术参数"
|
||||
- "说明"
|
||||
- "转换数据"
|
||||
- "新参数"
|
||||
primary_key_type: "NVARCHAR(50) NOT NULL"
|
||||
default_type: "NVARCHAR(500)"
|
||||
|
||||
# ================= 数据源映射 =================
|
||||
# 每个条目对应一个目标表
|
||||
# sources 中的顺序决定去重优先级：后出现的覆盖先出现的（同总排号时）
|
||||
tables:
|
||||
# ---------- 压力表 (PG) ----------
|
||||
- table: "PG_2022"
|
||||
sources:
|
||||
- file: "PG/生产执行卡2022.xlsm"
|
||||
sheet: "Sheet1"
|
||||
|
||||
- table: "PG_2023"
|
||||
sources:
|
||||
- file: "PG/生产执行卡2023（1-5月）.xlsm"
|
||||
sheet: "Sheet1"
|
||||
- file: "PG/生产执行卡2023(6月-.xlsm"
|
||||
sheet: "Sheet1"
|
||||
|
||||
- table: "PG_2024"
|
||||
sources:
|
||||
- file: "PG/生产执行卡2024 6月.xlsm"
|
||||
sheet: "重庆数据"
|
||||
- file: "PG/生产执行卡2024 6月.xlsm"
|
||||
sheet: "北京数据"
|
||||
|
||||
- table: "PG_2025"
|
||||
sources:
|
||||
- file: "PG/生产执行卡2025年.xlsm"
|
||||
sheet: "重庆数据"
|
||||
- file: "PG/生产执行卡2025年.xlsm"
|
||||
sheet: "北京数据"
|
||||
|
||||
- table: "PG_2026"
|
||||
sources:
|
||||
- file: "PG/生产执行卡2026年.xlsm"
|
||||
sheet: "重庆数据"
|
||||
- file: "PG/生产执行卡2026年.xlsm"
|
||||
sheet: "北京数据"
|
||||
|
||||
# ---------- 温度计 (TM) ----------
|
||||
- table: "TM_2022"
|
||||
sources:
|
||||
- file: "TM/生产执行卡(2022合同数据).xlsm"
|
||||
sheet: "数据"
|
||||
|
||||
- table: "TM_2023"
|
||||
sources:
|
||||
- file: "TM/生产执行卡(2023合同数据).xlsm"
|
||||
sheet: "数据"
|
||||
|
||||
- table: "TM_2024"
|
||||
sources:
|
||||
- file: "TM/生产执行卡(新版1).xlsm"
|
||||
sheet: "数据"
|
||||
|
||||
- table: "TM_2025"
|
||||
sources:
|
||||
- file: "TM/生产执行卡2025.xlsm"
|
||||
sheet: "数据"
|
||||
|
||||
- table: "TM_2026"
|
||||
sources:
|
||||
- file: "TM/生产执行卡2026.xlsm"
|
||||
sheet: "数据"
|
||||
```
|
||||
|
||||
**Step 2: 验证 YAML 可正确解析**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python -c "import yaml; cfg=yaml.safe_load(open('config.yaml',encoding='utf-8')); print(f'Tables: {len(cfg[\"tables\"])}, Schema: {cfg[\"schema\"]}')"
|
||||
```
|
||||
|
||||
Expected: `Tables: 10, Schema: executionCard`
|
||||
|
||||
---
|
||||
|
||||
### Task 3: 创建迁移脚本骨架 — 加载配置
|
||||
|
||||
**Files:**
|
||||
- Create: `migrate.py`
|
||||
|
||||
**Step 1: 编写 migrate.py 的配置加载部分**
|
||||
|
||||
```python
|
||||
"""
|
||||
Excel to SQL Server Migration Script
|
||||
Migrates production execution card data from Excel (.xlsm) to SQL Server.
|
||||
Configuration is loaded from config.yaml.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from collections import OrderedDict
|
||||
|
||||
import openpyxl
|
||||
import pyodbc
|
||||
import yaml
|
||||
|
||||
|
||||
def load_config(config_path=None):
|
||||
"""加载 YAML 配置文件"""
|
||||
if config_path is None:
|
||||
config_path = os.path.join(
|
||||
os.path.dirname(os.path.abspath(__file__)), 'config.yaml'
|
||||
)
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
|
||||
def get_connection_string(cfg):
|
||||
"""根据配置构建 pyodbc 连接字符串"""
|
||||
sql = cfg['sql_server']
|
||||
return (
|
||||
f"DRIVER={{{sql['driver']}}};"
|
||||
f"SERVER={sql['server']};"
|
||||
f"DATABASE={sql['database']};"
|
||||
f"UID={sql['username']};"
|
||||
f"PWD={sql['password']};"
|
||||
f"TrustServerCertificate={sql['TrustServerCertificate']};"
|
||||
)
|
||||
```
|
||||
|
||||
**Step 2: 验证脚本可导入无语法错误**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python -c "import ast; ast.parse(open('migrate.py').read()); print('Syntax OK')"
|
||||
```
|
||||
|
||||
Expected: `Syntax OK`
|
||||
|
||||
---
|
||||
|
||||
### Task 4: 实现 Excel 读取与数据合并逻辑
|
||||
|
||||
**Files:**
|
||||
- Modify: `migrate.py`
|
||||
|
||||
**Step 1: 编写 read_sheet() 和 merge_sources() 函数**
|
||||
|
||||
核心逻辑：
|
||||
|
||||
```python
|
||||
def read_sheet(file_path, sheet_name):
|
||||
"""读取单个工作表，返回 (列名列表, 数据行列表)。
|
||||
跳过总排号为空的行。"""
|
||||
|
||||
def merge_sources(sources):
|
||||
"""合并多个工作表的数据。
|
||||
- 列名取并集（按出现顺序）
|
||||
- 数据按总排号去重，后读覆盖先读
|
||||
- 处理重复列名（加 _2 后缀）
|
||||
返回 (unified_headers, merged_data_ordered_dict)"""
|
||||
```
|
||||
|
||||
关键处理：
|
||||
1. `read_sheet`: 用 openpyxl read_only 模式读取，第一行为表头，其余为数据。跳过总排号（第1列）为空的行。
|
||||
2. `merge_sources`: 遍历所有 source，收集所有列名并去重保留顺序。每个 source 的数据按其列名映射到统一列名集。用 OrderedDict 以总排号为 key 去重。
|
||||
|
||||
---
|
||||
|
||||
### Task 5: 实现 SQL Server Schema 创建与表创建
|
||||
|
||||
**Files:**
|
||||
- Modify: `migrate.py`
|
||||
|
||||
**Step 1: 编写 create_schema() 和 infer_sql_type() 和 create_table() 函数**
|
||||
|
||||
```python
|
||||
def get_connection(cfg):
|
||||
"""根据配置建立 SQL Server 连接，返回 pyodbc.Connection"""
|
||||
|
||||
def create_schema(cursor, schema_name):
|
||||
"""创建 schema（如不存在）"""
|
||||
|
||||
def infer_sql_type(col_name, type_rules, primary_key):
|
||||
"""根据 config.yaml 中的 type_rules 推断 SQL Server 数据类型"""
|
||||
|
||||
def create_table(cursor, table_name, headers, cfg):
|
||||
"""根据列名列表动态创建表。
|
||||
- id INT IDENTITY(1,1)
|
||||
- 总排号（由 cfg['primary_key'] 指定）为主键
|
||||
- 其余列根据 infer_sql_type 推断
|
||||
"""
|
||||
```
|
||||
|
||||
`infer_sql_type` 从 YAML 配置读取规则：
|
||||
```python
|
||||
def infer_sql_type(col_name, type_rules, primary_key):
|
||||
name = col_name
|
||||
# 主键列
|
||||
if name == primary_key or name.startswith('20BW'):
|
||||
return type_rules['primary_key_type']
|
||||
# 遍历类型规则
|
||||
if any(kw in name for kw in type_rules.get('datetime', [])):
|
||||
return 'DATETIME'
|
||||
for rule in type_rules.get('int', []):
|
||||
if rule.get('exact') and rule['name'] == name:
|
||||
return 'INT'
|
||||
elif not rule.get('exact') and rule['name'] in name:
|
||||
return 'INT'
|
||||
for rule in type_rules.get('decimal', []):
|
||||
if rule.get('exact') and rule['name'] == name:
|
||||
return 'DECIMAL(18,2)'
|
||||
elif not rule.get('exact') and rule['name'] in name:
|
||||
return 'DECIMAL(18,2)'
|
||||
if any(kw in name for kw in type_rules.get('long_text', [])):
|
||||
return 'NVARCHAR(MAX)'
|
||||
return type_rules.get('default_type', 'NVARCHAR(500)')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: 实现批量数据插入
|
||||
|
||||
**Files:**
|
||||
- Modify: `migrate.py`
|
||||
|
||||
**Step 1: 编写 batch_insert() 函数**
|
||||
|
||||
```python
|
||||
def batch_insert(cursor, table_name, headers, data_dict, cfg):
|
||||
"""批量插入数据到 SQL Server。
|
||||
- data_dict: OrderedDict，key=总排号, value=dict
|
||||
- batch_size 从 cfg['batch_size'] 读取
|
||||
- 使用 parameterized INSERT
|
||||
- 每 batch_size 行提交一次
|
||||
"""
|
||||
```
|
||||
|
||||
核心：
|
||||
- 构建 INSERT 语句：`INSERT INTO [executionCard].[{table_name}] ([col1], [col2], ...) VALUES (?, ?, ...)`
|
||||
- 参数化插入，pyodbc 自动处理类型转换
|
||||
- 空值传 `None`
|
||||
|
||||
---
|
||||
|
||||
### Task 7: 实现主流程与进度输出
|
||||
|
||||
**Files:**
|
||||
- Modify: `migrate.py`
|
||||
|
||||
**Step 1: 编写 migrate_table() 和 main() 函数**
|
||||
|
||||
```python
|
||||
def migrate_table(cursor, table_config, cfg, dry_run=False):
|
||||
"""迁移单个表：
|
||||
1. merge_sources 获取合并数据
|
||||
2. create_table 建表（若非 dry-run）
|
||||
3. batch_insert 写入数据（若非 dry-run）
|
||||
打印进度信息
|
||||
"""
|
||||
|
||||
def main():
|
||||
"""主入口：
|
||||
1. 解析命令行参数（--config, --dry-run）
|
||||
2. 加载 config.yaml
|
||||
3. 连接 SQL Server（若非 dry-run）
|
||||
4. 创建 schema（若非 dry-run）
|
||||
5. 遍历 cfg['tables']，逐表迁移
|
||||
6. 打印汇总
|
||||
"""
|
||||
```
|
||||
|
||||
进度输出格式：
|
||||
```
|
||||
[1/10] 迁移 PG_2022 ...
|
||||
读取: Excel/PG/生产执行卡2022.xlsm > Sheet1 → 52,854 行
|
||||
建表: [executionCard].[PG_2022] (39 列)
|
||||
插入: 52,854 行 ✓
|
||||
[2/10] 迁移 PG_2023 ...
|
||||
读取: Excel/PG/生产执行卡2023（1-5月）.xlsm > Sheet1 → 27,797 行
|
||||
读取: Excel/PG/生产执行卡2023(6月-.xlsm > Sheet1 → 30,646 行
|
||||
合并去重后: 58,443 行
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: 测试运行（dry-run 模式）
|
||||
|
||||
**Step 1: 确认 --dry-run 参数支持**
|
||||
|
||||
dry-run 模式下只读取数据、打印建表 SQL 和统计信息，不实际写入数据库。
|
||||
|
||||
**Step 2: 执行 dry-run 验证数据读取正确**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python migrate.py --dry-run
|
||||
```
|
||||
|
||||
Expected: 打印所有表的列名、行数、建表 SQL，无报错
|
||||
|
||||
---
|
||||
|
||||
### Task 9: 正式执行迁移
|
||||
|
||||
**Step 1: 运行迁移脚本**
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python migrate.py
|
||||
```
|
||||
|
||||
**Step 2: 验证数据**
|
||||
|
||||
对每张表执行验证查询：
|
||||
```sql
|
||||
SELECT COUNT(*) FROM executionCard.PG_2022;
|
||||
SELECT COUNT(*) FROM executionCard.PG_2023;
|
||||
-- ... 每张表
|
||||
-- 抽样检查
|
||||
SELECT TOP 5 * FROM executionCard.PG_2022;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 预估数据量
|
||||
|
||||
| 表名 | 预估行数 | 列数 |
|
||||
|------|---------|------|
|
||||
| executionCard.PG_2022 | ~52,854 | 38+1(id) |
|
||||
| executionCard.PG_2023 | ~58,443 | 38+1 |
|
||||
| executionCard.PG_2024 | ~64,004 | 46+1 |
|
||||
| executionCard.PG_2025 | ~70,093 | 47+1 |
|
||||
| executionCard.PG_2026 | ~29,232 | 46+1 |
|
||||
| executionCard.TM_2022 | ~10,719 | 68+1 |
|
||||
| executionCard.TM_2023 | ~10,929 | 84+1 |
|
||||
| executionCard.TM_2024 | ~15,870 | 76+1 |
|
||||
| executionCard.TM_2025 | ~13,335 | 78+1 |
|
||||
| executionCard.TM_2026 | ~5,408 | 77+1 |
|
||||
| **合计** | **~328,857** | |
|
||||
Reference in New Issue
Block a user