07ab64bf38
Migrates pressure gauge (PG) and thermometer (TM) data from .xlsm files into SQL Server executionCard schema, one table per year (PG_2022-2026, TM_2022-2026). Supports single/multi-table migration, column remapping, type inference, and dry-run mode via config.yaml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
491 lines
14 KiB
Markdown
491 lines
14 KiB
Markdown
# Excel to SQL Server Migration Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** 将 Excel 中压力表(PG)和温度计(TM)的生产执行卡数据迁移到 SQL Server 的 `executionCard` Schema 中，每年一张表。
|
||
|
||
**Architecture:** 单一 Python 脚本 + YAML 配置文件。使用 openpyxl 读取 Excel + pyodbc 写入 SQL Server。所有数据源映射、SQL Server 连接信息、类型推断规则等均由 `config.yaml` 管理，脚本仅负责执行逻辑。对每个目标表：读取所有源工作表 → 合并列名取并集 → 按总排号去重 → 动态建表 → 批量插入。
|
||
|
||
**Tech Stack:** Python 3, openpyxl, pyodbc, PyYAML, SQL Server (ODBC Driver 18)
|
||
|
||
---
|
||
|
||
## SQL Server 目标结构
|
||
|
||
- **Schema:** `executionCard`
|
||
- **表命名:** `executionCard.PG_YYYY` / `executionCard.TM_YYYY`
|
||
- **主键:** `总排号 NVARCHAR(50) PRIMARY KEY`
|
||
- **自增列:** `id INT IDENTITY(1,1)` （非主键，仅作辅助序号）
|
||
|
||
## 数据源映射
|
||
|
||
### 压力表 (PG)
|
||
|
||
| 目标表 | 文件 | 工作表 | 去重策略 |
|
||
|--------|------|--------|---------|
|
||
| PG_2022 | 生产执行卡2022.xlsm | Sheet1 | 直接导入 |
|
||
| PG_2023 | 生产执行卡2023（1-5月）.xlsm | Sheet1 | 先导入1-5月，再追加6月后数据，按总排号去重(保留后者) |
|
||
| PG_2023 | 生产执行卡2023(6月-.xlsm | Sheet1 | 同上 |
|
||
| PG_2024 | 生产执行卡2024 6月.xlsm | 重庆数据 + 北京数据 | 6月版为超集，直接用；重庆北京合并取列并集 |
|
||
| PG_2025 | 生产执行卡2025年.xlsm | 重庆数据 + 北京数据 | 合并取列并集 |
|
||
| PG_2026 | 生产执行卡2026年.xlsm | 重庆数据 + 北京数据 | 合并取列并集 |
|
||
|
||
### 温度计 (TM)
|
||
|
||
| 目标表 | 文件 | 工作表 |
|
||
|--------|------|--------|
|
||
| TM_2022 | 生产执行卡(2022合同数据).xlsm | 数据 |
|
||
| TM_2023 | 生产执行卡(2023合同数据).xlsm | 数据 |
|
||
| TM_2024 | 生产执行卡(新版1).xlsm | 数据 |
|
||
| TM_2025 | 生产执行卡2025.xlsm | 数据 |
|
||
| TM_2026 | 生产执行卡2026.xlsm | 数据 |
|
||
|
||
## 列名处理规则
|
||
|
||
1. 保留中文列名，SQL Server 中用方括号包裹（如 `[总排号]`）
|
||
2. 空列名 → 替换为 `col_N`（N 为列索引）
|
||
3. 同名列但不同位置（如 PG_2025 重庆的"订单号"在第3列 vs 2022 的"订单号"在第5列）→ 统一按列名对齐
|
||
4. 重复列名（如 TM 中出现了两次"数量"、"操作者"）→ 第二次出现的加后缀 `_2`
|
||
|
||
## 数据类型推断规则
|
||
|
||
按列名关键字自动推断，其余默认 `NVARCHAR(500)`：
|
||
|
||
| 关键字匹配 | SQL 类型 | 说明 |
|
||
|------------|----------|------|
|
||
| 日期、Date | DATETIME | 签订日期、交货日期、下单日期等 |
|
||
| 数量 | INT | 数值型 |
|
||
| 单价 | DECIMAL(18,2) | 金额型 |
|
||
| 备注、技术参数、说明、转换数据 | NVARCHAR(MAX) | 长文本 |
|
||
| 总排号 | NVARCHAR(50) NOT NULL PRIMARY KEY | 主键 |
|
||
| 其余 | NVARCHAR(500) | 默认文本 |
|
||
|
||
## 去重规则
|
||
|
||
- **同一表多来源时**：按 `总排号` 去重，保留**后读取**的记录（后导入覆盖先导入）
|
||
- **PG_2023**：先读1-5月文件，再读6月文件，重叠部分保留6月版本
|
||
- **PG_2024**：仅使用6月版文件（已是原版超集）
|
||
- **重庆+北京合并**：总排号前缀不同（C vs B），天然不重叠，直接合并
|
||
|
||
## 批量插入
|
||
|
||
- 每批 500 行
|
||
- 使用 parameterized insert（防注入 + 正确处理特殊字符）
|
||
- 插入时跳过总排号为空的行
|
||
|
||
---
|
||
|
||
## Tasks
|
||
|
||
### Task 1: 项目初始化 — 安装依赖、创建目录结构
|
||
|
||
**Files:**
|
||
- Create: `requirements.txt`
|
||
- Modify: `.venv` (install deps)
|
||
|
||
**Step 1: 创建 requirements.txt**
|
||
|
||
```txt
|
||
openpyxl>=3.1.0
|
||
pyodbc>=5.0.0
|
||
pyyaml>=6.0
|
||
```
|
||
|
||
**Step 2: 安装依赖**
|
||
|
||
```bash
|
||
.venv/Scripts/pip install -r requirements.txt
|
||
```
|
||
|
||
**Step 3: 验证安装**
|
||
|
||
```bash
|
||
.venv/Scripts/python -c "import openpyxl, pyodbc, yaml; print('OK')"
|
||
```
|
||
|
||
Expected: `OK`
|
||
|
||
---
|
||
|
||
### Task 2: 创建 YAML 配置文件
|
||
|
||
**Files:**
|
||
- Create: `config.yaml`
|
||
|
||
**Step 1: 编写 config.yaml**
|
||
|
||
所有配置集中在此文件，包括 SQL Server 连接、Schema、数据源映射、类型推断规则：
|
||
|
||
```yaml
|
||
# Excel to SQL Server Migration Configuration
|
||
|
||
# ================= SQL Server 连接 =================
|
||
sql_server:
|
||
driver: "ODBC Driver 18 for SQL Server"
|
||
server: "192.168.110.114"
|
||
database: "CompanyDB"
|
||
username: "peng"
|
||
password: "Cqbld123456."
|
||
TrustServerCertificate: "yes"
|
||
|
||
# ================= 目标 Schema =================
|
||
schema: "executionCard"
|
||
|
||
# ================= 批量插入参数 =================
|
||
batch_size: 500
|
||
|
||
# ================= 主键列名（用于去重） =================
|
||
primary_key: "总排号"
|
||
|
||
# ================= 数据类型推断规则 =================
|
||
type_rules:
|
||
# 列名包含关键字 → SQL 类型
|
||
datetime:
|
||
- "日期"
|
||
- "Date"
|
||
int:
|
||
- name: "数量"
|
||
exact: true
|
||
decimal:
|
||
- name: "单价"
|
||
exact: true
|
||
long_text:
|
||
- "备注"
|
||
- "技术参数"
|
||
- "说明"
|
||
- "转换数据"
|
||
- "新参数"
|
||
primary_key_type: "NVARCHAR(50) NOT NULL"
|
||
default_type: "NVARCHAR(500)"
|
||
|
||
# ================= 数据源映射 =================
|
||
# 每个条目对应一个目标表
|
||
# sources 中的顺序决定去重优先级：后出现的覆盖先出现的（同总排号时）
|
||
tables:
|
||
# ---------- 压力表 (PG) ----------
|
||
- table: "PG_2022"
|
||
sources:
|
||
- file: "PG/生产执行卡2022.xlsm"
|
||
sheet: "Sheet1"
|
||
|
||
- table: "PG_2023"
|
||
sources:
|
||
- file: "PG/生产执行卡2023（1-5月）.xlsm"
|
||
sheet: "Sheet1"
|
||
- file: "PG/生产执行卡2023(6月-.xlsm"
|
||
sheet: "Sheet1"
|
||
|
||
- table: "PG_2024"
|
||
sources:
|
||
- file: "PG/生产执行卡2024 6月.xlsm"
|
||
sheet: "重庆数据"
|
||
- file: "PG/生产执行卡2024 6月.xlsm"
|
||
sheet: "北京数据"
|
||
|
||
- table: "PG_2025"
|
||
sources:
|
||
- file: "PG/生产执行卡2025年.xlsm"
|
||
sheet: "重庆数据"
|
||
- file: "PG/生产执行卡2025年.xlsm"
|
||
sheet: "北京数据"
|
||
|
||
- table: "PG_2026"
|
||
sources:
|
||
- file: "PG/生产执行卡2026年.xlsm"
|
||
sheet: "重庆数据"
|
||
- file: "PG/生产执行卡2026年.xlsm"
|
||
sheet: "北京数据"
|
||
|
||
# ---------- 温度计 (TM) ----------
|
||
- table: "TM_2022"
|
||
sources:
|
||
- file: "TM/生产执行卡(2022合同数据).xlsm"
|
||
sheet: "数据"
|
||
|
||
- table: "TM_2023"
|
||
sources:
|
||
- file: "TM/生产执行卡(2023合同数据).xlsm"
|
||
sheet: "数据"
|
||
|
||
- table: "TM_2024"
|
||
sources:
|
||
- file: "TM/生产执行卡(新版1).xlsm"
|
||
sheet: "数据"
|
||
|
||
- table: "TM_2025"
|
||
sources:
|
||
- file: "TM/生产执行卡2025.xlsm"
|
||
sheet: "数据"
|
||
|
||
- table: "TM_2026"
|
||
sources:
|
||
- file: "TM/生产执行卡2026.xlsm"
|
||
sheet: "数据"
|
||
```
|
||
|
||
**Step 2: 验证 YAML 可正确解析**
|
||
|
||
```bash
|
||
.venv/Scripts/python -c "import yaml; cfg=yaml.safe_load(open('config.yaml',encoding='utf-8')); print(f'Tables: {len(cfg[\"tables\"])}, Schema: {cfg[\"schema\"]}')"
|
||
```
|
||
|
||
Expected: `Tables: 10, Schema: executionCard`
|
||
|
||
---
|
||
|
||
### Task 3: 创建迁移脚本骨架 — 加载配置
|
||
|
||
**Files:**
|
||
- Create: `migrate.py`
|
||
|
||
**Step 1: 编写 migrate.py 的配置加载部分**
|
||
|
||
```python
|
||
"""
|
||
Excel to SQL Server Migration Script
|
||
Migrates production execution card data from Excel (.xlsm) to SQL Server.
|
||
Configuration is loaded from config.yaml.
|
||
"""
|
||
|
||
import argparse
|
||
import os
|
||
import sys
|
||
from collections import OrderedDict
|
||
|
||
import openpyxl
|
||
import pyodbc
|
||
import yaml
|
||
|
||
|
||
def load_config(config_path=None):
|
||
"""加载 YAML 配置文件"""
|
||
if config_path is None:
|
||
config_path = os.path.join(
|
||
os.path.dirname(os.path.abspath(__file__)), 'config.yaml'
|
||
)
|
||
with open(config_path, 'r', encoding='utf-8') as f:
|
||
return yaml.safe_load(f)
|
||
|
||
|
||
def get_connection_string(cfg):
|
||
"""根据配置构建 pyodbc 连接字符串"""
|
||
sql = cfg['sql_server']
|
||
return (
|
||
f"DRIVER={{{sql['driver']}}};"
|
||
f"SERVER={sql['server']};"
|
||
f"DATABASE={sql['database']};"
|
||
f"UID={sql['username']};"
|
||
f"PWD={sql['password']};"
|
||
f"TrustServerCertificate={sql['TrustServerCertificate']};"
|
||
)
|
||
```
|
||
|
||
**Step 2: 验证脚本可导入无语法错误**
|
||
|
||
```bash
|
||
.venv/Scripts/python -c "import ast; ast.parse(open('migrate.py').read()); print('Syntax OK')"
|
||
```
|
||
|
||
Expected: `Syntax OK`
|
||
|
||
---
|
||
|
||
### Task 4: 实现 Excel 读取与数据合并逻辑
|
||
|
||
**Files:**
|
||
- Modify: `migrate.py`
|
||
|
||
**Step 1: 编写 read_sheet() 和 merge_sources() 函数**
|
||
|
||
核心逻辑：
|
||
|
||
```python
|
||
def read_sheet(file_path, sheet_name):
|
||
"""读取单个工作表，返回 (列名列表, 数据行列表)。
|
||
跳过总排号为空的行。"""
|
||
|
||
def merge_sources(sources):
|
||
"""合并多个工作表的数据。
|
||
- 列名取并集（按出现顺序）
|
||
- 数据按总排号去重，后读覆盖先读
|
||
- 处理重复列名（加 _2 后缀）
|
||
返回 (unified_headers, merged_data_ordered_dict)"""
|
||
```
|
||
|
||
关键处理：
|
||
1. `read_sheet`: 用 openpyxl read_only 模式读取，第一行为表头，其余为数据。跳过总排号（第1列）为空的行。
|
||
2. `merge_sources`: 遍历所有 source，收集所有列名并去重保留顺序。每个 source 的数据按其列名映射到统一列名集。用 OrderedDict 以总排号为 key 去重。
|
||
|
||
---
|
||
|
||
### Task 5: 实现 SQL Server Schema 创建与表创建
|
||
|
||
**Files:**
|
||
- Modify: `migrate.py`
|
||
|
||
**Step 1: 编写 create_schema() 和 infer_sql_type() 和 create_table() 函数**
|
||
|
||
```python
|
||
def get_connection(cfg):
|
||
"""根据配置建立 SQL Server 连接，返回 pyodbc.Connection"""
|
||
|
||
def create_schema(cursor, schema_name):
|
||
"""创建 schema（如不存在）"""
|
||
|
||
def infer_sql_type(col_name, type_rules, primary_key):
|
||
"""根据 config.yaml 中的 type_rules 推断 SQL Server 数据类型"""
|
||
|
||
def create_table(cursor, table_name, headers, cfg):
|
||
"""根据列名列表动态创建表。
|
||
- id INT IDENTITY(1,1)
|
||
- 总排号（由 cfg['primary_key'] 指定）为主键
|
||
- 其余列根据 infer_sql_type 推断
|
||
"""
|
||
```
|
||
|
||
`infer_sql_type` 从 YAML 配置读取规则：
|
||
```python
|
||
def infer_sql_type(col_name, type_rules, primary_key):
|
||
name = col_name
|
||
# 主键列
|
||
if name == primary_key or name.startswith('20BW'):
|
||
return type_rules['primary_key_type']
|
||
# 遍历类型规则
|
||
if any(kw in name for kw in type_rules.get('datetime', [])):
|
||
return 'DATETIME'
|
||
for rule in type_rules.get('int', []):
|
||
if rule.get('exact') and rule['name'] == name:
|
||
return 'INT'
|
||
elif not rule.get('exact') and rule['name'] in name:
|
||
return 'INT'
|
||
for rule in type_rules.get('decimal', []):
|
||
if rule.get('exact') and rule['name'] == name:
|
||
return 'DECIMAL(18,2)'
|
||
elif not rule.get('exact') and rule['name'] in name:
|
||
return 'DECIMAL(18,2)'
|
||
if any(kw in name for kw in type_rules.get('long_text', [])):
|
||
return 'NVARCHAR(MAX)'
|
||
return type_rules.get('default_type', 'NVARCHAR(500)')
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: 实现批量数据插入
|
||
|
||
**Files:**
|
||
- Modify: `migrate.py`
|
||
|
||
**Step 1: 编写 batch_insert() 函数**
|
||
|
||
```python
|
||
def batch_insert(cursor, table_name, headers, data_dict, cfg):
|
||
"""批量插入数据到 SQL Server。
|
||
- data_dict: OrderedDict，key=总排号, value=dict
|
||
- batch_size 从 cfg['batch_size'] 读取
|
||
- 使用 parameterized INSERT
|
||
- 每 batch_size 行提交一次
|
||
"""
|
||
```
|
||
|
||
核心：
|
||
- 构建 INSERT 语句：`INSERT INTO [executionCard].[{table_name}] ([col1], [col2], ...) VALUES (?, ?, ...)`
|
||
- 参数化插入，pyodbc 自动处理类型转换
|
||
- 空值传 `None`
|
||
|
||
---
|
||
|
||
### Task 7: 实现主流程与进度输出
|
||
|
||
**Files:**
|
||
- Modify: `migrate.py`
|
||
|
||
**Step 1: 编写 migrate_table() 和 main() 函数**
|
||
|
||
```python
|
||
def migrate_table(cursor, table_config, cfg, dry_run=False):
|
||
"""迁移单个表：
|
||
1. merge_sources 获取合并数据
|
||
2. create_table 建表（若非 dry-run）
|
||
3. batch_insert 写入数据（若非 dry-run）
|
||
打印进度信息
|
||
"""
|
||
|
||
def main():
|
||
"""主入口：
|
||
1. 解析命令行参数（--config, --dry-run）
|
||
2. 加载 config.yaml
|
||
3. 连接 SQL Server（若非 dry-run）
|
||
4. 创建 schema（若非 dry-run）
|
||
5. 遍历 cfg['tables']，逐表迁移
|
||
6. 打印汇总
|
||
"""
|
||
```
|
||
|
||
进度输出格式：
|
||
```
|
||
[1/10] 迁移 PG_2022 ...
|
||
读取: Excel/PG/生产执行卡2022.xlsm > Sheet1 → 52,854 行
|
||
建表: [executionCard].[PG_2022] (39 列)
|
||
插入: 52,854 行 ✓
|
||
[2/10] 迁移 PG_2023 ...
|
||
读取: Excel/PG/生产执行卡2023（1-5月）.xlsm > Sheet1 → 27,797 行
|
||
读取: Excel/PG/生产执行卡2023(6月-.xlsm > Sheet1 → 30,646 行
|
||
合并去重后: 58,443 行
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: 测试运行（dry-run 模式）
|
||
|
||
**Step 1: 确认 --dry-run 参数支持**
|
||
|
||
dry-run 模式下只读取数据、打印建表 SQL 和统计信息，不实际写入数据库。
|
||
|
||
**Step 2: 执行 dry-run 验证数据读取正确**
|
||
|
||
```bash
|
||
.venv/Scripts/python migrate.py --dry-run
|
||
```
|
||
|
||
Expected: 打印所有表的列名、行数、建表 SQL，无报错
|
||
|
||
---
|
||
|
||
### Task 9: 正式执行迁移
|
||
|
||
**Step 1: 运行迁移脚本**
|
||
|
||
```bash
|
||
.venv/Scripts/python migrate.py
|
||
```
|
||
|
||
**Step 2: 验证数据**
|
||
|
||
对每张表执行验证查询：
|
||
```sql
|
||
SELECT COUNT(*) FROM executionCard.PG_2022;
|
||
SELECT COUNT(*) FROM executionCard.PG_2023;
|
||
-- ... 每张表
|
||
-- 抽样检查
|
||
SELECT TOP 5 * FROM executionCard.PG_2022;
|
||
```
|
||
|
||
---
|
||
|
||
## 预估数据量
|
||
|
||
| 表名 | 预估行数 | 列数 |
|
||
|------|---------|------|
|
||
| executionCard.PG_2022 | ~52,854 | 38+1(id) |
|
||
| executionCard.PG_2023 | ~58,443 | 38+1 |
|
||
| executionCard.PG_2024 | ~64,004 | 46+1 |
|
||
| executionCard.PG_2025 | ~70,093 | 47+1 |
|
||
| executionCard.PG_2026 | ~29,232 | 46+1 |
|
||
| executionCard.TM_2022 | ~10,719 | 68+1 |
|
||
| executionCard.TM_2023 | ~10,929 | 84+1 |
|
||
| executionCard.TM_2024 | ~15,870 | 76+1 |
|
||
| executionCard.TM_2025 | ~13,335 | 78+1 |
|
||
| executionCard.TM_2026 | ~5,408 | 77+1 |
|
||
| **合计** | **~328,857** | |
|