From 1352f0875bd21240c7024341b85809921ce540a0 Mon Sep 17 00:00:00 2001 From: Misaka Server Date: Wed, 25 Mar 2026 15:03:18 +0800 Subject: [PATCH] Update SKILL.md for excel-report-converter: enhance description and workflow details for better clarity and usability --- skills/excel-report-converter/SKILL.md | 490 +++++++++++++------------ 1 file changed, 259 insertions(+), 231 deletions(-) diff --git a/skills/excel-report-converter/SKILL.md b/skills/excel-report-converter/SKILL.md index 104e3d0..31fb68e 100644 --- a/skills/excel-report-converter/SKILL.md +++ b/skills/excel-report-converter/SKILL.md @@ -1,262 +1,290 @@ --- name: excel-report-converter -description: Generate Python scripts to convert report-style Excel files to database-record format. Use for converting Excel files with multiple stacked reports into flat database tables, analyzing report structure, creating custom conversion scripts for specific Excel report formats, and transforming hierarchical report data (header + detail lines + footer) into normalized database records +description: 当用户需要创建Python脚本将报表形式的Excel数据转换为结构化数据表时使用此技能。当用户提到：将Excel报表转换为数据表、解析Excel报表结构、创建Excel数据转换脚本、从打印模板格式提取数据、或者有类似"请购单维护"、"订单打印模板"等报表格式需要转换为标准表格时，即使没有明确说"创建脚本"，也应该触发此技能。这个技能专门处理那些面向打印/展示的报表布局（数据分散在多行多列、有表头/明细/页脚区块）到标准关系型数据表的转换。 --- -# Excel Report Converter +# Excel报表数据转换技能 -Generate Python scripts to convert report-style Excel files (multiple reports per worksheet) into database-record format (flat table). +这个技能帮助你创建Python脚本，将各种报表形式的Excel数据转换为标准的数据表格式。 -## When to Use +## 适用场景 -Use this skill when: -- User provides an Excel file with multiple similar reports stacked vertically in one worksheet -- Each report has hierarchical structure: header information + detail data rows + footer information -- User wants to convert to database-record format where header fields are repeated for each detail row -- The report structure is consistent across all reports in the file +当你遇到以下情况时，使用此技能： -## Workflow +- **报表式布局**：数据不是标准的行列表格，而是分散在多个区域 +- **有表头/明细/页脚结构**：一个文档包含主信息、明细列表、汇总信息 +- **打印模板格式**：为打印设计的Excel文件，需要提取其中的数据 +- **多区块重复**：同一个Excel文件包含多个相同格式的报表区块 +- **字段映射复杂**：目标字段与源单元格位置有复杂的对应关系 +- **大数据量提取（性能敏感）**：需要快速提取数千行以上、包含大量单元格的复杂Excel报表 -### Step 1: Extract and Analyze Report Structure +--- -Use the `excel-to-markdown` skill to convert the Excel file to markdown format for analysis: +## 🚀 性能优化核心技术 (提速法则) +在处理中大型Excel报表时，传统的 `openpyxl` 逐个单元格读取方法会导致极严重的性能瓶颈。本技能强制采用以下高阶优化方案： -```bash -# Convert Excel to markdown to analyze structure -python3 scripts/excel_to_markdown.py input.xlsx -o /tmp/analysis.md --show-rows --show-cols -``` +1. **空间换时间（内存二维数组）**：禁止在循环中频繁调用 `sheet.cell(row, col).value`。必须使用 `sheet.iter_rows(values_only=True)` 将整表数据一次性读入 Python 的 `List[List]` 中。后续所有的查找全部基于内存列表索引进行，速度可提升数十倍。 +2. **预先构建映射字典**：将 `column_index_from_string` (字母转数字索引) 等固定操作移出循环，在脚本初始化时预先计算好映射字典。 +3. **Pandas 向量化降维打击**：在保存 Excel 自动调整列宽时，放弃 `openpyxl` 的逐单元格遍历，直接利用 Pandas 的底层 C 语言级别操作 `df[col].astype(str).map(len).max()` 秒算列宽。 +4. **避开 `read_only` 的“公式陷阱”**：绝不能为了加载速度盲目开启 `read_only=True`，这会导致由 Excel 隐式公式（如自动递增序号 `=A1+1`）生成的值返回 `None`。必须坚持使用默认加载模式配合 `data_only=True`，然后依靠“内存二维数组”来解决速度问题。 -Read the markdown file and identify: -1. **Report delimiters**: How to identify where each report starts/ends (e.g., specific title in column 1) -2. **Header structure**: Which rows contain header information and what fields are in each column -3. **Detail table**: Which row contains column headers and where detail data starts/ends -4. **Footer structure**: Which rows contain footer information and where the data is located -5. **Row separators**: Are there empty rows between reports? After the last row? +--- -### Step 2: Generate Conversion Script +## 工作流程 -Create a Python script with the following structure: +### 第一步：分析源Excel文件结构并构建内存视图 + +使用openpyxl读取Excel文件，并立即将其转换为内存二维数组以提升后续处理速度： ```python -#!/usr/bin/env python3 -""" -Convert report-style Excel files to database-record format -Customized for: [describe the report format] -""" +import openpyxl +from typing import List, Any -import argparse -import sys -from pathlib import Path -from openpyxl import load_workbook +# 1. 加载工作簿 (保留 data_only=True, 弃用 read_only=True 保证公式值完整) +wb = openpyxl.load_workbook(file_path, data_only=True) +sheet = wb.active + +# 2. 【核心提速机制】将整表数据一次性抽取为 Python 二维数组 +# 填充一行 [None]，并为每一行填充一列 [None]，使后续列表索引(1-based)与Excel坐标严格对齐 +excel_data = [[None]] +for row in sheet.iter_rows(values_only=True): + excel_data.append([None] + list(row)) +wb.close() # 释放文件句柄 + +# 3. 辅助读取函数 (替代慢速的 sheet.cell().value) +def get_val(data: List[List[Any]], row: int, col: int) -> Any: + try: + if row < len(data) and col < len(data[row]): + return data[row][col] + except IndexError: + pass + return None +```` + +### 第二步：识别报表区块 + +大多数报表式Excel有以下特征，按优先级在 `excel_data` 中进行检测： + +|**特征**|**检测方法**|**示例**| +|---|---|---| +|区块标题|A列包含特定关键词|"请购单维护"、"订单"| +|表头标签|冒号结尾的标签|"请购单号："、"日期："| +|明细表头|包含列名的行|"行号"、"物料编码"、"数量"| +|数据行|标签行下方连续的数据|非空的具体数据值| +|页脚行|包含"制单"、"审批"等|"制单人："、"审批人："| + +**识别规则**： + +1. 区块开始：通常在A列，包含报表类型名称 + +2. 表头区域：区块开始后3-6行，包含带冒号的字段标签 + +3. 明细表头：表头区域后，A列包含"行号"或类似列名 + +4. 明细数据：明细表头后，连续的非空行 + +5. 页脚区域：明细数据后，包含签名/审批信息 + + +### 第三步：设计数据结构 + +根据识别结果，设计输出表结构： + +**扁平化单表** + +- 每条明细记录携带完整的表头信息 + +- 适合数据分析和导出 + + +### 第四步：生成转换脚本 + +基于分析结果，生成包含以下函数的Python脚本： + +Python + +``` +# 必需的核心函数 +def find_sections(excel_data) + """识别所有报表区块的起始行""" + +def extract_header_data(excel_data, start_row) + """提取表头信息""" + +def extract_line_items(excel_data, header_row, section_end) + """提取明细行数据""" + +def extract_footer_data(excel_data, line_items_end) + """提取页脚信息""" + +def parse_section(excel_data, start_row, next_section) + """解析单个报表区块""" + +def parse_excel_file(file_path) + """读取Excel文件，转换为二维数组并解析所有区块""" + +def save_to_excel(data_df, output_path) + """保存转换结果，利用Pandas向量化计算列宽""" +``` + +## 字段提取模式 + +### 模式1：固定偏移量（利用预计算） + +当表头字段位置固定时使用： + +Python + +``` +# 提取表头信息 (从内存数组快速读取) +def extract_header_data(excel_data, start_row): + header = {} + header['请购单号'] = get_val(excel_data, start_row + 3, 2) # Row+3, B列(2) + return header +``` + +### 模式2：标签查找 + +当字段位置不固定但标签唯一时使用： + +Python + +``` +def find_field_by_label(excel_data, label, start_row, search_range=10): + """通过标签查找字段位置""" + max_row = len(excel_data) - 1 + for row in range(start_row, min(start_row + search_range, max_row + 1)): + # 假设标签在前10列中 + for col in range(1, min(11, len(excel_data[row]))): + cell_value = get_val(excel_data, row, col) + if cell_value and label in str(cell_value): + # 返回值的位置（通常在标签的右侧） + return get_val(excel_data, row, col + 1) + return None +``` + +## 明细行提取策略 + +### 策略：预计算列映射字典 (极速匹配) + +避免在双重循环中调用 `column_index_from_string`。 + +Python + +``` +from openpyxl.utils import column_index_from_string + +LINE_ITEM_COLUMNS = { + 'A': '行号', 'B': '排产号', 'C': '物料编码' +} + +# 全局预计算列索引 +LINE_ITEM_COLUMNS_IDX = { + column_index_from_string(col): field + for col, field in LINE_ITEM_COLUMNS.items() +} + +def extract_line_items(excel_data, header_row, section_end): + line_items = [] + for row in range(header_row + 1, section_end): + # ... 判断跳出逻辑 ... + item = {} + for col_num, field_name in LINE_ITEM_COLUMNS_IDX.items(): + item[field_name] = get_val(excel_data, row, col_num) + if any(item.values()): + line_items.append(item) + return line_items +``` + +## 处理特殊情况 + +### 1. 空区块处理 + +当某个区块没有明细数据时，仍需创建一条记录： + +Python + +``` +if not line_items: + # 创建一条空记录，保留表头和页脚信息 + record = {**header_data, **footer_data} + for field in LINE_ITEM_COLUMNS.values(): + record[field] = None + flat_records.append(record) +``` + +## 输出格式 + +### Excel格式（结合 Pandas 向量化提速） + +Python + +``` from openpyxl.utils import get_column_letter -class ReportParser: - def __init__(self, worksheet): - self.ws = worksheet - self.max_row = worksheet.max_row - self.max_col = worksheet.max_column +def save_to_excel(data_df, output_path): + """保存为Excel文件并极速调整列宽""" + with pd.ExcelWriter(output_path, engine='openpyxl') as writer: + data_df.to_excel(writer, sheet_name='转换结果', index=False) + worksheet = writer.sheets['转换结果'] - def find_report_blocks(self): - """Identify start and end rows for each report block""" - # Implement block detection logic - # Look for report delimiters (e.g., title in column 1) - # Exclude trailing empty rows - pass - - def parse_header(self, start_row): - """Extract header fields from the report header section""" - # Map cell positions to field names - pass - - def parse_detail_rows(self, start_row, end_row): - """Extract detail rows from the report detail section""" - # Identify column header row - # Extract data until empty row or footer starts - pass - - def parse_footer(self, end_row): - """Extract footer fields from the report footer section""" - # Map cell positions to field names - pass - - def parse_report(self, start_row, end_row): - """Parse complete report: header + details + footer""" - header = self.parse_header(start_row) - details = self.parse_detail_rows(start_row, end_row) - footer = self.parse_footer(end_row) - return header, details, footer - -def convert_to_database_format(input_file, output_file, sheet_name=None): - """Convert report format to database-record format""" - wb = load_workbook(input_file, data_only=True) - ws = wb[sheet_name] if sheet_name else wb.active - - parser = ReportParser(ws) - blocks = parser.find_report_blocks() - - # Collect all records - all_records = [] - for start_row, end_row in blocks: - header, details, footer = parser.parse_report(start_row, end_row) - # Merge header + each detail row + footer - for detail in details: - record = {**header, **detail, **footer} - all_records.append(record) - - # Write output - from openpyxl import Workbook - wb_out = Workbook() - ws_out = wb_out.active - - # Write header row - fields = list(all_records[0].keys()) - for col_idx, field_name in enumerate(fields, start=1): - ws_out.cell(row=1, column=col_idx, value=field_name) - - # Write data rows - for row_idx, record in enumerate(all_records, start=2): - for col_idx, field_name in enumerate(fields, start=1): - ws_out.cell(row=row_idx, column=col_idx, value=record.get(field_name)) - - wb_out.save(output_file) - -def main(): - parser = argparse.ArgumentParser(description='Convert report-style Excel to database-record format') - parser.add_argument('input_file', help='Input Excel file') - parser.add_argument('output_file', help='Output Excel file') - parser.add_argument('--sheet', help='Worksheet name (default: active)') - args = parser.parse_args() - - convert_to_database_format(args.input_file, args.output_file, args.sheet) - -if __name__ == '__main__': - main() + # 【提速】利用 Pandas 的向量化操作一次性算出最大列宽 + for idx, col in enumerate(data_df.columns): + max_len = max(data_df[col].astype(str).map(len).max() if not data_df.empty else 0, len(str(col))) + adjusted_width = min(max_len + 2, 50) + col_letter = get_column_letter(idx + 1) + worksheet.column_dimensions[col_letter].width = adjusted_width ``` -**Key Implementation Points:** +## 依赖库 -- **Column mapping**: Use fixed column indices (1-indexed) or use openpyxl's column letters -- **Empty row detection**: Check if all cells in a row are None/empty to identify boundaries -- **Date handling**: Excel dates may be numeric - use appropriate conversion if needed -- **Merged cells**: Check for merged cells in header/footer sections -- **Data types**: Preserve original data types (strings, numbers, dates) from source +脚本需要以下依赖，确保在运行前安装： -### Step 3: Test and Verify - -Run the generated script to convert the file: - -```bash -python3 scripts/custom_converter.py input.xlsx output.xlsx -``` - -Use `excel-to-markdown` to verify the output: - -```bash -# For large files, only verify first 10 records -python3 scripts/excel_to_markdown.py output.xlsx -o /tmp/verify.md --show-rows --show-cols --rows 1:11 -``` - -Read `/tmp/verify.md` and verify: -1. All records are extracted (count should match total detail rows across all reports) -2. Header fields are correctly populated for each record -3. Detail fields are correctly mapped -4. Footer fields are correctly populated -5. No data loss or corruption - -**Important:** For large output files with many records, use `--rows 1:11` to only convert and verify the first 10 data records (plus header row). This avoids processing time and memory issues when verifying large exports. - -### Step 4: Iterate if Needed - -If verification reveals issues: -1. Identify the specific problem (wrong column index, missing field, incorrect row detection) -2. Fix the script accordingly -3. Re-run the conversion -4. Re-verify with excel-to-markdown -5. Repeat until output is correct - -## Common Patterns - -### Report Block Detection - -Most report-style Excel files use one of these patterns: - -**Pattern A: Title-based delimiters** -``` -Row 1: Report Title -Row 2: [header data] -... -Row N: [footer data] -Row N+1: [empty or next report title] -``` - -Look for a specific value in column 1 (e.g., "离散备料计划", "Purchase Order") - -**Pattern B: Fixed-size reports** -All reports have the same number of rows. Calculate block size and divide evenly. - -### Header/Detail/Footer Separation - -**Typical structure:** -- Rows 1-4: Header information (labels and values in specific columns) -- Row 5: Empty separator -- Row 6: Detail table column headers -- Rows 7+: Detail data rows -- Row N-1: Footer row 1 (creator, date, approver) -- Row N: Footer row 2 (printer, print date) -- Row N+1: Empty separator or next report - -### Column Mapping Strategies - -**Strategy 1: Fixed column positions** -Use when report format is consistent: -```python -factory = ws.cell(row=2, column=3).value # C列 -order_no = ws.cell(row=2, column=9).value # I列 -``` - -**Strategy 2: Search by label** -Use when column positions may vary: -```python -# Find column by searching for label in first row -for col in range(1, max_col + 1): - if ws.cell(row=header_row, column=col).value == "Order No": - order_no_col = col - break -``` - -### Field Merging Strategy - -When creating database records: -1. Parse header fields once per report -2. Parse footer fields once per report -3. For each detail row, create a merged record: `{**header, **detail_row, **footer}` -4. This repeats header/footer fields for each detail line (database normalization) - -## Example Report Structure Analysis - -When analyzing the markdown output, look for: +Bash ``` -Row 1: | Report Title | | | ... -Row 2: | Field: | | Value | | Field: | | Value | ... -Row 3: | Field: | | Value | | Field: | | Value | ... -Row 4: | (empty or separator) -Row 5: | Col1 | Col2 | Col3 | ... (detail table headers) -Row 6: | val1 | val2 | val3 | ... (first detail row) -Row 7: | val1 | val2 | val3 | ... (second detail row) -... -Row N: | Creator: | | Name | | Date: | | 2025-01-01 | -Row N+1: | Printer: | | Name | | Print Date: | | 2025-01-02 | -Row N+2: | (empty) or next report title +pip install openpyxl pandas ``` -Map this structure to the script functions: -- `parse_header(start_row)`: Extract fields from rows 2-3 -- `parse_detail_rows(start_row, end_row)`: Extract rows 6+ until empty/footer -- `parse_footer(end_row)`: Extract fields from rows N to N+1 +## 验证清单 -## Resources +生成脚本后，验证以下内容： -The `excel-to-markdown` skill provides Excel-to-markdown conversion for structure analysis. +- [ ] 成功识别所有报表区块 +- [ ] 表头字段提取正确 +- [ ] 明细行数据完整 +- [ ] 页脚信息准确 +- [ ] 空区块得到正确处理 +- [ ] 输出文件格式正确 +- [ ] 数据类型准确（日期、数字等） +- [ ] 没有重复或遗漏的记录 -This skill does not include bundled scripts or references - each conversion script is generated dynamically based on the specific report format being analyzed. +## 调试技巧 + +当脚本出现问题时： + +1. **打印中间结果**：在每个函数中添加print语句，查看提取的数据 +2. **检查单元格值**：确认openpyxl读取的值与预期一致 +3. **验证索引**：确保行号和列号计算正确 +4. **分步测试**：先测试单个区块，确认正确后再处理全部 +5. **对比原文件**：在Excel中查看原始数据和提取结果的差异 + +## 常见问题 + +**Q: 为什么提取时有些序列号或公式计算的值变成了 `None`？** + +A: 这是因为在 `openpyxl` 中错误开启了 `read_only=True` 模式，导致依靠 Excel 公式生成的值无法正确读取缓存。**解决方案**：去掉 `read_only=True`，只保留 `data_only=True`，并使用二维数组提取法来保障速度。 + +**Q: 输出的数据需要进一步处理怎么办？** + +A: 脚本生成后，可以在将其转换为 Pandas DataFrame 之后，利用 Pandas 强大的生态添加数据清洗、验证、格式转换（如日期格式化）等功能。 + +## 最佳实践 + +1. **绝对优先使用内存二维数组**：直接摒弃 `sheet.cell().value` 的传统思维，这是报表转换脚本能商用的性能基石。 + +2. **预先计算，拒绝重复**：所有能够确定位置或索引关系的映射字典，全部放在全局作用域一次性计算完成。 + +3. **先分析，后编码**：花时间理解报表结构和分页/分块标识，比直接编码更高效。 + +4. **异常容错机制**：数据提取行要进行 `if any(item.values())` 判断，过滤纯空行；对于越界索引使用 `try-except` 包裹。 \ No newline at end of file