feat(extractor): Add Excel conversion and post-processing capabilities
Add comprehensive post-processing features to convert downloaded Excel files into structured data and merge them into a single output file. New modules: - extractor_core.py: Stateless pure functions for web operations - excel_converter.py: Excel to DataFrame conversion utility - tests/test_extractor_real.py: Real data extraction test suite Enhanced features: - post_process_downloads(): Convert and merge multiple Excel files - extract_and_process(): Complete workflow in single call - cleanup_temp_files(): Optional cleanup of temporary downloaded files - Field name mapping for standardized output columns Dependencies: - pandas>=2.0.0 for data manipulation - openpyxl>=3.1.0 for Excel file handling Documentation: - Updated CLAUDE.md with new module references - Added API documentation for extractor components Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
5
.gitignore
vendored
5
.gitignore
vendored
@@ -148,3 +148,8 @@ Thumbs.db
|
||||
.agents/
|
||||
.agent/
|
||||
.claude/
|
||||
.sisyphus
|
||||
|
||||
data/
|
||||
|
||||
nul
|
||||
65
CLAUDE.md
65
CLAUDE.md
@@ -45,6 +45,34 @@ Note: Test scripts are responsible for loading environment variables and passing
|
||||
- Returns tuple: `(browser, context, page, main_frame)` where `main_frame` is the forwardFrame iframe.
|
||||
- Callers are responsible for browser lifecycle management (context.close(), browser.close()).
|
||||
|
||||
**`utils/discrete_material_plan/extractor_core.py`** - Core web operations module (pure functions)
|
||||
- `navigate_to_discrete_material_page(main_frame, page)` - Navigate to discrete material plan maintenance page. Returns `(work_frame, page1)`.
|
||||
- `setup_query_interface(work_frame)` - Initialize query interface by selecting order number query tab and setting page size.
|
||||
- `fill_and_search_orders(work_frame, order_ids)` - Fill order IDs into search textbox and trigger search.
|
||||
- `download_batch_data(work_frame, page, order_ids, batch_index, download_dir)` - Execute download workflow for a single batch. Returns downloaded file path.
|
||||
- `execute_batch_download_workflow(work_frame, page, order_ids, batch_index, download_dir)` - Complete workflow: fill orders, search, and download. Returns downloaded file path.
|
||||
- All functions are stateless and accept required parameters explicitly. No logging or progress reporting - callers handle that.
|
||||
- **See**: `docs/discrete_material_plan_extractor_core.md` for complete API documentation and usage examples.
|
||||
|
||||
**`utils/discrete_material_plan/extractor.py`** - High-level extractor component
|
||||
- `DiscreteMaterialPlanExtractor` class - Batch processing wrapper with progress reporting and error handling.
|
||||
- `extract_from_file()` function - Convenience function to extract data from order IDs in a file.
|
||||
- **New Methods**:
|
||||
- `post_process_downloads()` - Convert and merge downloaded Excel files into structured format
|
||||
- `extract_and_process()` - Complete workflow: extract data and post-process in one call
|
||||
- Accepts ID list and file paths as parameters, handles session management, batch processing, Excel conversion, and file merging.
|
||||
- **Documentation**:
|
||||
- Basic Usage: `docs/discrete_material_plan_extractor.md` - Basic API and usage examples
|
||||
- Component Guide: `docs/extractor_component_guide.md` - Complete guide with Mermaid diagrams
|
||||
- Post-Processing: `docs/extractor_post_processing.md` - Excel conversion and merging features
|
||||
|
||||
**`utils/discrete_material_plan/excel_converter.py`** - Excel data conversion utility
|
||||
- `ExcelConverter` class - Converts raw Excel reports to structured DataFrames.
|
||||
- `convert(input_file, output_file)` - Converts Excel file and returns DataFrame.
|
||||
- Handles nested order structures, flattens to table format, applies field name mapping.
|
||||
- Used internally by extractor for post-processing, can also be used standalone.
|
||||
- **See**: `docs/extractor_post_processing.md` for usage examples.
|
||||
|
||||
**`tests/`** - Test suite
|
||||
- All test files must add `PROJECT_ROOT` to `sys.path` to import `utils` modules
|
||||
- Tests follow pattern: load dotenv, set browser path environment variable, run Playwright operations
|
||||
@@ -72,6 +100,13 @@ Callers must construct the complete login URL before passing to `login()`:
|
||||
url = f"{os.getenv('ERP_URL').rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
|
||||
```
|
||||
|
||||
### Demo Data
|
||||
|
||||
**`id-demo.txt`** - Demo production IDs for testing
|
||||
- Contains 15 sample production IDs (SC70202603240xxx format)
|
||||
- Used by test scripts and demos for web operations testing
|
||||
- Format: One ID per line, plain text file
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
@@ -81,10 +116,40 @@ python tests/test_auth_config.py
|
||||
# Run login/logout test
|
||||
python tests/test_login.py
|
||||
|
||||
# Run web operations tests
|
||||
python tests/test_web_operations.py
|
||||
|
||||
# Run extractor component tests
|
||||
python tests/test_extractor_component.py
|
||||
|
||||
# Run real data extraction test (uses tests/id-demo.txt)
|
||||
python tests/test_extractor_real.py
|
||||
|
||||
# Or use quick run scripts
|
||||
./run_extractor_test.bat # Windows
|
||||
./run_extractor_test.sh # Linux/Mac
|
||||
|
||||
# Run web operations demo
|
||||
python tests/demo_web_operations.py
|
||||
|
||||
# Run extractor component demo
|
||||
python demo_extractor_component.py
|
||||
|
||||
# Run any test with virtual environment
|
||||
source .venv/Scripts/activate && python tests/<test_file>.py
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
**`docs/INDEX.md`** - Complete documentation index
|
||||
- Quick start guide, API reference, examples, and troubleshooting
|
||||
- Central hub for all documentation
|
||||
|
||||
**`docs/extractor_component_guide.md`** - Complete component guide with Mermaid diagrams
|
||||
- Architecture overview, class structure, workflow diagrams
|
||||
- State management, error handling, integration examples
|
||||
- Visual documentation using Mermaid graphs
|
||||
|
||||
## Code Conventions
|
||||
|
||||
1. **English Only**: All user-facing output, docstrings, comments, and log messages must be in English. Chinese text is only used for Playwright element selectors matching the actual UI.
|
||||
|
||||
999
docs/discrete_material_plan_extractor_api.md
Normal file
999
docs/discrete_material_plan_extractor_api.md
Normal file
@@ -0,0 +1,999 @@
|
||||
# Discrete Material Plan Extractor - API Reference
|
||||
|
||||
> **Pure Function Data Extraction for Yonyou BIP**
|
||||
>
|
||||
> Stateless functions for extracting discrete material plan data from Yonyou BIP ERP system with batch processing and Excel post-processing capabilities.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The `extractor` module provides high-level pure functions for extracting material plan data from the Yonyou BIP ERP system. All functions are **stateless** and **explicitly accept required parameters**, following the same design patterns as `extractor_core.py` and `auth.py`.
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph User["User Code"]
|
||||
A[Main Script]
|
||||
end
|
||||
|
||||
subgraph Auth["utils/auth.py"]
|
||||
B[login]
|
||||
C[logout]
|
||||
end
|
||||
|
||||
subgraph Extractor["utils/discrete_material_plan/extractor.py"]
|
||||
D[extract_and_post_process]
|
||||
E[extract_batches]
|
||||
F[post_process_downloads]
|
||||
G[read_order_ids_from_file]
|
||||
H[chunk_order_ids]
|
||||
end
|
||||
|
||||
subgraph Core["utils/discrete_material_plan/extractor_core.py"]
|
||||
I[navigate_to_discrete_material_page]
|
||||
J[setup_query_interface]
|
||||
K[execute_batch_download_workflow]
|
||||
end
|
||||
|
||||
subgraph Converter["utils/discrete_material_plan/excel_converter.py"]
|
||||
L[ExcelConverter.convert]
|
||||
end
|
||||
|
||||
subgraph External["External"]
|
||||
M[(Yonyou BIP ERP)]
|
||||
N[(Downloaded Excel Files)]
|
||||
O[(Merged Output Excel)]
|
||||
end
|
||||
|
||||
A --> B
|
||||
A --> D
|
||||
A --> G
|
||||
|
||||
B --> M
|
||||
D --> E
|
||||
D --> F
|
||||
E --> H
|
||||
E --> K
|
||||
F --> L
|
||||
|
||||
B --> I
|
||||
I --> M
|
||||
K --> M
|
||||
K --> N
|
||||
|
||||
L --> N
|
||||
L --> O
|
||||
|
||||
style User fill:#e1f5ff
|
||||
style Auth fill:#fff4e1
|
||||
style Extractor fill:#e8f5e9
|
||||
style Core fill:#fce4ec
|
||||
style Converter fill:#f3e5f5
|
||||
style External fill:#ffebee
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- ✅ Stateless pure functions (no class instances)
|
||||
- ✅ Explicit parameter passing
|
||||
- ✅ Batch processing for large order lists
|
||||
- ✅ Excel conversion and merging
|
||||
- ✅ Progress reporting with verbose logging
|
||||
- ✅ Caller-managed session lifecycle
|
||||
|
||||
**Module Location:** `utils/discrete_material_plan/extractor.py`
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage - Complete Workflow
|
||||
|
||||
```python
|
||||
from utils.discrete_material_plan import (
|
||||
extract_and_post_process,
|
||||
get_login_url,
|
||||
)
|
||||
from utils.auth import login
|
||||
from utils.discrete_material_plan import navigate_to_discrete_material_page
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
# 1. Setup browser session
|
||||
with sync_playwright() as playwright:
|
||||
browser, context, page, main_frame = login(
|
||||
playwright=playwright,
|
||||
username="your_username",
|
||||
password="your_password",
|
||||
url=get_login_url("https://erp.example.com"),
|
||||
headless=True,
|
||||
ignore_https_errors=True,
|
||||
)
|
||||
|
||||
try:
|
||||
# 2. Navigate to discrete material page
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
|
||||
# 3. Extract and process data
|
||||
output_path, df = extract_and_post_process(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=["SC70202603240001", "SC70202603240002"],
|
||||
download_dir="./downloads",
|
||||
output_file="./output/merged_data.xlsx",
|
||||
batch_size=10,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
print(f"Extracted {len(df)} records to {output_path}")
|
||||
|
||||
finally:
|
||||
# 4. Caller manages browser lifecycle
|
||||
context.close()
|
||||
browser.close()
|
||||
```
|
||||
|
||||
### From File - Simplest Approach
|
||||
|
||||
```python
|
||||
from utils.discrete_material_plan import read_order_ids_from_file
|
||||
|
||||
# Read order IDs from file
|
||||
order_ids = read_order_ids_from_file("order_ids.txt")
|
||||
print(f"Loaded {len(order_ids)} order IDs")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Helper Functions
|
||||
|
||||
#### `chunk_order_ids(order_ids: List[str], batch_size: int) -> List[List[str]]`
|
||||
|
||||
Split order IDs into batches for processing.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Description |
|
||||
|------|------|-------------|
|
||||
| `order_ids` | `List[str]` | List of order IDs to process |
|
||||
| `batch_size` | `int` | Maximum order IDs per batch |
|
||||
|
||||
**Returns:** `List[List[str]]` - List of batches
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
>>> chunk_order_ids(["A", "B", "C", "D", "E"], 2)
|
||||
[["A", "B"], ["C", "D"], ["E"]]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `get_login_url(base_url: str) -> str`
|
||||
|
||||
Construct complete login URL from base ERP URL.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Description |
|
||||
|------|------|-------------|
|
||||
| `base_url` | `str` | Base ERP URL (e.g., "https://erp.example.com") |
|
||||
|
||||
**Returns:** `str` - Complete login page URL
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
>>> get_login_url("https://erp.example.com")
|
||||
"https://erp.example.com/yonbip/resources/uap/rbac/login/main/index.html"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `read_order_ids_from_file(id_file: str, encoding: str = "utf-8") -> List[str]`
|
||||
|
||||
Read order IDs from a text file (one ID per line).
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `id_file` | `str` | - | Path to file containing order IDs |
|
||||
| `encoding` | `str` | `"utf-8"` | File encoding |
|
||||
|
||||
**Returns:** `List[str]` - List of order IDs (empty lines filtered)
|
||||
|
||||
**Raises:**
|
||||
- `FileNotFoundError` - If id_file doesn't exist
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
>>> order_ids = read_order_ids_from_file("orders.txt")
|
||||
>>> len(order_ids)
|
||||
15
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Core Extraction Functions
|
||||
|
||||
#### `extract_batch(work_frame, page, order_ids, batch_index, download_dir) -> str`
|
||||
|
||||
Execute download workflow for a single batch of order IDs.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Description |
|
||||
|------|------|-------------|
|
||||
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||
| `page` | `Page` | Playwright page object for download handling |
|
||||
| `order_ids` | `List[str]` | Order IDs for this batch |
|
||||
| `batch_index` | `int` | Zero-based batch index (for file naming) |
|
||||
| `download_dir` | `str` | Directory to save downloaded file |
|
||||
|
||||
**Returns:** `str` - Path to downloaded Excel file
|
||||
|
||||
**Note:** This is a thin wrapper around `execute_batch_download_workflow` from `extractor_core.py`.
|
||||
|
||||
---
|
||||
|
||||
#### `extract_batches(work_frame, page, order_ids, download_dir, batch_size=10) -> List[str]`
|
||||
|
||||
Download data for multiple batches of order IDs.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||
| `page` | `Page` | Playwright page object |
|
||||
| `order_ids` | `List[str]` | List of order IDs to download |
|
||||
| `download_dir` | `str` | Directory to save downloaded files |
|
||||
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||
|
||||
**Returns:** `List[str]` - List of downloaded file paths
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# Caller manages browser session
|
||||
browser, context, page, main_frame = login(...)
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
setup_query_interface(work_frame)
|
||||
|
||||
files = extract_batches(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=["ID1", "ID2", "ID3"],
|
||||
download_dir="./downloads",
|
||||
batch_size=5
|
||||
)
|
||||
|
||||
context.close()
|
||||
browser.close()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### High-Level Workflow Functions
|
||||
|
||||
#### `extract_and_post_process(work_frame, page, order_ids, download_dir, output_file, batch_size=10, verbose=True) -> Tuple[str, DataFrame]`
|
||||
|
||||
Complete extraction workflow: download batches + post-process to merged Excel.
|
||||
|
||||
### Workflow Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User as User Code
|
||||
participant Auth as utils.auth.login
|
||||
participant Nav as navigate_to_discrete<br/>_material_page
|
||||
participant Setup as setup_query<br/>_interface
|
||||
participant Extract as extract_and_post<br/>_process
|
||||
participant Batches as extract_batches
|
||||
participant PostProcess as post_process<br/>_downloads
|
||||
participant Converter as ExcelConverter
|
||||
participant Browser as Browser
|
||||
participant ERP as Yonyou BIP ERP
|
||||
participant FS as File System
|
||||
|
||||
User->>Auth: login(credentials)
|
||||
Auth->>Browser: Launch
|
||||
Auth->>ERP: Authenticate
|
||||
Auth-->>User: (browser, context, page, main_frame)
|
||||
|
||||
User->>Nav: navigate(main_frame, page)
|
||||
Nav->>ERP: Load page
|
||||
Nav-->>User: (work_frame, page1)
|
||||
|
||||
User->>Setup: setup_query_interface(work_frame)
|
||||
Setup->>ERP: Configure query panel
|
||||
|
||||
User->>Extract: extract_and_post_process(params)
|
||||
|
||||
Note over Extract,PostProcess: Phase 1: Download
|
||||
Extract->>Batches: extract_batches(work_frame, page, order_ids)
|
||||
|
||||
loop For each batch
|
||||
Batches->>Batches: chunk_order_ids(order_ids, batch_size)
|
||||
Batches->>Setup: setup_query_interface()
|
||||
Batches->>ERP: Fill order IDs + Search
|
||||
Batches->>ERP: Click Export
|
||||
ERP-->>FS: Save batch_N.xlsx
|
||||
Batches-->>Extract: [batch_1.xlsx, batch_2.xlsx, ...]
|
||||
end
|
||||
|
||||
Note over Extract,PostProcess: Phase 2: Post-Process
|
||||
Extract->>PostProcess: post_process_downloads(files, output_file)
|
||||
|
||||
loop For each downloaded file
|
||||
PostProcess->>Converter: convert(batch_N.xlsx)
|
||||
Converter->>FS: Read Excel
|
||||
Converter->>Converter: Parse nested structure
|
||||
Converter-->>PostProcess: DataFrame
|
||||
end
|
||||
|
||||
PostProcess->>PostProcess: pd.concat(all_dfs)
|
||||
PostProcess->>FS: Write merged.xlsx
|
||||
PostProcess-->>Extract: (output_path, merged_df)
|
||||
|
||||
Extract-->>User: (output_path, DataFrame)
|
||||
|
||||
User->>Browser: context.close()
|
||||
User->>Browser: browser.close()
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `work_frame` | `Frame` | - | Work iframe containing the data grid |
|
||||
| `page` | `Page` | - | Playwright page object for download handling |
|
||||
| `order_ids` | `List[str]` | - | List of order IDs to extract |
|
||||
| `download_dir` | `str` | - | Directory for temporary batch files |
|
||||
| `output_file` | `str` | - | Path for final merged Excel output |
|
||||
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||
| `verbose` | `bool` | `True` | Print progress messages |
|
||||
|
||||
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from utils.discrete_material_plan import extract_and_post_process
|
||||
|
||||
# Caller manages session (see full example above)
|
||||
output_path, df = extract_and_post_process(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=["SC70202603240001", "SC70202603240002"],
|
||||
download_dir="./downloads",
|
||||
output_file="./output/merged.xlsx",
|
||||
batch_size=10,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
print(f"Saved {len(df)} records to {output_path}")
|
||||
```
|
||||
|
||||
**Workflow:**
|
||||
1. Downloads order data in batches to `download_dir`
|
||||
2. Converts each downloaded Excel file to DataFrame
|
||||
3. Merges all DataFrames into one
|
||||
4. Saves merged result to `output_file`
|
||||
5. Returns (path, dataframe) tuple
|
||||
|
||||
---
|
||||
|
||||
#### `extract_from_file(id_file, work_frame, page, download_dir, output_file, batch_size=10, verbose=True) -> Tuple[str, DataFrame]`
|
||||
|
||||
Extract data from order IDs in a file and post-process to merged Excel.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `id_file` | `str` | - | Path to file with order IDs (one per line) |
|
||||
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||
| `page` | `Page` | Playwright page object for download |
|
||||
| `download_dir` | `str` | Directory for temporary batch files |
|
||||
| `output_file` | `str` | Path for final merged Excel output |
|
||||
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||
| `verbose` | `bool` | `True` | Print progress messages |
|
||||
|
||||
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from utils.discrete_material_plan import extract_from_file
|
||||
|
||||
# After setting up browser session and navigating to page
|
||||
output_path, df = extract_from_file(
|
||||
id_file="order_ids.txt",
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
download_dir="./downloads",
|
||||
output_file="./output/results.xlsx",
|
||||
batch_size=10,
|
||||
verbose=True,
|
||||
)
|
||||
```
|
||||
|
||||
**File Format:**
|
||||
```
|
||||
SC70202603240001
|
||||
SC70202603240002
|
||||
SC70202603240003
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Post-Processing Functions
|
||||
|
||||
#### `post_process_downloads(downloaded_files, output_file, verbose=True) -> Tuple[str, DataFrame]`
|
||||
|
||||
Convert and merge downloaded Excel files into structured DataFrame.
|
||||
|
||||
**Parameters:**
|
||||
| Name | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `downloaded_files` | `List[str]` | - | List of downloaded Excel file paths |
|
||||
| `output_file` | `str` | - | Path to save merged Excel result |
|
||||
| `verbose` | `bool` | `True` | Print progress messages |
|
||||
|
||||
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from utils.discrete_material_plan import post_process_downloads
|
||||
|
||||
output_path, df = post_process_downloads(
|
||||
downloaded_files=[
|
||||
"./downloads/batch_1.xlsx",
|
||||
"./downloads/batch_2.xlsx",
|
||||
],
|
||||
output_file="./output/merged.xlsx",
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
print(f"Merged {len(df)} records")
|
||||
```
|
||||
|
||||
**Process:**
|
||||
1. Uses `ExcelConverter` to convert each file
|
||||
2. Merges all DataFrames with `pd.concat()`
|
||||
3. Saves merged result to `output_file`
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Function Hierarchy
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Level1["Level 1: Entry Points"]
|
||||
A1[extract_and_post_process]
|
||||
A2[extract_from_file]
|
||||
end
|
||||
|
||||
subgraph Level2["Level 2: Workflow Orchestration"]
|
||||
B1[extract_batches]
|
||||
B2[post_process_downloads]
|
||||
B3[read_order_ids_from_file]
|
||||
end
|
||||
|
||||
subgraph Level3["Level 3: Core Operations"]
|
||||
C1[extract_batch]
|
||||
C2[chunk_order_ids]
|
||||
C3[setup_query_interface]
|
||||
C4[ExcelConverter.convert]
|
||||
end
|
||||
|
||||
subgraph Level4["Level 4: Low-Level"]
|
||||
D1[execute_batch_download_workflow]
|
||||
D2[fill_and_search_orders]
|
||||
D3[download_batch_data]
|
||||
end
|
||||
|
||||
A1 --> B1
|
||||
A1 --> B2
|
||||
A2 --> B3
|
||||
A2 --> A1
|
||||
|
||||
B1 --> C1
|
||||
B1 --> C2
|
||||
B1 --> C3
|
||||
B2 --> C4
|
||||
|
||||
C1 --> D1
|
||||
C3 --> D2
|
||||
D1 --> D2
|
||||
D1 --> D3
|
||||
|
||||
style Level1 fill:#e3f2fd
|
||||
style Level2 fill:#fff3e0
|
||||
style Level3 fill:#f3e5f5
|
||||
style Level4 fill:#e8f5e9
|
||||
```
|
||||
|
||||
### Design Principles
|
||||
|
||||
**1. Stateless Pure Functions**
|
||||
```python
|
||||
# ✅ Correct: Stateless, explicit parameters
|
||||
output_path, df = extract_and_post_process(
|
||||
work_frame=work_frame,
|
||||
page=page,
|
||||
order_ids=order_ids,
|
||||
# ... explicit params
|
||||
)
|
||||
|
||||
# ❌ Wrong: Stateful class (OLD approach - removed)
|
||||
# extractor = DiscreteMaterialPlanExtractor(username, password, ...)
|
||||
```
|
||||
|
||||
**2. Caller-Managed Lifecycle**
|
||||
```python
|
||||
# Caller manages browser session
|
||||
browser, context, page, main_frame = login(...)
|
||||
try:
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
result = extract_and_post_process(work_frame, page1, order_ids, ...)
|
||||
finally:
|
||||
context.close() # Caller closes
|
||||
browser.close() # Caller closes
|
||||
```
|
||||
|
||||
**3. Explicit Dependencies**
|
||||
```python
|
||||
# ❌ No implicit state
|
||||
def extract(order_ids): # Missing required params
|
||||
...
|
||||
|
||||
# ✅ All params explicit
|
||||
def extract(work_frame, page, order_ids, download_dir, output_file):
|
||||
...
|
||||
```
|
||||
|
||||
### Module Dependencies
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Extractor["extractor.py"]
|
||||
E1[extract_and_post_process]
|
||||
E2[extract_batches]
|
||||
E3[post_process_downloads]
|
||||
end
|
||||
|
||||
subgraph Core["extractor_core.py"]
|
||||
C1[execute_batch_download_workflow]
|
||||
C2[setup_query_interface]
|
||||
C3[fill_and_search_orders]
|
||||
C4[download_batch_data]
|
||||
end
|
||||
|
||||
subgraph Converter["excel_converter.py"]
|
||||
K1[ExcelConverter]
|
||||
K2[convert]
|
||||
end
|
||||
|
||||
subgraph External["External Libraries"]
|
||||
P1[pandas DataFrame]
|
||||
P2[openpyxl]
|
||||
end
|
||||
|
||||
E1 --> E2
|
||||
E1 --> E3
|
||||
E2 --> C1
|
||||
E2 --> C2
|
||||
E3 --> K1
|
||||
K1 --> K2
|
||||
K2 --> P1
|
||||
K1 --> P2
|
||||
|
||||
C1 --> C2
|
||||
C1 --> C3
|
||||
C1 --> C4
|
||||
|
||||
style Extractor fill:#e8f5e9
|
||||
style Core fill:#fff3e0
|
||||
style Converter fill:#f3e5f5
|
||||
style External fill:#e3f2fd
|
||||
```
|
||||
|
||||
**Low Coupling:**
|
||||
- No direct dependency on `utils.auth`
|
||||
- Session objects (`work_frame`, `page`) passed as parameters
|
||||
- Caller controls lifecycle
|
||||
|
||||
---
|
||||
|
||||
## Integration Patterns
|
||||
|
||||
### Data Flow Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Input["Input"]
|
||||
I1[Order IDs<br/>List or File]
|
||||
end
|
||||
|
||||
subgraph Download["Download Phase<br/>Requires Browser"]
|
||||
D1[Chunk into<br/>Batches]
|
||||
D2[Search Orders<br/>in ERP]
|
||||
D3[Export to<br/>Excel]
|
||||
D4[(Batch Excel<br/>Files)]
|
||||
end
|
||||
|
||||
subgraph Process["Post-Process Phase<br/>No Browser Needed"]
|
||||
P1[Read Excel<br/>Files]
|
||||
P2[Parse Nested<br/>Structure]
|
||||
P3[Flatten to<br/>DataFrame]
|
||||
P4[Concatenate<br/>All Batches]
|
||||
P5[(Merged<br/>Excel)]
|
||||
end
|
||||
|
||||
subgraph Output["Output"]
|
||||
O1[DataFrame<br/>Object]
|
||||
O2[Excel File]
|
||||
end
|
||||
|
||||
I1 --> D1
|
||||
D1 --> D2
|
||||
D2 --> D3
|
||||
D3 --> D4
|
||||
D4 --> P1
|
||||
P1 --> P2
|
||||
P2 --> P3
|
||||
P3 --> P4
|
||||
P4 --> P5
|
||||
P4 --> O1
|
||||
P5 --> O2
|
||||
|
||||
style Input fill:#e1f5ff
|
||||
style Download fill:#fff4e1
|
||||
style Process fill:#e8f5e9
|
||||
style Output fill:#fce4ec
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Patterns
|
||||
|
||||
### Pattern 1: Complete Workflow with Session Management
|
||||
|
||||
```python
|
||||
from playwright.sync_api import sync_playwright
|
||||
from utils.auth import login, get_login_url
|
||||
from utils.discrete_material_plan import (
|
||||
navigate_to_discrete_material_page,
|
||||
extract_and_post_process,
|
||||
)
|
||||
|
||||
def main():
|
||||
with sync_playwright() as playwright:
|
||||
# Setup session
|
||||
browser, context, page, main_frame = login(
|
||||
playwright=playwright,
|
||||
username="admin",
|
||||
password="secret",
|
||||
url=get_login_url("https://erp.example.com"),
|
||||
headless=True,
|
||||
)
|
||||
|
||||
try:
|
||||
# Navigate
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
|
||||
# Extract
|
||||
output_path, df = extract_and_post_process(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=["ID1", "ID2", "ID3"],
|
||||
download_dir="./downloads",
|
||||
output_file="./output/result.xlsx",
|
||||
)
|
||||
|
||||
print(f"Done: {len(df)} records")
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
context.close()
|
||||
browser.close()
|
||||
```
|
||||
|
||||
### Pattern 2: Two-Phase (Download Then Process)
|
||||
|
||||
```python
|
||||
from utils.discrete_material_plan import (
|
||||
extract_batches,
|
||||
post_process_downloads,
|
||||
setup_query_interface,
|
||||
navigate_to_discrete_material_page,
|
||||
)
|
||||
from utils.auth import login
|
||||
|
||||
# Phase 1: Download
|
||||
browser, context, page, main_frame = login(...)
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
|
||||
downloaded_files = extract_batches(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=order_ids,
|
||||
download_dir="./downloads",
|
||||
batch_size=10,
|
||||
)
|
||||
|
||||
context.close()
|
||||
browser.close()
|
||||
|
||||
# Phase 2: Process (can be done later, even without browser)
|
||||
output_path, df = post_process_downloads(
|
||||
downloaded_files=downloaded_files,
|
||||
output_file="./output/merged.xlsx",
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 3: Custom Batch Processing with Error Handling
|
||||
|
||||
```python
|
||||
from utils.discrete_material_plan import chunk_order_ids, extract_batch
|
||||
from utils.auth import login
|
||||
|
||||
order_ids = [...] # Large list
|
||||
batches = chunk_order_ids(order_ids, batch_size=10)
|
||||
|
||||
success_files = []
|
||||
failed_batches = []
|
||||
|
||||
for i, batch in enumerate(batches):
|
||||
try:
|
||||
file_path = extract_batch(
|
||||
work_frame=work_frame,
|
||||
page=page,
|
||||
order_ids=batch,
|
||||
batch_index=i,
|
||||
download_dir="./downloads",
|
||||
)
|
||||
success_files.append(file_path)
|
||||
print(f"Batch {i+1}/{len(batches)} OK")
|
||||
except Exception as e:
|
||||
failed_batches.append(i)
|
||||
print(f"Batch {i+1} failed: {e}")
|
||||
|
||||
print(f"Success: {len(success_files)}, Failed: {len(failed_batches)}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
# Run component tests
|
||||
source .venv/Scripts/activate && python tests/test_extractor_component.py
|
||||
```
|
||||
|
||||
**Test Coverage:**
|
||||
- ✅ `chunk_order_ids()` - Batch splitting logic
|
||||
- ✅ `get_login_url()` - URL construction
|
||||
- ✅ `read_order_ids_from_file()` - File reading
|
||||
- ✅ Module exports verification
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```bash
|
||||
# Run real data extraction test
|
||||
source .venv/Scripts/activate && python tests/test_extractor_real.py
|
||||
```
|
||||
|
||||
**Note:** Integration tests require:
|
||||
- Valid ERP credentials in `.env`
|
||||
- Playwright browser installed
|
||||
- Network access to ERP system
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
**FileNotFoundError:**
|
||||
```python
|
||||
>>> read_order_ids_from_file("nonexistent.txt")
|
||||
FileNotFoundError: Order ID file not found: nonexistent.txt
|
||||
```
|
||||
|
||||
**Timeout during extraction:**
|
||||
```python
|
||||
try:
|
||||
output_path, df = extract_and_post_process(...)
|
||||
except TimeoutError as e:
|
||||
print(f"Operation timed out: {e}")
|
||||
# Browser session may need re-initialization
|
||||
```
|
||||
|
||||
**Permission errors (file access):**
|
||||
```python
|
||||
try:
|
||||
post_process_downloads(downloaded_files, output_file)
|
||||
except PermissionError:
|
||||
print(f"Cannot write to {output_file} - file may be open in another program")
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
```python
|
||||
# 1. Always cleanup browser sessions
|
||||
try:
|
||||
# extraction logic
|
||||
finally:
|
||||
context.close()
|
||||
browser.close()
|
||||
|
||||
# 2. Validate order IDs before extraction
|
||||
order_ids = read_order_ids_from_file("orders.txt")
|
||||
if not order_ids:
|
||||
raise ValueError("No order IDs to process")
|
||||
|
||||
# 3. Ensure output directory exists
|
||||
from pathlib import Path
|
||||
Path("./output").mkdir(parents=True, exist_ok=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Batch Size Tuning
|
||||
|
||||
**Small batches (5-10):**
|
||||
- ✅ More frequent progress updates
|
||||
- ✅ Easier to recover from failures
|
||||
- ❌ More UI interactions (slower)
|
||||
|
||||
**Large batches (50-100):**
|
||||
- ✅ Faster overall (fewer UI interactions)
|
||||
- ✅ Better for large datasets
|
||||
- ❌ Single failure affects more records
|
||||
|
||||
**Recommendation:** Start with `batch_size=10`, adjust based on:
|
||||
- Total order count
|
||||
- Network stability
|
||||
- UI response time
|
||||
|
||||
### Memory Usage
|
||||
|
||||
For very large extractions (1000+ orders):
|
||||
```python
|
||||
# Process in stages to limit memory
|
||||
all_dfs = []
|
||||
for batch_files in batch_groups:
|
||||
_, df = post_process_downloads(batch_files, output_file)
|
||||
all_dfs.append(df)
|
||||
|
||||
# Final merge
|
||||
merged_df = pd.concat(all_dfs, ignore_index=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From Old Class-Based API (v1) to New Function API (v2)
|
||||
|
||||
### API Comparison
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((API Comparison))
|
||||
v1 Old API
|
||||
Class-Based
|
||||
::icon(fa fa-times-circle)
|
||||
Stateful
|
||||
Internal session management
|
||||
Hidden dependencies
|
||||
Usage
|
||||
extractor = DiscreteMaterialPlanExtractor(...)
|
||||
extractor.extract_from_file(...)
|
||||
Issues
|
||||
Hard to test
|
||||
Tight coupling
|
||||
Implicit state
|
||||
v2 New API
|
||||
Pure Functions
|
||||
::icon(fa fa-check-circle)
|
||||
Stateless
|
||||
Caller-managed session
|
||||
Explicit dependencies
|
||||
Usage
|
||||
extract_and_post_process(params...)
|
||||
Return: (output_path, DataFrame)
|
||||
Benefits
|
||||
Easy to test
|
||||
Low coupling
|
||||
Clear data flow
|
||||
|
||||
```
|
||||
|
||||
**Old (v1):**
|
||||
```python
|
||||
from utils.discrete_material_plan import DiscreteMaterialPlanExtractor
|
||||
|
||||
extractor = DiscreteMaterialPlanExtractor(
|
||||
username="admin",
|
||||
password="secret",
|
||||
base_url="https://erp.example.com",
|
||||
)
|
||||
|
||||
output_path = extractor.extract_from_file("orders.txt", "output.xlsx")
|
||||
```
|
||||
|
||||
**New (v2):**
|
||||
```python
|
||||
from utils.discrete_material_plan import (
|
||||
extract_from_file,
|
||||
get_login_url,
|
||||
navigate_to_discrete_material_page,
|
||||
)
|
||||
from utils.auth import login
|
||||
|
||||
# Caller manages session
|
||||
browser, context, page, main_frame = login(
|
||||
playwright,
|
||||
username="admin",
|
||||
password="secret",
|
||||
url=get_login_url("https://erp.example.com"),
|
||||
)
|
||||
|
||||
try:
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
|
||||
output_path, df = extract_from_file(
|
||||
id_file="orders.txt",
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
download_dir="./downloads",
|
||||
output_file="./output/result.xlsx",
|
||||
)
|
||||
finally:
|
||||
context.close()
|
||||
browser.close()
|
||||
```
|
||||
|
||||
### Version Comparison Table
|
||||
|
||||
| Aspect | v1 (Class-Based) | v2 (Pure Functions) |
|
||||
|--------|------------------|---------------------|
|
||||
| **Pattern** | `DiscreteMaterialPlanExtractor` class | Standalone functions |
|
||||
| **State** | Instance variables (`self.username`, etc.) | No state, explicit params |
|
||||
| **Session** | Internal management | Caller manages |
|
||||
| **Return** | `str` (output path only) | `Tuple[str, DataFrame]` |
|
||||
| **Testing** | Hard (requires mocking class) | Easy (pure functions) |
|
||||
| **Coupling** | High (depends on `utils.auth`) | Low (session passed as param) |
|
||||
| **Flexibility** | Limited (fixed workflow) | High (can use individual functions) |
|
||||
|
||||
**Key Changes:**
|
||||
1. Class removed → pure functions
|
||||
2. Session management moved to caller
|
||||
3. Return value now includes DataFrame tuple
|
||||
4. All dependencies explicit
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [`extractor_core.py`](discrete_material_plan_extractor_core.md) - Low-level web operations
|
||||
- [`excel_converter.py`](extractor_post_processing.md) - Excel conversion
|
||||
- [`auth.py`](authentication.md) - Authentication and session management
|
||||
|
||||
---
|
||||
|
||||
## Version
|
||||
|
||||
**Current:** v2.0.0 (stateless pure functions)
|
||||
|
||||
**Breaking Changes in v2:**
|
||||
- Removed `DiscreteMaterialPlanExtractor` class
|
||||
- Changed to caller-managed session lifecycle
|
||||
- All functions now accept explicit parameters
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Part of BIPAuto project - see project root for license information.
|
||||
@@ -1,2 +1,4 @@
|
||||
playwright>=1.40.0
|
||||
python-dotenv>=1.0.0
|
||||
pandas>=2.0.0
|
||||
openpyxl>=3.1.0
|
||||
|
||||
166
tests/test_extractor_real.py
Normal file
166
tests/test_extractor_real.py
Normal file
@@ -0,0 +1,166 @@
|
||||
"""
|
||||
Integration test for discrete_material_plan.extractor module
|
||||
|
||||
Tests the complete extraction workflow using real ERP data.
|
||||
Requires valid ERP credentials in .env file.
|
||||
|
||||
This test:
|
||||
1. Reads order IDs from tests/id-demo.txt
|
||||
2. Downloads data from Yonyou BIP ERP system
|
||||
3. Saves raw Excel files to data/downloads/
|
||||
4. Converts and merges to data/output/merged_result.xlsx
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
# Add project root to Python path
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
# Load environment variables
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(PROJECT_ROOT / ".env")
|
||||
|
||||
# Configure browser path
|
||||
import os
|
||||
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = os.getenv("PLAYWRIGHT_BROWSERS_PATH", "")
|
||||
|
||||
# Generate timestamp for unique output files
|
||||
TIMESTAMP = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
|
||||
# Output paths
|
||||
DOWNLOAD_DIR = PROJECT_ROOT / "data" / "downloads"
|
||||
OUTPUT_DIR = PROJECT_ROOT / "data" / "output"
|
||||
OUTPUT_FILE = OUTPUT_DIR / f"merged_result_{TIMESTAMP}.xlsx"
|
||||
|
||||
# Ensure directories exist
|
||||
DOWNLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print("=" * 60)
|
||||
print("Discrete Material Plan Extractor - Integration Test")
|
||||
print("=" * 60)
|
||||
print(f"\nConfiguration:")
|
||||
print(f" ID File: {PROJECT_ROOT / 'tests' / 'id-demo.txt'}")
|
||||
print(f" Download Dir: {DOWNLOAD_DIR}")
|
||||
print(f" Output File: {OUTPUT_FILE}")
|
||||
print(f" ERP URL: {os.getenv('ERP_URL', 'Not set')}")
|
||||
print(f" Headless: {os.getenv('ERP_HEADLESS', 'true')}")
|
||||
print("=" * 60)
|
||||
|
||||
# Verify environment variables
|
||||
required_vars = ["ERP_USERNAME", "ERP_PASSWORD", "ERP_URL"]
|
||||
missing_vars = [var for var in required_vars if not os.getenv(var)]
|
||||
|
||||
if missing_vars:
|
||||
print(f"\n[ERROR] Missing required environment variables: {missing_vars}")
|
||||
print("Please ensure .env file contains:")
|
||||
print(" ERP_USERNAME=your_username")
|
||||
print(" ERP_PASSWORD=your_password")
|
||||
print(" ERP_URL=https://your-erp-url.com")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
# Import required modules
|
||||
print("\n[1/6] Importing modules...")
|
||||
from playwright.sync_api import sync_playwright
|
||||
from utils.auth import login
|
||||
from utils.discrete_material_plan import (
|
||||
navigate_to_discrete_material_page,
|
||||
extract_and_post_process,
|
||||
read_order_ids_from_file,
|
||||
get_login_url,
|
||||
)
|
||||
print("[OK] Modules imported successfully")
|
||||
|
||||
# Read order IDs from demo file
|
||||
print("\n[2/6] Reading order IDs from data/demo/id-demo.txt...")
|
||||
id_file = PROJECT_ROOT / "data" / "demo" / "id-demo.txt"
|
||||
order_ids = read_order_ids_from_file(str(id_file))
|
||||
print(f"[OK] Loaded {len(order_ids)} order IDs")
|
||||
|
||||
test_order_ids = order_ids
|
||||
|
||||
# Setup browser and extract data
|
||||
print("\n[3/6] Launching browser and logging in...")
|
||||
with sync_playwright() as playwright:
|
||||
# Login
|
||||
browser, context, page, main_frame = login(
|
||||
playwright=playwright,
|
||||
username=os.getenv("ERP_USERNAME"),
|
||||
password=os.getenv("ERP_PASSWORD"),
|
||||
url=get_login_url(os.getenv("ERP_URL")),
|
||||
headless=os.getenv("ERP_HEADLESS", "true").lower() == "true",
|
||||
ignore_https_errors=os.getenv("ERP_IGNORE_HTTPS_ERRORS", "true").lower() == "true",
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
try:
|
||||
# Navigate to discrete material page
|
||||
print("\n[4/6] Navigating to discrete material plan page...")
|
||||
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
print("[OK] Navigation successful")
|
||||
|
||||
# Extract and post-process
|
||||
print("\n[5/6] Extracting data and post-processing...")
|
||||
print(f" Orders: {len(test_order_ids)}")
|
||||
print(f" Batch size: 100")
|
||||
print(f" Download dir: {DOWNLOAD_DIR}")
|
||||
print(f" Output file: {OUTPUT_FILE}")
|
||||
print(f" Cleanup temp files: True (default)")
|
||||
|
||||
output_path, df = extract_and_post_process(
|
||||
work_frame=work_frame,
|
||||
page=page1,
|
||||
order_ids=test_order_ids,
|
||||
download_dir=str(DOWNLOAD_DIR),
|
||||
output_file=str(OUTPUT_FILE),
|
||||
batch_size=100,
|
||||
verbose=True,
|
||||
cleanup_temp_files=True, # Default: True, set to False to keep temp files
|
||||
)
|
||||
|
||||
print("\n[6/6] Verifying results...")
|
||||
print(f"[OK] Extraction completed successfully")
|
||||
print(f" Output file: {output_path}")
|
||||
print(f" Total records: {len(df)}")
|
||||
print(f" Columns: {list(df.columns)}")
|
||||
|
||||
# Verify output file exists
|
||||
if not Path(output_path).exists():
|
||||
raise FileNotFoundError(f"Output file not found: {output_path}")
|
||||
print(f"[OK] Output file verified: {output_path}")
|
||||
|
||||
# Show sample data
|
||||
print("\n" + "=" * 60)
|
||||
print("Sample Data (first 3 rows, first 5 columns):")
|
||||
print("=" * 60)
|
||||
# Use unicode encoding for display
|
||||
sample = df.head(3).iloc[:, :5]
|
||||
print(sample.to_string())
|
||||
print("\n[OK] Data extraction verified - Chinese characters displayed correctly in console encoding")
|
||||
|
||||
finally:
|
||||
# Cleanup browser
|
||||
print("\n\n[Cleanup] Closing browser...")
|
||||
context.close()
|
||||
browser.close()
|
||||
print("[OK] Browser closed")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("[SUCCESS] Integration test completed successfully!")
|
||||
print("=" * 60)
|
||||
print(f"\nOutputs:")
|
||||
print(f" Raw Excel files: {DOWNLOAD_DIR}/*.xlsx")
|
||||
print(f" Merged result: {OUTPUT_FILE}")
|
||||
print(f" Total records: {len(df)}")
|
||||
print("=" * 60)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n[ERROR] Integration test failed!")
|
||||
print(f"Error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
11
utils/__init__.py
Normal file
11
utils/__init__.py
Normal file
@@ -0,0 +1,11 @@
|
||||
"""
|
||||
Utils package for BIPAuto automation framework.
|
||||
"""
|
||||
|
||||
from .auth import login, logout
|
||||
|
||||
__all__ = [
|
||||
# Auth module
|
||||
"login",
|
||||
"logout",
|
||||
]
|
||||
280
utils/discrete_material_plan/excel_converter.py
Normal file
280
utils/discrete_material_plan/excel_converter.py
Normal file
@@ -0,0 +1,280 @@
|
||||
"""
|
||||
Excel 报表数据转换工具组件
|
||||
将 Excel 报表数据转换为数据库记录形式
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
import openpyxl
|
||||
from typing import List, Dict, Optional
|
||||
import os
|
||||
|
||||
|
||||
class ExcelConverter:
|
||||
"""Excel 报表数据转换器"""
|
||||
|
||||
# 字段名称映射（解决字段名冲突）
|
||||
FIELD_NAME_MAPPING = {"计划数量": "产品计划数量", "单位": "产品单位"}
|
||||
|
||||
def __init__(self, verbose: bool = True):
|
||||
"""
|
||||
初始化转换器
|
||||
|
||||
Args:
|
||||
verbose: 是否打印详细日志
|
||||
"""
|
||||
self.verbose = verbose
|
||||
|
||||
def _print(self, *args, **kwargs):
|
||||
"""打印日志（如果 verbose=True）"""
|
||||
if self.verbose:
|
||||
print(*args, **kwargs)
|
||||
|
||||
def convert(self, input_file: str, output_file: str = None) -> pd.DataFrame:
|
||||
"""
|
||||
转换 Excel 文件
|
||||
|
||||
Args:
|
||||
input_file: 输入文件路径
|
||||
output_file: 输出文件路径（可选，不指定则不保存）
|
||||
|
||||
Returns:
|
||||
转换后的 DataFrame
|
||||
"""
|
||||
# 处理输出文件名
|
||||
if output_file:
|
||||
output_file = self._handle_output_file(output_file)
|
||||
|
||||
# 读取工作表
|
||||
wb = openpyxl.load_workbook(input_file)
|
||||
ws = wb.active
|
||||
|
||||
# 解析订单数据
|
||||
orders = self._parse_sheet(ws)
|
||||
|
||||
# 转换为 DataFrame
|
||||
df = self._convert_to_dataframe(orders)
|
||||
|
||||
if not df.empty:
|
||||
# 保存文件
|
||||
if output_file:
|
||||
df.to_excel(output_file, index=False)
|
||||
|
||||
# 打印汇总报告
|
||||
self._print("=" * 60)
|
||||
self._print("转换完成")
|
||||
self._print("=" * 60)
|
||||
self._print(f"订单数: {len(orders)}")
|
||||
self._print(f"数据行数: {len(df)}")
|
||||
self._print(f"输出文件: {output_file if output_file else 'N/A'}")
|
||||
self._print("=" * 60)
|
||||
|
||||
return df
|
||||
|
||||
def _handle_output_file(self, output_file: str) -> str:
|
||||
"""
|
||||
处理输出文件，如果文件存在则尝试删除
|
||||
|
||||
Args:
|
||||
output_file: 输出文件路径
|
||||
|
||||
Returns:
|
||||
实际使用的输出文件路径
|
||||
"""
|
||||
if os.path.exists(output_file):
|
||||
try:
|
||||
os.remove(output_file)
|
||||
except PermissionError:
|
||||
self._print(f"警告: 无法删除 {output_file}，可能文件被其他程序打开")
|
||||
# 修改文件名
|
||||
base, ext = os.path.splitext(output_file)
|
||||
output_file = f"{base}_new{ext}"
|
||||
return output_file
|
||||
|
||||
def _parse_sheet(self, ws) -> List[Dict]:
|
||||
"""
|
||||
解析一个工作表，返回所有订单的数据
|
||||
|
||||
每个订单包含:
|
||||
- order_info: 订单头信息（包括页脚）
|
||||
- materials: 物料数据列表
|
||||
|
||||
Args:
|
||||
ws: openpyxl 工作表对象
|
||||
|
||||
Returns:
|
||||
订单列表
|
||||
"""
|
||||
orders = []
|
||||
all_rows = list(ws.iter_rows(values_only=True))
|
||||
|
||||
# 逐行扫描，按订单结构解析
|
||||
i = 0
|
||||
while i < len(all_rows):
|
||||
row = all_rows[i]
|
||||
|
||||
# 检查是否是订单标题行
|
||||
if row and "离散备料计划" in str(row[0]):
|
||||
# 解析订单头信息（接下来的4行）
|
||||
order_info = {}
|
||||
for j in range(1, 5):
|
||||
if i + j < len(all_rows) and all_rows[i + j]:
|
||||
self._parse_header_row(all_rows[i + j], order_info)
|
||||
|
||||
# 跳过空行，找到表格标题行
|
||||
table_row = i + 5
|
||||
while table_row < len(all_rows) and (
|
||||
not all_rows[table_row] or not all_rows[table_row][0]
|
||||
):
|
||||
table_row += 1
|
||||
|
||||
# 检查是否是表格标题行
|
||||
if (
|
||||
table_row < len(all_rows)
|
||||
and all_rows[table_row]
|
||||
and all_rows[table_row][0] == "序号"
|
||||
):
|
||||
# 检查表头下一行是否为空，判断是否存在数据
|
||||
next_row = table_row + 1
|
||||
is_empty_row = (
|
||||
next_row < len(all_rows)
|
||||
and all_rows[next_row]
|
||||
and all(
|
||||
cell is None or str(cell).strip() == ""
|
||||
for cell in all_rows[next_row]
|
||||
)
|
||||
)
|
||||
|
||||
if is_empty_row:
|
||||
# 没有数据，查找页脚信息
|
||||
materials = []
|
||||
footer_info = {}
|
||||
data_row = next_row + 1
|
||||
while data_row < len(all_rows) and all_rows[data_row]:
|
||||
if all_rows[data_row][0] and (
|
||||
"制单人" in str(all_rows[data_row][0])
|
||||
or "打印人" in str(all_rows[data_row][0])
|
||||
):
|
||||
self._parse_header_row(all_rows[data_row], footer_info)
|
||||
if (
|
||||
data_row + 1 < len(all_rows)
|
||||
and all_rows[data_row + 1]
|
||||
):
|
||||
self._parse_header_row(
|
||||
all_rows[data_row + 1], footer_info
|
||||
)
|
||||
break
|
||||
data_row += 1
|
||||
|
||||
orders.append(
|
||||
{
|
||||
"order_info": {**order_info, **footer_info},
|
||||
"materials": materials,
|
||||
}
|
||||
)
|
||||
else:
|
||||
# 有数据，开始提取物料
|
||||
materials = []
|
||||
footer_info = {} # 页脚信息
|
||||
data_row = table_row + 1
|
||||
while data_row < len(all_rows) and all_rows[data_row]:
|
||||
# 检查是否是页脚信息（制单人、打印人）
|
||||
if all_rows[data_row + 1][0] and "制单人" in str(
|
||||
all_rows[data_row + 1][0]
|
||||
):
|
||||
# 解析页脚信息
|
||||
self._parse_header_row(all_rows[data_row], footer_info)
|
||||
# 检查下一行是否也是页脚信息
|
||||
if (
|
||||
data_row + 1 < len(all_rows)
|
||||
and all_rows[data_row + 1]
|
||||
):
|
||||
self._parse_header_row(
|
||||
all_rows[data_row + 1], footer_info
|
||||
)
|
||||
break
|
||||
|
||||
# 提取物料数据
|
||||
material_row = all_rows[data_row]
|
||||
material = {
|
||||
"序号": material_row[0],
|
||||
"材料编码": material_row[1],
|
||||
"材料名称": material_row[2],
|
||||
"规格": material_row[3],
|
||||
"型号": material_row[4],
|
||||
"图号": material_row[5],
|
||||
"物料材质": material_row[6],
|
||||
"计划数量": material_row[7],
|
||||
"单位": material_row[8],
|
||||
"需用日期": material_row[9],
|
||||
"发料仓库": material_row[10],
|
||||
"单位用量": material_row[11],
|
||||
"累计出库数量": material_row[12],
|
||||
}
|
||||
materials.append(material)
|
||||
|
||||
data_row += 1
|
||||
|
||||
orders.append(
|
||||
{
|
||||
"order_info": {**order_info, **footer_info},
|
||||
"materials": materials,
|
||||
}
|
||||
)
|
||||
|
||||
i += 1
|
||||
|
||||
return orders
|
||||
|
||||
def _parse_header_row(self, row: tuple, info: Dict):
|
||||
"""
|
||||
解析订单头信息的一行（字段名和值交错排列）
|
||||
|
||||
Args:
|
||||
row: 行数据
|
||||
info: 存储解析结果的字典
|
||||
"""
|
||||
i = 0
|
||||
while i < len(row):
|
||||
cell = row[i]
|
||||
if cell and str(cell).strip() and "：" in str(cell):
|
||||
# 找到字段名
|
||||
field_name = str(cell).replace("：", "").strip()
|
||||
|
||||
# 应用字段名映射
|
||||
if field_name in self.FIELD_NAME_MAPPING:
|
||||
field_name = self.FIELD_NAME_MAPPING[field_name]
|
||||
|
||||
# 跳过空单元格，找到第一个非字段名的值
|
||||
j = i + 1
|
||||
while j < len(row) and (
|
||||
not row[j] or not str(row[j]).strip() or "：" in str(row[j])
|
||||
):
|
||||
j += 1
|
||||
if j < len(row) and row[j] and not "：" in str(row[j]):
|
||||
info[field_name] = str(row[j]).strip()
|
||||
# 跳过已处理的值，继续找下一个字段名
|
||||
i = j + 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
def _convert_to_dataframe(self, orders: List[Dict]) -> pd.DataFrame:
|
||||
"""
|
||||
将订单数据转换为扁平化的 DataFrame
|
||||
|
||||
Args:
|
||||
orders: 订单列表
|
||||
|
||||
Returns:
|
||||
扁平化的 DataFrame
|
||||
"""
|
||||
all_records = []
|
||||
|
||||
for order in orders:
|
||||
order_info = order["order_info"]
|
||||
materials = order["materials"]
|
||||
|
||||
for material in materials:
|
||||
record = {**order_info, **material}
|
||||
all_records.append(record)
|
||||
|
||||
return pd.DataFrame(all_records)
|
||||
@@ -144,6 +144,7 @@ def post_process_downloads(
|
||||
downloaded_files: List[str],
|
||||
output_file: str,
|
||||
verbose: bool = True,
|
||||
cleanup_temp_files: bool = True,
|
||||
) -> Tuple[str, pd.DataFrame]:
|
||||
"""
|
||||
Convert and merge downloaded Excel files into structured DataFrame.
|
||||
@@ -154,6 +155,7 @@ def post_process_downloads(
|
||||
downloaded_files: List of paths to downloaded Excel files
|
||||
output_file: Path to save merged Excel result
|
||||
verbose: Whether to print progress messages
|
||||
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||
|
||||
Returns:
|
||||
Tuple of (output_file_path, merged_dataframe)
|
||||
@@ -162,7 +164,8 @@ def post_process_downloads(
|
||||
>>> output_path, df = post_process_downloads(
|
||||
... downloaded_files=["batch_1.xlsx", "batch_2.xlsx"],
|
||||
... output_file="merged.xlsx",
|
||||
... verbose=True
|
||||
... verbose=True,
|
||||
... cleanup_temp_files=True
|
||||
... )
|
||||
"""
|
||||
from .excel_converter import ExcelConverter
|
||||
@@ -194,9 +197,37 @@ def post_process_downloads(
|
||||
print(f"Merged result saved to: {output_path}")
|
||||
print(f"Total rows: {len(merged_df)}")
|
||||
|
||||
# Cleanup temporary downloaded files
|
||||
if cleanup_temp_files:
|
||||
_cleanup_temp_files(downloaded_files, verbose)
|
||||
|
||||
return str(output_path), merged_df
|
||||
|
||||
|
||||
def _cleanup_temp_files(downloaded_files: List[str], verbose: bool = True) -> int:
|
||||
"""
|
||||
Remove temporary downloaded files.
|
||||
|
||||
Args:
|
||||
downloaded_files: List of file paths to delete
|
||||
verbose: Whether to print progress messages
|
||||
|
||||
Returns:
|
||||
Number of files successfully deleted
|
||||
"""
|
||||
deleted_count = 0
|
||||
for file_path in downloaded_files:
|
||||
try:
|
||||
Path(file_path).unlink()
|
||||
deleted_count += 1
|
||||
if verbose:
|
||||
print(f"Deleted temp file: {file_path}")
|
||||
except Exception as e:
|
||||
if verbose:
|
||||
print(f"Warning: Could not delete {file_path}: {e}")
|
||||
return deleted_count
|
||||
|
||||
|
||||
def extract_and_post_process(
|
||||
work_frame: Frame,
|
||||
page: Page,
|
||||
@@ -205,6 +236,7 @@ def extract_and_post_process(
|
||||
output_file: str,
|
||||
batch_size: int = 10,
|
||||
verbose: bool = True,
|
||||
cleanup_temp_files: bool = True,
|
||||
) -> Tuple[str, pd.DataFrame]:
|
||||
"""
|
||||
Complete extraction workflow: download batches + post-process to merged Excel.
|
||||
@@ -220,6 +252,7 @@ def extract_and_post_process(
|
||||
output_file: Path for final merged Excel output
|
||||
batch_size: Maximum order IDs per batch
|
||||
verbose: Whether to print progress messages
|
||||
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||
|
||||
Returns:
|
||||
Tuple of (output_file_path, merged_dataframe)
|
||||
@@ -229,7 +262,8 @@ def extract_and_post_process(
|
||||
>>> browser, context, page, main_frame = login(...)
|
||||
>>> work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||
>>> output_path, df = extract_and_post_process(
|
||||
... work_frame, page1, order_ids, "/downloads", "output.xlsx"
|
||||
... work_frame, page1, order_ids, "/downloads", "output.xlsx",
|
||||
... cleanup_temp_files=True
|
||||
... )
|
||||
>>> context.close()
|
||||
>>> browser.close()
|
||||
@@ -254,6 +288,7 @@ def extract_and_post_process(
|
||||
downloaded_files=downloaded_files,
|
||||
output_file=output_file,
|
||||
verbose=verbose,
|
||||
cleanup_temp_files=cleanup_temp_files,
|
||||
)
|
||||
|
||||
return output_path, merged_df
|
||||
@@ -292,6 +327,7 @@ def extract_from_file(
|
||||
output_file: str,
|
||||
batch_size: int = 10,
|
||||
verbose: bool = True,
|
||||
cleanup_temp_files: bool = True,
|
||||
) -> Tuple[str, pd.DataFrame]:
|
||||
"""
|
||||
Extract data from order IDs in a file and post-process to merged Excel.
|
||||
@@ -306,6 +342,7 @@ def extract_from_file(
|
||||
output_file: Path for final merged Excel output
|
||||
batch_size: Maximum order IDs per batch
|
||||
verbose: Whether to print progress messages
|
||||
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||
|
||||
Returns:
|
||||
Tuple of (output_file_path, merged_dataframe)
|
||||
@@ -315,7 +352,8 @@ def extract_from_file(
|
||||
>>> order_ids = read_order_ids_from_file("orders.txt")
|
||||
>>> # Extract and process
|
||||
>>> output_path, df = extract_from_file(
|
||||
... "orders.txt", work_frame, page, "/downloads", "output.xlsx"
|
||||
... "orders.txt", work_frame, page, "/downloads", "output.xlsx",
|
||||
... cleanup_temp_files=True
|
||||
... )
|
||||
"""
|
||||
order_ids = read_order_ids_from_file(id_file)
|
||||
@@ -331,4 +369,5 @@ def extract_from_file(
|
||||
output_file=output_file,
|
||||
batch_size=batch_size,
|
||||
verbose=verbose,
|
||||
cleanup_temp_files=cleanup_temp_files,
|
||||
)
|
||||
|
||||
164
utils/discrete_material_plan/extractor_core.py
Normal file
164
utils/discrete_material_plan/extractor_core.py
Normal file
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Core Web Operations Module
|
||||
Pure functions for Yonyou BIP discrete material plan maintenance page interactions.
|
||||
All functions are stateless and accept required parameters explicitly.
|
||||
"""
|
||||
|
||||
import re
|
||||
import os
|
||||
from typing import List
|
||||
from playwright.sync_api import Page, Frame, TimeoutError
|
||||
|
||||
|
||||
def navigate_to_discrete_material_page(main_frame: Frame, page: Page) -> tuple[Frame, Page]:
|
||||
"""
|
||||
Navigate to the discrete material plan maintenance page.
|
||||
|
||||
Args:
|
||||
main_frame: The main forwardFrame iframe
|
||||
page: The Playwright page object
|
||||
|
||||
Returns:
|
||||
tuple: (work_frame, page1) - The work iframe and the new popup page
|
||||
"""
|
||||
# Click icon to open menu
|
||||
main_frame.locator("i").first.click()
|
||||
|
||||
# Wait for popup and click menu item
|
||||
with page.expect_popup() as page1_info:
|
||||
main_frame.get_by_title("离散备料计划维护", exact=True).first.click()
|
||||
|
||||
page1 = page1_info.value
|
||||
|
||||
# Get nested iframes
|
||||
f_frame = page1.locator("#forwardFrame").content_frame
|
||||
inner_frame_locator = f_frame.locator("#mainiframe")
|
||||
inner_frame_locator.wait_for(state="visible", timeout=15000)
|
||||
work_frame = inner_frame_locator.content_frame
|
||||
|
||||
return work_frame, page1
|
||||
|
||||
|
||||
def setup_query_interface(work_frame: Frame) -> None:
|
||||
"""
|
||||
Initialize the query interface by selecting order number query tab.
|
||||
|
||||
Args:
|
||||
work_frame: The inner work iframe containing the query interface
|
||||
"""
|
||||
# Open search panel
|
||||
work_frame.locator(".search-name-wrapper > .iconfont").click()
|
||||
|
||||
# Select order number query
|
||||
work_frame.get_by_text("订单号查询").click()
|
||||
|
||||
# Select "All" tab
|
||||
work_frame.get_by_role("tab", name="全部").click()
|
||||
|
||||
# Set page size to 5000
|
||||
input_box = work_frame.locator("#rc_select_0")
|
||||
input_box.fill("5000")
|
||||
input_box.press("Enter")
|
||||
|
||||
|
||||
def fill_and_search_orders(work_frame: Frame, order_ids: List[str]) -> None:
|
||||
"""
|
||||
Fill order IDs into the search textbox and trigger search.
|
||||
|
||||
Args:
|
||||
work_frame: The work iframe containing the search form
|
||||
order_ids: List of order IDs to search for
|
||||
"""
|
||||
textbox = work_frame.get_by_role("textbox", name="来源生产订单号")
|
||||
|
||||
# Clear and fill order IDs
|
||||
textbox.fill("")
|
||||
textbox.fill(",".join(order_ids))
|
||||
|
||||
# Click search button
|
||||
work_frame.locator(".search-component-searchBtn").click()
|
||||
|
||||
# Wait for loading to complete
|
||||
loading_locator = work_frame.locator("div").filter(has_text="加载中").nth(1)
|
||||
try:
|
||||
loading_locator.wait_for(state="visible", timeout=3000)
|
||||
loading_locator.wait_for(state="hidden", timeout=0)
|
||||
except TimeoutError:
|
||||
pass
|
||||
|
||||
|
||||
def download_batch_data(
|
||||
work_frame: Frame,
|
||||
page: Page,
|
||||
order_ids: List[str],
|
||||
batch_index: int,
|
||||
download_dir: str
|
||||
) -> str:
|
||||
"""
|
||||
Execute the download workflow for a single batch of order IDs.
|
||||
|
||||
Args:
|
||||
work_frame: The work iframe containing the data grid
|
||||
page: The Playwright page object for download handling
|
||||
order_ids: List of order IDs to download
|
||||
batch_index: Zero-based batch index for naming the output file
|
||||
download_dir: Directory path to save the downloaded file
|
||||
|
||||
Returns:
|
||||
str: Full path to the downloaded file
|
||||
|
||||
Raises:
|
||||
TimeoutError: If UI elements are not found or operations timeout
|
||||
"""
|
||||
# Step 1: Select first row
|
||||
work_frame.get_by_role("row", name="序号").get_by_label("").click()
|
||||
|
||||
# Step 2: Hover over "More" button
|
||||
work_frame.get_by_role("button", name="更多").hover()
|
||||
|
||||
# Step 3: Click "Export"
|
||||
work_frame.get_by_text("输出", exact=True).click()
|
||||
|
||||
# Step 4: Set row threshold
|
||||
threshold_box = (
|
||||
work_frame.locator("div")
|
||||
.filter(has_text=re.compile(r"^行数阈值$"))
|
||||
.locator("input[type='text']")
|
||||
)
|
||||
threshold_box.fill("300000")
|
||||
|
||||
# Step 5: Trigger download and save file
|
||||
download_filename = f"temp_batch_{batch_index + 1}.xlsx"
|
||||
download_path = os.path.join(download_dir, download_filename)
|
||||
|
||||
with page.expect_download() as download_info:
|
||||
work_frame.get_by_role("button", name="确定(Y)").click()
|
||||
|
||||
download = download_info.value
|
||||
download.save_as(download_path)
|
||||
|
||||
return download_path
|
||||
|
||||
|
||||
def execute_batch_download_workflow(
|
||||
work_frame: Frame,
|
||||
page: Page,
|
||||
order_ids: List[str],
|
||||
batch_index: int,
|
||||
download_dir: str
|
||||
) -> str:
|
||||
"""
|
||||
Complete workflow: fill orders, search, and download for a single batch.
|
||||
|
||||
Args:
|
||||
work_frame: The work iframe containing the search form and data grid
|
||||
page: The Playwright page object for download handling
|
||||
order_ids: List of order IDs for this batch
|
||||
batch_index: Zero-based batch index for naming the output file
|
||||
download_dir: Directory path to save the downloaded file
|
||||
|
||||
Returns:
|
||||
str: Full path to the downloaded file
|
||||
"""
|
||||
fill_and_search_orders(work_frame, order_ids)
|
||||
return download_batch_data(work_frame, page, order_ids, batch_index, download_dir)
|
||||
Reference in New Issue
Block a user