admin/BIPAuto
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(extractor): Add Excel conversion and post-processing capabilities

Browse Source 
Add comprehensive post-processing features to convert downloaded Excel files
into structured data and merge them into a single output file.

New modules:
- extractor_core.py: Stateless pure functions for web operations
- excel_converter.py: Excel to DataFrame conversion utility
- tests/test_extractor_real.py: Real data extraction test suite

Enhanced features:
- post_process_downloads(): Convert and merge multiple Excel files
- extract_and_process(): Complete workflow in single call
- cleanup_temp_files(): Optional cleanup of temporary downloaded files
- Field name mapping for standardized output columns

Dependencies:
- pandas>=2.0.0 for data manipulation
- openpyxl>=3.1.0 for Excel file handling

Documentation:
- Updated CLAUDE.md with new module references
- Added API documentation for extractor components

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
2026-03-27 14:34:56 +08:00
parent 1b984f5cfd
commit c3bbc919a5
9 changed files with 1734 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
CLAUDE.md
View File

@@ -45,6 +45,34 @@ Note: Test scripts are responsible for loading environment variables and passing
- Returns tuple: `(browser, context, page, main_frame)` where `main_frame` is the forwardFrame iframe.
- Callers are responsible for browser lifecycle management (context.close(), browser.close()).
**`utils/discrete_material_plan/extractor_core.py`** - Core web operations module (pure functions)
- `navigate_to_discrete_material_page(main_frame, page)` - Navigate to discrete material plan maintenance page. Returns `(work_frame, page1)`.
- `setup_query_interface(work_frame)` - Initialize query interface by selecting order number query tab and setting page size.
- `fill_and_search_orders(work_frame, order_ids)` - Fill order IDs into search textbox and trigger search.
- `download_batch_data(work_frame, page, order_ids, batch_index, download_dir)` - Execute download workflow for a single batch. Returns downloaded file path.
- `execute_batch_download_workflow(work_frame, page, order_ids, batch_index, download_dir)` - Complete workflow: fill orders, search, and download. Returns downloaded file path.
- All functions are stateless and accept required parameters explicitly. No logging or progress reporting - callers handle that.
- **See**: `docs/discrete_material_plan_extractor_core.md` for complete API documentation and usage examples.
**`utils/discrete_material_plan/extractor.py`** - High-level extractor component
- `DiscreteMaterialPlanExtractor` class - Batch processing wrapper with progress reporting and error handling.
- `extract_from_file()` function - Convenience function to extract data from order IDs in a file.
- **New Methods**:
- `post_process_downloads()` - Convert and merge downloaded Excel files into structured format
- `extract_and_process()` - Complete workflow: extract data and post-process in one call
- Accepts ID list and file paths as parameters, handles session management, batch processing, Excel conversion, and file merging.
- **Documentation**:
- Basic Usage: `docs/discrete_material_plan_extractor.md` - Basic API and usage examples
- Component Guide: `docs/extractor_component_guide.md` - Complete guide with Mermaid diagrams
- Post-Processing: `docs/extractor_post_processing.md` - Excel conversion and merging features
**`utils/discrete_material_plan/excel_converter.py`** - Excel data conversion utility
- `ExcelConverter` class - Converts raw Excel reports to structured DataFrames.
- `convert(input_file, output_file)` - Converts Excel file and returns DataFrame.
- Handles nested order structures, flattens to table format, applies field name mapping.
- Used internally by extractor for post-processing, can also be used standalone.
- **See**: `docs/extractor_post_processing.md` for usage examples.
**`tests/`** - Test suite
- All test files must add `PROJECT_ROOT` to `sys.path` to import `utils` modules
- Tests follow pattern: load dotenv, set browser path environment variable, run Playwright operations
@@ -72,6 +100,13 @@ Callers must construct the complete login URL before passing to `login()`:
url = f"{os.getenv('ERP_URL').rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
```
### Demo Data
**`id-demo.txt`** - Demo production IDs for testing
- Contains 15 sample production IDs (SC70202603240xxx format)
- Used by test scripts and demos for web operations testing
- Format: One ID per line, plain text file
## Common Commands
```bash
@@ -81,10 +116,40 @@ python tests/test_auth_config.py
# Run login/logout test
python tests/test_login.py
# Run web operations tests
python tests/test_web_operations.py
# Run extractor component tests
python tests/test_extractor_component.py
# Run real data extraction test (uses tests/id-demo.txt)
python tests/test_extractor_real.py
# Or use quick run scripts
./run_extractor_test.bat # Windows
./run_extractor_test.sh # Linux/Mac
# Run web operations demo
python tests/demo_web_operations.py
# Run extractor component demo
python demo_extractor_component.py
# Run any test with virtual environment
source .venv/Scripts/activate && python tests/<test_file>.py
```
## Documentation
**`docs/INDEX.md`** - Complete documentation index
- Quick start guide, API reference, examples, and troubleshooting
- Central hub for all documentation
**`docs/extractor_component_guide.md`** - Complete component guide with Mermaid diagrams
- Architecture overview, class structure, workflow diagrams
- State management, error handling, integration examples
- Visual documentation using Mermaid graphs
## Code Conventions
1. **English Only**: All user-facing output, docstrings, comments, and log messages must be in English. Chinese text is only used for Playwright element selectors matching the actual UI.