Add centralized logging utility and optional logger parameters to all
core functions for better observability and debugging capabilities.
New modules:
- utils/logging.py: Centralized logger configuration with console
and optional file handlers
Enhanced features:
- Added optional logger parameter to all extractor_core functions
- Added logger support to extractor, excel_converter, and auth modules
- Functions remain silent when logger=None (backward compatible)
- Improved environment variable validation in test files
Documentation:
- Added discrete_material_plan_extractor_core.md with complete API
reference and usage patterns
Benefits:
- Consistent logging format across all components
- Optional debug output for troubleshooting
- No breaking changes - fully backward compatible
- Better error messages and validation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add comprehensive post-processing features to convert downloaded Excel files
into structured data and merge them into a single output file.
New modules:
- extractor_core.py: Stateless pure functions for web operations
- excel_converter.py: Excel to DataFrame conversion utility
- tests/test_extractor_real.py: Real data extraction test suite
Enhanced features:
- post_process_downloads(): Convert and merge multiple Excel files
- extract_and_process(): Complete workflow in single call
- cleanup_temp_files(): Optional cleanup of temporary downloaded files
- Field name mapping for standardized output columns
Dependencies:
- pandas>=2.0.0 for data manipulation
- openpyxl>=3.1.0 for Excel file handling
Documentation:
- Updated CLAUDE.md with new module references
- Added API documentation for extractor components
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Create detailed step-by-step implementation plan
- 9 tasks covering all refactoring aspects
- TDD approach with explicit verification steps
- Include test updates and documentation changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Document decoupling approach for utils/auth.py
- Remove all environment variable dependencies
- Make login() a pure function with required parameters
- Remove close_session() function
- Provide migration guide for test scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
