feat: add automatic database import after ERP data extraction
- Add DataImportService for reading Excel and importing to database - Extend DiscreteMaterialPlanDAO with deleteBySourceNumbers and batchInsert - Auto-trigger database write after successful Excel merge - Support batch delete by SourceNumber and batch insert (1000/batch) - Update ExtractorPage UI to show import results - Fix SQL Server query to handle undefined recordset for DELETE/INSERT Co-Authored-By: Claude (glm-5) <noreply@anthropic.com>
This commit is contained in:
119
IMPLEMENTATION_PLAN.md
Normal file
119
IMPLEMENTATION_PLAN.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Implementation Plan: Auto-import Extracted Data to Database
|
||||
|
||||
## Overview
|
||||
Implement automatic database import after ERP data extraction completes. The merged Excel file will be read and written to the `dbo_DiscreteMaterialPlanData` table.
|
||||
|
||||
## Requirements
|
||||
- **Trigger**: Automatic after extraction completes
|
||||
- **Delete Strategy**: Batch delete by `SourceNumber` before insert
|
||||
- **Batch Insert**: 1000 records per batch
|
||||
- **Field Mapping**: 28 Excel fields → database columns (skip 打印人, 打印日期, BOMVersion)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
ExtractorService
|
||||
│
|
||||
├── extract() → download + merge Excel
|
||||
│
|
||||
└── NEW: importToDatabase(mergedFile)
|
||||
│
|
||||
▼
|
||||
DataImportService
|
||||
│
|
||||
├── readExcelFile() → records + sourceNumbers
|
||||
├── deleteExistingRecords(sourceNumbers)
|
||||
└── batchInsert(records, batchSize=1000)
|
||||
│
|
||||
▼
|
||||
DiscreteMaterialPlanDAO
|
||||
├── deleteBySourceNumbers()
|
||||
└── batchInsert()
|
||||
```
|
||||
|
||||
## Field Mapping
|
||||
|
||||
| Excel Header | Database Column | Notes |
|
||||
|-------------|-----------------|-------|
|
||||
| 工厂 | Factory | |
|
||||
| 备料状态 | MaterialStatus | |
|
||||
| 备料计划单号 | PlanNumber | |
|
||||
| 来源单号 | SourceNumber | **Deletion key** |
|
||||
| 备料类型 | MaterialType | |
|
||||
| 产品编码 | ProductCode | |
|
||||
| 产品名称 | ProductName | |
|
||||
| 产品计划数量 | ProductPlanQuantity | decimal |
|
||||
| 产品单位 | ProductUnit | |
|
||||
| 用料部门 | UseDepartment | |
|
||||
| 备注 | Remark | |
|
||||
| 制单人 | Creator | |
|
||||
| 制单日期 | CreateDate | date |
|
||||
| 审批人 | Approver | |
|
||||
| 审批日期 | ApproveDate | date |
|
||||
| 序号 | SequenceNumber | int |
|
||||
| 材料编码 | MaterialCode | |
|
||||
| 材料名称 | MaterialName | |
|
||||
| 规格 | Specification | |
|
||||
| 型号 | Model | |
|
||||
| 图号 | DrawingNumber | |
|
||||
| 物料材质 | MaterialQuality | |
|
||||
| 计划数量 | PlanQuantity | decimal |
|
||||
| 单位 | Unit | |
|
||||
| 需用日期 | RequiredDate | date |
|
||||
| 发料仓库 | Warehouse | |
|
||||
| 单位用量 | UnitUsage | decimal |
|
||||
| 累计出库数量 | CumulativeOutputQuantity | decimal |
|
||||
| 打印人 | ❌ SKIP | Not in DB |
|
||||
| 打印日期 | ❌ SKIP | Not in DB |
|
||||
| - | BOMVersion | SKIP (no source) |
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
### 1. NEW: `src/main/services/database/data-importer.ts`
|
||||
Main import service with:
|
||||
- `importFromExcel(filePath)` - Main entry point
|
||||
- `readExcelFile(filePath)` - Parse Excel using ExcelJS
|
||||
- Map Excel columns to database fields
|
||||
- Return records and unique SourceNumbers
|
||||
|
||||
### 2. MODIFY: `src/main/services/database/discrete-material-plan-dao.ts`
|
||||
Add methods:
|
||||
- `deleteBySourceNumbers(sourceNumbers: string[])` - Batch delete
|
||||
- `batchInsert(records: MaterialPlanRecord[], batchSize: number)` - Batch insert
|
||||
|
||||
### 3. MODIFY: `src/main/services/erp/extractor.ts`
|
||||
- After successful merge, call `importToDatabase(mergedFile)`
|
||||
- Add import results to `ExtractorResult`
|
||||
|
||||
### 4. MODIFY: `src/main/types/extractor.types.ts`
|
||||
Add types:
|
||||
```typescript
|
||||
export interface ImportResult {
|
||||
success: boolean
|
||||
recordsImported: number
|
||||
recordsDeleted: number
|
||||
errors: string[]
|
||||
}
|
||||
|
||||
export interface ExtractorResult {
|
||||
// existing fields...
|
||||
importResult?: ImportResult
|
||||
}
|
||||
```
|
||||
|
||||
### 5. MODIFY: `src/renderer/src/pages/ExtractorPage.tsx`
|
||||
- Display import results
|
||||
- Show records deleted/imported counts
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Extend `DiscreteMaterialPlanDAO` with insert/delete methods
|
||||
2. Create `DataImportService`
|
||||
3. Integrate into `ExtractorService`
|
||||
4. Update types
|
||||
5. Update UI
|
||||
|
||||
## Testing Plan
|
||||
1. Unit test DAO methods
|
||||
2. Integration test with sample Excel file
|
||||
3. E2E test extraction → import flow
|
||||
304
src/main/services/database/data-importer.ts
Normal file
304
src/main/services/database/data-importer.ts
Normal file
@@ -0,0 +1,304 @@
|
||||
/**
|
||||
* Data Import Service
|
||||
*
|
||||
* Reads Excel files and imports data to the DiscreteMaterialPlanData table.
|
||||
* Workflow:
|
||||
* 1. Read Excel file
|
||||
* 2. Extract unique SourceNumbers
|
||||
* 3. Delete existing records by SourceNumber
|
||||
* 4. Batch insert new records
|
||||
*/
|
||||
|
||||
import path from 'path'
|
||||
import { createLogger } from '../logger'
|
||||
import { DiscreteMaterialPlanDAO, type MaterialPlanRecord } from './discrete-material-plan-dao'
|
||||
|
||||
const log = createLogger('DataImportService')
|
||||
|
||||
/**
|
||||
* Excel column header to database field mapping
|
||||
*/
|
||||
const EXCEL_TO_DB_MAPPING: Record<string, keyof MaterialPlanRecord> = {
|
||||
'工厂': 'factory',
|
||||
'备料状态': 'materialStatus',
|
||||
'备料计划单号': 'planNumber',
|
||||
'来源单号': 'sourceNumber',
|
||||
'备料类型': 'materialType',
|
||||
'产品编码': 'productCode',
|
||||
'产品名称': 'productName',
|
||||
'产品计划数量': 'productPlanQuantity',
|
||||
'产品单位': 'productUnit',
|
||||
'用料部门': 'useDepartment',
|
||||
'备注': 'remark',
|
||||
'制单人': 'creator',
|
||||
'制单日期': 'createDate',
|
||||
'审批人': 'approver',
|
||||
'审批日期': 'approveDate',
|
||||
'序号': 'sequenceNumber',
|
||||
'材料编码': 'materialCode',
|
||||
'材料名称': 'materialName',
|
||||
'规格': 'specification',
|
||||
'型号': 'model',
|
||||
'图号': 'drawingNumber',
|
||||
'物料材质': 'materialQuality',
|
||||
'计划数量': 'planQuantity',
|
||||
'单位': 'unit',
|
||||
'需用日期': 'requiredDate',
|
||||
'发料仓库': 'warehouse',
|
||||
'单位用量': 'unitUsage',
|
||||
'累计出库数量': 'cumulativeOutputQuantity'
|
||||
// Note: '打印人', '打印日期' are skipped (not in DB)
|
||||
// Note: 'BOMVersion' is skipped (not in Excel)
|
||||
}
|
||||
|
||||
/**
|
||||
* Import result
|
||||
*/
|
||||
export interface ImportResult {
|
||||
success: boolean
|
||||
recordsRead: number
|
||||
recordsDeleted: number
|
||||
recordsImported: number
|
||||
uniqueSourceNumbers: number
|
||||
errors: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* DataImportService class
|
||||
*/
|
||||
export class DataImportService {
|
||||
private dao: DiscreteMaterialPlanDAO
|
||||
|
||||
constructor() {
|
||||
this.dao = new DiscreteMaterialPlanDAO()
|
||||
}
|
||||
|
||||
/**
|
||||
* Import data from Excel file to database
|
||||
* @param filePath - Path to the Excel file
|
||||
* @param batchSize - Number of records per insert batch (default: 1000)
|
||||
* @returns Import result with statistics
|
||||
*/
|
||||
async importFromExcel(filePath: string, batchSize = 1000): Promise<ImportResult> {
|
||||
const result: ImportResult = {
|
||||
success: false,
|
||||
recordsRead: 0,
|
||||
recordsDeleted: 0,
|
||||
recordsImported: 0,
|
||||
uniqueSourceNumbers: 0,
|
||||
errors: []
|
||||
}
|
||||
|
||||
try {
|
||||
log.info('Starting import from Excel', { filePath, batchSize })
|
||||
|
||||
// Step 1: Read Excel file
|
||||
log.info('Reading Excel file...')
|
||||
const { records, sourceNumbers } = await this.readExcelFile(filePath)
|
||||
result.recordsRead = records.length
|
||||
result.uniqueSourceNumbers = sourceNumbers.size
|
||||
|
||||
log.info('Excel read completed', {
|
||||
recordsRead: result.recordsRead,
|
||||
uniqueSourceNumbers: result.uniqueSourceNumbers
|
||||
})
|
||||
|
||||
if (records.length === 0) {
|
||||
result.success = true
|
||||
result.errors.push('Excel file contains no data records')
|
||||
return result
|
||||
}
|
||||
|
||||
// Step 2: Delete existing records by SourceNumber
|
||||
log.info('Deleting existing records...', {
|
||||
sourceNumberCount: sourceNumbers.size
|
||||
})
|
||||
|
||||
const sourceNumberArray = Array.from(sourceNumbers)
|
||||
result.recordsDeleted = await this.dao.deleteBySourceNumbers(sourceNumberArray)
|
||||
|
||||
log.info('Existing records deleted', {
|
||||
recordsDeleted: result.recordsDeleted
|
||||
})
|
||||
|
||||
// Step 3: Batch insert new records
|
||||
log.info('Inserting new records...', {
|
||||
recordCount: records.length,
|
||||
batchSize
|
||||
})
|
||||
|
||||
result.recordsImported = await this.dao.batchInsert(records, batchSize)
|
||||
|
||||
log.info('Records imported successfully', {
|
||||
recordsImported: result.recordsImported
|
||||
})
|
||||
|
||||
result.success = true
|
||||
} catch (error) {
|
||||
const errorMsg = error instanceof Error ? error.message : String(error)
|
||||
result.errors.push(`Import failed: ${errorMsg}`)
|
||||
log.error('Import failed', { error: errorMsg })
|
||||
} finally {
|
||||
// Disconnect DAO
|
||||
try {
|
||||
await this.dao.disconnect()
|
||||
} catch (e) {
|
||||
log.warn('Error disconnecting DAO', {
|
||||
error: e instanceof Error ? e.message : String(e)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Read Excel file and extract records
|
||||
* @param filePath - Path to the Excel file
|
||||
* @returns Records and unique SourceNumbers
|
||||
*/
|
||||
private async readExcelFile(
|
||||
filePath: string
|
||||
): Promise<{ records: MaterialPlanRecord[]; sourceNumbers: Set<string> }> {
|
||||
const records: MaterialPlanRecord[] = []
|
||||
const sourceNumbers = new Set<string>()
|
||||
|
||||
// Dynamic import ExcelJS
|
||||
const ExcelJSModule = await import('exceljs')
|
||||
const ExcelJS = (ExcelJSModule as any).default || ExcelJSModule
|
||||
|
||||
const workbook = new ExcelJS.Workbook()
|
||||
await workbook.xlsx.readFile(filePath)
|
||||
|
||||
// Get first worksheet
|
||||
const worksheet = workbook.worksheets[0]
|
||||
if (!worksheet) {
|
||||
throw new Error('Excel file has no worksheets')
|
||||
}
|
||||
|
||||
// Get header row to map column indices
|
||||
const headerRow = worksheet.getRow(1)
|
||||
const columnMapping = this.buildColumnMapping(headerRow)
|
||||
|
||||
log.debug('Column mapping built', {
|
||||
columnCount: Object.keys(columnMapping).length
|
||||
})
|
||||
|
||||
// Iterate through data rows (starting from row 2)
|
||||
worksheet.eachRow((row: any, rowNumber: number) => {
|
||||
if (rowNumber === 1) return // Skip header row
|
||||
|
||||
try {
|
||||
const record = this.buildRecordFromRow(row, columnMapping)
|
||||
if (record) {
|
||||
records.push(record)
|
||||
if (record.sourceNumber) {
|
||||
sourceNumbers.add(record.sourceNumber)
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
log.warn('Failed to parse row', {
|
||||
rowNumber,
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
return { records, sourceNumbers }
|
||||
}
|
||||
|
||||
/**
|
||||
* Build column index to field name mapping from header row
|
||||
*/
|
||||
private buildColumnMapping(headerRow: any): Map<number, keyof MaterialPlanRecord> {
|
||||
const mapping = new Map<number, keyof MaterialPlanRecord>()
|
||||
|
||||
headerRow.eachCell((cell: any, colNumber: number) => {
|
||||
const headerText = cell.text?.toString().trim()
|
||||
if (headerText && EXCEL_TO_DB_MAPPING[headerText]) {
|
||||
mapping.set(colNumber, EXCEL_TO_DB_MAPPING[headerText])
|
||||
}
|
||||
})
|
||||
|
||||
return mapping
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a MaterialPlanRecord from an Excel row
|
||||
*/
|
||||
private buildRecordFromRow(
|
||||
row: any,
|
||||
columnMapping: Map<number, keyof MaterialPlanRecord>
|
||||
): MaterialPlanRecord | null {
|
||||
const record: Partial<MaterialPlanRecord> = {}
|
||||
|
||||
row.eachCell((cell: any, colNumber: number) => {
|
||||
const fieldName = columnMapping.get(colNumber)
|
||||
if (!fieldName) return
|
||||
|
||||
const value = this.parseCellValue(cell, fieldName)
|
||||
record[fieldName] = value as any
|
||||
})
|
||||
|
||||
// Validate required fields
|
||||
if (!record.planNumber) {
|
||||
return null // Skip records without PlanNumber
|
||||
}
|
||||
|
||||
return record as MaterialPlanRecord
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse cell value based on field type
|
||||
*/
|
||||
private parseCellValue(cell: any, fieldName: keyof MaterialPlanRecord): any {
|
||||
const text = cell.text?.toString().trim()
|
||||
const value = cell.value
|
||||
|
||||
// Return null for empty cells
|
||||
if (!text || text === '') {
|
||||
return null
|
||||
}
|
||||
|
||||
// Handle numeric fields
|
||||
const numericFields: (keyof MaterialPlanRecord)[] = [
|
||||
'productPlanQuantity',
|
||||
'sequenceNumber',
|
||||
'planQuantity',
|
||||
'unitUsage',
|
||||
'cumulativeOutputQuantity'
|
||||
]
|
||||
|
||||
if (numericFields.includes(fieldName)) {
|
||||
const num = parseFloat(text)
|
||||
return isNaN(num) ? null : num
|
||||
}
|
||||
|
||||
// Handle date fields
|
||||
const dateFields: (keyof MaterialPlanRecord)[] = [
|
||||
'createDate',
|
||||
'approveDate',
|
||||
'requiredDate'
|
||||
]
|
||||
|
||||
if (dateFields.includes(fieldName)) {
|
||||
// ExcelJS returns date as Date object if recognized
|
||||
if (value instanceof Date) {
|
||||
return value
|
||||
}
|
||||
// Try to parse date string
|
||||
const date = new Date(text)
|
||||
return isNaN(date.getTime()) ? null : date
|
||||
}
|
||||
|
||||
// Handle string fields
|
||||
return text
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a DataImportService instance
|
||||
*/
|
||||
export function createDataImportService(): DataImportService {
|
||||
return new DataImportService()
|
||||
}
|
||||
@@ -383,6 +383,224 @@ export class DiscreteMaterialPlanDAO {
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== DELETE OPERATIONS ====================
|
||||
|
||||
/**
|
||||
* Delete records by SourceNumber list
|
||||
* Uses batch processing for large lists
|
||||
* @param sourceNumbers - List of SourceNumber values to delete
|
||||
* @returns Number of records deleted
|
||||
*/
|
||||
async deleteBySourceNumbers(sourceNumbers: string[]): Promise<number> {
|
||||
if (!sourceNumbers || sourceNumbers.length === 0) {
|
||||
return 0
|
||||
}
|
||||
|
||||
try {
|
||||
const dbService = await this.getDatabaseService()
|
||||
const tableName = this.getTableName()
|
||||
const isSqlServer = dbService.type === 'sqlserver'
|
||||
const batchSize = 2000
|
||||
let totalDeleted = 0
|
||||
|
||||
// Get unique source numbers
|
||||
const uniqueSourceNumbers = [...new Set(sourceNumbers.filter(Boolean))]
|
||||
|
||||
for (let i = 0; i < uniqueSourceNumbers.length; i += batchSize) {
|
||||
const batch = uniqueSourceNumbers.slice(i, i + batchSize)
|
||||
const placeholders = this.buildPlaceholders(batch.length, isSqlServer)
|
||||
|
||||
const sqlString = `
|
||||
DELETE FROM ${tableName}
|
||||
WHERE SourceNumber IN (${placeholders})
|
||||
`
|
||||
|
||||
const result = await dbService.query(sqlString, batch)
|
||||
totalDeleted += result.rowCount || 0
|
||||
|
||||
log.debug('Deleted batch', {
|
||||
batch: i / batchSize + 1,
|
||||
count: result.rowCount
|
||||
})
|
||||
}
|
||||
|
||||
log.info('Deleted records by source numbers', {
|
||||
totalDeleted,
|
||||
sourceNumberCount: uniqueSourceNumbers.length
|
||||
})
|
||||
|
||||
return totalDeleted
|
||||
} catch (error) {
|
||||
log.error('Delete by source numbers error', {
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
})
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== INSERT OPERATIONS ====================
|
||||
|
||||
/**
|
||||
* Insert records in batches
|
||||
* @param records - List of MaterialPlanRecord to insert
|
||||
* @param batchSize - Number of records per batch (default: 1000)
|
||||
* @returns Number of records inserted
|
||||
*/
|
||||
async batchInsert(records: MaterialPlanRecord[], batchSize = 1000): Promise<number> {
|
||||
if (!records || records.length === 0) {
|
||||
return 0
|
||||
}
|
||||
|
||||
try {
|
||||
const dbService = await this.getDatabaseService()
|
||||
const tableName = this.getTableName()
|
||||
const isSqlServer = dbService.type === 'sqlserver'
|
||||
let totalInserted = 0
|
||||
|
||||
// Process in batches
|
||||
for (let i = 0; i < records.length; i += batchSize) {
|
||||
const batch = records.slice(i, i + batchSize)
|
||||
const inserted = await this.insertBatch(dbService, tableName, batch, isSqlServer)
|
||||
totalInserted += inserted
|
||||
|
||||
log.debug('Inserted batch', {
|
||||
batch: Math.floor(i / batchSize) + 1,
|
||||
count: inserted
|
||||
})
|
||||
}
|
||||
|
||||
log.info('Batch insert completed', {
|
||||
totalInserted,
|
||||
batchSize
|
||||
})
|
||||
|
||||
return totalInserted
|
||||
} catch (error) {
|
||||
log.error('Batch insert error', {
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
})
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert a single batch of records
|
||||
*/
|
||||
private async insertBatch(
|
||||
dbService: IDatabaseService,
|
||||
tableName: string,
|
||||
records: MaterialPlanRecord[],
|
||||
isSqlServer: boolean
|
||||
): Promise<number> {
|
||||
if (records.length === 0) {
|
||||
return 0
|
||||
}
|
||||
|
||||
// Build column list (excluding id)
|
||||
const columns = [
|
||||
'Factory', 'MaterialStatus', 'PlanNumber', 'SourceNumber', 'MaterialType',
|
||||
'ProductCode', 'ProductName', 'ProductUnit', 'ProductPlanQuantity',
|
||||
'UseDepartment', 'Remark', 'Creator', 'CreateDate', 'Approver', 'ApproveDate',
|
||||
'SequenceNumber', 'MaterialCode', 'MaterialName', 'Specification', 'Model',
|
||||
'DrawingNumber', 'MaterialQuality', 'PlanQuantity', 'Unit', 'RequiredDate',
|
||||
'Warehouse', 'UnitUsage', 'CumulativeOutputQuantity'
|
||||
]
|
||||
|
||||
// Build parameterized insert
|
||||
const values: any[] = []
|
||||
const rowPlaceholders: string[] = []
|
||||
|
||||
records.forEach((record, rowIndex) => {
|
||||
const rowValues = this.buildRowValues(record, columns, rowIndex, isSqlServer, values)
|
||||
rowPlaceholders.push(`(${rowValues.join(',')})`)
|
||||
})
|
||||
|
||||
const sqlString = `
|
||||
INSERT INTO ${tableName} (${columns.join(', ')})
|
||||
VALUES ${rowPlaceholders.join(', ')}
|
||||
`
|
||||
|
||||
const result = await dbService.query(sqlString, values)
|
||||
return result.rowCount || records.length
|
||||
}
|
||||
|
||||
/**
|
||||
* Build parameter values for a single row
|
||||
*/
|
||||
private buildRowValues(
|
||||
record: MaterialPlanRecord,
|
||||
columns: string[],
|
||||
rowIndex: number,
|
||||
isSqlServer: boolean,
|
||||
values: any[]
|
||||
): string[] {
|
||||
return columns.map((col) => {
|
||||
const value = this.getColumnValue(record, col)
|
||||
values.push(value)
|
||||
|
||||
if (isSqlServer) {
|
||||
return `@p${values.length - 1}`
|
||||
} else {
|
||||
return '?'
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the value for a specific column from the record
|
||||
*/
|
||||
private getColumnValue(record: MaterialPlanRecord, column: string): any {
|
||||
const columnMapping: Record<string, keyof MaterialPlanRecord> = {
|
||||
Factory: 'factory',
|
||||
MaterialStatus: 'materialStatus',
|
||||
PlanNumber: 'planNumber',
|
||||
SourceNumber: 'sourceNumber',
|
||||
MaterialType: 'materialType',
|
||||
ProductCode: 'productCode',
|
||||
ProductName: 'productName',
|
||||
ProductUnit: 'productUnit',
|
||||
ProductPlanQuantity: 'productPlanQuantity',
|
||||
UseDepartment: 'useDepartment',
|
||||
Remark: 'remark',
|
||||
Creator: 'creator',
|
||||
CreateDate: 'createDate',
|
||||
Approver: 'approver',
|
||||
ApproveDate: 'approveDate',
|
||||
SequenceNumber: 'sequenceNumber',
|
||||
MaterialCode: 'materialCode',
|
||||
MaterialName: 'materialName',
|
||||
Specification: 'specification',
|
||||
Model: 'model',
|
||||
DrawingNumber: 'drawingNumber',
|
||||
MaterialQuality: 'materialQuality',
|
||||
PlanQuantity: 'planQuantity',
|
||||
Unit: 'unit',
|
||||
RequiredDate: 'requiredDate',
|
||||
Warehouse: 'warehouse',
|
||||
UnitUsage: 'unitUsage',
|
||||
CumulativeOutputQuantity: 'cumulativeOutputQuantity'
|
||||
}
|
||||
|
||||
const key = columnMapping[column]
|
||||
if (!key) {
|
||||
return null
|
||||
}
|
||||
|
||||
const value = record[key]
|
||||
|
||||
// Handle null/undefined
|
||||
if (value === null || value === undefined) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Handle empty strings for string fields
|
||||
if (typeof value === 'string' && value.trim() === '') {
|
||||
return null
|
||||
}
|
||||
|
||||
return value
|
||||
}
|
||||
|
||||
// ==================== UTILITY METHODS ====================
|
||||
|
||||
/**
|
||||
|
||||
@@ -92,8 +92,8 @@ export class SqlServerService implements IDatabaseService {
|
||||
|
||||
const result = await request.query(sqlString)
|
||||
|
||||
// Convert recordset to array of objects
|
||||
const rows = result.recordset as Record<string, unknown>[]
|
||||
// Convert recordset to array of objects (may be undefined for DELETE/INSERT/UPDATE)
|
||||
const rows = (result.recordset as Record<string, unknown>[]) || []
|
||||
// Extract column names from the first row if available
|
||||
const columns = rows.length > 0 ? Object.keys(rows[0]) : []
|
||||
|
||||
|
||||
@@ -3,7 +3,8 @@ import fs from 'fs/promises'
|
||||
import { ExtractorCore } from './extractor-core'
|
||||
import { ErpAuthService } from './erp-auth'
|
||||
import { ExcelParser } from '../excel/excel-parser'
|
||||
import type { ExtractorInput, ExtractorResult } from '../../types/extractor.types'
|
||||
import type { ExtractorInput, ExtractorResult, ImportResult } from '../../types/extractor.types'
|
||||
import { DataImportService } from '../database/data-importer'
|
||||
|
||||
/**
|
||||
* ERP Data Extractor Service
|
||||
@@ -70,6 +71,17 @@ export class ExtractorService {
|
||||
|
||||
// Always clean up temporary files regardless of merge success
|
||||
await this.cleanupTempFiles(result.downloadedFiles)
|
||||
|
||||
// Auto-import to database if merge was successful
|
||||
if (result.mergedFile) {
|
||||
input.onProgress?.('正在写入数据库...', 98)
|
||||
const importResult = await this.importToDatabase(result.mergedFile)
|
||||
result.importResult = importResult
|
||||
|
||||
if (!importResult.success && importResult.errors.length > 0) {
|
||||
result.errors.push(...importResult.errors)
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'Unknown error'
|
||||
@@ -270,3 +282,40 @@ export class ExtractorService {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Import merged Excel data to database
|
||||
* @param filePath - Path to the merged Excel file
|
||||
* @returns Import result with statistics
|
||||
*/
|
||||
private async importToDatabase(filePath: string): Promise<ImportResult> {
|
||||
console.log(`[Extractor] Starting database import from: ${filePath}`)
|
||||
|
||||
const importService = new DataImportService()
|
||||
|
||||
try {
|
||||
const result = await importService.importFromExcel(filePath, 1000)
|
||||
|
||||
console.log(`[Extractor] Import completed`, {
|
||||
success: result.success,
|
||||
recordsRead: result.recordsRead,
|
||||
recordsDeleted: result.recordsDeleted,
|
||||
recordsImported: result.recordsImported
|
||||
})
|
||||
|
||||
return result
|
||||
} catch (error) {
|
||||
const errorMsg = error instanceof Error ? error.message : String(error)
|
||||
console.error(`[Extractor] Import failed: ${errorMsg}`)
|
||||
|
||||
return {
|
||||
success: false,
|
||||
recordsRead: 0,
|
||||
recordsDeleted: 0,
|
||||
recordsImported: 0,
|
||||
uniqueSourceNumbers: 0,
|
||||
errors: [errorMsg]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,11 +6,25 @@ export interface ExtractorInput {
|
||||
onProgress?: (message: string, progress: number) => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of database import operation
|
||||
*/
|
||||
export interface ImportResult {
|
||||
success: boolean
|
||||
recordsRead: number
|
||||
recordsDeleted: number
|
||||
recordsImported: number
|
||||
uniqueSourceNumbers: number
|
||||
errors: string[]
|
||||
}
|
||||
|
||||
export interface ExtractorResult {
|
||||
downloadedFiles: string[]
|
||||
mergedFile: string | null
|
||||
recordCount: number
|
||||
errors: string[]
|
||||
/** Database import result (only populated if mergedFile was created) */
|
||||
importResult?: ImportResult
|
||||
}
|
||||
|
||||
export interface OrderInfo {
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
import React, { useState, useEffect } from 'react'
|
||||
import { Download, Play, Terminal } from 'lucide-react'
|
||||
import { Download, Play, Terminal, Database } from 'lucide-react'
|
||||
|
||||
// Import result type (matches the type from main process)
|
||||
interface ImportResult {
|
||||
success: boolean
|
||||
recordsRead: number
|
||||
recordsDeleted: number
|
||||
recordsImported: number
|
||||
uniqueSourceNumbers: number
|
||||
errors: string[]
|
||||
}
|
||||
|
||||
// Extractor result type (matches the type from main process)
|
||||
interface ExtractorResult {
|
||||
@@ -7,6 +17,7 @@ interface ExtractorResult {
|
||||
mergedFile: string | null
|
||||
recordCount: number
|
||||
errors: string[]
|
||||
importResult?: ImportResult
|
||||
}
|
||||
|
||||
interface ExtractorProgress {
|
||||
@@ -212,6 +223,54 @@ const ExtractorPage: React.FC = () => {
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Database import results */}
|
||||
{result?.importResult && (
|
||||
<div className="bg-white rounded-xl shadow-sm border border-slate-200 p-6 flex flex-col gap-4">
|
||||
<h3 className="font-semibold text-lg border-b pb-2 flex items-center gap-2">
|
||||
<Database size={20} className={result.importResult.success ? 'text-emerald-600' : 'text-red-500'} />
|
||||
<span className={result.importResult.success ? 'text-emerald-600' : 'text-red-500'}>
|
||||
数据库写入结果
|
||||
</span>
|
||||
</h3>
|
||||
<div className="grid grid-cols-4 gap-4">
|
||||
<div className="bg-slate-50 p-4 rounded-lg border border-slate-100 flex flex-col items-center justify-center">
|
||||
<span className="text-slate-500 text-sm">读取记录</span>
|
||||
<span className="text-2xl font-bold text-slate-800">
|
||||
{result.importResult.recordsRead}
|
||||
</span>
|
||||
</div>
|
||||
<div className="bg-slate-50 p-4 rounded-lg border border-slate-100 flex flex-col items-center justify-center">
|
||||
<span className="text-slate-500 text-sm">删除旧记录</span>
|
||||
<span className="text-2xl font-bold text-amber-600">
|
||||
{result.importResult.recordsDeleted}
|
||||
</span>
|
||||
</div>
|
||||
<div className="bg-slate-50 p-4 rounded-lg border border-slate-100 flex flex-col items-center justify-center">
|
||||
<span className="text-slate-500 text-sm">写入新记录</span>
|
||||
<span className="text-2xl font-bold text-emerald-600">
|
||||
{result.importResult.recordsImported}
|
||||
</span>
|
||||
</div>
|
||||
<div className="bg-slate-50 p-4 rounded-lg border border-slate-100 flex flex-col items-center justify-center">
|
||||
<span className="text-slate-500 text-sm">来源单号数</span>
|
||||
<span className="text-2xl font-bold text-blue-600">
|
||||
{result.importResult.uniqueSourceNumbers}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
{result.importResult.errors.length > 0 && (
|
||||
<div className="bg-red-50 p-4 rounded-lg border border-red-200">
|
||||
<span className="text-red-600 text-sm font-medium block mb-1">错误信息</span>
|
||||
<ul className="text-sm text-red-500 list-disc list-inside">
|
||||
{result.importResult.errors.map((err, idx) => (
|
||||
<li key={idx}>{err}</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="bg-slate-900 rounded-xl shadow-lg border border-slate-700 overflow-hidden flex flex-col h-[500px]">
|
||||
<div className="bg-slate-800 px-4 py-2 flex items-center justify-between border-b border-slate-700">
|
||||
<div className="flex items-center gap-2 text-slate-400 text-sm">
|
||||
|
||||
119
tests/unit/data-importer.test.ts
Normal file
119
tests/unit/data-importer.test.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* Unit tests for DataImportService
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
|
||||
import { DataImportService } from '../../../src/main/services/database/data-importer'
|
||||
|
||||
// Mock the logger
|
||||
vi.mock('../../../src/main/services/logger', () => ({
|
||||
createLogger: () => ({
|
||||
info: vi.fn(),
|
||||
debug: vi.fn(),
|
||||
warn: vi.fn(),
|
||||
error: vi.fn()
|
||||
})
|
||||
}))
|
||||
|
||||
// Mock the database factory
|
||||
vi.mock('../../../src/main/services/database', () => ({
|
||||
create: vi.fn().mockResolvedValue({
|
||||
type: 'mysql',
|
||||
isConnected: () => true,
|
||||
query: vi.fn().mockResolvedValue({ rows: [], rowCount: 0 }),
|
||||
disconnect: vi.fn().mockResolvedValue(undefined)
|
||||
})
|
||||
}))
|
||||
|
||||
// Create mock worksheet
|
||||
const mockWorksheet = {
|
||||
getRow: vi.fn().mockReturnValue({
|
||||
eachCell: vi.fn((callback) => {
|
||||
callback({ text: '工厂' }, 1)
|
||||
callback({ text: '来源单号' }, 2)
|
||||
callback({ text: '备料计划单号' }, 3)
|
||||
})
|
||||
}),
|
||||
eachRow: vi.fn((callback) => {
|
||||
// Skip header row (rowNumber 1)
|
||||
// Add data rows
|
||||
callback(
|
||||
{
|
||||
eachCell: vi.fn((cellCallback) => {
|
||||
cellCallback({ text: '工厂A', value: '工厂A' }, 1)
|
||||
cellCallback({ text: 'PO-001', value: 'PO-001' }, 2)
|
||||
cellCallback({ text: 'PLAN-001', value: 'PLAN-001' }, 3)
|
||||
}),
|
||||
text: '工厂A,PO-001,PLAN-001'
|
||||
},
|
||||
2
|
||||
)
|
||||
callback(
|
||||
{
|
||||
eachCell: vi.fn((cellCallback) => {
|
||||
cellCallback({ text: '工厂A', value: '工厂A' }, 1)
|
||||
cellCallback({ text: 'PO-002', value: 'PO-002' }, 2)
|
||||
cellCallback({ text: 'PLAN-002', value: 'PLAN-002' }, 3)
|
||||
}),
|
||||
text: '工厂A,PO-002,PLAN-002'
|
||||
},
|
||||
3
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
// Mock ExcelJS with a proper class constructor
|
||||
class MockWorkbook {
|
||||
worksheets = [mockWorksheet]
|
||||
xlsx = {
|
||||
readFile: vi.fn().mockResolvedValue(undefined)
|
||||
}
|
||||
}
|
||||
|
||||
vi.mock('exceljs', () => {
|
||||
return {
|
||||
default: {
|
||||
Workbook: MockWorkbook
|
||||
},
|
||||
Workbook: MockWorkbook
|
||||
}
|
||||
})
|
||||
|
||||
describe('DataImportService', () => {
|
||||
let service: DataImportService
|
||||
|
||||
beforeEach(() => {
|
||||
service = new DataImportService()
|
||||
vi.clearAllMocks()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
describe('importFromExcel', () => {
|
||||
it('should return result with expected structure', async () => {
|
||||
const result = await service.importFromExcel('/path/to/test.xlsx', 1000)
|
||||
|
||||
// Check that we got a result with expected structure
|
||||
expect(result).toHaveProperty('success')
|
||||
expect(result).toHaveProperty('recordsRead')
|
||||
expect(result).toHaveProperty('recordsDeleted')
|
||||
expect(result).toHaveProperty('recordsImported')
|
||||
expect(result).toHaveProperty('uniqueSourceNumbers')
|
||||
expect(result).toHaveProperty('errors')
|
||||
})
|
||||
|
||||
it('should return records read count', async () => {
|
||||
const result = await service.importFromExcel('/path/to/test.xlsx', 1000)
|
||||
|
||||
expect(result.recordsRead).toBeGreaterThanOrEqual(0)
|
||||
})
|
||||
|
||||
it('should return unique source numbers count', async () => {
|
||||
const result = await service.importFromExcel('/path/to/test.xlsx', 1000)
|
||||
|
||||
expect(result.uniqueSourceNumbers).toBeGreaterThanOrEqual(0)
|
||||
})
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user