14 Commits
v1.13.0 ... dev

Author SHA1 Message Date
Misaka
cf9976f605 fix: add AT and ZONE to PostgreSQL SQL keywords for prepareSql
The prepareSql function quotes any word not in SQL_KEYWORDS as an
identifier. Since AT and ZONE were missing from the set, the expression
(NOW() AT TIME ZONE 'UTC') was mangled into (NOW() "AT" TIME "ZONE"
'UTC'), causing INSERT failures on PostgreSQL.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 22:00:06 +08:00
Misaka
a6c2e2ccc2 fix: use explicit UTC timestamp in PostgreSQL dialect
PostgreSQL's CURRENT_TIMESTAMP returns session-local time, unlike
SYSUTCDATETIME() (SQL Server) and UTC_TIMESTAMP() (MySQL) which
explicitly return UTC. Switch to (NOW() AT TIME ZONE 'UTC') to
keep operation history timestamps consistent across all databases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 21:48:28 +08:00
Misaka_Company
21089e8b40 perf(import): bypass intermediate Excel file in extraction pipeline
Replace the Extract → Write Excel → Read Excel → Import DB flow with
direct record-to-database persistence. The extractor now builds
MaterialPlanRecord[] from parsed orders and imports them without the
round-trip through a merged Excel file.

Key changes:
- Add importFromRecords() to DataImportService for record-based import
- Add SQL Server OPENJSON batch insert and atomic replace operations
  in DiscreteMaterialPlanDAO for efficient bulk writes
- Extract common import logic into private importRecords() method
- Configure explicit request/connection timeouts for SQL Server
- Add unit tests for direct record import path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 17:11:30 +08:00
Misaka_Company
f36c88aa89 fix(extractor): display operation time in local timezone
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 13:53:54 +08:00
Misaka_Company
dbb8e4904e perf: optimize order resolution history writes 2026-04-28 13:39:08 +08:00
Misaka_Company
1ffa0650a6 1.14.1 2026-04-28 12:17:15 +08:00
Misaka_Company
72dba32a52 docs: add release notes for version 1.14.1
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 12:17:11 +08:00
Misaka_Company
98865f5d7e fix(cleaner): preserve production IDs in operation history
The 总排号 field was always empty because getCleanerData() resolved
production IDs to order numbers before passing them to the cleaner,
losing the original inputs. Now originalInputs are carried through
the full chain so the resolver can properly set productionId.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 11:57:47 +08:00
Misaka_Company
5ff99cdd0f refactor: use dot notation for table name config (schema.table instead of schema_table)
Replace underscore-based table name splitting with dot-based splitting
to match the standard schema.tablename format, removing MySQL compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-28 11:10:03 +08:00
Misaka
664f26d63f fix(cleaner): guard missing session refresh config 2026-04-24 19:11:21 +08:00
Misaka_Company
e39fc87869 1.14.0 2026-04-24 15:50:33 +08:00
Misaka_Company
4b071e4331 docs: add release notes for version 1.14.0 2026-04-24 15:50:29 +08:00
Misaka_Company
28a632d0a7 Merge branch 'dev-log-enhance' into dev 2026-04-24 15:41:23 +08:00
Misaka_Company
ac43790127 Add cleaner session refresh and ERP diagnostics 2026-04-24 15:40:47 +08:00
38 changed files with 1700 additions and 260 deletions

View File

@@ -65,6 +65,7 @@ orderResolution:
cleaner: cleaner:
queryBatchSize: 100 queryBatchSize: 100
processConcurrency: 1 processConcurrency: 1
sessionRefreshOrderThreshold: 160 # 会在 batch 边界检查;达到或超过阈值后,在当前 batch 完成后重建浏览器会话
logging: logging:
level: info level: info

View File

@@ -188,21 +188,26 @@ flowchart LR
**表名转换逻辑**: **表名转换逻辑**:
```typescript ```typescript
// MySQL: dbo_MaterialsToBeDeleted // 输入格式: dbo.MaterialsToBeDeleted
// SQL Server: [dbo].[MaterialsToBeDeleted] // SQL Server: [dbo].[MaterialsToBeDeleted]
function getTableName(mysqlTableName: string): string { // PostgreSQL: "dbo"."MaterialsToBeDeleted"
const dbType = process.env.DB_TYPE?.toLowerCase() function getValidationTableName(dottedTableName: string): string {
if (dbType === 'sqlserver' || dbType === 'mssql') { const configManager = ConfigManager.getInstance()
// 找到第一个下划线分割schema和表名 const dbType = configManager.getDatabaseType()
const firstUnderscoreIndex = mysqlTableName.indexOf('_')
if (firstUnderscoreIndex > 0) { const dotIndex = dottedTableName.indexOf('.')
const schema = mysqlTableName.substring(0, firstUnderscoreIndex) if (dotIndex > 0) {
const tableName = mysqlTableName.substring(firstUnderscoreIndex + 1) const schema = dottedTableName.substring(0, dotIndex)
const tableName = dottedTableName.substring(dotIndex + 1)
if (dbType === 'sqlserver') {
return `[${schema}].[${tableName}]` return `[${schema}].[${tableName}]`
} }
return `[dbo].[${mysqlTableName}]` return `"${schema}"."${tableName}"`
} }
return mysqlTableName if (dbType === 'sqlserver') {
return `[dbo].[${dottedTableName}]`
}
return `"public"."${dottedTableName}"`
} }
``` ```

View File

@@ -541,7 +541,7 @@ graph LR
### 10.2 默认配置示例 ### 10.2 默认配置示例
```env ```env
DB_TABLE_NAME=productionContractData_26 年压力表合同数据 DB_TABLE_NAME=ERPAuto.vw_productionContractData
DB_FIELD_PRODUCTION_ID=总排号 DB_FIELD_PRODUCTION_ID=总排号
DB_FIELD_ORDER_NUMBER=生产订单号 DB_FIELD_ORDER_NUMBER=生产订单号
``` ```

16
docs/releases/1.14.0.md Normal file
View File

@@ -0,0 +1,16 @@
# 1.14.0
## 核心功能
- 清理任务新增长批次自动重建会话机制,在大批量订单处理中可按批次边界自动重开浏览器并继续执行,降低长时间运行中断的概率。
- 会话重建阈值支持配置,可结合批次大小灵活调整重建时机,避免在批次处理中途打断流程。
## 问题修复
- 修复无头模式下大批量清理任务更容易在后半程停滞的问题,提升真实数据处理场景下的完成率。
- 优化清理流程在登录态异常场景下的识别能力,减少详情页长时间等待后才暴露问题的情况。
## 改进
- 新增 ERP 页面状态诊断日志,可区分登录页、首页、查询页、详情页等关键页面状态,方便快速定位异常发生位置。
- 补充详情页打开、页面跳转、弹窗创建和并发等待等关键日志,排查间歇性问题时可以更快还原现场。

10
docs/releases/1.14.1.md Normal file
View File

@@ -0,0 +1,10 @@
# 1.14.1
## 问题修复
- 修复清理任务操作历史中「总排号」始终为空的问题,原始生产编号现在能正确保留并显示在历史记录中。
- 修复会话重建配置缺失时清理任务可能异常中断的问题,提升配置不完整场景下的运行稳定性。
## 改进
- 表名配置统一使用 `schema.tablename` 标准点分写法,与数据库标准格式保持一致。

View File

@@ -180,7 +180,7 @@ validation:
# 订单号解析配置 # 订单号解析配置
orderResolution: orderResolution:
tableName: 'productionContractData_26 年压力表合同数据' tableName: 'ERPAuto.vw_productionContractData'
productionIdField: '总排号' productionIdField: '总排号'
orderNumberField: '生产订单号' orderNumberField: '生产订单号'
``` ```

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{ {
"name": "erpauto", "name": "erpauto",
"version": "1.13.0", "version": "1.14.1",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "erpauto", "name": "erpauto",
"version": "1.13.0", "version": "1.14.1",
"hasInstallScript": true, "hasInstallScript": true,
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "^3.929.0", "@aws-sdk/client-s3": "^3.929.0",

View File

@@ -1,6 +1,6 @@
{ {
"name": "erpauto", "name": "erpauto",
"version": "1.13.0", "version": "1.14.1",
"description": "An Electron application with React and TypeScript", "description": "An Electron application with React and TypeScript",
"main": "./out/main/index.js", "main": "./out/main/index.js",
"author": "example.com", "author": "example.com",

View File

@@ -126,7 +126,9 @@ export function registerExtractorHandlers(): void {
sendLog(sender, 'info', '正在解析订单号...') sendLog(sender, 'info', '正在解析订单号...')
const resolver = new OrderNumberResolver(dbService) const resolver = new OrderNumberResolver(dbService)
const resolutionStart = Date.now()
const mappings = await resolver.resolve(input.orderNumbers) const mappings = await resolver.resolve(input.orderNumbers)
const resolutionDurationMs = Date.now() - resolutionStart
// Get valid order numbers and warnings // Get valid order numbers and warnings
const validOrderNumbers = resolver.getValidOrderNumbers(mappings) const validOrderNumbers = resolver.getValidOrderNumbers(mappings)
@@ -146,7 +148,16 @@ export function registerExtractorHandlers(): void {
) )
} }
log.info('Resolved order numbers', { count: validOrderNumbers.length }) log.info('Resolved order numbers', {
inputCount: input.orderNumbers.length,
count: validOrderNumbers.length,
durationMs: resolutionDurationMs
})
sendLog(
sender,
'info',
`订单号解析完成:${validOrderNumbers.length}/${input.orderNumbers.length} 个有效,耗时 ${(resolutionDurationMs / 1000).toFixed(2)}`
)
// Initialize operation history recording // Initialize operation history recording
const currentUser = SessionManager.getInstance().getUserInfo() const currentUser = SessionManager.getInstance().getUserInfo()
@@ -155,6 +166,7 @@ export function registerExtractorHandlers(): void {
// Save order records to history (preserve productionId -> orderNumber mapping) // Save order records to history (preserve productionId -> orderNumber mapping)
if (currentUser) { if (currentUser) {
const historyInsertStart = Date.now()
const orderRecords = mappings.map((m) => ({ const orderRecords = mappings.map((m) => ({
productionId: m.productionId || null, productionId: m.productionId || null,
orderNumber: m.orderNumber || m.input orderNumber: m.orderNumber || m.input
@@ -165,9 +177,11 @@ export function registerExtractorHandlers(): void {
currentUser.username, currentUser.username,
orderRecords orderRecords
) )
const historyInsertDurationMs = Date.now() - historyInsertStart
log.info('Operation history batch created', { log.info('Operation history batch created', {
batchId, batchId,
recordCount: orderRecords.length recordCount: orderRecords.length,
durationMs: historyInsertDurationMs
}) })
} }

View File

@@ -14,7 +14,8 @@ export const CleanerInputSchema = z.object({
materialCodes: z.array(z.string().min(1, 'Material code cannot be empty')), materialCodes: z.array(z.string().min(1, 'Material code cannot be empty')),
dryRun: z.boolean(), dryRun: z.boolean(),
queryBatchSize: z.number().int().min(1).max(100).optional().default(100), queryBatchSize: z.number().int().min(1).max(100).optional().default(100),
processConcurrency: z.number().int().min(1).max(20).optional().default(1) processConcurrency: z.number().int().min(1).max(20).optional().default(1),
sessionRefreshOrderThreshold: z.number().int().positive().optional().default(160)
// Note: onProgress is a function, not validated via Zod // Note: onProgress is a function, not validated via Zod
}) })

View File

@@ -27,6 +27,7 @@ import type { InsertMaterialDetailInput, InsertOrderInput } from '../../types/cl
import type { OrderMapping } from '../../types/order-resolver.types' import type { OrderMapping } from '../../types/order-resolver.types'
const log = createLogger('CleanerApplicationService') const log = createLogger('CleanerApplicationService')
const DEFAULT_SESSION_REFRESH_ORDER_THRESHOLD = 160
export class CleanerApplicationService { export class CleanerApplicationService {
async runCleaner( async runCleaner(
@@ -65,7 +66,8 @@ export class CleanerApplicationService {
} }
const resolver = new OrderNumberResolver(dbService) const resolver = new OrderNumberResolver(dbService)
const mappings = await resolver.resolve(input.orderNumbers) const inputsToResolve = input.originalInputs?.length ? input.originalInputs : input.orderNumbers
const mappings = await resolver.resolve(inputsToResolve)
const validOrderNumbers = resolver.getValidOrderNumbers(mappings) const validOrderNumbers = resolver.getValidOrderNumbers(mappings)
const warnings = resolver.getWarnings(mappings) const warnings = resolver.getWarnings(mappings)
@@ -148,6 +150,8 @@ export class CleanerApplicationService {
log.info('Login successful') log.info('Login successful')
const totalOrders = validOrderNumbers.length const totalOrders = validOrderNumbers.length
const effectiveSessionRefreshOrderThreshold =
this.resolveSessionRefreshOrderThreshold(input, configManager)
this.sendProgress(eventSender, 'ERP 登录成功', (1 / (1 + totalOrders)) * 100, { this.sendProgress(eventSender, 'ERP 登录成功', (1 / (1 + totalOrders)) * 100, {
phase: 'login', phase: 'login',
currentOrderIndex: 0, currentOrderIndex: 0,
@@ -159,6 +163,7 @@ export class CleanerApplicationService {
const modifiedInput: CleanerInput = { const modifiedInput: CleanerInput = {
...input, ...input,
orderNumbers: validOrderNumbers, orderNumbers: validOrderNumbers,
sessionRefreshOrderThreshold: effectiveSessionRefreshOrderThreshold,
onProgress: (message, progress, extra) => { onProgress: (message, progress, extra) => {
this.sendProgress(eventSender, message, progress ?? 0, extra) this.sendProgress(eventSender, message, progress ?? 0, extra)
} }
@@ -168,7 +173,8 @@ export class CleanerApplicationService {
batchId, batchId,
orderCount: validOrderNumbers.length, orderCount: validOrderNumbers.length,
queryBatchSize: input.queryBatchSize ?? 100, queryBatchSize: input.queryBatchSize ?? 100,
processConcurrency: input.processConcurrency ?? 1 processConcurrency: input.processConcurrency ?? 1,
sessionRefreshOrderThreshold: effectiveSessionRefreshOrderThreshold
}) })
let cleaner = new CleanerService(authService) let cleaner = new CleanerService(authService)
@@ -450,18 +456,34 @@ export class CleanerApplicationService {
: result.errors.length > 0 : result.errors.length > 0
? AuditStatus.FAILURE ? AuditStatus.FAILURE
: AuditStatus.SUCCESS : AuditStatus.SUCCESS
const effectiveSessionRefreshOrderThreshold = this.resolveSessionRefreshOrderThreshold(
input,
ConfigManager.getInstance()
)
logAuditWithCurrentUser(AuditAction.CLEAN, 'MATERIAL_PLAN', status, { logAuditWithCurrentUser(AuditAction.CLEAN, 'MATERIAL_PLAN', status, {
orderCount, orderCount,
dryRun: input.dryRun ?? false, dryRun: input.dryRun ?? false,
queryBatchSize: input.queryBatchSize ?? 100, queryBatchSize: input.queryBatchSize ?? 100,
processConcurrency: input.processConcurrency ?? 1, processConcurrency: input.processConcurrency ?? 1,
sessionRefreshOrderThreshold: effectiveSessionRefreshOrderThreshold,
materialsDeleted: result.materialsDeleted, materialsDeleted: result.materialsDeleted,
materialsSkipped: result.materialsSkipped, materialsSkipped: result.materialsSkipped,
errorCount: result.errors.length errorCount: result.errors.length
}) })
} }
private resolveSessionRefreshOrderThreshold(
input: CleanerInput,
configManager: Pick<ConfigManager, 'getConfig'>
): number {
return (
input.sessionRefreshOrderThreshold ??
configManager.getConfig().cleaner?.sessionRefreshOrderThreshold ??
DEFAULT_SESSION_REFRESH_ORDER_THRESHOLD
)
}
/** /**
* Save attempt results to database: update order statuses, insert material details, * Save attempt results to database: update order statuses, insert material details,
* and update execution status. * and update execution status.

View File

@@ -99,7 +99,8 @@ const DEFAULT_CONFIG: FullConfig = {
}, },
cleaner: { cleaner: {
queryBatchSize: 100, queryBatchSize: 100,
processConcurrency: 1 processConcurrency: 1,
sessionRefreshOrderThreshold: 160
}, },
orderResolution: { orderResolution: {
tableName: '', tableName: '',

View File

@@ -96,45 +96,12 @@ export class DataImportService {
// Step 1: Read Excel file // Step 1: Read Excel file
log.info('Reading Excel file...') log.info('Reading Excel file...')
const { records, sourceNumbers } = await this.readExcelFile(filePath) const { records, sourceNumbers } = await this.readExcelFile(filePath)
result.recordsRead = records.length
result.uniqueSourceNumbers = sourceNumbers.size
log.info('Excel read completed', { log.info('Excel read completed', {
recordsRead: result.recordsRead, recordsRead: records.length,
uniqueSourceNumbers: result.uniqueSourceNumbers uniqueSourceNumbers: sourceNumbers.size
}) })
if (records.length === 0) { await this.importRecords(records, batchSize, result)
result.success = true
result.errors.push('Excel file contains no data records')
return result
}
// Step 2: Delete existing records by SourceNumber
log.info('Deleting existing records...', {
sourceNumberCount: sourceNumbers.size
})
const sourceNumberArray = Array.from(sourceNumbers)
result.recordsDeleted = await this.dao.deleteBySourceNumbers(sourceNumberArray)
log.info('Existing records deleted', {
recordsDeleted: result.recordsDeleted
})
// Step 3: Batch insert new records
log.info('Inserting new records...', {
recordCount: records.length,
batchSize
})
result.recordsImported = await this.dao.batchInsert(records, batchSize)
log.info('Records imported successfully', {
recordsImported: result.recordsImported
})
result.success = true
} catch (error) { } catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error) const errorMsg = error instanceof Error ? error.message : String(error)
result.errors.push(`Import failed: ${errorMsg}`) result.errors.push(`Import failed: ${errorMsg}`)
@@ -167,6 +134,92 @@ export class DataImportService {
return result return result
} }
/**
* Import already parsed records to database.
*
* This is the preferred path for extraction: the downloader/parser already has
* structured rows, so database persistence should not require writing and
* reading an intermediate Excel file.
*/
async importFromRecords(records: MaterialPlanRecord[], batchSize = 1000): Promise<ImportResult> {
const result: ImportResult = {
success: false,
recordsRead: 0,
recordsDeleted: 0,
recordsImported: 0,
uniqueSourceNumbers: 0,
errors: []
}
try {
log.info('Starting import from parsed records', {
recordCount: records.length,
batchSize
})
return await this.importRecords(records, batchSize, result)
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error)
result.errors.push(`Import failed: ${errorMsg}`)
log.error('Import from records failed', { error: errorMsg })
return result
} finally {
try {
await this.dao.disconnect()
} catch (e) {
log.warn('Error disconnecting DAO', {
error: e instanceof Error ? e.message : String(e)
})
}
logAuditWithCurrentUser(
AuditAction.DATA_IMPORT,
'MATERIAL_PLAN',
result.success ? AuditStatus.SUCCESS : AuditStatus.FAILURE,
{
recordsRead: result.recordsRead,
recordsDeleted: result.recordsDeleted,
recordsImported: result.recordsImported,
uniqueSourceNumbers: result.uniqueSourceNumbers,
errorCount: result.errors.length
}
)
}
}
private async importRecords(
records: MaterialPlanRecord[],
batchSize: number,
result: ImportResult
): Promise<ImportResult> {
const sourceNumbers = new Set(records.map((record) => record.sourceNumber).filter(Boolean))
result.recordsRead = records.length
result.uniqueSourceNumbers = sourceNumbers.size
if (records.length === 0) {
result.success = true
result.errors.push('No data records to import')
return result
}
// Step 1: Replace existing records by SourceNumber
log.info('Replacing existing records...', {
sourceNumberCount: sourceNumbers.size
})
const replaceResult = await this.dao.replaceBySourceNumbers(records, batchSize)
result.recordsDeleted = replaceResult.deleted
result.recordsImported = replaceResult.inserted
log.info('Records replaced successfully', {
recordsDeleted: result.recordsDeleted,
recordsImported: result.recordsImported
})
result.success = true
return result
}
/** /**
* Read Excel file and extract records * Read Excel file and extract records
* @param filePath - Path to the Excel file * @param filePath - Path to the Excel file

View File

@@ -26,7 +26,7 @@ export class PostgreSqlDialect implements SqlDialect {
} }
currentTimestamp(): string { currentTimestamp(): string {
return 'CURRENT_TIMESTAMP' return "(NOW() AT TIME ZONE 'UTC')"
} }
upsert(params: { upsert(params: {

View File

@@ -13,6 +13,7 @@ import { createDialect, type SqlDialect } from './dialects'
import { createLogger, getRequestId, trackDuration } from '../logger' import { createLogger, getRequestId, trackDuration } from '../logger'
const log = createLogger('DiscreteMaterialPlanDAO') const log = createLogger('DiscreteMaterialPlanDAO')
const SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE = 25
/** /**
* Material plan record interface * Material plan record interface
@@ -562,9 +563,16 @@ export class DiscreteMaterialPlanDAO {
const tableName = this.getTableName() const tableName = this.getTableName()
const dialect = this.getDialect() const dialect = this.getDialect()
// SQL Server has a limit of 2100 parameters per query if (dbService.type === 'sqlserver') {
// Each record has 28 columns, so max rows per batch = 2100 / 28 = 75 return await this.batchInsertSqlServerJson(
// Leave some margin for query overhead dbService,
tableName,
records,
batchSize,
batchId
)
}
const columnsPerRow = 28 const columnsPerRow = 28
const effectiveBatchSize = Math.min(batchSize, dialect.maxBatchRows(columnsPerRow)) const effectiveBatchSize = Math.min(batchSize, dialect.maxBatchRows(columnsPerRow))
const totalBatches = Math.ceil(records.length / effectiveBatchSize) const totalBatches = Math.ceil(records.length / effectiveBatchSize)
@@ -626,6 +634,237 @@ export class DiscreteMaterialPlanDAO {
} }
} }
async replaceBySourceNumbers(
records: MaterialPlanRecord[],
batchSize = 1000
): Promise<{ deleted: number; inserted: number }> {
if (!records || records.length === 0) {
return { deleted: 0, inserted: 0 }
}
const dbService = await this.getDatabaseService()
const sourceNumbers = [...new Set(records.map((record) => record.sourceNumber).filter(Boolean))]
if (dbService.type === 'sqlserver') {
return await this.replaceSqlServerJson(dbService, records, sourceNumbers)
}
const deleted = await this.deleteBySourceNumbers(sourceNumbers)
const inserted = await this.batchInsert(records, batchSize)
return { deleted, inserted }
}
private async replaceSqlServerJson(
dbService: IDatabaseService,
records: MaterialPlanRecord[],
sourceNumbers: string[]
): Promise<{ deleted: number; inserted: number }> {
const tableName = this.getTableName()
const columns = this.getInsertColumns()
const withColumns = this.getSqlServerJsonWithColumns(columns)
const quotedColumns = columns.map((column) => `[${column}]`).join(', ')
const recordsBySourceNumber = this.groupRecordsBySourceNumber(records)
const totalBatches = Math.ceil(
sourceNumbers.length / SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE
)
let totalDeleted = 0
let totalInserted = 0
log.info('SQL Server JSON replace started', {
tableName,
operationType: 'REPLACE',
totalSourceNumbers: sourceNumbers.length,
totalRecords: records.length,
sourceNumberBatchSize: SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE,
totalBatches
})
for (
let offset = 0;
offset < sourceNumbers.length;
offset += SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE
) {
const sourceNumberBatch = sourceNumbers.slice(
offset,
offset + SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE
)
const batchNumber = Math.floor(offset / SQLSERVER_REPLACE_SOURCE_NUMBER_BATCH_SIZE) + 1
const recordBatch = sourceNumberBatch.flatMap(
(sourceNumber) => recordsBySourceNumber.get(sourceNumber) || []
)
const jsonRows = recordBatch.map((record) => this.buildJsonRow(record, columns))
const sqlString = `
DECLARE @deleted int = 0;
DECLARE @inserted int = 0;
BEGIN TRY
BEGIN TRANSACTION;
DELETE target
FROM ${tableName} AS target
INNER JOIN OPENJSON(@p0)
WITH (SourceNumber nvarchar(100) '$') AS source
ON target.SourceNumber = source.SourceNumber;
SET @deleted = @@ROWCOUNT;
INSERT INTO ${tableName} (${quotedColumns})
SELECT ${quotedColumns}
FROM OPENJSON(@p1)
WITH (
${withColumns}
);
SET @inserted = @@ROWCOUNT;
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
IF @@TRANCOUNT > 0
ROLLBACK TRANSACTION;
THROW;
END CATCH;
SELECT @deleted AS deletedCount, @inserted AS insertedCount;
`
const result = await trackDuration(
async () =>
await dbService.query(sqlString, [
JSON.stringify(sourceNumberBatch),
JSON.stringify(jsonRows)
]),
{
operationName: 'DiscreteMaterialPlanDAO.replaceSqlServerJsonBatch',
context: {
tableName,
operationType: 'REPLACE',
batchNumber,
totalBatches,
sourceNumberCount: sourceNumberBatch.length,
recordCount: recordBatch.length
}
}
)
const stats = result.result.rows[0] || {}
totalDeleted += Number(stats.deletedCount || 0)
totalInserted += Number(stats.insertedCount || recordBatch.length)
log.debug('SQL Server JSON replace batch completed', {
tableName,
batchNumber,
totalBatches,
sourceNumberCount: sourceNumberBatch.length,
recordCount: recordBatch.length
})
}
log.info('SQL Server JSON replace completed', {
tableName,
operationType: 'REPLACE',
totalDeleted,
totalInserted,
totalBatches
})
return {
deleted: totalDeleted,
inserted: totalInserted
}
}
private groupRecordsBySourceNumber(
records: MaterialPlanRecord[]
): Map<string, MaterialPlanRecord[]> {
const groups = new Map<string, MaterialPlanRecord[]>()
for (const record of records) {
if (!record.sourceNumber) {
continue
}
const existing = groups.get(record.sourceNumber) || []
existing.push(record)
groups.set(record.sourceNumber, existing)
}
return groups
}
private async batchInsertSqlServerJson(
dbService: IDatabaseService,
tableName: string,
records: MaterialPlanRecord[],
batchSize: number,
batchId: string
): Promise<number> {
const columns = this.getInsertColumns()
const effectiveBatchSize = Math.max(1, batchSize)
const totalBatches = Math.ceil(records.length / effectiveBatchSize)
let totalInserted = 0
log.info('SQL Server JSON batch insert started', {
tableName,
operationType: 'INSERT',
requestId: batchId,
totalRecords: records.length,
effectiveBatchSize,
totalBatches
})
for (let i = 0; i < records.length; i += effectiveBatchSize) {
const batch = records.slice(i, i + effectiveBatchSize)
const batchNumber = Math.floor(i / effectiveBatchSize) + 1
const jsonRows = batch.map((record) => this.buildJsonRow(record, columns))
const withColumns = this.getSqlServerJsonWithColumns(columns)
const quotedColumns = columns.map((column) => `[${column}]`).join(', ')
const sqlString = `
INSERT INTO ${tableName} (${quotedColumns})
SELECT ${quotedColumns}
FROM OPENJSON(@p0)
WITH (
${withColumns}
)
`
const result = await trackDuration(
async () => await dbService.query(sqlString, [JSON.stringify(jsonRows)]),
{
operationName: 'DiscreteMaterialPlanDAO.insertBatchSqlServerJson',
context: {
tableName,
operationType: 'INSERT',
batchId,
batchNumber,
totalBatches,
recordCount: batch.length
}
}
)
totalInserted += result.result.rowCount || batch.length
log.debug('Inserted SQL Server JSON batch', {
batch: batchNumber,
totalBatches,
count: batch.length,
batchId
})
}
log.info('SQL Server JSON batch insert completed', {
tableName,
operationType: 'INSERT',
requestId: batchId,
totalInserted,
batchSize: effectiveBatchSize,
totalBatches
})
return totalInserted
}
/** /**
* Insert a single batch of records with tracking * Insert a single batch of records with tracking
*/ */
@@ -641,37 +880,7 @@ export class DiscreteMaterialPlanDAO {
return 0 return 0
} }
// Build column list (excluding id) const columns = this.getInsertColumns()
const columns = [
'Factory',
'MaterialStatus',
'PlanNumber',
'SourceNumber',
'MaterialType',
'ProductCode',
'ProductName',
'ProductUnit',
'ProductPlanQuantity',
'UseDepartment',
'Remark',
'Creator',
'CreateDate',
'Approver',
'ApproveDate',
'SequenceNumber',
'MaterialCode',
'MaterialName',
'Specification',
'Model',
'DrawingNumber',
'MaterialQuality',
'PlanQuantity',
'Unit',
'RequiredDate',
'Warehouse',
'UnitUsage',
'CumulativeOutputQuantity'
]
// Build parameterized insert // Build parameterized insert
const values: any[] = [] const values: any[] = []
@@ -730,6 +939,91 @@ export class DiscreteMaterialPlanDAO {
}) })
} }
private getInsertColumns(): string[] {
return [
'Factory',
'MaterialStatus',
'PlanNumber',
'SourceNumber',
'MaterialType',
'ProductCode',
'ProductName',
'ProductUnit',
'ProductPlanQuantity',
'UseDepartment',
'Remark',
'Creator',
'CreateDate',
'Approver',
'ApproveDate',
'SequenceNumber',
'MaterialCode',
'MaterialName',
'Specification',
'Model',
'DrawingNumber',
'MaterialQuality',
'PlanQuantity',
'Unit',
'RequiredDate',
'Warehouse',
'UnitUsage',
'CumulativeOutputQuantity'
]
}
private buildJsonRow(record: MaterialPlanRecord, columns: string[]): Record<string, unknown> {
const row: Record<string, unknown> = {}
for (const column of columns) {
const value = this.getColumnValue(record, column)
row[column] = value instanceof Date ? value.toISOString() : value
}
return row
}
private getSqlServerJsonWithColumns(columns: string[]): string {
return columns
.map((column) => `[${column}] ${this.getSqlServerJsonColumnType(column)} '$.${column}'`)
.join(',\n ')
}
private getSqlServerJsonColumnType(column: string): string {
const columnTypes: Record<string, string> = {
Factory: 'nvarchar(100)',
MaterialStatus: 'nvarchar(50)',
PlanNumber: 'nvarchar(100)',
SourceNumber: 'nvarchar(100)',
MaterialType: 'nvarchar(100)',
ProductCode: 'nvarchar(100)',
ProductName: 'nvarchar(255)',
ProductUnit: 'nvarchar(50)',
ProductPlanQuantity: 'decimal(18,4)',
UseDepartment: 'nvarchar(100)',
Remark: 'nvarchar(500)',
Creator: 'nvarchar(100)',
CreateDate: 'datetime2',
Approver: 'nvarchar(100)',
ApproveDate: 'datetime2',
SequenceNumber: 'int',
MaterialCode: 'nvarchar(100)',
MaterialName: 'nvarchar(255)',
Specification: 'nvarchar(255)',
Model: 'nvarchar(255)',
DrawingNumber: 'nvarchar(100)',
MaterialQuality: 'nvarchar(100)',
PlanQuantity: 'decimal(18,4)',
Unit: 'nvarchar(50)',
RequiredDate: 'datetime2',
Warehouse: 'nvarchar(100)',
UnitUsage: 'decimal(18,6)',
CumulativeOutputQuantity: 'decimal(18,4)'
}
return columnTypes[column] || 'nvarchar(max)'
}
/** /**
* Get the value for a specific column from the record * Get the value for a specific column from the record
*/ */
@@ -777,6 +1071,10 @@ export class DiscreteMaterialPlanDAO {
return null return null
} }
if (value instanceof Date && Number.isNaN(value.getTime())) {
return null
}
// Handle empty strings for string fields // Handle empty strings for string fields
if (typeof value === 'string' && value.trim() === '') { if (typeof value === 'string' && value.trim() === '') {
return null return null

View File

@@ -126,35 +126,47 @@ export class ExtractorOperationHistoryDAO {
recordCount: records.length recordCount: records.length
}) })
for (const record of records) { const columnsPerRecord = 5
const batchSize = Math.max(1, dialect.maxBatchRows(columnsPerRecord))
for (let offset = 0; offset < records.length; offset += batchSize) {
const batch = records.slice(offset, offset + batchSize)
try { try {
const valuesSql: string[] = []
const params: (string | number | null)[] = []
batch.forEach((record, index) => {
const paramOffset = index * columnsPerRecord
valuesSql.push(
`(${dialect.param(paramOffset)}, ${dialect.param(paramOffset + 1)}, ${dialect.param(paramOffset + 2)}, ${dialect.param(paramOffset + 3)}, ${dialect.param(paramOffset + 4)}, ${dialect.currentTimestamp()}, 'pending')`
)
params.push(batchId, userId, username, record.productionId || null, record.orderNumber)
})
const sqlString = ` const sqlString = `
INSERT INTO ${tableName} INSERT INTO ${tableName}
(BatchId, UserId, Username, ProductionId, OrderNumber, OperationTime, Status) (BatchId, UserId, Username, ProductionId, OrderNumber, OperationTime, Status)
VALUES VALUES
(${dialect.param(0)}, ${dialect.param(1)}, ${dialect.param(2)}, ${dialect.param(3)}, ${dialect.param(4)}, ${dialect.currentTimestamp()}, 'pending') ${valuesSql.join(',\n ')}
` `
await trackDuration( await trackDuration(async () => await dbService.query(sqlString, params), {
async () => operationName: 'ExtractorOperationHistoryDAO.insertBatchRecords',
await dbService.query(sqlString, [ context: {
batchId, tableName,
userId, operationType: 'INSERT',
username, batchId,
record.productionId || null, batchOffset: offset,
record.orderNumber batchCount: batch.length
]),
{
operationName: 'ExtractorOperationHistoryDAO.insertBatchRecords',
context: { tableName, operationType: 'INSERT', batchId }
} }
) })
} catch (error) { } catch (error) {
log.error('Error inserting individual record', { log.error('Error inserting record batch', {
tableName, tableName,
operationType: 'INSERT', operationType: 'INSERT',
requestId, requestId,
batchId, batchId,
orderNumber: record.orderNumber, batchOffset: offset,
batchCount: batch.length,
error: error instanceof Error ? error.message : String(error) error: error instanceof Error ? error.message : String(error)
}) })
} }

View File

@@ -339,7 +339,11 @@ const SQL_KEYWORDS = new Set([
'IF', 'IF',
'CURRENT_TIMESTAMP', 'CURRENT_TIMESTAMP',
'NOW', 'NOW',
'GETDATE' 'GETDATE',
// ==================== Timezone Expression ====================
'AT',
'ZONE'
]) ])
/** /**

View File

@@ -38,6 +38,8 @@ export class SqlServerService implements IDatabaseService {
user: this.config.user, user: this.config.user,
password: this.config.password, password: this.config.password,
database: this.config.database, database: this.config.database,
requestTimeout: 60000,
connectionTimeout: 15000,
options: { options: {
encrypt: this.config.options?.encrypt ?? false, encrypt: this.config.options?.encrypt ?? false,
trustServerCertificate: this.config.options?.trustServerCertificate ?? false trustServerCertificate: this.config.options?.trustServerCertificate ?? false

View File

@@ -8,9 +8,10 @@ import type {
OrderCleanDetail OrderCleanDetail
} from '../../types/cleaner.types' } from '../../types/cleaner.types'
import type { ErpSession } from '../../types/erp.types' import type { ErpSession } from '../../types/erp.types'
import type { FrameLocator, Locator, Page } from 'playwright' import type { BrowserContext, FrameLocator, Locator, Page } from 'playwright'
import { createLogger, run, trackDuration } from '../logger' import { createLogger, run, trackDuration } from '../logger'
import { capturePageContext } from './erp-error-context' import { capturePageContext } from './erp-error-context'
import { capturePageState } from './page-state'
const log = createLogger('CleanerService') const log = createLogger('CleanerService')
@@ -18,6 +19,7 @@ const DEFAULT_QUERY_BATCH_SIZE = 100
const MAX_QUERY_BATCH_SIZE = 100 const MAX_QUERY_BATCH_SIZE = 100
const DEFAULT_PROCESS_CONCURRENCY = 1 const DEFAULT_PROCESS_CONCURRENCY = 1
const MAX_PROCESS_CONCURRENCY = 20 const MAX_PROCESS_CONCURRENCY = 20
const DEFAULT_SESSION_REFRESH_ORDER_THRESHOLD = 160
interface RetryResult { interface RetryResult {
retriedOrders: number retriedOrders: number
@@ -67,37 +69,86 @@ class ConcurrencyTracker {
private activeWorkers = 0 private activeWorkers = 0
private waitQueue = 0 private waitQueue = 0
private mutexWaitCount = 0 private mutexWaitCount = 0
private currentOwnerWorkerId: number | null = null
private currentOwnerOrderNumber: string | null = null
private waitStartedAt = new Map<number, number>()
workerStarted() { workerStarted(workerId: number, orderNumber: string) {
this.activeWorkers++ this.activeWorkers++
log.verbose('[CONCURRENCY] Worker started', { log.verbose('[CONCURRENCY] Worker started', {
workerId,
orderNumber,
activeWorkers: this.activeWorkers, activeWorkers: this.activeWorkers,
waitQueue: this.waitQueue, waitQueue: this.waitQueue,
waitingForPopupMutex: this.mutexWaitCount > 0 waitingForPopupMutex: this.mutexWaitCount > 0
}) })
} }
workerCompleted() { workerCompleted(workerId: number, orderNumber: string) {
this.activeWorkers-- this.activeWorkers--
log.verbose('[CONCURRENCY] Worker completed', { log.verbose('[CONCURRENCY] Worker completed', {
workerId,
orderNumber,
activeWorkers: this.activeWorkers, activeWorkers: this.activeWorkers,
queueRemaining: this.waitQueue queueRemaining: this.waitQueue
}) })
} }
waitingForMutex() { waitingForMutex(workerId: number, orderNumber: string) {
this.mutexWaitCount++ this.mutexWaitCount++
this.waitQueue++
this.waitStartedAt.set(workerId, Date.now())
log.warn('[CONCURRENCY] Worker waiting for popup mutex', { log.warn('[CONCURRENCY] Worker waiting for popup mutex', {
workerId,
orderNumber,
mutexWaitCount: this.mutexWaitCount, mutexWaitCount: this.mutexWaitCount,
activeWorkers: this.activeWorkers activeWorkers: this.activeWorkers,
queueDepth: this.waitQueue,
currentOwnerWorkerId: this.currentOwnerWorkerId,
currentOwnerOrderNumber: this.currentOwnerOrderNumber
}) })
} }
acquiredMutex() { acquiredMutex(workerId: number, orderNumber: string) {
const waitStartedAt = this.waitStartedAt.get(workerId)
const waitDurationMs = waitStartedAt ? Date.now() - waitStartedAt : 0
this.waitStartedAt.delete(workerId)
this.mutexWaitCount-- this.mutexWaitCount--
this.waitQueue = Math.max(0, this.waitQueue - 1)
this.currentOwnerWorkerId = workerId
this.currentOwnerOrderNumber = orderNumber
log.verbose('[CONCURRENCY] Worker acquired popup mutex', { log.verbose('[CONCURRENCY] Worker acquired popup mutex', {
workerId,
orderNumber,
mutexWaitCount: this.mutexWaitCount, mutexWaitCount: this.mutexWaitCount,
activeWorkers: this.activeWorkers activeWorkers: this.activeWorkers,
waitDurationMs,
queueDepth: this.waitQueue
})
if (waitDurationMs > 5000) {
log.warn('[CONCURRENCY] Popup mutex slow wait', {
workerId,
orderNumber,
waitDurationMs,
activeWorkers: this.activeWorkers,
queueDepth: this.waitQueue,
currentOwnerWorkerId: this.currentOwnerWorkerId,
currentOwnerOrderNumber: this.currentOwnerOrderNumber
})
}
}
releasedMutex(workerId: number, orderNumber: string) {
if (this.currentOwnerWorkerId === workerId) {
this.currentOwnerWorkerId = null
this.currentOwnerOrderNumber = null
}
log.verbose('[CONCURRENCY] Worker released popup mutex', {
workerId,
orderNumber,
activeWorkers: this.activeWorkers,
queueDepth: this.waitQueue
}) })
} }
} }
@@ -148,20 +199,20 @@ export function getMissingOrders(inputOrders: string[], processedOrders: Set<str
export async function runWithConcurrency<T, R>( export async function runWithConcurrency<T, R>(
items: T[], items: T[],
concurrency: number, concurrency: number,
worker: (item: T, index: number) => Promise<R> worker: (item: T, index: number, workerId: number) => Promise<R>
): Promise<R[]> { ): Promise<R[]> {
const results = new Array<R>(items.length) const results = new Array<R>(items.length)
const limit = Math.max(1, Math.trunc(concurrency)) const limit = Math.max(1, Math.trunc(concurrency))
let cursor = 0 let cursor = 0
const runners = Array.from({ length: Math.min(limit, items.length) }, async () => { const runners = Array.from({ length: Math.min(limit, items.length) }, async (_, workerId) => {
while (true) { while (true) {
const current = cursor const current = cursor
cursor += 1 cursor += 1
if (current >= items.length) { if (current >= items.length) {
return return
} }
results[current] = await worker(items[current], current) results[current] = await worker(items[current], current, workerId)
} }
}) })
@@ -262,6 +313,10 @@ export class CleanerService {
1, 1,
MAX_PROCESS_CONCURRENCY MAX_PROCESS_CONCURRENCY
) )
const sessionRefreshOrderThreshold =
input.sessionRefreshOrderThreshold && input.sessionRefreshOrderThreshold > 0
? Math.trunc(input.sessionRefreshOrderThreshold)
: DEFAULT_SESSION_REFRESH_ORDER_THRESHOLD
log.info('Starting cleaner', { log.info('Starting cleaner', {
totalOrders, totalOrders,
@@ -269,6 +324,7 @@ export class CleanerService {
dryRun, dryRun,
queryBatchSize, queryBatchSize,
processConcurrency, processConcurrency,
sessionRefreshOrderThreshold,
orderNumbers: input.orderNumbers, orderNumbers: input.orderNumbers,
materialCodes: input.materialCodes materialCodes: input.materialCodes
}) })
@@ -281,12 +337,13 @@ export class CleanerService {
const session = this.authService.getSession() const session = this.authService.getSession()
const navigation = await this.navigateToCleanerPage(session) const navigation = await this.navigateToCleanerPage(session)
popupPage = navigation.popupPage popupPage = navigation.popupPage
const { workFrame } = navigation let { workFrame } = navigation
await this.setupQueryInterface(workFrame) await this.setupQueryInterface(workFrame, popupPage)
const orderBatches = createBatches(input.orderNumbers, queryBatchSize) const orderBatches = createBatches(input.orderNumbers, queryBatchSize)
const popupMutex = new AsyncMutex() const popupMutex = new AsyncMutex()
let ordersProcessedSinceLogin = 0
const progressState: ProgressState = { const progressState: ProgressState = {
ordersStarted: 0, ordersStarted: 0,
ordersCompleted: 0, ordersCompleted: 0,
@@ -329,7 +386,7 @@ export class CleanerService {
await trackDuration( await trackDuration(
async () => { async () => {
// Phase 1: Query orders // Phase 1: Query orders
await trackDuration(async () => await this.queryOrders(workFrame, batchOrders), { await trackDuration(async () => await this.queryOrders(workFrame, popupPage!, batchOrders), {
operationName: 'query', operationName: 'query',
message: '执行订单查询', message: '执行订单查询',
slowThresholdMs: 3000, slowThresholdMs: 3000,
@@ -345,7 +402,7 @@ export class CleanerService {
// Phase 3: Collect query results // Phase 3: Collect query results
const collectResult = await trackDuration( const collectResult = await trackDuration(
async () => await this.collectQueryResultRows(workFrame), async () => await this.collectQueryResultRows(workFrame, popupPage!),
{ {
operationName: 'collect_results', operationName: 'collect_results',
message: '收集查询结果', message: '收集查询结果',
@@ -358,20 +415,29 @@ export class CleanerService {
// Phase 4: Process all orders in batch // Phase 4: Process all orders in batch
await trackDuration( await trackDuration(
async () => { async () => {
await runWithConcurrency(queriedRows, processConcurrency, async (row) => { await runWithConcurrency(queriedRows, processConcurrency, async (row, _index, workerId) => {
const { rowIndex, orderNumber } = row const { rowIndex, orderNumber } = row
// [新增] Worker 开始追踪 // [新增] Worker 开始追踪
tracker.workerStarted() tracker.workerStarted(workerId, orderNumber)
try { try {
const openedDetailPage = await popupMutex.runExclusive(async () => { const openedDetailPage = await popupMutex.runExclusive(async () => {
// [新增] Mutex 等待追踪 // [新增] Mutex 等待追踪
tracker.waitingForMutex() tracker.waitingForMutex(workerId, orderNumber)
const page = await this.openDetailPageFromRow(workFrame, popupPage!, rowIndex) try {
// [新增] Mutex 获取追踪 const page = await this.openDetailPageFromRow(workFrame, popupPage!, rowIndex, {
tracker.acquiredMutex() orderNumber,
return page orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
workerId
})
// [新增] Mutex 获取追踪
tracker.acquiredMutex(workerId, orderNumber)
return page
} finally {
tracker.releasedMutex(workerId, orderNumber)
}
}) })
let detail: OrderCleanDetail let detail: OrderCleanDetail
@@ -408,7 +474,7 @@ export class CleanerService {
result.uncertainDeletions += detail.uncertainDeletions result.uncertainDeletions += detail.uncertainDeletions
} finally { } finally {
// [新增] Worker 完成追踪 // [新增] Worker 完成追踪
tracker.workerCompleted() tracker.workerCompleted(workerId, orderNumber)
} }
}) })
@@ -446,6 +512,51 @@ export class CleanerService {
// [新增] 清理健康检查定时器 // [新增] 清理健康检查定时器
clearInterval(healthCheckInterval) clearInterval(healthCheckInterval)
ordersProcessedSinceLogin += batchOrders.length
const remainingBatches = orderBatches.length - (batchIndex + 1)
log.info('[SESSION_REFRESH_CHECK] 批次完成,检查是否需要重建会话', {
batchIndex: batchIndex + 1,
totalBatches: orderBatches.length,
batchSize: batchOrders.length,
ordersProcessedSinceLogin,
threshold: sessionRefreshOrderThreshold,
remainingBatches
})
if (remainingBatches > 0 && ordersProcessedSinceLogin >= sessionRefreshOrderThreshold) {
log.info('[SESSION_REFRESH_TRIGGERED] 达到阈值,准备重建会话', {
batchIndex: batchIndex + 1,
totalBatches: orderBatches.length,
batchSize: batchOrders.length,
ordersProcessedSinceLogin,
threshold: sessionRefreshOrderThreshold,
remainingBatches
})
const refreshedNavigation = await this.refreshSessionAtBatchBoundary({
batchIndex: batchIndex + 1,
totalBatches: orderBatches.length,
batchSize: batchOrders.length,
threshold: sessionRefreshOrderThreshold,
ordersProcessedSinceLogin,
totalOrders,
completedOrders: progressState.ordersCompleted
})
popupPage = refreshedNavigation.popupPage
workFrame = refreshedNavigation.workFrame
ordersProcessedSinceLogin = 0
} else if (remainingBatches === 0 && ordersProcessedSinceLogin >= sessionRefreshOrderThreshold) {
log.info('[SESSION_REFRESH_SKIPPED] 已达到阈值但无剩余批次,跳过重建', {
batchIndex: batchIndex + 1,
totalBatches: orderBatches.length,
batchSize: batchOrders.length,
ordersProcessedSinceLogin,
threshold: sessionRefreshOrderThreshold,
remainingBatches
})
}
} }
const retryResult = await this.retryFailedOrders({ const retryResult = await this.retryFailedOrders({
@@ -556,6 +667,10 @@ export class CleanerService {
elapsedMs: Date.now() - navStartTime, elapsedMs: Date.now() - navStartTime,
popupOpened: !!popupPage popupOpened: !!popupPage
}) })
await this.logPageStateSnapshot(popupPage, 'nav.popup_opened', {
level: 'info',
elapsedMs: Date.now() - navStartTime
})
// Step 3: Get forward frame // Step 3: Get forward frame
log.debug('[导航 Step 3] 获取 forwardFrame 框架') log.debug('[导航 Step 3] 获取 forwardFrame 框架')
@@ -612,6 +727,10 @@ export class CleanerService {
log.debug('[导航 Step 5] 等待页面就绪标志', { selector: '#hot-key-head_list', timeout: 30000 }) log.debug('[导航 Step 5] 等待页面就绪标志', { selector: '#hot-key-head_list', timeout: 30000 })
await workFrame.locator('#hot-key-head_list').waitFor({ state: 'visible', timeout: 30000 }) await workFrame.locator('#hot-key-head_list').waitFor({ state: 'visible', timeout: 30000 })
const totalNavTime = Date.now() - navStartTime const totalNavTime = Date.now() - navStartTime
await this.logPageStateSnapshot(popupPage, 'nav.cleaner_page_ready', {
level: 'info',
elapsedMs: totalNavTime
})
log.info('[导航完成] 已导航到清理页面', { log.info('[导航完成] 已导航到清理页面', {
totalNavTimeMs: totalNavTime, totalNavTimeMs: totalNavTime,
isSlow: totalNavTime > 5000 isSlow: totalNavTime > 5000
@@ -620,9 +739,13 @@ export class CleanerService {
return { popupPage, workFrame } return { popupPage, workFrame }
} }
private async setupQueryInterface(innerFrame: FrameLocator): Promise<void> { private async setupQueryInterface(innerFrame: FrameLocator, popupPage: Page): Promise<void> {
const setupStartTime = Date.now() const setupStartTime = Date.now()
log.debug('[查询界面设置开始] 准备配置查询界面') log.debug('[查询界面设置开始] 准备配置查询界面')
await this.logPageStateSnapshot(popupPage, 'query.setup.start', {
level: 'debug',
elapsedMs: 0
})
// Step 1: Click search icon // Step 1: Click search icon
log.debug('[查询设置 Step 1] 点击搜索图标') log.debug('[查询设置 Step 1] 点击搜索图标')
@@ -647,6 +770,10 @@ export class CleanerService {
await inputEl.fill('5000') await inputEl.fill('5000')
await inputEl.press('Enter') await inputEl.press('Enter')
const totalSetupTime = Date.now() - setupStartTime const totalSetupTime = Date.now() - setupStartTime
await this.logPageStateSnapshot(popupPage, 'query.setup.ready', {
level: 'info',
elapsedMs: totalSetupTime
})
log.debug('[查询设置完成] 查询界面配置完毕', { log.debug('[查询设置完成] 查询界面配置完毕', {
totalSetupTimeMs: totalSetupTime, totalSetupTimeMs: totalSetupTime,
queryLimit: 5000, queryLimit: 5000,
@@ -654,7 +781,11 @@ export class CleanerService {
}) })
} }
private async queryOrders(workFrame: FrameLocator, orderNumbers: string[]): Promise<void> { private async queryOrders(
workFrame: FrameLocator,
popupPage: Page,
orderNumbers: string[]
): Promise<void> {
const queryStartTime = Date.now() const queryStartTime = Date.now()
log.debug('[订单查询开始]', { log.debug('[订单查询开始]', {
orderCount: orderNumbers.length, orderCount: orderNumbers.length,
@@ -663,6 +794,10 @@ export class CleanerService {
.concat(orderNumbers.length > 5 ? [`... (${orderNumbers.length - 5} more)`] : []), .concat(orderNumbers.length > 5 ? [`... (${orderNumbers.length - 5} more)`] : []),
isPreview: orderNumbers.length > 5 isPreview: orderNumbers.length > 5
}) })
await this.logPageStateSnapshot(popupPage, 'query.before_submit', {
level: 'debug',
elapsedMs: 0
})
const textbox = workFrame.getByRole('textbox', { name: '生产订单号' }) const textbox = workFrame.getByRole('textbox', { name: '生产订单号' })
log.debug('[订单查询] 准备填入订单号') log.debug('[订单查询] 准备填入订单号')
@@ -678,9 +813,16 @@ export class CleanerService {
elapsedMs: Date.now() - queryStartTime, elapsedMs: Date.now() - queryStartTime,
orderCount: orderNumbers.length orderCount: orderNumbers.length
}) })
await this.logPageStateSnapshot(popupPage, 'query.after_submit', {
level: 'info',
elapsedMs: Date.now() - queryStartTime
})
} }
private async collectQueryResultRows(workFrame: FrameLocator): Promise<QueryResultRow[]> { private async collectQueryResultRows(
workFrame: FrameLocator,
popupPage: Page
): Promise<QueryResultRow[]> {
const collectStartTime = Date.now() const collectStartTime = Date.now()
log.debug('[查询结果收集开始] 准备读取查询结果表格') log.debug('[查询结果收集开始] 准备读取查询结果表格')
@@ -707,6 +849,10 @@ export class CleanerService {
} }
const totalCollectTime = Date.now() - collectStartTime const totalCollectTime = Date.now() - collectStartTime
await this.logPageStateSnapshot(popupPage, 'query.results.collected', {
level: 'info',
elapsedMs: totalCollectTime
})
log.info('[查询结果收集完成]', { log.info('[查询结果收集完成]', {
totalRowsScanned: rowCount, totalRowsScanned: rowCount,
validOrderCount, validOrderCount,
@@ -754,37 +900,111 @@ export class CleanerService {
private async openDetailPageFromRow( private async openDetailPageFromRow(
workFrame: FrameLocator, workFrame: FrameLocator,
popupPage: Page, popupPage: Page,
rowIndex: number rowIndex: number,
options?: {
orderNumber?: string
orderIndex?: number
orderPosition?: string
workerId?: number
}
): Promise<Page> { ): Promise<Page> {
const openStartTime = Date.now()
const row = workFrame.locator('tbody tr').nth(rowIndex) const row = workFrame.locator('tbody tr').nth(rowIndex)
await row.waitFor({ state: 'visible', timeout: 15000 }) await row.waitFor({ state: 'visible', timeout: 15000 })
log.info('[NAV_EVENT] 准备从查询结果打开详情页', {
step: 'detail.open.from_query_row',
rowIndex,
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId
})
await this.logPageStateSnapshot(popupPage, 'detail.open.from_query_row.before_click', {
level: 'debug',
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId,
elapsedMs: Date.now() - openStartTime
})
const moreButton = row.locator('a.row-more').first() const moreButton = row.locator('a.row-more').first()
await moreButton.scrollIntoViewIfNeeded() await moreButton.scrollIntoViewIfNeeded()
const detailPagePromise = popupPage.waitForEvent('popup') const detailPagePromise = popupPage.waitForEvent('popup')
await moreButton.click() await moreButton.click()
await this.clickMaterialPlanMenu(workFrame) await this.clickMaterialPlanMenu(workFrame, popupPage, options)
return await detailPagePromise const detailPage = await detailPagePromise
log.info('[POPUP_EVENT] 详情页弹窗已创建', {
step: 'detail.popup.opened',
rowIndex,
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId,
popupUrl: detailPage.url(),
elapsedMs: Date.now() - openStartTime
})
await this.logPageStateSnapshot(detailPage, 'detail.popup.opened', {
level: 'info',
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId,
elapsedMs: Date.now() - openStartTime
})
return detailPage
} }
private async openDetailPageFromCurrentQuery( private async openDetailPageFromCurrentQuery(
workFrame: FrameLocator, workFrame: FrameLocator,
popupPage: Page popupPage: Page,
orderNumber?: string
): Promise<Page> { ): Promise<Page> {
const openStartTime = Date.now()
const firstRow = workFrame.locator('tbody tr').first() const firstRow = workFrame.locator('tbody tr').first()
await firstRow.waitFor({ state: 'visible', timeout: 10000 }) await firstRow.waitFor({ state: 'visible', timeout: 10000 })
log.info('[NAV_EVENT] 准备从当前查询结果打开详情页', {
step: 'detail.open.from_current_query',
orderNumber
})
await this.logPageStateSnapshot(popupPage, 'detail.open.from_current_query.before_click', {
level: 'debug',
orderNumber,
elapsedMs: Date.now() - openStartTime
})
const moreButton = firstRow.locator('a.row-more').first() const moreButton = firstRow.locator('a.row-more').first()
const detailPagePromise = popupPage.waitForEvent('popup') const detailPagePromise = popupPage.waitForEvent('popup')
await moreButton.click() await moreButton.click()
await this.clickMaterialPlanMenu(workFrame) await this.clickMaterialPlanMenu(workFrame, popupPage, { orderNumber })
return await detailPagePromise const detailPage = await detailPagePromise
log.info('[POPUP_EVENT] 详情页弹窗已创建', {
step: 'detail.popup.opened.retry',
orderNumber,
popupUrl: detailPage.url(),
elapsedMs: Date.now() - openStartTime
})
await this.logPageStateSnapshot(detailPage, 'detail.popup.opened.retry', {
level: 'info',
orderNumber,
elapsedMs: Date.now() - openStartTime
})
return detailPage
} }
private async clickMaterialPlanMenu(workFrame: FrameLocator): Promise<void> { private async clickMaterialPlanMenu(
workFrame: FrameLocator,
popupPage: Page,
options?: {
orderNumber?: string
orderIndex?: number
orderPosition?: string
workerId?: number
}
): Promise<void> {
const candidates = [ const candidates = [
workFrame.locator('li:visible, a:visible, span:visible, div:visible').filter({ workFrame.locator('li:visible, a:visible, span:visible, div:visible').filter({
hasText: /^备料计划$/ hasText: /^备料计划$/
@@ -798,7 +1018,21 @@ export class CleanerService {
const target = candidate.last() const target = candidate.last()
try { try {
await target.waitFor({ state: 'visible', timeout: 2000 }) await target.waitFor({ state: 'visible', timeout: 2000 })
log.info('[NAV_EVENT] 点击备料计划菜单', {
step: 'detail.menu.material_plan',
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId
})
await target.click() await target.click()
await this.logPageStateSnapshot(popupPage, 'detail.menu.material_plan.clicked', {
level: 'debug',
orderNumber: options?.orderNumber,
orderIndex: options?.orderIndex,
orderPosition: options?.orderPosition,
workerId: options?.workerId
})
return return
} catch { } catch {
// Try next locator candidate // Try next locator candidate
@@ -811,6 +1045,106 @@ export class CleanerService {
throw new Error('无法定位”备料计划”菜单项(可能菜单结构已变化)') throw new Error('无法定位”备料计划”菜单项(可能菜单结构已变化)')
} }
private getBrowserContext(): BrowserContext | undefined {
try {
return this.authService.getSession().context
} catch {
return undefined
}
}
private async logPageStateSnapshot(
page: Page,
step: string,
options: {
level?: 'info' | 'warn' | 'error' | 'debug'
orderNumber?: string
orderIndex?: number
orderPosition?: string
workerId?: number
elapsedMs?: number
includeFrameHierarchy?: boolean
includeBodyTextPreview?: boolean
} = {}
) {
const pageState = await capturePageState(page, this.getBrowserContext(), {
includeFrameHierarchy: options.includeFrameHierarchy,
includeBodyTextPreview: options.includeBodyTextPreview
})
const payload = {
step,
orderNumber: options.orderNumber,
orderIndex: options.orderIndex,
orderPosition: options.orderPosition,
workerId: options.workerId,
elapsedMs: options.elapsedMs,
...pageState
}
const level = options.level ?? 'info'
log[level]('[PAGE_STATE] 页面状态快照', payload)
return payload
}
private async ensureNotRedirectedToLoginPage(
page: Page,
step: string,
expectedOrderNumber: string | undefined,
progressState: ProgressState,
elapsedMs: number
): Promise<void> {
const pageState = await capturePageState(page, this.getBrowserContext(), {
includeFrameHierarchy: true,
includeBodyTextPreview: true
})
if (!pageState.isCasLoginRedirect && pageState.pageKind !== 'login') {
return
}
log.error('[SESSION_LOST] 会话跳转到 CAS 登录页', {
step,
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs,
detectedBy: pageState.isCasLoginRedirect ? 'url_match' : 'login_form_detected',
...pageState
})
throw new Error('ERP 会话已跳转到登录页')
}
private async refreshSessionAtBatchBoundary(params: {
batchIndex: number
totalBatches: number
batchSize: number
threshold: number
ordersProcessedSinceLogin: number
totalOrders: number
completedOrders: number
}): Promise<{ popupPage: Page; workFrame: FrameLocator }> {
const refreshStartTime = Date.now()
log.info('[SESSION_REFRESH_START] 开始关闭浏览器并重新登录', {
...params
})
await this.authService.close()
const session = await this.authService.login()
const navigation = await this.navigateToCleanerPage(session)
await this.setupQueryInterface(navigation.workFrame, navigation.popupPage)
log.info('[SESSION_REFRESH_SUCCESS] 会话重建成功', {
...params,
elapsedMs: Date.now() - refreshStartTime
})
return navigation
}
private async processDetailPage(params: { private async processDetailPage(params: {
detailPage: Page detailPage: Page
deleteSet: Set<string> deleteSet: Set<string>
@@ -861,16 +1195,40 @@ export class CleanerService {
} }
try { try {
await this.logPageStateSnapshot(detailPage, 'detail.page.opened', {
level: 'debug',
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs: Date.now() - processStartTime
})
// Step 1: Access forward frame // Step 1: Access forward frame
log.debug('[详情页面 Step 1] 准备访问 forwardFrame') log.debug('[详情页面 Step 1] 准备访问 forwardFrame')
await this.ensureNotRedirectedToLoginPage(
detailPage,
'detail.step1.forwardFrame',
expectedOrderNumber,
progressState,
Date.now() - processStartTime
)
const detailMainFrame = detailPage.locator('#forwardFrame') const detailMainFrame = detailPage.locator('#forwardFrame')
const dFrame = await detailMainFrame.contentFrame() const dFrame = await detailMainFrame.contentFrame()
if (!dFrame) { if (!dFrame) {
const errorMsg = '无法访问详情页面的 forwardFrame' const pageState = await this.logPageStateSnapshot(detailPage, 'detail.step1.forwardFrame', {
log.error('[详情页面失败] forwardFrame 访问失败', { level: 'error',
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs: Date.now() - processStartTime, elapsedMs: Date.now() - processStartTime,
pageUrl: detailPage.url(), includeFrameHierarchy: true,
includeBodyTextPreview: true
})
const errorMsg = '无法访问详情页面的 forwardFrame'
log.error('[DETAIL_PAGE_INVALID] 详情页未建立', {
failureKind: 'forward_frame_missing',
...pageState,
contextData: await capturePageContext( contextData: await capturePageContext(
detailPage, detailPage,
undefined, undefined,
@@ -888,15 +1246,49 @@ export class CleanerService {
// Step 2: Access inner frame // Step 2: Access inner frame
log.debug('[详情页面 Step 2] 等待并获取 mainiframe 内部框架', { timeout: 30000 }) log.debug('[详情页面 Step 2] 等待并获取 mainiframe 内部框架', { timeout: 30000 })
await this.ensureNotRedirectedToLoginPage(
detailPage,
'detail.step2.mainiframe',
expectedOrderNumber,
progressState,
Date.now() - processStartTime
)
const detailInnerLocator = dFrame.locator('#mainiframe') const detailInnerLocator = dFrame.locator('#mainiframe')
await detailInnerLocator.waitFor({ state: 'visible', timeout: 30000 }) try {
await detailInnerLocator.waitFor({ state: 'visible', timeout: 30000 })
} catch (error) {
const pageState = await this.logPageStateSnapshot(detailPage, 'detail.step2.mainiframe', {
level: 'error',
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs: Date.now() - processStartTime,
includeFrameHierarchy: true,
includeBodyTextPreview: true
})
log.error('[DETAIL_PAGE_TIMEOUT] 详情页等待超时', {
failureKind: pageState.isCasLoginRedirect ? 'redirected_to_cas' : 'mainiframe_missing',
...pageState,
error: error instanceof Error ? error.message : String(error)
})
throw error
}
const detailInnerFrame = await detailInnerLocator.contentFrame() const detailInnerFrame = await detailInnerLocator.contentFrame()
if (!detailInnerFrame) { if (!detailInnerFrame) {
const errorMsg = '无法访问详情页面的内部框架' const pageState = await this.logPageStateSnapshot(detailPage, 'detail.step2.detailInnerFrame', {
log.error('[详情页面失败] 内部框架访问失败', { level: 'error',
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs: Date.now() - processStartTime, elapsedMs: Date.now() - processStartTime,
pageUrl: detailPage.url(), includeFrameHierarchy: true,
includeBodyTextPreview: true
})
const errorMsg = '无法访问详情页面的内部框架'
log.error('[DETAIL_PAGE_INVALID] 详情页未建立', {
failureKind: 'mainiframe_missing',
...pageState,
contextData: await capturePageContext( contextData: await capturePageContext(
detailPage, detailPage,
undefined, undefined,
@@ -917,9 +1309,36 @@ export class CleanerService {
selector: '离散备料计划维护', selector: '离散备料计划维护',
timeout: 30000 timeout: 30000
}) })
await detailInnerFrame await this.ensureNotRedirectedToLoginPage(
.getByText(/^离散备料计划维护:/) detailPage,
.waitFor({ state: 'visible', timeout: 30000 }) 'detail.step3.header',
expectedOrderNumber,
progressState,
Date.now() - processStartTime
)
try {
await detailInnerFrame.getByText(/^离散备料计划维护:/).waitFor({
state: 'visible',
timeout: 30000
})
} catch (error) {
const pageState = await this.logPageStateSnapshot(detailPage, 'detail.step3.header', {
level: 'error',
orderNumber: expectedOrderNumber,
orderIndex: progressState.ordersStarted,
orderPosition: `${progressState.ordersStarted + 1}/${progressState.totalOrders}`,
elapsedMs: Date.now() - processStartTime,
includeFrameHierarchy: true,
includeBodyTextPreview: true
})
log.error('[DETAIL_PAGE_TIMEOUT] 详情页等待超时', {
failureKind: pageState.isCasLoginRedirect ? 'redirected_to_cas' : 'detail_header_missing',
expectedMarker: '离散备料计划维护:',
...pageState,
error: error instanceof Error ? error.message : String(error)
})
throw error
}
log.debug('[详情页面 Step 3 完成] 页面标题已显示', { log.debug('[详情页面 Step 3 完成] 页面标题已显示', {
elapsedMs: Date.now() - processStartTime elapsedMs: Date.now() - processStartTime
}) })
@@ -1743,7 +2162,7 @@ export class CleanerService {
try { try {
// Step 1: Re-query order // Step 1: Re-query order
log.debug('[重试查询] 重新查询订单', { orderNumber, attempt }) log.debug('[重试查询] 重新查询订单', { orderNumber, attempt })
await this.queryOrders(workFrame, [orderNumber]) await this.queryOrders(workFrame, popupPage, [orderNumber])
await this.waitForLoading(workFrame) await this.waitForLoading(workFrame)
log.debug('[重试查询完成] 查询加载完成', { log.debug('[重试查询完成] 查询加载完成', {
orderNumber, orderNumber,
@@ -1768,7 +2187,11 @@ export class CleanerService {
// Step 3: Open detail page // Step 3: Open detail page
log.debug('[重试详情] 打开订单详情页', { orderNumber, attempt }) log.debug('[重试详情] 打开订单详情页', { orderNumber, attempt })
const detailPage = await this.openDetailPageFromCurrentQuery(workFrame, popupPage) const detailPage = await this.openDetailPageFromCurrentQuery(
workFrame,
popupPage,
orderNumber
)
log.debug('[重试详情] 详情页已打开', { log.debug('[重试详情] 详情页已打开', {
orderNumber, orderNumber,
attempt, attempt,

View File

@@ -3,6 +3,7 @@ import type { ErpConfig, ErpSession } from '../../types/erp.types'
import { createLogger } from '../logger' import { createLogger } from '../logger'
import { capturePageContext } from './erp-error-context' import { capturePageContext } from './erp-error-context'
import { attachPageDiagnostics, attachContextDiagnostics } from './page-diagnostics' import { attachPageDiagnostics, attachContextDiagnostics } from './page-diagnostics'
import { capturePageState } from './page-state'
const log = createLogger('ErpAuthService') const log = createLogger('ErpAuthService')
@@ -64,6 +65,10 @@ export class ErpAuthService {
await page.goto(loginUrl) await page.goto(loginUrl)
log.debug('已导航到登录页面') log.debug('已导航到登录页面')
log.info('[PAGE_STATE] 页面状态快照', {
step: 'auth.login_page_loaded',
...(await capturePageState(page, context))
})
// Wait for page to load // Wait for page to load
await page.waitForLoadState('domcontentloaded', { timeout: PAGE_LOAD_TIMEOUT }) await page.waitForLoadState('domcontentloaded', { timeout: PAGE_LOAD_TIMEOUT })
@@ -156,6 +161,10 @@ export class ErpAuthService {
}) })
await this.waitForLoginResult(mainFrame as unknown as import('playwright').Frame) await this.waitForLoginResult(mainFrame as unknown as import('playwright').Frame)
log.info('[PAGE_STATE] 页面状态快照', {
step: 'auth.login_result_confirmed',
...(await capturePageState(page, context, { includeFrameHierarchy: true }))
})
// Create session with mainFrame (Python returns main_frame as part of login result) // Create session with mainFrame (Python returns main_frame as part of login result)
this.session = { this.session = {

View File

@@ -10,6 +10,7 @@ import type {
LogLevel LogLevel
} from '../../types/extractor.types' } from '../../types/extractor.types'
import { DataImportService } from '../database/data-importer' import { DataImportService } from '../database/data-importer'
import type { MaterialPlanRecord } from '../database/discrete-material-plan-dao'
import { createLogger, withRequestContext, getRequestId } from '../logger' import { createLogger, withRequestContext, getRequestId } from '../logger'
import { trackDuration } from '../logger/performance-monitor' import { trackDuration } from '../logger/performance-monitor'
@@ -112,16 +113,18 @@ export class ExtractorService {
// Always clean up temporary files regardless of merge success // Always clean up temporary files regardless of merge success
await this.cleanupTempFiles(result.downloadedFiles, input.orderNumbers) await this.cleanupTempFiles(result.downloadedFiles, input.orderNumbers)
// Auto-import to database if merge was successful // Auto-import parsed records directly. The merged Excel file is an archive artifact,
if (result.mergedFile) { // not the source for persistence.
if (mergeResult.records.length > 0) {
const importProgress = (1 + totalBatches + 1) * progressPerPoint const importProgress = (1 + totalBatches + 1) * progressPerPoint
input.onProgress?.('正在写入数据库...', importProgress, { input.onProgress?.('正在写入数据库...', importProgress, {
phase: 'importing', phase: 'importing',
totalBatches totalBatches
}) })
const importResult = await this.importToDatabaseWithLogging( const importResult = await this.importRecordsToDatabaseWithLogging(
result.mergedFile, mergeResult.records,
input.onLog input.onLog,
result.mergedFile
) )
result.importResult = importResult result.importResult = importResult
@@ -168,9 +171,10 @@ export class ExtractorService {
recordCount: number recordCount: number
error?: string error?: string
orderRecordCounts: Array<{ orderNumber: string; recordCount: number }> orderRecordCounts: Array<{ orderNumber: string; recordCount: number }>
records: MaterialPlanRecord[]
}> { }> {
if (filePaths.length === 0) { if (filePaths.length === 0) {
return { mergedFile: null, recordCount: 0, orderRecordCounts: [] } return { mergedFile: null, recordCount: 0, orderRecordCounts: [], records: [] }
} }
log.info('Starting merge', { fileCount: filePaths.length, orderCount: orderNumbers.length }) log.info('Starting merge', { fileCount: filePaths.length, orderCount: orderNumbers.length })
@@ -219,10 +223,11 @@ export class ExtractorService {
} }
log.info('Merge summary', { orderCount: allOrders.length, recordCount }) log.info('Merge summary', { orderCount: allOrders.length, recordCount })
const records = this.buildMaterialPlanRecords(allOrders)
if (recordCount === 0) { if (recordCount === 0) {
log.warn('No records found in any downloaded files', { orderNumbers }) log.warn('No records found in any downloaded files', { orderNumbers })
return { mergedFile: null, recordCount: 0, orderRecordCounts } return { mergedFile: null, recordCount: 0, orderRecordCounts, records }
} }
// Generate output filename with timestamp // Generate output filename with timestamp
@@ -238,7 +243,7 @@ export class ExtractorService {
log.info('Saving merged file', { outputPath }) log.info('Saving merged file', { outputPath })
await this.saveMergedOrders(allOrders, outputPath) await this.saveMergedOrders(allOrders, outputPath)
log.info('Merged file saved successfully', { recordCount }) log.info('Merged file saved successfully', { recordCount })
return { mergedFile: outputPath, recordCount, orderRecordCounts } return { mergedFile: outputPath, recordCount, orderRecordCounts, records }
} catch (error) { } catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error) const errorMsg = error instanceof Error ? error.message : String(error)
const errorStack = error instanceof Error ? error.stack : '' const errorStack = error instanceof Error ? error.stack : ''
@@ -253,6 +258,7 @@ export class ExtractorService {
mergedFile: null, mergedFile: null,
recordCount, recordCount,
orderRecordCounts, orderRecordCounts,
records,
error: `保存合并文件失败:${errorMsg}` error: `保存合并文件失败:${errorMsg}`
} }
} }
@@ -370,6 +376,78 @@ export class ExtractorService {
log.debug('File saved successfully', { outputPath }) log.debug('File saved successfully', { outputPath })
} }
private buildMaterialPlanRecords(
orders: Array<{ orderInfo: any; materials: any[] }>
): MaterialPlanRecord[] {
const records: MaterialPlanRecord[] = []
for (const order of orders) {
const { orderInfo, materials } = order
for (const material of materials) {
records.push({
factory: this.toText(orderInfo.factory),
materialStatus: this.toText(orderInfo.materialStatus),
planNumber: this.toText(orderInfo.planNumber),
sourceNumber: this.toText(orderInfo.productionOrder),
materialType: this.toText(orderInfo.materialType),
productCode: this.toText(orderInfo.productCode),
productName: this.toText(orderInfo.productName),
productPlanQuantity: this.toNumber(orderInfo.plannedQuantity),
productUnit: this.toText(orderInfo.unit),
useDepartment: this.toText(orderInfo.department),
remark: this.toText(orderInfo.remark),
creator: this.toText(orderInfo.creator),
createDate: this.toDate(orderInfo.createDate),
approver: this.toText(orderInfo.approver),
approveDate: this.toDate(orderInfo.approveDate),
sequenceNumber: this.toNumber(material.sequence),
materialCode: this.toText(material.materialCode),
materialName: this.toText(material.materialName),
specification: this.toText(material.specification),
model: this.toText(material.model),
drawingNumber: this.toText(material.drawingNumber),
materialQuality: this.toText(material.material),
planQuantity: this.toNumber(material.quantity),
unit: this.toText(material.unit),
requiredDate: this.toDate(material.requiredDate),
warehouse: this.toText(material.warehouse),
unitUsage: this.toNumber(material.unitUsage),
cumulativeOutputQuantity: this.toNumber(material.cumulativeOutboundQty),
bomVersion: ''
})
}
}
return records
}
private toText(value: unknown): string {
if (value === null || value === undefined) {
return ''
}
return String(value).trim()
}
private toNumber(value: unknown): number {
if (value === null || value === undefined || value === '') {
return 0
}
const parsed = Number(value)
return Number.isFinite(parsed) ? parsed : 0
}
private toDate(value: unknown): Date {
if (value instanceof Date) {
return value
}
if (value === null || value === undefined || value === '') {
return new Date(NaN)
}
const parsed = new Date(String(value))
return parsed
}
/** /**
* Clean up temporary batch files after merging * Clean up temporary batch files after merging
* @param filePaths - Array of temporary file paths to delete * @param filePaths - Array of temporary file paths to delete
@@ -458,4 +536,70 @@ export class ExtractorService {
return trackedResult.result return trackedResult.result
} }
private async importRecordsToDatabaseWithLogging(
records: MaterialPlanRecord[],
onLog?: (level: LogLevel, message: string) => void,
archiveFilePath?: string | null
): Promise<ImportResult> {
log.info('Starting database import from parsed records', {
recordCount: records.length,
archiveFilePath
})
onLog?.('info', `开始导入数据到数据库...`)
const trackedResult = await trackDuration(
async () => {
const importService = new DataImportService()
try {
const result = await importService.importFromRecords(records, 1000)
log.info('Import completed', {
success: result.success,
recordsRead: result.recordsRead,
recordsDeleted: result.recordsDeleted,
recordsImported: result.recordsImported
})
if (result.success) {
onLog?.(
'success',
`导入完成:读取 ${result.recordsRead} 条,删除 ${result.recordsDeleted} 条,导入 ${result.recordsImported}`
)
} else if (result.errors.length > 0) {
result.errors.forEach((err) => onLog?.('error', err))
}
return result
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error)
log.error('Import failed', {
error: errorMsg,
archiveFilePath,
downloadDir: this.downloadDir
})
onLog?.('error', `导入失败:${errorMsg}`)
return {
success: false,
recordsRead: 0,
recordsDeleted: 0,
recordsImported: 0,
uniqueSourceNumbers: 0,
errors: [errorMsg]
}
}
},
{
operationName: 'Database Import',
context: {
recordCount: records.length,
archiveFilePath
}
}
)
return trackedResult.result
}
} }

View File

@@ -32,6 +32,8 @@ const PRODUCTION_ID_PATTERN = /^\d{2}[A-Z]\d{1,6}$/i
*/ */
const ORDER_NUMBER_PATTERN = /^SC\d{14}$/i const ORDER_NUMBER_PATTERN = /^SC\d{14}$/i
const RESOLUTION_QUERY_BATCH_SIZE = 1000
/** /**
* Database table and field names * Database table and field names
* Loaded from config.yaml via ConfigManager * Loaded from config.yaml via ConfigManager
@@ -40,7 +42,7 @@ export function getDbConfig() {
const configManager = ConfigManager.getInstance() const configManager = ConfigManager.getInstance()
const config = configManager.getConfig() const config = configManager.getConfig()
return { return {
TABLE_NAME: config.orderResolution.tableName || 'productionContractData_26 年压力表合同数据', TABLE_NAME: config.orderResolution.tableName || 'ERPAuto.vw_productionContractData',
FIELD_PRODUCTION_ID: config.orderResolution.productionIdField || '总排号', FIELD_PRODUCTION_ID: config.orderResolution.productionIdField || '总排号',
FIELD_ORDER_NUMBER: config.orderResolution.orderNumberField || '生产订单号' FIELD_ORDER_NUMBER: config.orderResolution.orderNumberField || '生产订单号'
} }
@@ -56,37 +58,38 @@ export class OrderNumberResolver {
this.dbService = dbService this.dbService = dbService
} }
private chunk<T>(items: T[], size: number): T[][] {
const chunks: T[][] = []
for (let index = 0; index < items.length; index += size) {
chunks.push(items.slice(index, index + size))
}
return chunks
}
/** /**
* Get table name based on database type * Get table name based on database type
* Converts schema_tablename format to database-specific quoting: * Converts schema.tablename format to database-specific quoting:
* - SQL Server: [schema].[tablename] * - SQL Server: [schema].[tablename]
* - PostgreSQL: "schema"."tablename" * - PostgreSQL: "schema"."tablename"
* - MySQL: schema_tablename (as-is) * e.g., ERPAuto.vw_productionContractData ->
* e.g., productionContractData_26年压力表合同数据 -> * SQL Server: [ERPAuto].[vw_productionContractData]
* SQL Server: [productionContractData].[26年压力表合同数据] * PostgreSQL: "ERPAuto"."vw_productionContractData"
* PostgreSQL: "productionContractData"."26年压力表合同数据"
* MySQL: productionContractData_26年压力表合同数据
*/ */
private getTableName(tableName: string): string { private getTableName(tableName: string): string {
if (this.dbService.type === 'sqlserver' || this.dbService.type === 'postgresql') { const dotIndex = tableName.indexOf('.')
// Find the FIRST underscore to split schema and table name if (dotIndex > 0) {
// This handles patterns like: schema_tablename const schema = tableName.substring(0, dotIndex)
const firstUnderscoreIndex = tableName.indexOf('_') const actualTableName = tableName.substring(dotIndex + 1)
if (firstUnderscoreIndex > 0) {
const schema = tableName.substring(0, firstUnderscoreIndex)
const actualTableName = tableName.substring(firstUnderscoreIndex + 1)
if (this.dbService.type === 'sqlserver') {
return `[${schema}].[${actualTableName}]`
}
return `"${schema}"."${actualTableName}"`
}
// If no underscore found, default schema
if (this.dbService.type === 'sqlserver') { if (this.dbService.type === 'sqlserver') {
return `[dbo].[${tableName}]` return `[${schema}].[${actualTableName}]`
} }
return `"public"."${tableName}"` return `"${schema}"."${actualTableName}"`
} }
return tableName // No dot found — use default schema
if (this.dbService.type === 'sqlserver') {
return `[dbo].[${tableName}]`
}
return `"public"."${tableName}"`
} }
/** /**
@@ -163,36 +166,36 @@ export class OrderNumberResolver {
// P1: Deduplicate input productionIds to avoid redundant queries // P1: Deduplicate input productionIds to avoid redundant queries
const uniqueProductionIds = [...new Set(productionIds)] const uniqueProductionIds = [...new Set(productionIds)]
// Use parameterized query to prevent SQL injection
const placeholders = uniqueProductionIds.map((_, i) => `@p${i}`).join(', ')
const params = uniqueProductionIds
let sql: string
if (this.dbService.type === 'sqlserver') {
// P0: Use DISTINCT to prevent duplicates from one-to-many relationships
// 使用 COLLATE 指定不区分大小写的排序规则
sql = `SELECT DISTINCT [${dbConfig.FIELD_PRODUCTION_ID}], [${dbConfig.FIELD_ORDER_NUMBER}] FROM ${tableName} WHERE [${dbConfig.FIELD_PRODUCTION_ID}] COLLATE SQL_Latin1_General_CP1_CI_AS IN (${placeholders})`
} else if (this.dbService.type === 'postgresql') {
// PostgreSQL: 使用双引号保护中文标识符UPPER 实现不区分大小写
// 注意getTableName() 已返回带双引号的表名,不应再加引号
const pgPlaceholders = uniqueProductionIds.map((_, i) => `UPPER($${i + 1})`).join(', ')
sql = `SELECT DISTINCT "${dbConfig.FIELD_PRODUCTION_ID}", "${dbConfig.FIELD_ORDER_NUMBER}" FROM ${tableName} WHERE UPPER("${dbConfig.FIELD_PRODUCTION_ID}") IN (${pgPlaceholders})`
} else {
const idPlaceholders = uniqueProductionIds.map(() => 'UPPER(?)').join(', ')
// P0: Use DISTINCT to prevent duplicates from one-to-many relationships
// MySQL: 使用 UPPER 确保不区分大小写
sql = `SELECT DISTINCT \`${dbConfig.FIELD_PRODUCTION_ID}\`, \`${dbConfig.FIELD_ORDER_NUMBER}\` FROM \`${tableName}\` WHERE UPPER(\`${dbConfig.FIELD_PRODUCTION_ID}\`) IN (${idPlaceholders})`
}
const result = await this.dbService.query(sql, params)
const mappings = new Map<string, string>() const mappings = new Map<string, string>()
for (const row of result.rows) { const batches = this.chunk(uniqueProductionIds, RESOLUTION_QUERY_BATCH_SIZE)
const keys = Object.keys(row)
const prodId = row[keys[0]] as string for (const batch of batches) {
const orderNum = row[keys[1]] as string let sql: string
if (prodId && orderNum) { const params = batch
mappings.set(prodId, orderNum)
if (this.dbService.type === 'sqlserver') {
const placeholders = batch.map((_, i) => `@p${i}`).join(', ')
// P0: Use DISTINCT to prevent duplicates from one-to-many relationships.
sql = `SELECT DISTINCT [${dbConfig.FIELD_PRODUCTION_ID}], [${dbConfig.FIELD_ORDER_NUMBER}] FROM ${tableName} WHERE [${dbConfig.FIELD_PRODUCTION_ID}] COLLATE SQL_Latin1_General_CP1_CI_AS IN (${placeholders})`
} else if (this.dbService.type === 'postgresql') {
// PostgreSQL: 使用双引号保护中文标识符UPPER 实现不区分大小写。
const pgPlaceholders = batch.map((_, i) => `UPPER($${i + 1})`).join(', ')
sql = `SELECT DISTINCT "${dbConfig.FIELD_PRODUCTION_ID}", "${dbConfig.FIELD_ORDER_NUMBER}" FROM ${tableName} WHERE UPPER("${dbConfig.FIELD_PRODUCTION_ID}") IN (${pgPlaceholders})`
} else {
const idPlaceholders = batch.map(() => 'UPPER(?)').join(', ')
// MySQL: 使用 UPPER 确保不区分大小写。
sql = `SELECT DISTINCT \`${dbConfig.FIELD_PRODUCTION_ID}\`, \`${dbConfig.FIELD_ORDER_NUMBER}\` FROM \`${tableName}\` WHERE UPPER(\`${dbConfig.FIELD_PRODUCTION_ID}\`) IN (${idPlaceholders})`
}
const result = await this.dbService.query(sql, params)
for (const row of result.rows) {
const keys = Object.keys(row)
const prodId = row[keys[0]] as string
const orderNum = row[keys[1]] as string
if (prodId && orderNum) {
mappings.set(prodId, orderNum)
}
} }
} }
@@ -242,13 +245,14 @@ export class OrderNumberResolver {
// Build results while preserving original input order // Build results while preserving original input order
// Note: Multiple productionIDs mapping to the same order number is VALID (not an error) // Note: Multiple productionIDs mapping to the same order number is VALID (not an error)
const results: OrderMapping[] = [] const results: OrderMapping[] = []
const processedInputs = new Set<string>()
for (const input of inputs) { for (const input of inputs) {
// Skip if this exact input was already processed // Skip if this exact input was already processed
const alreadyProcessed = results.some((r) => r.input === input) if (processedInputs.has(input)) {
if (alreadyProcessed) {
continue continue
} }
processedInputs.add(input)
const mapping: OrderMapping = { input, resolved: false } const mapping: OrderMapping = { input, resolved: false }

View File

@@ -0,0 +1,214 @@
import type { BrowserContext, Page } from 'playwright'
export type ErpPageKind = 'login' | 'home' | 'query' | 'detail' | 'cas_login' | 'unknown'
export interface CapturePageStateOptions {
includeFrameHierarchy?: boolean
includeBodyTextPreview?: boolean
bodyTextPreviewLength?: number
}
export interface ErpPageState {
pageUrl?: string
pageTitle?: string
pageKind: ErpPageKind
hasForwardFrame: boolean
hasMainIframe: boolean
hasLoginForm: boolean
hasWorkbenchMarker: boolean
hasQueryMarker: boolean
hasDetailHeader: boolean
isCasLoginRedirect: boolean
frameCount?: number
popupCount?: number
visibleMarkers?: string[]
frameHierarchy?: Array<{ name: string; url: string }>
bodyTextPreview?: string
}
async function safePageUrl(page: Page): Promise<string | undefined> {
try {
return page.url()
} catch {
return undefined
}
}
async function safePageTitle(page: Page): Promise<string | undefined> {
try {
const title = await page.title()
return title.slice(0, 200)
} catch {
return undefined
}
}
async function safeFrameCount(page: Page): Promise<number | undefined> {
try {
return page.frames().length
} catch {
return undefined
}
}
async function safePopupCount(context?: BrowserContext): Promise<number | undefined> {
try {
return context?.pages().length
} catch {
return undefined
}
}
async function safeLocatorExists(locator: ReturnType<Page['locator']>): Promise<boolean> {
try {
return (await locator.count()) > 0
} catch {
return false
}
}
async function safeBodyPreview(page: Page, maxLength: number): Promise<string | undefined> {
try {
const text = await page.locator('body').innerText({ timeout: 1000 })
return text.replace(/\s+/g, ' ').trim().slice(0, maxLength)
} catch {
return undefined
}
}
export async function capturePageState(
page: Page,
context?: BrowserContext,
options: CapturePageStateOptions = {}
): Promise<ErpPageState> {
const pageUrl = await safePageUrl(page)
const pageTitle = await safePageTitle(page)
const isCasLoginRedirect = !!pageUrl?.includes('euc.yonyoucloud.com/cas/login')
let hasForwardFrame = false
let hasMainIframe = false
let hasLoginForm = false
let hasWorkbenchMarker = false
let hasQueryMarker = false
let hasDetailHeader = false
try {
hasForwardFrame = await safeLocatorExists(page.locator('#forwardFrame'))
} catch {
hasForwardFrame = false
}
let forwardFrame: Awaited<ReturnType<ReturnType<Page['locator']>['contentFrame']>> | null = null
if (hasForwardFrame) {
try {
forwardFrame = await page.locator('#forwardFrame').contentFrame()
} catch {
forwardFrame = null
}
}
if (forwardFrame) {
try {
hasMainIframe = (await forwardFrame.locator('#mainiframe').count()) > 0
} catch {
hasMainIframe = false
}
try {
hasWorkbenchMarker = (await forwardFrame.locator('.nc-workbench-icon').count()) > 0
} catch {
hasWorkbenchMarker = false
}
try {
hasLoginForm =
(await forwardFrame.getByRole('textbox', { name: '用户名' }).count()) > 0 ||
(await forwardFrame.getByRole('textbox', { name: '密码' }).count()) > 0
} catch {
hasLoginForm = false
}
}
let innerFrame: Awaited<
ReturnType<ReturnType<NonNullable<typeof forwardFrame>['locator']>['contentFrame']>
> | null = null
if (forwardFrame && hasMainIframe) {
try {
innerFrame = await forwardFrame.locator('#mainiframe').contentFrame()
} catch {
innerFrame = null
}
}
if (innerFrame) {
try {
hasQueryMarker =
(await innerFrame.getByText('订单号查询').count()) > 0 ||
(await innerFrame.locator('#rc_select_0').count()) > 0
} catch {
hasQueryMarker = false
}
try {
hasDetailHeader = (await innerFrame.getByText(/^离散备料计划维护:/).count()) > 0
} catch {
hasDetailHeader = false
}
}
if (!hasLoginForm) {
try {
hasLoginForm =
(await page.getByRole('textbox', { name: '用户名' }).count()) > 0 ||
(await page.getByRole('textbox', { name: '密码' }).count()) > 0
} catch {
hasLoginForm = false
}
}
const visibleMarkers: string[] = []
if (isCasLoginRedirect) visibleMarkers.push('cas_login_url')
if (hasLoginForm) visibleMarkers.push('login_form')
if (hasWorkbenchMarker) visibleMarkers.push('workbench_icon')
if (hasQueryMarker) visibleMarkers.push('query_marker')
if (hasDetailHeader) visibleMarkers.push('detail_header')
if (hasForwardFrame) visibleMarkers.push('forwardFrame')
if (hasMainIframe) visibleMarkers.push('mainiframe')
let pageKind: ErpPageKind = 'unknown'
if (isCasLoginRedirect) pageKind = 'cas_login'
else if (hasLoginForm) pageKind = 'login'
else if (hasDetailHeader) pageKind = 'detail'
else if (hasQueryMarker) pageKind = 'query'
else if (hasWorkbenchMarker) pageKind = 'home'
const state: ErpPageState = {
pageUrl,
pageTitle,
pageKind,
hasForwardFrame,
hasMainIframe,
hasLoginForm,
hasWorkbenchMarker,
hasQueryMarker,
hasDetailHeader,
isCasLoginRedirect,
frameCount: await safeFrameCount(page),
popupCount: await safePopupCount(context),
visibleMarkers
}
if (options.includeFrameHierarchy) {
try {
state.frameHierarchy = page.frames().map((frame) => ({ name: frame.name(), url: frame.url() }))
} catch {
state.frameHierarchy = undefined
}
}
if (options.includeBodyTextPreview) {
state.bodyTextPreview = await safeBodyPreview(page, options.bodyTextPreviewLength ?? 500)
}
return state
}

View File

@@ -41,7 +41,7 @@ export async function getSourceNumbersFromInputs(
} }
if (productionIds.length > 0) { if (productionIds.length > 0) {
const contractTableName = getValidationTableName('productionContractData_26年压力表合同数据') const contractTableName = getValidationTableName('ERPAuto.vw_productionContractData')
const batchSize = 2000 const batchSize = 2000
if (dbType === 'sqlserver') { if (dbType === 'sqlserver') {

View File

@@ -236,6 +236,7 @@ export class ValidationApplicationService {
): Promise<{ ): Promise<{
success: boolean success: boolean
orderNumbers?: string[] orderNumbers?: string[]
originalInputs?: string[]
materialCodes?: string[] materialCodes?: string[]
error?: string error?: string
}> { }> {
@@ -288,6 +289,7 @@ export class ValidationApplicationService {
return { return {
success: true, success: true,
orderNumbers, orderNumbers,
originalInputs: sharedIds,
materialCodes materialCodes
} }
} catch (error) { } catch (error) {
@@ -325,7 +327,7 @@ export class ValidationApplicationService {
} }
private async loadTypeKeywords(dbService: ValidationDatabaseService): Promise<TypeKeyword[]> { private async loadTypeKeywords(dbService: ValidationDatabaseService): Promise<TypeKeyword[]> {
const typeKeywordTableName = getValidationTableName('dbo_MaterialsTypeToBeDeleted') const typeKeywordTableName = getValidationTableName('dbo.MaterialsTypeToBeDeleted')
const sql = ` const sql = `
SELECT MaterialName, ManagerName SELECT MaterialName, ManagerName
FROM ${typeKeywordTableName} FROM ${typeKeywordTableName}
@@ -341,7 +343,7 @@ export class ValidationApplicationService {
private async loadMarkedCodes( private async loadMarkedCodes(
dbService: ValidationDatabaseService dbService: ValidationDatabaseService
): Promise<Map<string, string>> { ): Promise<Map<string, string>> {
const markedTableName = getValidationTableName('dbo_MaterialsToBeDeleted') const markedTableName = getValidationTableName('dbo.MaterialsToBeDeleted')
const sql = ` const sql = `
SELECT MaterialCode, ManagerName SELECT MaterialCode, ManagerName
FROM ${markedTableName} FROM ${markedTableName}
@@ -416,7 +418,7 @@ export class ValidationApplicationService {
try { try {
dbService = await createValidationDatabaseService() dbService = await createValidationDatabaseService()
const detailTableName = getValidationTableName('dbo_DiscreteMaterialPlanData') const detailTableName = getValidationTableName('dbo.DiscreteMaterialPlanData')
const enrichedMaterials: MaterialRecordSummary[] = [] const enrichedMaterials: MaterialRecordSummary[] = []
log.info(`Enriching ${materials.length} materials with details`) log.info(`Enriching ${materials.length} materials with details`)
@@ -501,7 +503,7 @@ export class ValidationApplicationService {
selectedManagers: string[], selectedManagers: string[],
orderNumbers: string[] orderNumbers: string[]
): Promise<string[]> { ): Promise<string[]> {
const markedTableName = getValidationTableName('dbo_MaterialsToBeDeleted') const markedTableName = getValidationTableName('dbo.MaterialsToBeDeleted')
// Admin with selected managers: filter MaterialsToBeDeleted by ManagerName IN (selectedManagers) // Admin with selected managers: filter MaterialsToBeDeleted by ManagerName IN (selectedManagers)
if (isAdmin && selectedManagers && selectedManagers.length > 0) { if (isAdmin && selectedManagers && selectedManagers.length > 0) {

View File

@@ -52,25 +52,22 @@ export async function createValidationDatabaseService(): Promise<ValidationDatab
return mysqlService return mysqlService
} }
export function getValidationTableName(mysqlTableName: string): string { export function getValidationTableName(dottedTableName: string): string {
const configManager = ConfigManager.getInstance() const configManager = ConfigManager.getInstance()
const dbType = configManager.getDatabaseType() const dbType = configManager.getDatabaseType()
if (dbType === 'sqlserver' || dbType === 'postgresql') { const dotIndex = dottedTableName.indexOf('.')
const firstUnderscoreIndex = mysqlTableName.indexOf('_') if (dotIndex > 0) {
if (firstUnderscoreIndex > 0) { const schema = dottedTableName.substring(0, dotIndex)
const schema = mysqlTableName.substring(0, firstUnderscoreIndex) const tableName = dottedTableName.substring(dotIndex + 1)
const tableName = mysqlTableName.substring(firstUnderscoreIndex + 1)
if (dbType === 'sqlserver') {
return `[${schema}].[${tableName}]`
}
return `"${schema}"."${tableName}"`
}
if (dbType === 'sqlserver') { if (dbType === 'sqlserver') {
return `[dbo].[${mysqlTableName}]` return `[${schema}].[${tableName}]`
} }
return `"public"."${mysqlTableName}"` return `"${schema}"."${tableName}"`
} }
// No dot found — use default schema
return mysqlTableName if (dbType === 'sqlserver') {
return `[dbo].[${dottedTableName}]`
}
return `"public"."${dottedTableName}"`
} }

View File

@@ -13,11 +13,13 @@ export interface CleanerProgress {
export interface CleanerInput { export interface CleanerInput {
orderNumbers: string[] orderNumbers: string[]
originalInputs?: string[]
materialCodes: string[] materialCodes: string[]
dryRun: boolean dryRun: boolean
headless?: boolean headless?: boolean
queryBatchSize?: number queryBatchSize?: number
processConcurrency?: number processConcurrency?: number
sessionRefreshOrderThreshold?: number
onProgress?: (message: string, progress?: number, extra?: Partial<CleanerProgress>) => void onProgress?: (message: string, progress?: number, extra?: Partial<CleanerProgress>) => void
} }

View File

@@ -117,7 +117,8 @@ export const validationConfigSchema = z.object({
*/ */
export const cleanerConfigSchema = z.object({ export const cleanerConfigSchema = z.object({
queryBatchSize: z.number().int().min(1).max(100).default(100), queryBatchSize: z.number().int().min(1).max(100).default(100),
processConcurrency: z.number().int().min(1).max(20).default(1) processConcurrency: z.number().int().min(1).max(20).default(1),
sessionRefreshOrderThreshold: z.number().int().positive().default(160)
}) })
export type CleanerConfig = z.infer<typeof cleanerConfigSchema> export type CleanerConfig = z.infer<typeof cleanerConfigSchema>

View File

@@ -62,12 +62,12 @@ const formatDateTime = (dateStr: string) => {
return dateStr // Return original if invalid return dateStr // Return original if invalid
} }
// Use UTC methods to display the time as stored in database (without timezone conversion) // Use local time for display
const year = date.getUTCFullYear() const year = date.getFullYear()
const month = String(date.getUTCMonth() + 1).padStart(2, '0') const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getUTCDate()).padStart(2, '0') const day = String(date.getDate()).padStart(2, '0')
const hours = String(date.getUTCHours()).padStart(2, '0') const hours = String(date.getHours()).padStart(2, '0')
const minutes = String(date.getUTCMinutes()).padStart(2, '0') const minutes = String(date.getMinutes()).padStart(2, '0')
return `${year}-${month}-${day} ${hours}:${minutes}` return `${year}-${month}-${day} ${hours}:${minutes}`
} }

View File

@@ -10,6 +10,7 @@ import type { CleanerExportItem, MaterialBatchChange } from './helpers'
interface CleanerDataPayload { interface CleanerDataPayload {
success?: boolean success?: boolean
orderNumbers?: string[] orderNumbers?: string[]
originalInputs?: string[]
materialCodes?: string[] materialCodes?: string[]
} }
@@ -61,7 +62,8 @@ export async function loadCleanerConfig(): Promise<CleanerConfigResult | null> {
return { return {
queryBatchSize: result.data.queryBatchSize, queryBatchSize: result.data.queryBatchSize,
processConcurrency: result.data.processConcurrency processConcurrency: result.data.processConcurrency,
sessionRefreshOrderThreshold: result.data.sessionRefreshOrderThreshold
} }
} }
@@ -120,6 +122,7 @@ export async function runCleanerExecution(params: {
headless: boolean headless: boolean
queryBatchSize: number queryBatchSize: number
processConcurrency: number processConcurrency: number
sessionRefreshOrderThreshold: number
selectedManagers: string[] selectedManagers: string[]
}): Promise<CleanerReportData> { }): Promise<CleanerReportData> {
const cleanerDataResult = await window.electron.validation.getCleanerData({ const cleanerDataResult = await window.electron.validation.getCleanerData({
@@ -145,11 +148,13 @@ export async function runCleanerExecution(params: {
const response = await window.electron.cleaner.runCleaner({ const response = await window.electron.cleaner.runCleaner({
orderNumbers: orderNumberList, orderNumbers: orderNumberList,
originalInputs: cleanerData?.originalInputs,
materialCodes: materialCodeList, materialCodes: materialCodeList,
dryRun: params.dryRun, dryRun: params.dryRun,
headless: params.headless, headless: params.headless,
queryBatchSize: params.queryBatchSize, queryBatchSize: params.queryBatchSize,
processConcurrency: params.processConcurrency processConcurrency: params.processConcurrency,
sessionRefreshOrderThreshold: params.sessionRefreshOrderThreshold
}) })
const cleanerRunData = response.success ? (response.data as CleanerRunPayload | null) : null const cleanerRunData = response.success ? (response.data as CleanerRunPayload | null) : null

View File

@@ -60,6 +60,7 @@ export interface CleanerInitializationResult {
export interface CleanerConfigResult { export interface CleanerConfigResult {
queryBatchSize: number queryBatchSize: number
processConcurrency: number processConcurrency: number
sessionRefreshOrderThreshold: number
} }
// Cleaner operation history types (mirrors preload/index.d.ts) // Cleaner operation history types (mirrors preload/index.d.ts)

View File

@@ -62,6 +62,7 @@ export function useCleaner() {
const [headless, setHeadless] = useState(() => getStoredBoolean('cleaner_headless', true)) const [headless, setHeadless] = useState(() => getStoredBoolean('cleaner_headless', true))
const [queryBatchSize, setQueryBatchSize] = useState(100) const [queryBatchSize, setQueryBatchSize] = useState(100)
const [processConcurrency, setProcessConcurrency] = useState(1) const [processConcurrency, setProcessConcurrency] = useState(1)
const [sessionRefreshOrderThreshold, setSessionRefreshOrderThreshold] = useState(160)
const [showSettingsMenu, setShowSettingsMenu] = useState(false) const [showSettingsMenu, setShowSettingsMenu] = useState(false)
// Inline editing state for manager field (Admin only) // Inline editing state for manager field (Admin only)
@@ -138,6 +139,7 @@ export function useCleaner() {
if (result) { if (result) {
setQueryBatchSize(result.queryBatchSize) setQueryBatchSize(result.queryBatchSize)
setProcessConcurrency(result.processConcurrency) setProcessConcurrency(result.processConcurrency)
setSessionRefreshOrderThreshold(result.sessionRefreshOrderThreshold)
} }
} catch (err) { } catch (err) {
logger.error('Failed to load cleaner config', { logger.error('Failed to load cleaner config', {
@@ -377,6 +379,7 @@ export function useCleaner() {
headless, headless,
queryBatchSize, queryBatchSize,
processConcurrency, processConcurrency,
sessionRefreshOrderThreshold,
selectedManagers: Array.from(selectedManagers) selectedManagers: Array.from(selectedManagers)
}) })
setReportData(result) setReportData(result)
@@ -440,6 +443,8 @@ export function useCleaner() {
setQueryBatchSize, setQueryBatchSize,
processConcurrency, processConcurrency,
setProcessConcurrency, setProcessConcurrency,
sessionRefreshOrderThreshold,
setSessionRefreshOrderThreshold,
updateProcessConcurrency, updateProcessConcurrency,
showSettingsMenu, showSettingsMenu,
setShowSettingsMenu, setShowSettingsMenu,

View File

@@ -46,8 +46,8 @@ describe('PostgreSqlDialect', () => {
}) })
describe('currentTimestamp', () => { describe('currentTimestamp', () => {
it('should return CURRENT_TIMESTAMP', () => { it('should return explicit UTC timestamp expression', () => {
expect(dialect.currentTimestamp()).toBe('CURRENT_TIMESTAMP') expect(dialect.currentTimestamp()).toBe("(NOW() AT TIME ZONE 'UTC')")
}) })
}) })

View File

@@ -279,6 +279,14 @@ describe('CleanerApplicationService', () => {
expect(result.ordersProcessed).toBe(1) expect(result.ordersProcessed).toBe(1)
}) })
it('should fall back to default session refresh threshold when cleaner config is missing', async () => {
const eventSender: any = { send: vi.fn() }
await service.runCleaner(eventSender, makeInput())
expect(lastCleanerInput?.sessionRefreshOrderThreshold).toBe(160)
})
it('should close ERP browser on success', async () => { it('should close ERP browser on success', async () => {
await service.runCleaner({ send: vi.fn() } as any, makeInput()) await service.runCleaner({ send: vi.fn() } as any, makeInput())

View File

@@ -0,0 +1,104 @@
import { describe, it, expect, beforeEach, vi } from 'vitest'
import {
DiscreteMaterialPlanDAO,
type MaterialPlanRecord
} from '../../../../src/main/services/database/discrete-material-plan-dao'
import type { IDatabaseService } from '../../../../src/main/services/database'
vi.mock('../../../../src/main/services/logger', () => ({
createLogger: vi.fn(() => ({
info: vi.fn(),
debug: vi.fn(),
warn: vi.fn(),
error: vi.fn()
})),
getRequestId: vi.fn(() => 'test-request-id'),
trackDuration: vi.fn(async (fn) => ({ result: await fn(), durationMs: 1, isSlow: false }))
}))
vi.mock('../../../../src/main/services/database', () => ({
create: vi.fn()
}))
function createRecord(sourceNumber: string, index: number): MaterialPlanRecord {
return {
factory: '工厂A',
materialStatus: '已审批',
planNumber: `PLAN-${index}`,
sourceNumber,
materialType: '标准',
productCode: 'P001',
productName: '产品A',
productUnit: 'PCS',
productPlanQuantity: 1,
useDepartment: '',
remark: '',
creator: '',
createDate: new Date('2026-04-28T00:00:00Z'),
approver: '',
approveDate: new Date('2026-04-28T00:00:00Z'),
sequenceNumber: index,
materialCode: `MAT-${index}`,
materialName: '物料A',
specification: '',
model: '',
drawingNumber: '',
materialQuality: '',
planQuantity: 1,
unit: 'PCS',
requiredDate: new Date('2026-04-28T00:00:00Z'),
warehouse: '',
unitUsage: 1,
cumulativeOutputQuantity: 0,
bomVersion: ''
}
}
describe('DiscreteMaterialPlanDAO', () => {
let mockDbService: IDatabaseService
beforeEach(async () => {
vi.clearAllMocks()
mockDbService = {
type: 'sqlserver',
connect: vi.fn(),
disconnect: vi.fn(),
isConnected: vi.fn(() => true),
query: vi.fn(async (_sql, params = []) => {
const rows = JSON.parse(params[1] || '[]')
return {
rows: [{ deletedCount: 0, insertedCount: rows.length }],
columns: ['deletedCount', 'insertedCount'],
rowCount: 1
}
}),
transaction: vi.fn()
}
const database = await import('../../../../src/main/services/database')
vi.mocked(database.create).mockResolvedValue(mockDbService)
})
it('splits SQL Server replace operations by source number batches', async () => {
const records = Array.from({ length: 151 }, (_, index) =>
createRecord(`SC-${String(index).padStart(4, '0')}`, index)
)
const dao = new DiscreteMaterialPlanDAO()
const result = await dao.replaceBySourceNumbers(records, 1000)
expect(result).toEqual({ deleted: 0, inserted: 151 })
expect(mockDbService.query).toHaveBeenCalledTimes(7)
const firstParams = vi.mocked(mockDbService.query).mock.calls[0][1] || []
const sixthParams = vi.mocked(mockDbService.query).mock.calls[5][1] || []
const seventhParams = vi.mocked(mockDbService.query).mock.calls[6][1] || []
expect(JSON.parse(firstParams[0])).toHaveLength(25)
expect(JSON.parse(sixthParams[0])).toHaveLength(25)
expect(JSON.parse(seventhParams[0])).toHaveLength(1)
expect(JSON.parse(firstParams[1])).toHaveLength(25)
expect(JSON.parse(sixthParams[1])).toHaveLength(25)
expect(JSON.parse(seventhParams[1])).toHaveLength(1)
})
})

View File

@@ -97,6 +97,14 @@ describe('ExtractorService', () => {
recordsImported: 0, recordsImported: 0,
uniqueSourceNumbers: 0, uniqueSourceNumbers: 0,
errors: [] errors: []
} as ImportResult),
importFromRecords: vi.fn().mockResolvedValue({
success: true,
recordsRead: 0,
recordsDeleted: 0,
recordsImported: 0,
uniqueSourceNumbers: 0,
errors: []
} as ImportResult) } as ImportResult)
} }
@@ -173,6 +181,65 @@ describe('ExtractorService', () => {
expect(Array.isArray(result.errors)).toBe(true) expect(Array.isArray(result.errors)).toBe(true)
}) })
it('should import parsed records directly instead of re-reading merged Excel', async () => {
mockExtractorCoreInstance.downloadAllBatches.mockResolvedValue({
downloadedFiles: ['./file1.xlsx'],
errors: []
})
mockExcelParserInstance.parse = vi.fn().mockImplementation(() => {
mockExcelParserInstance._lastOrders = [
{
orderInfo: {
factory: '工厂A',
planNumber: 'PLAN001',
productionOrder: 'ORD001',
productCode: 'P001',
productName: '产品A',
plannedQuantity: '10',
unit: 'PCS'
},
materials: [
{
sequence: 1,
materialCode: 'MAT001',
materialName: '物料A',
quantity: 2,
unit: 'PCS'
}
]
}
]
return Promise.resolve()
})
mockDataImportInstance.importFromRecords.mockResolvedValue({
success: true,
recordsRead: 1,
recordsDeleted: 0,
recordsImported: 1,
uniqueSourceNumbers: 1,
errors: []
} as ImportResult)
const service = new ExtractorService(mockAuthService, './test-downloads')
vi.spyOn(service as any, 'saveMergedOrders').mockResolvedValue(undefined)
const result = await service.extract({
orderNumbers: ['ORD001'],
onProgress: vi.fn(),
onLog: vi.fn()
})
expect(result.importResult?.success).toBe(true)
expect(mockDataImportInstance.importFromRecords).toHaveBeenCalledTimes(1)
expect(mockDataImportInstance.importFromExcel).not.toHaveBeenCalled()
expect(mockDataImportInstance.importFromRecords.mock.calls[0][0][0]).toMatchObject({
planNumber: 'PLAN001',
sourceNumber: 'ORD001',
materialCode: 'MAT001',
planQuantity: 2
})
})
}) })
describe('mergeFiles()', () => { describe('mergeFiles()', () => {
@@ -196,6 +263,7 @@ describe('ExtractorService', () => {
] ]
const service = new ExtractorService(mockAuthService, './test-downloads') const service = new ExtractorService(mockAuthService, './test-downloads')
vi.spyOn(service as any, 'saveMergedOrders').mockResolvedValue(undefined)
// @ts-ignore - accessing private method for testing // @ts-ignore - accessing private method for testing
const result = await service.mergeFiles(['./file1.xlsx'], ['ORD001']) const result = await service.mergeFiles(['./file1.xlsx'], ['ORD001'])
@@ -231,6 +299,7 @@ describe('ExtractorService', () => {
}) })
const service = new ExtractorService(mockAuthService, './test-downloads') const service = new ExtractorService(mockAuthService, './test-downloads')
vi.spyOn(service as any, 'saveMergedOrders').mockResolvedValue(undefined)
// @ts-ignore - accessing private method for testing // @ts-ignore - accessing private method for testing
const result = await service.mergeFiles( const result = await service.mergeFiles(

View File

@@ -173,6 +173,25 @@ describe('OrderNumberResolver', () => {
// Should be optimized to query unique values only // Should be optimized to query unique values only
expect(mockDbService.query).toHaveBeenCalledTimes(1) expect(mockDbService.query).toHaveBeenCalledTimes(1)
}) })
it('splits large mapping queries into bounded batches', async () => {
const largeInput = Array.from({ length: 1001 }, (_, i) => `22A${i}`)
vi.mocked(mockDbService.query).mockImplementation(async (_sql, params = []) => ({
rows: params.map((prodId, i) => ({
总排号: prodId,
: `SC7020260212${String(i).padStart(5, '0')}`
})),
columns: ['总排号', '生产订单号'],
rowCount: params.length
}))
await resolver.mapProductionIdsToOrderNumbers(largeInput)
expect(mockDbService.query).toHaveBeenCalledTimes(2)
expect(vi.mocked(mockDbService.query).mock.calls[0][1]).toHaveLength(1000)
expect(vi.mocked(mockDbService.query).mock.calls[1][1]).toHaveLength(1)
})
}) })
describe('error handling', () => { describe('error handling', () => {

View File

@@ -125,31 +125,25 @@ describe('ValidationDatabaseService', () => {
}) })
describe('getValidationTableName', () => { describe('getValidationTableName', () => {
it('returns table name unchanged for mysql', async () => { it('converts schema.table to [schema].[table] for sqlserver', async () => {
currentDbType = 'mysql'
const mod = await import('../../../../src/main/services/validation/validation-database')
expect(mod.getValidationTableName('MaterialsToBeDeleted')).toBe('MaterialsToBeDeleted')
})
it('converts schema_table to [schema].[table] for sqlserver', async () => {
currentDbType = 'sqlserver' currentDbType = 'sqlserver'
const mod = await import('../../../../src/main/services/validation/validation-database') const mod = await import('../../../../src/main/services/validation/validation-database')
expect(mod.getValidationTableName('dbo_Materials')).toBe('[dbo].[Materials]') expect(mod.getValidationTableName('dbo.Materials')).toBe('[dbo].[Materials]')
}) })
it('wraps nameless table in [dbo].[name] for sqlserver', async () => { it('wraps dotless table in [dbo].[name] for sqlserver', async () => {
currentDbType = 'sqlserver' currentDbType = 'sqlserver'
const mod = await import('../../../../src/main/services/validation/validation-database') const mod = await import('../../../../src/main/services/validation/validation-database')
expect(mod.getValidationTableName('Materials')).toBe('[dbo].[Materials]') expect(mod.getValidationTableName('Materials')).toBe('[dbo].[Materials]')
}) })
it('converts schema_table to "schema"."table" for postgresql', async () => { it('converts schema.table to "schema"."table" for postgresql', async () => {
currentDbType = 'postgresql' currentDbType = 'postgresql'
const mod = await import('../../../../src/main/services/validation/validation-database') const mod = await import('../../../../src/main/services/validation/validation-database')
expect(mod.getValidationTableName('public_Materials')).toBe('"public"."Materials"') expect(mod.getValidationTableName('public.Materials')).toBe('"public"."Materials"')
}) })
it('wraps nameless table in "public"."name" for postgresql', async () => { it('wraps dotless table in "public"."name" for postgresql', async () => {
currentDbType = 'postgresql' currentDbType = 'postgresql'
const mod = await import('../../../../src/main/services/validation/validation-database') const mod = await import('../../../../src/main/services/validation/validation-database')
expect(mod.getValidationTableName('Materials')).toBe('"public"."Materials"') expect(mod.getValidationTableName('Materials')).toBe('"public"."Materials"')