feat(cleaner): add outer-level retry on fatal crash with execution ID

When CleanerService hits a fatal error (browser crash, timeout), the outer catch now sets result.crashed=true. CleanerApplicationService detects this, closes the dead browser session, re-logs into ERP, and re-runs all orders once. An execution ID (CLN-yyyyMMddHHmmss-XXXX) generated at startup ensures report files are deduplicated across retries. Reports now display execution ID and app version. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 10:07:29 +08:00
parent 8b173890fa
commit e1d55b8b39
7 changed files with 246 additions and 12 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -45,3 +45,7 @@ nul

 # TypeScript incremental compilation cache
 *.tsbuildinfo
+
+# temporary files
+tmp/
+temp/
--- a/docs/plans/2026-04-13-cleaner-outer-retry-design.md
+++ b/docs/plans/2026-04-13-cleaner-outer-retry-design.md
@@ -0,0 +1,151 @@
+# Cleaner 外层重试机制设计
+
+## 背景
+
+当 CleanerService.performCleanup 的主循环抛出未捕获异常时（如查询超时、浏览器崩溃），代码进入 outer catch 块，直接返回 partial result。位于 try 块后半段的订单级重试逻辑（retryFailedOrders）永远没有机会执行。
+
+典型场景：211 个订单中处理到第 80 个时，查询列表页等待表格行超时 → Cleaner failed → 浏览器被关闭 → 剩余 131 个订单未处理 → 无重试。
+
+## 设计决策
+
+| 决策项 | 选择 | 理由 |
+|---|---|---|
+| 重试层级 | CleanerApplicationService | 崩溃后浏览器不可用，必须重新登录 |
+| 重试范围 | 全部订单重新跑 | 简单可靠，物料删除是幂等操作 |
+| 最大重试次数 | 1 次 | 覆盖瞬态故障，不过度消耗时间 |
+| 触发条件 | result.crashed === true | 仅 outer catch 触发时才重试 |
+| 报告去重 | 执行 ID | 用户点击执行时生成，重试不变 |
+
+## 变更清单
+
+### 1. CleanerResult 新增字段
+
+**文件**: `src/main/types/cleaner.types.ts`
+
+```typescript
+export interface CleanerResult {
+  // ... 现有字段
+  crashed?: boolean  // true = outer catch triggered, 流程级崩溃
+}
+```
+
+同步更新 `src/shared/types/cleaner.types.ts`（如有独立定义）和 preload 暴露的类型声明。
+
+### 2. CleanerService 标记崩溃
+
+**文件**: `src/main/services/erp/cleaner.ts`，line 375 的 catch 块
+
+```typescript
+} catch (error) {
+  const message = error instanceof Error ? error.message : 'Unknown error'
+  log.error('Cleaner failed', { ... })
+  result.errors.push(`Clean failed: ${message}`)
+  result.crashed = true  // ← 新增
+}
+```
+
+### 3. CleanerApplicationService 重试逻辑
+
+**文件**: `src/main/services/cleaner/cleaner-application-service.ts`
+
+在 `runCleaner()` 中，`cleaner.clean()` 返回后增加重试判断：
+
+```
+runCleaner(eventSender, input) {
+  const executionId = generateExecutionId()  // 用户点击时生成
+  const startTime = Date.now()
+
+  // 1. 获取 ERP 配置、数据库连接、订单解析（不变）
+  // 2. 登录 ERP（不变）
+
+  let result = await cleaner.clean(modifiedInput)
+
+  // === 外层重试 ===
+  if (result.crashed) {
+    log.warn('检测到流程级崩溃，准备外层重试', { executionId })
+
+    await authService.close()  // 关闭不可用的浏览器
+    authService = new ErpAuthService({...})
+    await authService.login()  // 重新登录
+
+    cleaner = new CleanerService(authService)
+    result = await cleaner.clean(modifiedInput)  // 全部订单重新跑
+  }
+
+  // 3. 生成报告（使用 executionId 作为文件名一部分，避免重复）
+  await this.generateAndUploadReport(input, result, startTime, executionId)
+  return result
+}
+```
+
+### 4. 执行 ID 生成规则
+
+格式: `CLN-{yyyyMMddHHmmss}-{4位随机字母}`
+
+示例: `CLN-20260410112930-A7FK`
+
+生成时机: `runCleaner()` 入口处，在 ERP 登录之前。重试时同一个 executionId 不变。
+
+用途:
+- 报告文件名: `cleaner-report-CLN-20260410112930-A7FK.md`
+- RustFS 存储路径中包含该 ID，重试时覆盖同一文件
+- 报告内容中显示该 ID
+
+### 5. 报告增强
+
+**文件**: `src/main/services/report/cleaner-report-generator.ts`
+
+在执行摘要表格中新增字段:
+
+```markdown
+| 项目             | 值                                |
+| ---------------- | --------------------------------- |
+| **执行 ID**      | `CLN-20260410112930-A7FK`         |  ← 新增
+| **应用版本**     | `1.11.1`                          |  ← 新增
+| **执行时间**     | `2026-04-10 11:29:30`             |
+| **执行模式**     | `正式执行`                        |
+| ...              | ...                               |
+```
+
+- **执行 ID**: 从 ReportOptions 传入
+- **应用版本**: `app.getVersion()`，沿用 logger 中已有的获取方式
+
+**ReportOptions 变更**:
+
+```typescript
+export interface ReportOptions {
+  dryRun: boolean
+  username: string
+  startTime: number
+  endTime: number
+  executionId: string   // ← 新增
+  appVersion: string    // ← 新增
+}
+```
+
+**报告文件名变更**:
+
+```
+旧: cleaner-report-2026-04-10-03-30-12.md
+新: cleaner-report-CLN-20260410112930-A7FK.md
+```
+
+重试时同一个 executionId 生成相同的文件名，本地文件和 RustFS 上传都会覆盖旧报告，无需额外去重逻辑。
+
+### 6. 进度通知增强
+
+重试时向前端发送进度通知，让用户知道正在重试:
+
+```typescript
+this.sendProgress(eventSender, '流程崩溃，正在重新登录并重试...', 0, {
+  phase: 'retry',
+  ...
+})
+```
+
+## 不涉及的部分
+
+- 前端 UI 变更（后续可单独做，展示重试状态）
+- IPC channel 变更
+- 内层重试逻辑（订单级/物料级）不变
+- 数据库 schema 变更
--- a/src/main/services/cleaner/cleaner-application-service.ts
+++ b/src/main/services/cleaner/cleaner-application-service.ts
@@ -1,4 +1,5 @@
 import type { WebContents } from 'electron'
+import { app } from 'electron'
 import type { IDatabaseService } from '../../types/database.types'
 import { ErpAuthService } from '../erp/erp-auth'
 import { CleanerService } from '../erp/cleaner'
@@ -27,8 +28,21 @@ import type {

 const log = createLogger('CleanerApplicationService')

+function generateExecutionId(): string {
+  const now = new Date()
+  const date = `${now.getFullYear()}${String(now.getMonth() + 1).padStart(2, '0')}${String(now.getDate()).padStart(2, '0')}`
+  const time = `${String(now.getHours()).padStart(2, '0')}${String(now.getMinutes()).padStart(2, '0')}${String(now.getSeconds()).padStart(2, '0')}`
+  const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+  let suffix = ''
+  for (let i = 0; i < 4; i++) {
+    suffix += chars.charAt(Math.floor(Math.random() * chars.length))
+  }
+  return `CLN-${date}${time}-${suffix}`
+}
+
 export class CleanerApplicationService {
  async runCleaner(eventSender: WebContents, input: CleanerInput): Promise<CleanerResult> {
+    const executionId = generateExecutionId()
    const startTime = Date.now()
    let authService: ErpAuthService | null = null
    let dbService: IDatabaseService | null = null
@@ -104,7 +118,6 @@ export class CleanerApplicationService {
        totalMaterialsInOrder: 0
      })

-      const cleaner = new CleanerService(authService)
      const modifiedInput: CleanerInput = {
        ...input,
        orderNumbers: validOrderNumbers,
@@ -114,12 +127,68 @@ export class CleanerApplicationService {
      }

      log.info('Starting cleaning', {
+        executionId,
        orderCount: validOrderNumbers.length,
        queryBatchSize: input.queryBatchSize ?? 100,
        processConcurrency: input.processConcurrency ?? 1
      })

-      const result = await cleaner.clean(modifiedInput)
+      let cleaner = new CleanerService(authService)
+      let result = await cleaner.clean(modifiedInput)
+
+      // Outer retry: re-login and re-run all orders on fatal crash
+      if (result.crashed) {
+        log.warn('检测到流程级崩溃，准备外层重试', { executionId })
+
+        this.sendProgress(eventSender, '流程崩溃，正在重新登录并重试...', 0, {
+          phase: 'retry',
+          currentOrderIndex: 0,
+          totalOrders,
+          currentMaterialIndex: 0,
+          totalMaterialsInOrder: 0
+        })
+
+        try {
+          await authService.close()
+        } catch {
+          // Browser may already be dead, ignore close errors
+        }
+        authService = null
+
+        authService = new ErpAuthService({
+          url: erpConfig.url,
+          username: erpConfig.username,
+          password: erpConfig.password,
+          headless: input.headless ?? true
+        })
+
+        try {
+          await authService.login()
+          log.info('Outer retry: re-login successful', { executionId })
+        } catch (loginError) {
+          log.error('Outer retry: re-login failed', {
+            executionId,
+            error: loginError instanceof Error ? loginError.message : String(loginError)
+          })
+          // Return the original crash result if re-login fails
+          result.errors.push(`外层重试登录失败: ${loginError instanceof Error ? loginError.message : String(loginError)}`)
+          if (warnings.length > 0) {
+            result.errors = [...warnings, ...result.errors]
+          }
+          await this.recordCleanupAudit(validOrderNumbers.length, input, result)
+          await this.generateAndUploadReport(input, result, startTime, executionId)
+          return result
+        }
+
+        cleaner = new CleanerService(authService)
+        result = await cleaner.clean(modifiedInput)
+        log.info('Outer retry completed', {
+          executionId,
+          processedCount: result.ordersProcessed,
+          errorCount: result.errors.length,
+          crashed: result.crashed
+        })
+      }

      if (warnings.length > 0) {
        result.errors = [...warnings, ...result.errors]
@@ -134,12 +203,13 @@ export class CleanerApplicationService {
      })

      log.info('Cleaning completed', {
+        executionId,
        processedCount: result.ordersProcessed,
        errorCount: result.errors.length
      })

      await this.recordCleanupAudit(validOrderNumbers.length, input, result)
-      await this.generateAndUploadReport(input, result, startTime)
+      await this.generateAndUploadReport(input, result, startTime, executionId)

      return result
    } finally {
@@ -300,7 +370,8 @@ export class CleanerApplicationService {
  private async generateAndUploadReport(
    input: CleanerInput,
    result: CleanerResult,
-    startTime: number
+    startTime: number,
+    executionId: string
  ): Promise<void> {
    try {
      const endTime = Date.now()
@@ -312,7 +383,9 @@ export class CleanerApplicationService {
        dryRun: input.dryRun ?? false,
        username,
        startTime,
-        endTime
+        endTime,
+        executionId,
+        appVersion: app.getVersion()
      })
      log.info('Report generated', { path: reportPath })

--- a/src/main/services/erp/cleaner.ts
+++ b/src/main/services/erp/cleaner.ts
@@ -384,6 +384,7 @@ export class CleanerService {
        ...(popupPage ? await capturePageContext(popupPage, undefined, 'cleaner.outerCatch') : {})
      })
      result.errors.push(`Clean failed: ${message}`)
+      result.crashed = true
    } finally {
      if (popupPage) {
        try {
--- a/src/main/services/report/cleaner-report-generator.ts
+++ b/src/main/services/report/cleaner-report-generator.ts
@@ -16,6 +16,8 @@ export interface ReportOptions {
  username: string
  startTime: number
  endTime: number
+  executionId: string
+  appVersion: string
 }

 interface OrderStats {
@@ -41,7 +43,7 @@ export class CleanerReportGenerator {
  }

  async generateReport(result: CleanerResult, options: ReportOptions): Promise<string> {
-    const filePath = this.getReportFilePath()
+    const filePath = this.getReportFilePath(options.executionId)
    log.info('Generating cleaner report', { path: filePath })

    const stats = this.calculateOrderStats(result)
@@ -53,10 +55,8 @@ export class CleanerReportGenerator {
    return filePath
  }

-  private getReportFilePath(): string {
-    const now = new Date()
-    const timestamp = now.toISOString().replace(/[:.]/g, '-').slice(0, -5).replace('T', '-')
-    const fileName = `cleaner-report-${timestamp}.md`
+  private getReportFilePath(executionId: string): string {
+    const fileName = `cleaner-report-${executionId}.md`
    return path.join(this.reportDir, fileName)
  }

@@ -87,6 +87,8 @@ export class CleanerReportGenerator {
    lines.push('')
    lines.push('| 项目           | 值                                |')
    lines.push('| -------------- | --------------------------------- |')
+    lines.push(`| **执行 ID**    | \`${options.executionId}\``)
+    lines.push(`| **应用版本**   | \`${options.appVersion}\``)
    lines.push(`| **执行时间**   | \`${this.formatDateTime(options.endTime)}\``)
    lines.push(`| **执行模式**   | \`${options.dryRun ? '模拟运行 (Dry Run)' : '正式执行'}\``)
    lines.push(`| **操作用户**   | \`${options.username}\``)
--- a/src/main/types/cleaner.types.ts
+++ b/src/main/types/cleaner.types.ts
@@ -1,4 +1,4 @@
-export type CleanerPhase = 'login' | 'processing' | 'complete'
+export type CleanerPhase = 'login' | 'processing' | 'complete' | 'retry'

 export interface CleanerProgress {
  message: string
@@ -33,6 +33,8 @@ export interface CleanerResult {
  // Deletion verification statistics
  materialsFailed: number
  uncertainDeletions: number
+  // Outer retry: true when outer catch triggered (browser crash / fatal timeout)
+  crashed?: boolean
 }

 export interface SkippedMaterial {
--- a/src/renderer/src/hooks/cleaner/types.ts
+++ b/src/renderer/src/hooks/cleaner/types.ts
@@ -35,7 +35,7 @@ export interface CleanerProgress {
  currentMaterialIndex: number
  totalMaterialsInOrder: number
  currentOrderNumber?: string
-  phase: 'login' | 'processing' | 'complete'
+  phase: 'login' | 'processing' | 'complete' | 'retry'
 }

 export interface CleanerReportData {
@@ -47,6 +47,7 @@ export interface CleanerReportData {
  successfulRetries?: number
  materialsFailed?: number
  uncertainDeletions?: number
+  crashed?: boolean
 }

 export interface CleanerInitializationResult {