Skip to content

S
skills
Commit 1027cb42 authored by allen.wang's avatar allen.wang
Browse files
Options

fix：优化报表效果

parent d90a2b4d
Show whitespace changes
Inline Side-by-side
Showing with 6266 additions and 766 deletions
vip-report/SKILL.md
View file @ 1027cb42
---
name: vip-report
description: 协调 VIP 月报所有页的来源与渲染，通过固定配置与模板确保生成版与基线完全一致。
description: Coordinate VIP monthly report data collection, asset generation, PowerPoint replacement, and final comparison so the generated report stays aligned with the template.
---
# VIP Report
## 概览
## Overview
这个总 skill 串联 VIP 月报的数据采集、素材生成、模板替换、比对验证四步，确保最终 `Report.pptx` 一模一样。
This skill coordinates the end-to-end VIP monthly report workflow:
## 数据来源约定
- collect source data
- generate slide assets
- replace template content
- compare the generated PPT with the baseline/template
- 所有页的来源映射记录在 `C:\workspace\cursor\output\vip-report\slide-source-map.yaml`，按 slide_code 映射到 Tableau view、MySQL 表或第三方系统。
- 依赖 Tableau (`tableau.charleskeith.cn`) 时统一走 `config.yaml` 里的账号（目前是 `ec_user01`），优先尝试下载/导出，若必须截屏则按照脚本裁切。
- 某些页面会引用 `vip.com`，凭据不驻留仓库；调用 `config.yaml` 中的 `vip.login_endpoint` 让第三方系统自动处理登录。
- 需要数据库的页面使用测试库 `ckc_cep_db_test`（`config.yaml` 的 `mysql` 段），会参考 `s11-source-validation.md`、`s11-sql-hypothesis.sql` 等文件保持口径。
- S11 的活动口径已切换到 `oms_shop_report`：`memo` 维护活动/非活动，`single='Y'` 维护独家。
## Source Rules
## 关键路径
- Slide-to-source mapping lives in `C:\workspace\cursor\output\vip-report\slide-source-map.yaml`.
- Tableau source of truth is `tableau.charleskeith.cn`, using the credentials in `config.yaml`.
- Some pages may depend on `vip.com`; if needed, use `vip.login_endpoint` from config rather than hardcoding login logic.
- Test MySQL for VIP-report remains the `config.yaml` `mysql` section unless the task explicitly asks for production verification.
- Production MySQL default for future VIP-report sessions:
`jdbc:mysql://erp.charleskeith.cn:30004/ckc_cep_db?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true`
username: `allen`
password: `wangjun@123`
- When a future session says “生产地址 mysql” for VIP-report, default to the production MySQL above.
- S11 campaign logic is maintained separately through memo / validation files and must stay consistent with its confirmed source logic.
- 配置文件：`C:\Users\niuniu\.codex\vip-report\config.yaml`
- 配置样例：`C:\Users\niuniu\.codex\vip-report\config.yaml.example`
- 模板 PPT：`C:\Users\niuniu\Desktop\Report.pptx`
- slide 来源表：`C:\workspace\cursor\output\vip-report\slide-source-map.yaml`
- 素材与输出目录：`C:\workspace\cursor\output\vip-report`
## Key Paths
## 渲染策略
- Config: `C:\Users\niuniu\.codex\vip-report\config.yaml`
- Config example: `C:\Users\niuniu\.codex\vip-report\config.yaml.example`
- Template PPT: `C:\Users\niuniu\Desktop\Report.pptx`
- Slide source map: `C:\workspace\cursor\output\vip-report\slide-source-map.yaml`
- Output root: `C:\workspace\cursor\output\vip-report`
- 目前 Windows 可以成功创建 `PowerPoint.Application`，建议优先用 COM 改写模板（如 `render_template_com.ps1`）。
- `OpenXML` 只在无法运行 COM 或需包级比较时作为补充手段。
- 所有替换完成后必须运行 `compare_pptx.py`（`render-ops.*.json` 也可辅助）确认差异限定在预期的 `ppt/media/image*.png`、`ppt/slides/slide*.xml` 中。
## Rendering Strategy
## 子技能与流程
- On Windows, prefer COM-based rendering when PowerPoint automation is available.
- Use OpenXML only as a fallback or for package-level comparisons.
- After replacement, compare with `compare_pptx.py` and any `render-ops.*.json` outputs to confirm only expected files changed.
- `/vip-report overview`：整理 `slide-source-map.yaml`、`shape-inventory.json`、`template-manifest.json`，确认每页 shape_id/名称。
- `/vip-report data-sync`：按 slide 逐个源头从 Tableau 或 SQL 拉数据并生成 `render-ops.*.json`。
- `/vip-report presentation`：调用 `vip-report-render.ps1` 等脚本把素材替回 `Report.pptx` 并输出比对结果。
- 当前拆出的专题 skill：`/vip-report monthly-sales`、`/vip-report inventory-monthly`、`/vip-report top-products`、`/vip-report campaign-s11`、`/vip-report warehouse-100060`，它们分别对应 `slide-source-map.yaml` 中 S02-S03、S04-S10、S11、S13 等页的来源。
## Sub-skills
## 辅助命令
- `/vip-report overview`
- `/vip-report data-sync`
- `/vip-report presentation`
- Specialized report skills for monthly sales, inventory monthly, top products, campaign S11, and warehouse 100060
- 生成模板 manifest（用于审查 slide 结构）：`powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-manifest.ps1`
- 生成 shape inventory（查 shape_id/名称）：`powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-inventory.ps1`
- 生成基线副本与比对（验证模板/生成文件差异）：`powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-baseline.ps1`
- 生成 S02/S03 素材并渲染：`powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-monthly-sales-sync.ps1 -Render`
## Helper Commands
## 备注
- Manifest: `powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-manifest.ps1`
- Shape inventory: `powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-inventory.ps1`
- Baseline compare: `powershell -ExecutionPolicy Bypass -File C:\Users\niuniu\.codex\skills\vip-report\bin\vip-report-baseline.ps1`
- Tableau 登录凭据只保存在 `config.yaml`，请勿复制到技能文档。
- vip.com 登录（若展开）只需配置 `vip.login_endpoint` 让第三方服务处理。
- 每次渲染完成后务必与模板做 `compare_pptx.py` 比较，确认差异仅限预期替换的图片和 `docProps/*`、`ppt/slides/*.xml`。
## Notes
- Keep Tableau / MySQL / vip.com credentials in config or approved skill docs only.
- Do not silently change business filtering logic when validating report mismatches.
- Always verify generated output against the template before calling the task done.
vip-report/bin/vip-report-campaign-s11-sync.ps1
View file @ 1027cb42
......@@ -13,15 +13,27 @@ param(
[int]$ShopId = 20,
[switch]$DisableStrictJanuaryCheck,
[switch]$Render,
[string]$OutputDir = "",
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-campaign-s11-live.pptx",
[string]$CompareOutputPath = "C:\workspace\cursor\output\vip-report\generated-campaign-s11-live.compare.json"
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$opsPath = "C:\workspace\cursor\output\vip-report\render-ops.campaign-s11.live.json"
$resolvedOutputDir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { "C:\workspace\cursor\output\vip-report" }
$opsPath = Join-Path $resolvedOutputDir "render-ops.campaign-s11.live.json"
$templatePath = "C:\Users\niuniu\Desktop\Report.pptx"
if (-not (Test-Path -LiteralPath $resolvedOutputDir)) {
New-Item -ItemType Directory -Path $resolvedOutputDir -Force | Out-Null
}
if ($OutputDir -and $OutputPath -eq "C:\workspace\cursor\output\vip-report\generated-campaign-s11-live.pptx") {
$OutputPath = Join-Path $resolvedOutputDir "generated-campaign-s11-live.pptx"
}
if ($OutputDir -and $CompareOutputPath -eq "C:\workspace\cursor\output\vip-report\generated-campaign-s11-live.compare.json") {
$CompareOutputPath = Join-Path $resolvedOutputDir "generated-campaign-s11-live.compare.json"
}
$pythonArgs = @(
"$root\scripts\sync_campaign_s11_assets.py",
"--config", "$ConfigPath",
......@@ -40,6 +52,7 @@ if ($CompareYear -gt 0) {
if ($DisableStrictJanuaryCheck) {
$pythonArgs += @("--disable-strict-january-check")
}
$pythonArgs += @("--output-dir", "$resolvedOutputDir")
python @pythonArgs
if ($LASTEXITCODE -ne 0) {
......
vip-report/bin/vip-report-inventory-monthly-sync.ps1
View file @ 1027cb42
......@@ -4,12 +4,18 @@ param(
[string]$ReportMonth = "",
[int]$ReportYear = 0,
[int]$CompareYear = 0,
[string]$OutputDir = "",
[switch]$DisableTemplateLock
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$opsPath = "C:\workspace\cursor\output\vip-report\render-ops.inventory-monthly.live.json"
$resolvedOutputDir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { "C:\workspace\cursor\output\vip-report" }
$opsPath = Join-Path $resolvedOutputDir "render-ops.inventory-monthly.live.json"
if (-not (Test-Path -LiteralPath $resolvedOutputDir)) {
New-Item -ItemType Directory -Path $resolvedOutputDir -Force | Out-Null
}
$pythonArgs = @(
"$root\scripts\sync_inventory_monthly_assets.py",
......@@ -28,6 +34,7 @@ if ($CompareYear -gt 0) {
if ($DisableTemplateLock.IsPresent) {
$pythonArgs += "--disable-template-lock"
}
$pythonArgs += @("--output-dir", "$resolvedOutputDir")
python @pythonArgs
Write-Output $opsPath
vip-report/bin/vip-report-monthly-sales-sync.ps1
View file @ 1027cb42
......@@ -5,16 +5,27 @@ param(
[int]$ReportYear = 0,
[int]$CompareYear = 0,
[switch]$Render,
[string]$OutputDir = "",
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-monthly-sales-live.pptx",
[string]$CompareOutputPath = "C:\workspace\cursor\output\vip-report\generated-monthly-sales-live.compare.json"
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$workdir = "C:\workspace\cursor"
$opsPath = "C:\workspace\cursor\output\vip-report\render-ops.monthly-sales.live.json"
$resolvedOutputDir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { "C:\workspace\cursor\output\vip-report" }
$opsPath = Join-Path $resolvedOutputDir "render-ops.monthly-sales.live.json"
$templatePath = "C:\Users\niuniu\Desktop\Report.pptx"
if (-not (Test-Path -LiteralPath $resolvedOutputDir)) {
New-Item -ItemType Directory -Path $resolvedOutputDir -Force | Out-Null
}
if ($OutputDir -and $OutputPath -eq "C:\workspace\cursor\output\vip-report\generated-monthly-sales-live.pptx") {
$OutputPath = Join-Path $resolvedOutputDir "generated-monthly-sales-live.pptx"
}
if ($OutputDir -and $CompareOutputPath -eq "C:\workspace\cursor\output\vip-report\generated-monthly-sales-live.compare.json") {
$CompareOutputPath = Join-Path $resolvedOutputDir "generated-monthly-sales-live.compare.json"
}
# Forward optional report-period args; config.yaml defaults are used when omitted.
$pythonArgs = @(
"$root\scripts\sync_monthly_sales_assets.py",
......@@ -30,6 +41,7 @@ if ($ReportYear -gt 0) {
if ($CompareYear -gt 0) {
$pythonArgs += @("--compare-year", "$CompareYear")
}
$pythonArgs += @("--output-dir", "$resolvedOutputDir")
python @pythonArgs
if ($Render) {
......
vip-report/bin/vip-report-render.ps1
View file @ 1027cb42
param(
[string]$TemplatePath = "C:\Users\niuniu\Desktop\Report.pptx",
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-from-template.pptx",
[string]$TemplatePath = "",
[string]$OutputPath = "",
[string]$OperationsPath = ""
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
function Resolve-ProjectPath {
param([string]$PathValue)
if (-not $PathValue) {
return $PathValue
}
if ([System.IO.Path]::IsPathRooted($PathValue)) {
return $PathValue
}
return [System.IO.Path]::GetFullPath((Join-Path $root $PathValue))
}
if (-not $TemplatePath) {
$TemplatePath = "Report.pptx"
}
if (-not $OutputPath) {
$OutputPath = "output\generated-from-template.pptx"
}
$TemplatePath = Resolve-ProjectPath $TemplatePath
$OutputPath = Resolve-ProjectPath $OutputPath
$OperationsPath = Resolve-ProjectPath $OperationsPath
powershell -ExecutionPolicy Bypass -File "$root\scripts\render_template_com.ps1" -TemplatePath "$TemplatePath" -OutputPath "$OutputPath" -OperationsPath "$OperationsPath"
vip-report/bin/vip-report-tableau-sync.ps1
View file @ 1027cb42
param(
[string]$ConfigPath = "C:\Users\niuniu\.codex\vip-report\config.yaml",
[string]$Slides = "S02,S03,S04,S05,S06,S07,S08,S09,S10,S13",
[string]$ReportMonth = "",
[string]$ConfigPath = "",
[Parameter(Mandatory = $true)]
[string]$ReportMonth,
[Parameter(Mandatory = $true)]
[string]$Slides,
[int]$ReportYear = 0,
[int]$CompareYear = 0,
[switch]$Render,
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-tableau-all.pptx",
[string]$CompareOutputPath = "C:\workspace\cursor\output\vip-report\generated-tableau-all.compare.json"
[ValidateSet("single", "cumulative")]
[string]$MonthlySalesMode = "single",
[int]$Retries = 3,
[int]$RetryDelaySeconds = 8,
[string]$TemplatePath = "",
[string]$OutputDir = "",
[string]$OutputPath = "",
[switch]$NoRender
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$templatePath = "C:\Users\niuniu\Desktop\Report.pptx"
$mergedOpsPath = "C:\workspace\cursor\output\vip-report\render-ops.tableau.all.live.json"
$outputDir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { Join-Path $root "output" }
$mergedOpsPath = Join-Path $outputDir "render-ops.tableau-selection.live.json"
$commandStartedAt = Get-Date
function Resolve-GroupSlides {
param(
[string[]]$RequestedSlides,
[string[]]$GroupSlides
)
# 按 GroupSlides 的既定顺序过滤，保证输出稳定，避免顺序导致后续差异。
$resolved = ""
foreach ($slide in $GroupSlides) {
if ($RequestedSlides -contains $slide) {
if ($resolved) {
$resolved += ","
}
$resolved += $slide
if (-not $ConfigPath) {
$ConfigPath = Join-Path $root "config.yaml"
}
if (-not $TemplatePath) {
$TemplatePath = Join-Path $root "Report.pptx"
}
if (-not $OutputPath) {
$OutputPath = Join-Path $outputDir "generated-tableau-selection.pptx"
}
if (-not (Test-Path -LiteralPath $ConfigPath)) {
throw "ConfigPath not found: $ConfigPath"
}
if (-not (Test-Path -LiteralPath $TemplatePath) -and -not $NoRender.IsPresent) {
throw "TemplatePath not found: $TemplatePath"
}
if (-not (Test-Path -LiteralPath $outputDir)) {
New-Item -ItemType Directory -Path $outputDir -Force | Out-Null
}
function Resolve-PythonCommand {
if (Get-Command py -ErrorAction SilentlyContinue) {
return @("py", "-3")
}
if (Get-Command python -ErrorAction SilentlyContinue) {
return @("python")
}
return $resolved
throw "Python launcher not found. Please install py or python."
}
$pythonCommand = Resolve-PythonCommand
function Format-Elapsed {
param([TimeSpan]$Duration)
return "{0:N1}s" -f [Math]::Round($Duration.TotalSeconds, 1)
}
function Invoke-SyncScript {
function Write-StageLog {
param(
[string]$Scope,
[string]$Message
)
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
Write-Host ("[{0}] [{1}] {2}" -f $timestamp, $Scope, $Message)
}
function Invoke-PythonScriptWithRetry {
param(
[string]$ScriptPath,
[string]$SyncSlides
[string[]]$ScriptArgs,
[string]$StageName = "",
[string[]]$SlideCodes = @(),
[int]$MaxAttempts = 1,
[int]$SleepSeconds = 8
)
if (-not $SyncSlides) {
$stageLabel = if ($StageName) { $StageName } else { [System.IO.Path]::GetFileNameWithoutExtension($ScriptPath) }
$slideLabel = if ($SlideCodes -and $SlideCodes.Count -gt 0) { $SlideCodes -join "," } else { "-" }
$stageStartedAt = Get-Date
Write-StageLog -Scope "stage" -Message ("start {0} slides={1}" -f $stageLabel, $slideLabel)
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
$attemptStartedAt = Get-Date
Write-StageLog -Scope "stage" -Message ("run {0} attempt={1}/{2} slides={3}" -f $stageLabel, $attempt, $MaxAttempts, $slideLabel)
if ($pythonCommand.Count -eq 2) {
& $pythonCommand[0] $pythonCommand[1] $ScriptPath @ScriptArgs
} else {
& $pythonCommand[0] $ScriptPath @ScriptArgs
}
if ($LASTEXITCODE -eq 0) {
$attemptElapsed = (Get-Date) - $attemptStartedAt
$stageElapsed = (Get-Date) - $stageStartedAt
Write-StageLog -Scope "timing" -Message ("attempt {0} {1}: {2}" -f $attempt, $stageLabel, (Format-Elapsed -Duration $attemptElapsed))
Write-StageLog -Scope "timing" -Message ("stage {0} slides={1}: {2}" -f $stageLabel, $slideLabel, (Format-Elapsed -Duration $stageElapsed))
return
}
$invokeParams = @{
ConfigPath = $ConfigPath
Slides = $SyncSlides
if ($attempt -lt $MaxAttempts) {
$attemptElapsed = (Get-Date) - $attemptStartedAt
Write-StageLog -Scope "timing" -Message ("attempt {0} {1} failed after {2}" -f $attempt, $stageLabel, (Format-Elapsed -Duration $attemptElapsed))
Write-StageLog -Scope "stage" -Message ("retry {0} after {1}s" -f $stageLabel, $SleepSeconds)
Start-Sleep -Seconds $SleepSeconds
}
if ($ReportMonth) {
$invokeParams.ReportMonth = $ReportMonth
}
if ($ReportYear -gt 0) {
$invokeParams.ReportYear = $ReportYear
throw "Script failed after $MaxAttempts attempts: $ScriptPath"
}
function Normalize-SlideCode {
param([string]$SlideRaw)
$text = $SlideRaw.Trim().ToUpper()
if (-not $text) {
return $null
}
if ($CompareYear -gt 0) {
$invokeParams.CompareYear = $CompareYear
if ($text -match "^S?(\d{1,2})$") {
$number = [int]$Matches[1]
if ($number -lt 1 -or $number -gt 99) {
throw "Invalid slide number: $SlideRaw"
}
return ("S{0:d2}" -f $number)
}
throw "Invalid slide token: $SlideRaw"
}
function Get-CodePointKey {
param([string]$Value)
return (($Value.ToCharArray() | ForEach-Object { "{0:X4}" -f [int][char]$_ }) -join "-")
}
function Normalize-ReportMonthArgument {
param([string]$RawMonth)
$text = $RawMonth.Trim()
if (-not $text) {
throw "ReportMonth is required."
}
if ($text -match "^0?([1-9]|1[0-2])$") {
return ([string]([int]$Matches[1]))
}
$monthCodeMap = @{
"4E00-6708" = "1"
"4E8C-6708" = "2"
"4E09-6708" = "3"
"56DB-6708" = "4"
"4E94-6708" = "5"
"516D-6708" = "6"
"4E03-6708" = "7"
"516B-6708" = "8"
"4E5D-6708" = "9"
"5341-6708" = "10"
"5341-4E00-6708" = "11"
"5341-4E8C-6708" = "12"
}
$global:LASTEXITCODE = 0
& $ScriptPath @invokeParams | Out-Null
if ($LASTEXITCODE -ne 0) {
throw "Sync script failed: $ScriptPath (ExitCode=$LASTEXITCODE)"
$codePointKey = Get-CodePointKey -Value $text
if ($monthCodeMap.ContainsKey($codePointKey)) {
return $monthCodeMap[$codePointKey]
}
throw "Invalid ReportMonth: $RawMonth. Use month labels or 1..12."
}
function Resolve-GroupSlides {
param(
[string[]]$RequestedSlides,
[string[]]$GroupSlides
)
$resolved = @()
foreach ($slide in $GroupSlides) {
if ($RequestedSlides -contains $slide) {
$resolved += $slide
}
}
return $resolved
}
function Merge-Operations {
......@@ -67,6 +191,8 @@ function Merge-Operations {
)
$merged = @{
replace_text = @()
replace_tables = @()
replace_charts = @()
replace_images = @()
}
......@@ -74,64 +200,174 @@ function Merge-Operations {
if (-not (Test-Path -LiteralPath $opsPath)) {
continue
}
$ops = Get-Content -LiteralPath $opsPath -Encoding UTF8 | ConvertFrom-Json
$ops = Get-Content -LiteralPath $opsPath -Encoding UTF8 -Raw | ConvertFrom-Json
if ($ops.PSObject.Properties.Name -contains "replace_text") {
$merged.replace_text += @($ops.replace_text)
}
if ($ops.PSObject.Properties.Name -contains "replace_tables") {
$merged.replace_tables += @($ops.replace_tables)
}
if ($ops.PSObject.Properties.Name -contains "replace_charts") {
$merged.replace_charts += @($ops.replace_charts)
}
if ($ops.PSObject.Properties.Name -contains "replace_images") {
$merged.replace_images += @($ops.replace_images)
}
}
$outDir = Split-Path -Parent $OutputOpsPath
if ($outDir -and -not (Test-Path -LiteralPath $outDir)) {
New-Item -ItemType Directory -Path $outDir -Force | Out-Null
}
$json = $merged | ConvertTo-Json -Depth 20
$json = $merged | ConvertTo-Json -Depth 100
$utf8NoBom = New-Object System.Text.UTF8Encoding($false)
[System.IO.File]::WriteAllText($OutputOpsPath, $json, $utf8NoBom)
}
$requestedSlides = @(
$Slides.Split(",") `
| ForEach-Object { $_.Trim().ToUpper() } `
| Where-Object { $_ -ne "" }
)
$reportMonthCliValue = Normalize-ReportMonthArgument -RawMonth $ReportMonth
$normalizedSlides = @()
$seenSlides = @{}
foreach ($raw in ($Slides -split ",")) {
$slideCode = Normalize-SlideCode -SlideRaw $raw
if ($slideCode -and -not $seenSlides.ContainsKey($slideCode)) {
$normalizedSlides += $slideCode
$seenSlides[$slideCode] = $true
}
}
$monthlySlides = Resolve-GroupSlides -RequestedSlides $requestedSlides -GroupSlides @("S02", "S03")
$inventorySlides = Resolve-GroupSlides -RequestedSlides $requestedSlides -GroupSlides @("S04", "S05", "S06", "S07", "S08")
$topSlides = Resolve-GroupSlides -RequestedSlides $requestedSlides -GroupSlides @("S09", "S10")
$warehouseSlides = Resolve-GroupSlides -RequestedSlides $requestedSlides -GroupSlides @("S13")
if (-not $normalizedSlides) {
throw "No slides provided."
}
if (-not $monthlySlides -and -not $inventorySlides -and -not $topSlides -and -not $warehouseSlides) {
throw "No Tableau slides selected. Allowed: S02,S03,S04,S05,S06,S07,S08,S09,S10,S13"
# Supported slide whitelist (including S05 Bags).
$allowedSlides = @("S02", "S03", "S04", "S05", "S06", "S07", "S08", "S09", "S10", "S11", "S13")
$invalidSlides = @($normalizedSlides | Where-Object { $allowedSlides -notcontains $_ })
if ($invalidSlides.Count -gt 0) {
throw "Unsupported slides: $($invalidSlides -join ', '). Allowed: $($allowedSlides -join ', ')"
}
Invoke-SyncScript -ScriptPath "$root\bin\vip-report-monthly-sales-sync.ps1" -SyncSlides $monthlySlides
Invoke-SyncScript -ScriptPath "$root\bin\vip-report-inventory-monthly-sync.ps1" -SyncSlides $inventorySlides
Invoke-SyncScript -ScriptPath "$root\bin\vip-report-top-products-sync.ps1" -SyncSlides $topSlides
Invoke-SyncScript -ScriptPath "$root\bin\vip-report-warehouse-100060-sync.ps1" -SyncSlides $warehouseSlides
$monthlySlides = Resolve-GroupSlides -RequestedSlides $normalizedSlides -GroupSlides @("S02", "S03")
# S05 belongs to inventory group and uses sync_inventory_monthly_assets.py.
$inventorySlides = Resolve-GroupSlides -RequestedSlides $normalizedSlides -GroupSlides @("S04", "S05", "S06", "S07", "S08")
$topSlides = Resolve-GroupSlides -RequestedSlides $normalizedSlides -GroupSlides @("S09", "S10")
$campaignSlides = Resolve-GroupSlides -RequestedSlides $normalizedSlides -GroupSlides @("S11")
$warehouseSlides = Resolve-GroupSlides -RequestedSlides $normalizedSlides -GroupSlides @("S13")
$opsPaths = @()
if ($monthlySlides) {
$opsPaths += "C:\workspace\cursor\output\vip-report\render-ops.monthly-sales.live.json"
}
if ($inventorySlides) {
$opsPaths += "C:\workspace\cursor\output\vip-report\render-ops.inventory-monthly.live.json"
Push-Location $root
try {
if ($monthlySlides.Count -gt 0) {
$args = @(
"--config", $ConfigPath,
"--slides", ($monthlySlides -join ","),
"--report-month", $reportMonthCliValue
)
if ($ReportYear -gt 0) {
$args += @("--report-year", "$ReportYear")
}
if ($CompareYear -gt 0) {
$args += @("--compare-year", "$CompareYear")
}
if ($MonthlySalesMode -eq "single") {
$args += "--single-month"
}
$args += @("--output-dir", $outputDir)
Invoke-PythonScriptWithRetry -ScriptPath (Join-Path $root "scripts\sync_monthly_sales_assets.py") -ScriptArgs $args -StageName "monthly-sales" -SlideCodes $monthlySlides -MaxAttempts $Retries -SleepSeconds $RetryDelaySeconds
$opsPaths += Join-Path $outputDir "render-ops.monthly-sales.live.json"
}
if ($inventorySlides.Count -gt 0) {
$args = @(
"--config", $ConfigPath,
"--slides", ($inventorySlides -join ","),
"--report-month", $reportMonthCliValue
)
if ($ReportYear -gt 0) {
$args += @("--report-year", "$ReportYear")
}
if ($CompareYear -gt 0) {
$args += @("--compare-year", "$CompareYear")
}
$args += @("--output-dir", $outputDir)
$inventorySlideList = @($inventorySlides)
$inventoryRetries = if ($inventorySlideList.Count -eq 1) { 1 } else { $Retries }
Invoke-PythonScriptWithRetry -ScriptPath (Join-Path $root "scripts\sync_inventory_monthly_assets.py") -ScriptArgs $args -StageName "inventory-monthly" -SlideCodes $inventorySlides -MaxAttempts $inventoryRetries -SleepSeconds $RetryDelaySeconds
$opsPaths += Join-Path $outputDir "render-ops.inventory-monthly.live.json"
}
if ($topSlides.Count -gt 0) {
$args = @(
"--config", $ConfigPath,
"--slides", ($topSlides -join ","),
"--report-month", $reportMonthCliValue
)
if ($ReportYear -gt 0) {
$args += @("--report-year", "$ReportYear")
}
if ($CompareYear -gt 0) {
$args += @("--compare-year", "$CompareYear")
}
$args += @("--output-dir", $outputDir)
Invoke-PythonScriptWithRetry -ScriptPath (Join-Path $root "scripts\sync_top_products_assets.py") -ScriptArgs $args -StageName "top-products" -SlideCodes $topSlides -MaxAttempts $Retries -SleepSeconds $RetryDelaySeconds
$opsPaths += Join-Path $outputDir "render-ops.top-products.live.json"
}
if ($campaignSlides.Count -gt 0) {
$args = @(
"--config", $ConfigPath,
"--slides", ($campaignSlides -join ","),
"--report-month", $reportMonthCliValue
)
if ($ReportYear -gt 0) {
$args += @("--report-year", "$ReportYear")
}
if ($CompareYear -gt 0) {
$args += @("--compare-year", "$CompareYear")
}
$args += @("--output-dir", $outputDir)
Invoke-PythonScriptWithRetry -ScriptPath (Join-Path $root "scripts\sync_campaign_s11_assets.py") -ScriptArgs $args -StageName "campaign-s11" -SlideCodes $campaignSlides -MaxAttempts 1 -SleepSeconds $RetryDelaySeconds
$opsPaths += Join-Path $outputDir "render-ops.campaign-s11.live.json"
}
if ($warehouseSlides.Count -gt 0) {
$args = @(
"--config", $ConfigPath,
"--slides", ($warehouseSlides -join ","),
"--report-month", $reportMonthCliValue
)
if ($ReportYear -gt 0) {
$args += @("--report-year", "$ReportYear")
}
if ($CompareYear -gt 0) {
$args += @("--compare-year", "$CompareYear")
}
$args += @("--output-dir", $outputDir)
Invoke-PythonScriptWithRetry -ScriptPath (Join-Path $root "scripts\sync_warehouse_100060_assets.py") -ScriptArgs $args -StageName "warehouse-100060" -SlideCodes $warehouseSlides -MaxAttempts $Retries -SleepSeconds $RetryDelaySeconds
$opsPaths += Join-Path $outputDir "render-ops.warehouse-100060.live.json"
}
}
if ($topSlides) {
$opsPaths += "C:\workspace\cursor\output\vip-report\render-ops.top-products.live.json"
finally {
Pop-Location
}
if ($warehouseSlides) {
$opsPaths += "C:\workspace\cursor\output\vip-report\render-ops.warehouse-100060.live.json"
if ($opsPaths.Count -eq 0) {
throw "No operations generated for slides: $($normalizedSlides -join ',')"
}
Merge-Operations -OpsPaths $opsPaths -OutputOpsPath $mergedOpsPath
Write-StageLog -Scope "stage" -Message ("merged operations -> {0}" -f $mergedOpsPath)
if ($Render) {
powershell -ExecutionPolicy Bypass -File "$root\bin\vip-report-render.ps1" -TemplatePath "$templatePath" -OutputPath "$OutputPath" -OperationsPath "$mergedOpsPath" | Out-Null
python "$root\scripts\compare_pptx.py" "$templatePath" "$OutputPath" --output "$CompareOutputPath"
Write-Output $OutputPath
Write-Output $CompareOutputPath
} else {
if ($NoRender.IsPresent) {
$totalElapsed = (Get-Date) - $commandStartedAt
Write-StageLog -Scope "timing" -Message ("total tableau sync: {0}" -f (Format-Elapsed -Duration $totalElapsed))
Write-Output $mergedOpsPath
exit 0
}
$renderStartedAt = Get-Date
Write-StageLog -Scope "stage" -Message ("start render output={0}" -f $OutputPath)
powershell -ExecutionPolicy Bypass -File (Join-Path $root "bin\vip-report-render.ps1") -TemplatePath $TemplatePath -OutputPath $OutputPath -OperationsPath $mergedOpsPath | Out-Null
if ($LASTEXITCODE -ne 0) {
throw "vip-report-render.ps1 failed with exit code $LASTEXITCODE"
}
$renderElapsed = (Get-Date) - $renderStartedAt
Write-StageLog -Scope "timing" -Message ("stage render: {0}" -f (Format-Elapsed -Duration $renderElapsed))
$totalElapsed = (Get-Date) - $commandStartedAt
Write-StageLog -Scope "timing" -Message ("total tableau sync: {0}" -f (Format-Elapsed -Duration $totalElapsed))
Write-Output $OutputPath
vip-report/bin/vip-report-top-products-sync.ps1
View file @ 1027cb42
param(
[string]$ConfigPath = "C:\Users\niuniu\.codex\vip-report\config.yaml",
[string]$ConfigPath = "",
[string]$Slides = "S09,S10",
[string]$ReportMonth = "",
[int]$ReportYear = 0,
[int]$CompareYear = 0,
[switch]$Render,
[string]$OutputDir = "",
[string]$TemplatePath = "",
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-top-products.pptx",
[string]$CompareOutputPath = "C:\workspace\cursor\output\vip-report\generated-top-products.compare.json"
[string]$OutputPath = "",
[string]$CompareOutputPath = ""
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$workdir = "C:\workspace\cursor"
$opsPath = "C:\workspace\cursor\output\vip-report\render-ops.top-products.live.json"
$fallbackTemplatePath = Join-Path $root "output\Report.clean.pptx"
$workdir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { Join-Path $root "output" }
$opsPath = Join-Path $workdir "render-ops.top-products.live.json"
$workspaceRoot = Split-Path -Parent (Split-Path -Parent $root)
$workspacePython = Join-Path $workspaceRoot "tools\python.exe"
function Resolve-ProjectPath {
param([string]$PathValue)
if (-not $PathValue) {
return $PathValue
}
if ([System.IO.Path]::IsPathRooted($PathValue)) {
return $PathValue
}
return [System.IO.Path]::GetFullPath((Join-Path $root $PathValue))
}
function Resolve-PythonCommand {
if (Test-Path -LiteralPath $workspacePython) {
return $workspacePython
}
$pythonCmd = Get-Command python -ErrorAction SilentlyContinue
if ($pythonCmd) {
return $pythonCmd.Source
}
throw "Python executable not found. Checked PATH and $workspacePython"
}
if (-not $ConfigPath) {
$ConfigPath = "config.yaml"
}
if (-not $TemplatePath) {
$TemplatePath = $fallbackTemplatePath
$TemplatePath = "Report.pptx"
}
if (-not $OutputPath) {
$OutputPath = if ($OutputDir) { Join-Path $workdir "generated-top-products.pptx" } else { "output\generated-top-products.pptx" }
}
if (-not $CompareOutputPath) {
$CompareOutputPath = if ($OutputDir) { Join-Path $workdir "generated-top-products.compare.json" } else { "output\generated-top-products.compare.json" }
}
$ConfigPath = Resolve-ProjectPath $ConfigPath
$TemplatePath = Resolve-ProjectPath $TemplatePath
$OutputPath = Resolve-ProjectPath $OutputPath
$CompareOutputPath = Resolve-ProjectPath $CompareOutputPath
if (-not (Test-Path -LiteralPath $workdir)) {
New-Item -ItemType Directory -Path $workdir -Force | Out-Null
}
if (-not (Test-Path -LiteralPath $ConfigPath)) {
throw "ConfigPath not found: $ConfigPath"
}
if (-not (Test-Path -LiteralPath $TemplatePath)) {
throw "TemplatePath not found: $TemplatePath"
}
$pythonExe = Resolve-PythonCommand
$pythonArgs = @(
"$root\scripts\sync_top_products_assets.py",
"--config", "$ConfigPath",
......@@ -38,15 +89,16 @@ if ($ReportYear -gt 0) {
if ($CompareYear -gt 0) {
$pythonArgs += @("--compare-year", "$CompareYear")
}
$pythonArgs += @("--output-dir", "$workdir")
python @pythonArgs
& $pythonExe @pythonArgs
if ($LASTEXITCODE -ne 0) {
throw "sync_top_products_assets.py failed with exit code $LASTEXITCODE"
}
if ($Render) {
powershell -ExecutionPolicy Bypass -File "$root\bin\vip-report-render.ps1" -TemplatePath "$TemplatePath" -OutputPath "$OutputPath" -OperationsPath "$opsPath" | Out-Null
python "$root\scripts\compare_pptx.py" "$TemplatePath" "$OutputPath" --output "$CompareOutputPath"
& $pythonExe "$root\scripts\compare_pptx.py" "$TemplatePath" "$OutputPath" --output "$CompareOutputPath"
if ($LASTEXITCODE -ne 0) {
throw "compare_pptx.py failed with exit code $LASTEXITCODE"
}
......
vip-report/bin/vip-report-warehouse-100060-sync.ps1
View file @ 1027cb42
......@@ -5,15 +5,25 @@ param(
[int]$ReportYear = 0,
[int]$CompareYear = 0,
[switch]$Render,
[string]$OutputDir = "",
[string]$OutputPath = "C:\workspace\cursor\output\vip-report\generated-warehouse-100060-live.pptx",
[string]$CompareOutputPath = "C:\workspace\cursor\output\vip-report\generated-warehouse-100060-live.compare.json"
)
$ErrorActionPreference = "Stop"
$root = Split-Path -Parent $PSScriptRoot
$workdir = "C:\workspace\cursor"
$opsPath = "C:\workspace\cursor\output\vip-report\render-ops.warehouse-100060.live.json"
$resolvedOutputDir = if ($OutputDir) { [System.IO.Path]::GetFullPath($OutputDir) } else { "C:\workspace\cursor\output\vip-report" }
$opsPath = Join-Path $resolvedOutputDir "render-ops.warehouse-100060.live.json"
$localConfigPath = Join-Path $root "config.yaml"
if (-not (Test-Path -LiteralPath $resolvedOutputDir)) {
New-Item -ItemType Directory -Path $resolvedOutputDir -Force | Out-Null
}
if ($OutputDir -and $OutputPath -eq "C:\workspace\cursor\output\vip-report\generated-warehouse-100060-live.pptx") {
$OutputPath = Join-Path $resolvedOutputDir "generated-warehouse-100060-live.pptx"
}
if ($OutputDir -and $CompareOutputPath -eq "C:\workspace\cursor\output\vip-report\generated-warehouse-100060-live.compare.json") {
$CompareOutputPath = Join-Path $resolvedOutputDir "generated-warehouse-100060-live.compare.json"
}
if (-not (Test-Path -LiteralPath $ConfigPath) -and (Test-Path -LiteralPath $localConfigPath)) {
$ConfigPath = $localConfigPath
}
......@@ -42,6 +52,7 @@ if ($ReportYear -gt 0) {
if ($CompareYear -gt 0) {
$pythonArgs += @("--compare-year", "$CompareYear")
}
$pythonArgs += @("--output-dir", "$resolvedOutputDir")
python @pythonArgs
if ($Render) {
......
vip-report/config.yaml 0 → 100644
View file @ 1027cb42
tableau:
base_url: "http://tableau.charleskeith.cn"
username: "ec_user01"
password: "1!Cu&euRo17b6yu@R40gr7"
vip:
login_endpoint: ""
enabled: false
mysql:
host: "erp.charleskeith.cn"
port: 30004
database: "ckc_cep_db"
username: "allen"
password: "wangjun@123"
paths:
template_pptx: "./Report.pptx"
slide_source_map: "./output/slide-source-map.yaml"
workdir: "./output"
render:
mode: "com"
baseline_output: "./output/generated-baseline.pptx"
manifest_output: "./output/template-manifest.json"
inventory_output: "./output/shape-inventory.json"
compare_output: "./output/baseline-compare.json"
report:
# 月份支持输入中文月份（如：二月）或数字（如：2/02），脚本会统一处理。
month_cn: "一月"
year: 2026
compare_year: 2025
vip-report/data-sync/SKILL.md
View file @ 1027cb42
......@@ -7,11 +7,11 @@ description: Use when fetching or validating VIP report source data from Tableau
## Focus
这个子 skill 负责“拿什么数据”：
This skill covers source-data work before rendering:
- 读取配置中的 Tableau / MySQL / vip.com 登录入口
- 按 `slide-source-map.yaml` 的 source group 拉取数据
- 把中间结果落成可复用的 JSON / CSV / 图片
- read Tableau / MySQL / vip.com endpoints and credentials
- fetch data by slide source group
- persist reusable intermediate JSON / CSV / image outputs
## Inputs
......@@ -24,14 +24,19 @@ description: Use when fetching or validating VIP report source data from Tableau
- `C:\workspace\cursor\output\vip-report\data\*.csv`
- `C:\workspace\cursor\output\vip-report\assets\*.png`
## Current Scope
## Guardrails
- 已确认的非 `vip.com` 页优先
- `S11` 仅实现已确认来源的部分
- `vip.com` 页保留接口位，等 `login_endpoint` 接入后再补
- Read credentials from config or approved skill docs only.
- Every fetched dataset must remain traceable to `slide-source-map.yaml`.
- Default test DB is `ckc_cep_db_test`.
- Default production MySQL for VIP-report validation:
`jdbc:mysql://erp.charleskeith.cn:30004/ckc_cep_db?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true`
username: `allen`
password: `wangjun@123`
- When the user explicitly asks for production verification, use the production MySQL above unless they override it.
## Guardrails
## Current Scope
- 账号密码只从配置文件读取
- 所有数据都要能追溯回 `slide-source-map.yaml`
- 测试库默认使用 `ckc_cep_db_test`
- Tableau-first validation and capture
- MySQL validation for report consistency checks
- `vip.com` kept as an extension point until `login_endpoint` is required
vip-report/render-ops.tableau.all.live.json 0 → 100644
View file @ 1027cb42
{
"replace_images": [
{
"slide": 2,
"shape_id": 5,
"shape_name": "图片 4",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\monthly-sales\\s02_monthly_sales_chart.png"
},
{
"slide": 2,
"shape_id": 7,
"shape_name": "图片 6",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\monthly-sales\\s02_monthly_sales_overview_table.png"
},
{
"slide": 2,
"shape_id": 8,
"shape_name": "图片 7",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\monthly-sales\\s02_monthly_sales_summary_strip.png"
},
{
"slide": 3,
"shape_id": 2,
"shape_name": "图片 1",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\monthly-sales\\s03_kpi_lfl_quad.png"
},
{
"slide": 4,
"shape_id": 2,
"shape_name": "图片 1",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s04_category_overall_top.png"
},
{
"slide": 4,
"shape_id": 4,
"shape_name": "图片 3",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s04_category_overall_mid.png"
},
{
"slide": 4,
"shape_id": 5,
"shape_name": "图片 4",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s04_category_overall_bottom.png"
},
{
"slide": 5,
"shape_id": 2,
"shape_name": "图片 1",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s05_bags_top.png"
},
{
"slide": 5,
"shape_id": 3,
"shape_name": "图片 2",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s05_bags_bottom.png"
},
{
"slide": 6,
"shape_id": 3,
"shape_name": "图片 2",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s06_shoes_top.png"
},
{
"slide": 6,
"shape_id": 7,
"shape_name": "图片 6",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s06_shoes_bottom.png"
},
{
"slide": 7,
"shape_id": 3,
"shape_name": "图片 2",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s07_discount.png"
},
{
"slide": 8,
"shape_id": 2,
"shape_name": "图片 1",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\inventory-monthly\\s08_subcategory.png"
},
{
"slide": 10,
"shape_id": 2,
"shape_name": "Picture 1",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_products_chart.png"
},
{
"slide": 10,
"shape_id": 3,
"shape_name": "Picture 2",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_01_CK1-60361556.png"
},
{
"slide": 10,
"shape_id": 4,
"shape_name": "Picture 2",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_02_CK1-60280377.png"
},
{
"slide": 10,
"shape_id": 6,
"shape_name": "Picture 4",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_03_CK1-60361352.png"
},
{
"slide": 10,
"shape_id": 7,
"shape_name": "Picture 6",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_04_CK1-70381169.png"
},
{
"slide": 10,
"shape_id": 8,
"shape_name": "Picture 8",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_05_CK1-70920197.png"
},
{
"slide": 10,
"shape_id": 9,
"shape_name": "Picture 10",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_06_CK1-61720177.png"
},
{
"slide": 10,
"shape_id": 10,
"shape_name": "Picture 12",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_07_CK1-70381133.png"
},
{
"slide": 10,
"shape_id": 11,
"shape_name": "Picture 14",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_08_CK1-60361613.png"
},
{
"slide": 10,
"shape_id": 13,
"shape_name": "Picture 16",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_09_CK1-60280245-1.png"
},
{
"slide": 10,
"shape_id": 14,
"shape_name": "Picture 18",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\top-products\\s10_top_product_10_CK1-70900382-2.png"
},
{
"slide": 13,
"shape_id": 5,
"shape_name": "图片 4",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\warehouse-100060\\s13_top.png"
},
{
"slide": 13,
"shape_id": 6,
"shape_name": "图片 5",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\warehouse-100060\\s13_left.png"
},
{
"slide": 13,
"shape_id": 10,
"shape_name": "图片 9",
"image_path": "C:\\workspace\\skills\\vip-report\\output\\assets\\warehouse-100060\\s13_mid.png"
}
],
"replace_text": [
]
}
vip-report/scripts/compare_pptx.py
View file @ 1027cb42
......@@ -8,10 +8,12 @@ from pathlib import Path
def sha256_bytes(data: bytes) -> str:
"""函数说明：封装sha256_bytes方法的核心处理流程。"""
return hashlib.sha256(data).hexdigest()
def package_index(path: Path) -> dict[str, str]:
"""函数说明：封装package_index方法的核心处理流程。"""
index: dict[str, str] = {}
with zipfile.ZipFile(path) as archive:
for name in sorted(archive.namelist()):
......@@ -20,6 +22,7 @@ def package_index(path: Path) -> dict[str, str]:
def compare_packages(left: Path, right: Path) -> dict:
"""函数说明：封装compare_packages方法的核心处理流程。"""
left_index = package_index(left)
right_index = package_index(right)
......@@ -44,10 +47,12 @@ def compare_packages(left: Path, right: Path) -> dict:
def main() -> None:
"""函数说明：脚本主入口，负责串联参数解析与执行流程。"""
parser = argparse.ArgumentParser(description="Compare two pptx packages entry-by-entry.")
parser.add_argument("left", help="Left pptx file")
parser.add_argument("right", help="Right pptx file")
parser.add_argument("--output", help="Optional json output path")
# 关键行注释：解析命令行参数，生成本次运行配置。
args = parser.parse_args()
result = compare_packages(Path(args.left), Path(args.right))
......@@ -60,5 +65,6 @@ def main() -> None:
print(payload)
# 关键行注释：脚本直接运行时，从这里进入主流程。
if __name__ == "__main__":
main()
vip-report/scripts/extract_pptx_manifest.py
View file @ 1027cb42
......@@ -17,15 +17,18 @@ NS = {
def natural_slide_number(name: str) -> int:
"""函数说明：封装natural_slide_number方法的核心处理流程。"""
match = re.search(r"(\d+)", name)
return int(match.group(1)) if match else 0
def read_xml(root: Path, relative_path: str) -> etree._ElementTree:
"""函数说明：封装read_xml方法的核心处理流程。"""
return etree.parse(str(root / relative_path))
def rel_map(rel_tree: etree._ElementTree) -> dict[str, dict[str, str]]:
"""函数说明：封装rel_map方法的核心处理流程。"""
mapping: dict[str, dict[str, str]] = {}
for node in rel_tree.xpath("//pr:Relationship", namespaces=NS):
mapping[node.get("Id")] = {
......@@ -36,6 +39,7 @@ def rel_map(rel_tree: etree._ElementTree) -> dict[str, dict[str, str]]:
def chart_workbook_target(root: Path, chart_target: str) -> str | None:
"""函数说明：封装chart_workbook_target方法的核心处理流程。"""
chart_path = Path("ppt/slides") / chart_target
chart_tree = read_xml(root, str(chart_path))
external = chart_tree.xpath("//c:externalData", namespaces=NS)
......@@ -52,6 +56,7 @@ def chart_workbook_target(root: Path, chart_target: str) -> str | None:
def extract_manifest(unpacked_root: Path) -> dict:
"""函数说明：封装extract_manifest方法的核心处理流程。"""
ppt_root = unpacked_root / "ppt"
slides_dir = ppt_root / "slides"
notes_dir = ppt_root / "notesSlides"
......@@ -130,9 +135,11 @@ def extract_manifest(unpacked_root: Path) -> dict:
def main() -> None:
"""函数说明：脚本主入口，负责串联参数解析与执行流程。"""
parser = argparse.ArgumentParser(description="Extract a machine-readable manifest from an unpacked PPTX.")
parser.add_argument("unpacked_root", help="Root directory of the unpacked pptx package")
parser.add_argument("--output", help="Optional output json path. Prints to stdout when omitted.")
# 关键行注释：解析命令行参数，生成本次运行配置。
args = parser.parse_args()
manifest = extract_manifest(Path(args.unpacked_root))
......@@ -145,5 +152,6 @@ def main() -> None:
print(payload)
# 关键行注释：脚本直接运行时，从这里进入主流程。
if __name__ == "__main__":
main()
vip-report/scripts/inspect_template_inventory.ps1
View file @ 1027cb42
......@@ -5,6 +5,8 @@ param(
$ErrorActionPreference = "Stop"
# 关键行注释：这里是关键处理节点。
# 函数说明：封装Get-ShapeNode方法的核心处理流程。
function Get-ShapeNode {
param(
[Parameter(Mandatory = $true)]$Shape,
......@@ -12,6 +14,7 @@ function Get-ShapeNode {
)
$text = $null
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
if ($Shape.HasTextFrame -eq -1 -and $Shape.TextFrame.HasText -eq -1) {
$text = $Shape.TextFrame.TextRange.Text -replace "`r`n", " "
......@@ -24,6 +27,7 @@ function Get-ShapeNode {
$chartCount = 0
$imageCount = 0
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
if ($Shape.HasChart -eq -1) {
$chartCount = 1
......@@ -35,6 +39,7 @@ function Get-ShapeNode {
$imageCount = 1
}
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$groupCount = $Shape.GroupItems.Count
if ($groupCount -gt 0) {
......@@ -49,6 +54,7 @@ function Get-ShapeNode {
}
$hasTable = $false
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$hasTable = ($Shape.HasTable -eq -1)
} catch {
......@@ -77,6 +83,7 @@ function Get-ShapeNode {
$ppt = $null
$pres = $null
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$ppt = New-Object -ComObject PowerPoint.Application
$pres = $ppt.Presentations.Open($TemplatePath, -1, 0, 0)
......@@ -94,6 +101,7 @@ try {
}
$title = $null
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
if ($slide.Shapes.HasTitle -eq -1) {
$title = $slide.Shapes.Title.TextFrame.TextRange.Text -replace "`r`n", " "
......
vip-report/scripts/render_baseline.py
View file @ 1027cb42
......@@ -6,9 +6,11 @@ from pathlib import Path
def main() -> None:
"""函数说明：脚本主入口，负责串联参数解析与执行流程。"""
parser = argparse.ArgumentParser(description="Create a byte-identical baseline render by copying the PPTX template.")
parser.add_argument("template", help="Template pptx path")
parser.add_argument("output", help="Output pptx path")
# 关键行注释：解析命令行参数，生成本次运行配置。
args = parser.parse_args()
template = Path(args.template)
......@@ -18,5 +20,6 @@ def main() -> None:
print(output)
# 关键行注释：脚本直接运行时，从这里进入主流程。
if __name__ == "__main__":
main()
vip-report/scripts/render_template_com.ps1
View file @ 1027cb42
......@@ -6,9 +6,12 @@ param(
$ErrorActionPreference = "Stop"
# 关键行注释：这里是关键处理节点。
# 函数说明：封装Get-ShapeChildren方法的核心处理流程。
function Get-ShapeChildren {
param([Parameter(Mandatory = $true)]$Shape)
$children = @()
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$count = $Shape.GroupItems.Count
for ($i = 1; $i -le $count; $i++) {
......@@ -21,6 +24,8 @@ function Get-ShapeChildren {
return $children
}
# 关键行注释：这里是关键处理节点。
# 函数说明：封装Find-Shape方法的核心处理流程。
function Find-Shape {
param(
[Parameter(Mandatory = $true)]$Slide,
......@@ -46,6 +51,7 @@ function Find-Shape {
}
$text = $null
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
if ($shape.HasTextFrame -eq -1 -and $shape.TextFrame.HasText -eq -1) {
$text = $shape.TextFrame.TextRange.Text
......@@ -66,6 +72,8 @@ function Find-Shape {
return $null
}
# 关键行注释：这里是关键处理节点。
# 函数说明：封装Set-ShapeText方法的核心处理流程。
function Set-ShapeText {
param(
[Parameter(Mandatory = $true)]$Shape,
......@@ -79,6 +87,8 @@ function Set-ShapeText {
$Shape.TextFrame.TextRange.Text = $NewText
}
# 关键行注释：这里是关键处理节点。
# 函数说明：封装Replace-PictureShape方法的核心处理流程。
function Replace-PictureShape {
param(
[Parameter(Mandatory = $true)]$Slide,
......@@ -102,6 +112,7 @@ function Replace-PictureShape {
while ($newShape.ZOrderPosition -lt $z) {
$newShape.ZOrder(1) | Out-Null
}
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$newShape.Name = $name
} catch {
......@@ -159,6 +170,7 @@ if ($replaceText.Count -eq 0 -and $replaceTables.Count -eq 0 -and $replaceCharts
$ppt = $null
$pres = $null
# 关键行注释：进入核心执行区，并在 finally 中释放 PowerPoint 资源。
try {
$ppt = New-Object -ComObject PowerPoint.Application
$pres = $ppt.Presentations.Open($OutputPath, 0, 0, 0)
......
vip-report/scripts/sync_campaign_s11_assets.py
View file @ 1027cb42
......@@ -112,6 +112,7 @@ S11_TABLE2_SHAPE = {
"shape_id": 3,
"shape_name": "表格 2",
}
S11_TABLE2_TOTAL_ROWS = 18
S11_TABLE3_SHAPE = {
"slide": 11,
"shape_id": 4,
......@@ -145,10 +146,12 @@ class ReportPeriod:
@property
def ym(self) -> str:
"""函数说明：封装ym方法的核心处理流程。"""
return f"{self.report_year}-{self.month_num:02d}"
@property
def compare_ym(self) -> str:
"""函数说明：封装compare_ym方法的核心处理流程。"""
return f"{self.compare_year}-{self.month_num:02d}"
......@@ -163,11 +166,12 @@ class CampaignWindow:
@property
def label(self) -> str:
"""函数说明：封装label方法的核心处理流程。"""
return f"{self.current_start.month}.{self.current_start.day}-{self.current_end.month}.{self.current_end.day}"
def normalize_month_label(raw_month: Any) -> str:
"""把 1/01/1月/一月 等输入统一为中文月份，避免 SQL 组装分支膨胀。"""
"""函数说明：封装normalize_month_label方法的核心处理流程。"""
if raw_month is None:
return "一月"
if isinstance(raw_month, int):
......@@ -182,10 +186,12 @@ def normalize_month_label(raw_month: Any) -> str:
def month_label_to_number(month_label: str) -> int:
"""函数说明：封装month_label_to_number方法的核心处理流程。"""
return MONTH_CN_TO_NUM.get(month_label, 1)
def parse_args() -> argparse.Namespace:
"""函数说明：封装parse_args方法的核心处理流程。"""
parser = argparse.ArgumentParser(description="同步 S11 Campaign 数据来源并生成校验产物。")
parser.add_argument(
"--config",
......@@ -225,11 +231,16 @@ def parse_args() -> argparse.Namespace:
action="store_true",
help="关闭 2026-01 模板口径强校验。",
)
parser.add_argument(
"--output-dir",
default="",
help="Override the output/work directory used for generated assets and ops.",
)
return parser.parse_args()
def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> ReportPeriod:
"""周期优先级：CLI > config.yaml > 默认值。"""
"""函数说明：封装resolve_report_period方法的核心处理流程。"""
report_cfg = config.get("report", {})
month_label = normalize_month_label(args.report_month or report_cfg.get("month_cn", "一月"))
month_num = month_label_to_number(month_label)
......@@ -244,6 +255,7 @@ def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> R
def parse_ymd(raw: Any) -> date | None:
"""函数说明：封装parse_ymd方法的核心处理流程。"""
if raw is None:
return None
text = str(raw).strip()
......@@ -256,6 +268,7 @@ def parse_ymd(raw: Any) -> date | None:
def to_decimal(value: Any) -> Decimal:
"""函数说明：封装to_decimal方法的核心处理流程。"""
if value is None:
return Decimal("0")
if isinstance(value, Decimal):
......@@ -270,6 +283,7 @@ def to_decimal(value: Any) -> Decimal:
def to_int(value: Any) -> int:
"""函数说明：封装to_int方法的核心处理流程。"""
if value is None:
return 0
try:
......@@ -279,17 +293,19 @@ def to_int(value: Any) -> int:
def round_half_up(value: Decimal) -> int:
"""函数说明：封装round_half_up方法的核心处理流程。"""
return int(value.quantize(Decimal("1"), rounding=ROUND_HALF_UP))
def safe_ratio(cur: Decimal, prev: Decimal) -> Decimal:
"""函数说明：封装safe_ratio方法的核心处理流程。"""
if prev == 0:
return Decimal("0")
return (cur - prev) / prev
def ratio_string(left: int, right: int) -> str:
"""把两段量转成展示口径（如 11:89），供文案和校验共用。"""
"""函数说明：封装ratio_string方法的核心处理流程。"""
total = left + right
if total <= 0:
return "0:0"
......@@ -299,11 +315,13 @@ def ratio_string(left: int, right: int) -> str:
def ratio_percent_text(cur: Decimal, prev: Decimal) -> str:
"""函数说明：封装ratio_percent_text方法的核心处理流程。"""
pct = round_half_up(safe_ratio(cur, prev) * Decimal("100"))
return f"{pct}%"
def pct_of_total_text(value: Decimal, total: Decimal) -> str:
"""函数说明：封装pct_of_total_text方法的核心处理流程。"""
if total == 0:
return "0%"
pct = round_half_up(value * Decimal("100") / total)
......@@ -311,26 +329,32 @@ def pct_of_total_text(value: Decimal, total: Decimal) -> str:
def format_int_with_comma(value: int) -> str:
"""函数说明：封装format_int_with_comma方法的核心处理流程。"""
return f"{value:,}"
def format_plain_int(value: int) -> str:
"""函数说明：封装format_plain_int方法的核心处理流程。"""
return str(value)
def format_sales_2_with_comma(value: Decimal) -> str:
"""函数说明：封装format_sales_2_with_comma方法的核心处理流程。"""
return f"{value.quantize(Decimal('0.01')):,.2f}"
def format_sales_int_with_comma(value: Decimal) -> str:
"""函数说明：封装format_sales_int_with_comma方法的核心处理流程。"""
return f"{round_half_up(value):,}"
def format_sales_2_plain(value: Decimal) -> str:
"""函数说明：封装format_sales_2_plain方法的核心处理流程。"""
return f"{value.quantize(Decimal('0.01')):.2f}"
def format_sales_trim_plain(value: Decimal) -> str:
"""函数说明：封装format_sales_trim_plain方法的核心处理流程。"""
normalized = value.quantize(Decimal("0.01")).normalize()
text = format(normalized, "f")
if "." in text:
......@@ -339,23 +363,21 @@ def format_sales_trim_plain(value: Decimal) -> str:
def fetch_one(cursor: pymysql.cursors.Cursor, sql: str, params: tuple[Any, ...]) -> dict[str, Any]:
"""函数说明：封装fetch_one方法的核心处理流程。"""
cursor.execute(sql, params)
row = cursor.fetchone() or {}
return dict(row)
def fetch_all(cursor: pymysql.cursors.Cursor, sql: str, params: tuple[Any, ...]) -> list[dict[str, Any]]:
"""函数说明：封装fetch_all方法的核心处理流程。"""
cursor.execute(sql, params)
rows = cursor.fetchall() or []
return [dict(row) for row in rows]
def infer_campaign_window(shop_rows: list[dict[str, Any]], period: ReportPeriod) -> CampaignWindow:
"""从 oms_shop_report 行里推导活动窗口：
- activitybegin 视为当期起始；
- activityend 视为同比期结束；
- 当期结束使用 activityend 的月日 + report_year 重建。
"""
"""函数说明：封装infer_campaign_window方法的核心处理流程。"""
start_candidates = [parse_ymd(item.get("activitybegin")) for item in shop_rows]
end_candidates = [parse_ymd(item.get("activityend")) for item in shop_rows]
starts = [item for item in start_candidates if item is not None]
......@@ -388,8 +410,23 @@ def infer_campaign_window(shop_rows: list[dict[str, Any]], period: ReportPeriod)
)
def filter_shop_rows_by_window(
shop_rows: list[dict[str, Any]],
start_date: date,
end_date: date,
) -> list[dict[str, Any]]:
"""函数说明：按 activitybegin/activityend 精确筛选活动窗口，避免同月混入同比快照。"""
filtered: list[dict[str, Any]] = []
for row in shop_rows:
activity_begin = parse_ymd(row.get("activitybegin"))
activity_end = parse_ymd(row.get("activityend"))
if activity_begin == start_date and activity_end == end_date:
filtered.append(row)
return filtered
def load_s11_narrative_template(workdir: Path) -> str:
"""从 shape-inventory 读取 S11 文案模板；读取失败时回退到内置文案骨架。"""
"""函数说明：封装load_s11_narrative_template方法的核心处理流程。"""
inventory_path = workdir / "shape-inventory.json"
if inventory_path.exists():
try:
......@@ -421,7 +458,7 @@ def rewrite_s11_narrative(
activity_ratio_qty: str,
shoes_sales_lfl_pct: int,
) -> str:
"""按固定槽位替换 S11 文案核心数字，确保版式与语气保持模板风格。"""
"""函数说明：封装rewrite_s11_narrative方法的核心处理流程。"""
text = template_text
text = re.sub(r"销售\d+w", f"销售{sales_10k}w", text, count=1)
......@@ -447,6 +484,7 @@ def rewrite_s11_narrative(
def build_table_cells(rows: list[list[str]]) -> list[dict[str, Any]]:
"""函数说明：封装build_table_cells方法的核心处理流程。"""
cells: list[dict[str, Any]] = []
for row_index, row in enumerate(rows, start=1):
for col_index, value in enumerate(row, start=1):
......@@ -470,6 +508,7 @@ def choose_cn_category_for_detail(
level3_to_cn: dict[str, str],
) -> str:
# 优先级：同款 article 映射 > level3+level4 > level3 > fallback。
"""函数说明：封装choose_cn_category_for_detail方法的核心处理流程。"""
if articlecode in article_to_cn and article_to_cn[articlecode] in TABLE3_CATEGORY_SET:
return article_to_cn[articlecode]
if (level3, level4) in combo_to_cn and combo_to_cn[(level3, level4)] in TABLE3_CATEGORY_SET:
......@@ -483,7 +522,7 @@ def choose_cn_category_for_detail(
def load_font(size: int, *, bold: bool = False) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
"""优先加载系统中文字体，保证导出的 TOP10 图片中文不乱码。"""
"""函数说明：封装load_font方法的核心处理流程。"""
candidates: list[str] = []
if bold:
candidates.extend(
......@@ -508,12 +547,13 @@ def load_font(size: int, *, bold: bool = False) -> ImageFont.FreeTypeFont | Imag
def measure_text(draw: ImageDraw.ImageDraw, text: str, font: ImageFont.ImageFont) -> tuple[int, int]:
"""函数说明：封装measure_text方法的核心处理流程。"""
bbox = draw.textbbox((0, 0), text, font=font)
return bbox[2] - bbox[0], bbox[3] - bbox[1]
def candidate_image_codes(article: str) -> list[str]:
"""按款号生成候选图片编码，优先全码，再尝试去掉短尾缀。"""
"""函数说明：封装candidate_image_codes方法的核心处理流程。"""
article = article.strip()
if not article:
return []
......@@ -527,7 +567,7 @@ def candidate_image_codes(article: str) -> list[str]:
def load_article_image(article: str, *, image_cache_dir: Path) -> Image.Image | None:
"""从 api.charleskeith.cn 下载并缓存商品图，失败则返回 None。"""
"""函数说明：封装load_article_image方法的核心处理流程。"""
image_cache_dir.mkdir(parents=True, exist_ok=True)
for code in candidate_image_codes(article):
cache_path = image_cache_dir / f"{code}.jpg"
......@@ -553,7 +593,7 @@ def load_article_image(article: str, *, image_cache_dir: Path) -> Image.Image |
def render_top10_panel_image(top_rows: list[dict[str, Any]], output_path: Path, *, image_cache_dir: Path) -> None:
"""生成 S11 右侧 TOP10 图块（按当期数据库排序结果）。"""
"""函数说明：封装render_top10_panel_image方法的核心处理流程。"""
width = 1000
height = 1980
margin = 14
......@@ -662,7 +702,7 @@ def render_top10_panel_image(top_rows: list[dict[str, Any]], output_path: Path,
def nice_percent_axis_bounds(values_pct: list[float], *, floor_min: float = -10.0, ceil_max: float = 50.0) -> tuple[float, float]:
"""给 LFL 百分比轴做温和的上下界，保证点位清晰且不被截断。"""
"""函数说明：封装nice_percent_axis_bounds方法的核心处理流程。"""
if not values_pct:
return floor_min, ceil_max
min_v = min(values_pct)
......@@ -682,7 +722,7 @@ def render_lfl_chart_image(
compare_values: list[float],
lfl_values: list[float],
) -> None:
"""生成 S11 的 LFL 图（柱形 + 红点，不连线）。"""
"""函数说明：封装render_lfl_chart_image方法的核心处理流程。"""
categories = ["BAGS", "SHOES"]
x = np.arange(len(categories))
width = 0.32
......@@ -693,7 +733,9 @@ def render_lfl_chart_image(
color_cur = "#6E76E8"
color_prev = "#F1B7DF"
color_lfl = "#FF2D2D"
color_lfl_positive = "#FF2D2D"
color_lfl_negative = "#00B050"
color_lfl_zero = "#4A4A4A"
bars_cur = ax1.bar(x - width / 2, current_values, width, color=color_cur, label="Y-26")
bars_prev = ax1.bar(x + width / 2, compare_values, width, color=color_prev, label="Y-25")
......@@ -718,14 +760,24 @@ def render_lfl_chart_image(
ax2.spines["left"].set_visible(False)
ax2.spines["right"].set_color("#D0D0D0")
# 关键：仅画红点，不画连线。
ax2.scatter(x, pct_values, color=color_lfl, s=28, zorder=6, label="LFL")
lfl_colors = []
for idx, pct in enumerate(pct_values):
if pct > 0:
point_color = color_lfl_positive
elif pct < 0:
point_color = color_lfl_negative
else:
point_color = color_lfl_zero
lfl_colors.append(point_color)
# 关键：仅画点，不画连线；正值红色，负值绿色。
ax2.scatter(x, pct_values, color=lfl_colors, s=28, zorder=6, label="LFL")
for idx, pct in enumerate(pct_values):
label_color = lfl_colors[idx]
ax2.text(
x[idx] + 0.05,
pct,
f"{round(pct):.0f}%",
color="#4A4A4A",
color=label_color,
fontsize=9,
va="center",
ha="left",
......@@ -739,7 +791,120 @@ def render_lfl_chart_image(
legend_handles = [
bars_cur[0],
bars_prev[0],
Line2D([0], [0], marker="o", color="w", markerfacecolor=color_lfl, markersize=6, label="LFL"),
Line2D([0], [0], marker="o", color="w", markerfacecolor=color_lfl_positive, markersize=6, label="LFL"),
]
ax1.legend(
handles=legend_handles,
labels=["Y-26", "Y-25", "LFL"],
loc="lower center",
bbox_to_anchor=(0.5, -0.28),
ncol=3,
frameon=False,
fontsize=8,
handlelength=1.2,
columnspacing=0.8,
)
output_path.parent.mkdir(parents=True, exist_ok=True)
fig.tight_layout()
fig.savefig(output_path, facecolor="white")
plt.close(fig)
def render_lfl_chart_image_with_line(
*,
output_path: Path,
title: str,
current_values: list[float],
compare_values: list[float],
lfl_values: list[float],
) -> None:
"""Render the S11 LFL combo chart with a connector line between BAGS and SHOES."""
categories = ["BAGS", "SHOES"]
x = np.arange(len(categories))
width = 0.32
fig, ax1 = plt.subplots(figsize=(6.5, 3.5), dpi=220)
fig.patch.set_facecolor("white")
ax1.set_facecolor("white")
color_cur = "#6E76E8"
color_prev = "#F1B7DF"
color_lfl_positive = "#FF2D2D"
color_lfl_negative = "#00B050"
color_lfl_zero = "#4A4A4A"
bars_cur = ax1.bar(x - width / 2, current_values, width, color=color_cur, label="Y-26")
bars_prev = ax1.bar(x + width / 2, compare_values, width, color=color_prev, label="Y-25")
ax1.set_xticks(x)
ax1.set_xticklabels(categories, fontsize=9)
ax1.tick_params(axis="y", labelsize=8, colors="#6B6B6B")
ax1.tick_params(axis="x", labelsize=8, colors="#6B6B6B")
ax1.spines["top"].set_visible(False)
ax1.spines["right"].set_visible(False)
ax1.spines["left"].set_color("#D0D0D0")
ax1.spines["bottom"].set_color("#D0D0D0")
ax1.yaxis.grid(False)
ax1.set_title(title, fontsize=12, fontweight="bold", color="#3F3F3F", pad=10)
ax2 = ax1.twinx()
pct_values = [item * 100.0 for item in lfl_values]
lower, upper = nice_percent_axis_bounds(pct_values)
ax2.set_ylim(lower, upper)
ax2.tick_params(axis="y", labelsize=8, colors="#6B6B6B")
ax2.spines["top"].set_visible(False)
ax2.spines["left"].set_visible(False)
ax2.spines["right"].set_color("#D0D0D0")
lfl_colors: list[str] = []
for pct in pct_values:
if pct > 0:
lfl_colors.append(color_lfl_positive)
elif pct < 0:
lfl_colors.append(color_lfl_negative)
else:
lfl_colors.append(color_lfl_zero)
if all(pct > 0 for pct in pct_values):
lfl_line_color = color_lfl_positive
elif all(pct < 0 for pct in pct_values):
lfl_line_color = color_lfl_negative
elif all(pct == 0 for pct in pct_values):
lfl_line_color = color_lfl_zero
else:
lfl_line_color = "#7A7A7A"
ax2.plot(x, pct_values, color=lfl_line_color, linewidth=1.2, zorder=5)
ax2.scatter(x, pct_values, color=lfl_colors, s=28, zorder=6, label="LFL")
for idx, pct in enumerate(pct_values):
ax2.text(
x[idx] + 0.05,
pct,
f"{round(pct):.0f}%",
color=lfl_colors[idx],
fontsize=9,
va="center",
ha="left",
)
ticks = np.linspace(lower, upper, num=7)
ax2.set_yticks(ticks)
ax2.set_yticklabels([f"{int(round(v))}%" for v in ticks])
legend_handles = [
bars_cur[0],
bars_prev[0],
Line2D(
[0],
[0],
marker="o",
color=color_lfl_positive,
markerfacecolor=color_lfl_positive,
markersize=6,
linewidth=1.2,
label="LFL",
),
]
ax1.legend(
handles=legend_handles,
......@@ -760,12 +925,7 @@ def render_lfl_chart_image(
def run_sync(args: argparse.Namespace) -> dict[str, Any]:
"""S11 同步主流程：
1) 拉取 oms_shop_report / oms_daily_report / oms_daily_report_detail；
2) 计算表格、图表、文案所需全部指标；
3) 对一月模板做强校验；
4) 输出 manifest 与 render-ops。
"""
"""函数说明：封装run_sync方法的核心处理流程。"""
config_path = Path(args.config)
config = yaml.safe_load(config_path.read_text(encoding="utf-8"))
period = resolve_report_period(args, config)
......@@ -774,7 +934,7 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
raise SystemExit("当前脚本只支持 S11，请传 --slides S11")
mysql_cfg = config["mysql"]
workdir = Path(config["paths"]["workdir"]).resolve()
workdir = Path(args.output_dir).resolve() if args.output_dir else Path(config["paths"]["workdir"]).resolve()
data_dir = workdir / "data" / "campaign-s11"
data_dir.mkdir(parents=True, exist_ok=True)
......@@ -811,7 +971,30 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
if not shop_rows:
raise SystemExit(f"oms_shop_report 未找到 {period.ym} 数据。")
campaign_window = infer_campaign_window(shop_rows, period)
raw_shop_rows = list(shop_rows)
campaign_window = infer_campaign_window(raw_shop_rows, period)
current_window_shop_rows = filter_shop_rows_by_window(
raw_shop_rows,
campaign_window.current_start,
campaign_window.current_end,
)
if current_window_shop_rows:
shop_rows = current_window_shop_rows
compare_window_shop_rows = filter_shop_rows_by_window(
compare_shop_rows,
campaign_window.compare_start,
campaign_window.compare_end,
)
if not compare_window_shop_rows:
compare_window_shop_rows = filter_shop_rows_by_window(
raw_shop_rows,
campaign_window.compare_start,
campaign_window.compare_end,
)
if compare_window_shop_rows:
compare_shop_rows = compare_window_shop_rows
summary_cur = fetch_one(
cursor,
......@@ -991,8 +1174,12 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
}
)
top_rows.sort(key=lambda item: item["sales"], reverse=True)
top_rows = top_rows[:20]
top_rows_by_sales = sorted(top_rows, key=lambda item: item["sales"], reverse=True)[:20]
top_rows = sorted(
top_rows,
key=lambda item: (item["qty"], item["sales"]),
reverse=True,
)[:20]
top10_rows = top_rows[:10]
asset_dir = workdir / "assets" / "campaign-s11"
......@@ -1243,8 +1430,6 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
pct_of_total_text(ess_sales_val, total_sales),
]
)
else:
table2_rows.append(["", "", "", "", "", ""])
table2_rows.append(
[
level1,
......@@ -1255,6 +1440,9 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
pct_of_total_text(non_sales_val, total_sales),
]
)
if not show_ess_rows:
# Keep TTL on the last visible row of the level1 block when ESS is absent.
table2_rows.append([level1, "", "", "", "", ""])
table2_rows.append(
[
level1,
......@@ -1297,8 +1485,6 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
pct_of_total_text(ess_sales, total_sales),
]
)
else:
table2_rows.append(["", "", "", "", "", ""])
table2_rows.append(
[
"非活动款",
......@@ -1309,6 +1495,9 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
pct_of_total_text(non_activity_only_sales, total_sales),
]
)
if not show_ess_rows:
# Leave the last summary item anchored to the template's bottom row.
table2_rows.append(["", "", "", "", "", ""])
table2_rows.append(
[
"独家款",
......@@ -1319,6 +1508,8 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
pct_of_total_text(exclusive_sales, total_sales),
]
)
while len(table2_rows) < S11_TABLE2_TOTAL_ROWS:
table2_rows.append(["", "", "", "", "", ""])
# 表3（三级分类）
table3_rows = [
......@@ -1342,15 +1533,28 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
]
)
current_table3_total_qty = sum(
Decimal(to_int(current_table3_by_category[name]["qty"])) for name in TABLE3_CATEGORY_ORDER
)
current_table3_total_sales = sum(
to_decimal(current_table3_by_category[name]["sales"]) for name in TABLE3_CATEGORY_ORDER
)
compare_table3_total_qty = sum(
Decimal(to_int(compare_table3_by_category[name]["qty"])) for name in TABLE3_CATEGORY_ORDER
)
compare_table3_total_sales = sum(
to_decimal(compare_table3_by_category[name]["sales"]) for name in TABLE3_CATEGORY_ORDER
)
table3_rows.append(
[
"TTL",
format_plain_int(round_half_up(summary_cur_qty)),
format_plain_int(round_half_up(summary_prev_qty)),
ratio_percent_text(summary_cur_qty, summary_prev_qty),
format_sales_trim_plain(summary_cur_sales),
format_sales_trim_plain(summary_prev_sales),
ratio_percent_text(summary_cur_sales, summary_prev_sales),
format_plain_int(round_half_up(current_table3_total_qty)),
format_plain_int(round_half_up(compare_table3_total_qty)),
ratio_percent_text(current_table3_total_qty, compare_table3_total_qty),
format_sales_trim_plain(current_table3_total_sales),
format_sales_trim_plain(compare_table3_total_sales),
ratio_percent_text(current_table3_total_sales, compare_table3_total_sales),
]
)
......@@ -1400,7 +1604,7 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
]
# 图表在 COM 下直接写 Series 会受限，改为按数据库值生成图片替换。
render_lfl_chart_image(
render_lfl_chart_image_with_line(
output_path=qty_lfl_chart_image_path,
title="QTY LFL",
current_values=[float(current_bags_qty), float(current_shoes_qty)],
......@@ -1410,7 +1614,7 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
float(safe_ratio(Decimal(current_shoes_qty), Decimal(compare_shoes_qty))),
],
)
render_lfl_chart_image(
render_lfl_chart_image_with_line(
output_path=sales_lfl_chart_image_path,
title="SALES LFL",
current_values=[float(current_bags_sales), float(current_shoes_sales)],
......@@ -1513,8 +1717,11 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
for memo, bucket in sorted(by_memo.items(), key=lambda item: item[0])
},
"top10_generated_image": str(top10_image_path),
"top10_articles_by_sales": top10_rows,
"top20_articles_by_sales": top_rows,
"top_rank_basis": "qty_desc_sales_desc",
"top10_articles_by_qty": top10_rows,
"top20_articles_by_qty": top_rows,
"top10_articles_by_sales": top_rows_by_sales[:10],
"top20_articles_by_sales": top_rows_by_sales,
},
"daily_report_summary": {
"current": {
......@@ -1605,10 +1812,12 @@ def run_sync(args: argparse.Namespace) -> dict[str, Any]:
def main() -> None:
"""函数说明：脚本主入口，负责串联参数解析与执行流程。"""
args = parse_args()
result = run_sync(args)
print(json.dumps(result, ensure_ascii=False))
# 关键行注释：脚本直接运行时，从这里进入主流程。
if __name__ == "__main__":
main()
vip-report/scripts/sync_inventory_monthly_assets.py
View file @ 1027cb42
This source diff could not be displayed because it is too large. You can view the blob instead.
vip-report/scripts/sync_monthly_sales_assets.py
View file @ 1027cb42
from __future__ import annotations
from __future__ import annotations
import argparse
import json
......@@ -11,6 +11,7 @@ from typing import Any
from PIL import Image
import yaml
from tableau_export import build_export_image_js, normalize_image_size
NPX_EXECUTABLE = shutil.which("npx.cmd") or shutil.which("npx") or "npx"
......@@ -19,9 +20,7 @@ SESSION_NAME = "vip-report-monthly-sales"
VIEWPORT = {"width": 1400, "height": 3400}
def normalize_month_label(raw_month: Any) -> str:
"""兼容 2/02/2月/二月 等输入，统一转为 Tableau 使用的中文月份。"""
month_cn_map = {
MONTH_CN_MAP = {
1: "一月",
2: "二月",
3: "三月",
......@@ -34,82 +33,160 @@ def normalize_month_label(raw_month: Any) -> str:
10: "十月",
11: "十一月",
12: "十二月",
}
}
ORDERED_MONTHS = [MONTH_CN_MAP[index] for index in range(1, 13)]
CAPTURE_EXECUTION_ORDER = [
# S02 fixed two-step download order requested by product flow:
# 1) Overview (after applying filters) -> download image
# 2) StoreSalesinDetail -> download image
"overview",
"store_sales_detail",
# S03 dependencies keep their natural order afterwards.
"overview_s03",
"store_kpi_lfl",
]
def normalize_month_label(raw_month: Any) -> str:
"""Normalize report month input to the Chinese month label used by the report."""
if raw_month is None:
return "一月"
return MONTH_CN_MAP[1]
if isinstance(raw_month, int):
return month_cn_map.get(raw_month, "一月")
return MONTH_CN_MAP.get(raw_month, MONTH_CN_MAP[1])
text = str(raw_month).strip()
if text in ORDERED_MONTHS:
return text
match = re.fullmatch(r"0?([1-9]|1[0-2])(?:月)?", text)
if match:
return month_cn_map[int(match.group(1))]
return MONTH_CN_MAP[int(match.group(1))]
return text
def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> tuple[str, int, int]:
"""按 CLI > config.yaml > 默认值 的优先级确定报表周期。"""
"""Resolve the report month/year inputs from CLI args and config."""
report_cfg = config.get("report", {})
report_month = normalize_month_label(args.report_month or report_cfg.get("month_cn", "一月"))
report_month = normalize_month_label(args.report_month or report_cfg.get("month_cn", MONTH_CN_MAP[1]))
report_year = int(args.report_year or report_cfg.get("year", 2026))
report_compare_year = int(
args.compare_year or report_cfg.get("compare_year", report_year - 1)
)
report_compare_year = int(args.compare_year or report_cfg.get("compare_year", report_year - 1))
return report_month, report_year, report_compare_year
def log_progress(scope: str, message: str) -> None:
"""Print a compact progress line for long-running stages."""
print(f"[{scope}] {message}", flush=True)
def log_timing(label: str, started_at: float) -> None:
"""Print a compact timing line."""
elapsed_seconds = time.monotonic() - started_at
print(f"[timing] {label}: {elapsed_seconds:.1f}s", flush=True)
def build_capture_slide_codes(filtered_assets: list[dict[str, Any]]) -> dict[str, list[str]]:
"""Map each capture_id to the slide codes that depend on it."""
capture_slide_codes: dict[str, set[str]] = {}
for asset in filtered_assets:
capture_slide_codes.setdefault(asset["capture_id"], set()).add(asset["slide_code"])
return {capture_id: sorted(slide_codes) for capture_id, slide_codes in capture_slide_codes.items()}
def describe_capture(
capture_id: str,
capture_spec: dict[str, Any],
capture_slide_codes: dict[str, list[str]],
) -> str:
"""Build a readable page label for logging."""
slide_codes = "/".join(capture_slide_codes.get(capture_id, [])) or "-"
return f"{slide_codes} {capture_id} ({capture_spec['activate_sheet']})"
def build_cumulative_month_values(report_month: str) -> list[str]:
"""S02 采用累计口径：二月=一月+二月，三月=一月+二月+三月。"""
ordered_months = [
"一月",
"二月",
"三月",
"四月",
"五月",
"六月",
"七月",
"八月",
"九月",
"十月",
"十一月",
"十二月",
"""Build cumulative month labels up to the report month."""
if report_month not in ORDERED_MONTHS:
return [report_month]
index = ORDERED_MONTHS.index(report_month)
return ORDERED_MONTHS[: index + 1]
def resolve_month_index(report_month: str) -> int:
"""Resolve the normalized month label into a month index."""
normalized = normalize_month_label(report_month)
if normalized not in ORDERED_MONTHS:
return 1
return ORDERED_MONTHS.index(normalized) + 1
def build_month_value_variants(report_month: str, *, cumulative: bool) -> list[list[str]]:
"""Build robust Tableau month filter value candidates."""
month_index = resolve_month_index(report_month)
month_indexes = list(range(1, month_index + 1)) if cumulative else [month_index]
month_labels = [normalize_month_label(index) for index in month_indexes]
return [
[str(index) for index in month_indexes],
[f"{index:02d}" for index in month_indexes],
month_labels,
]
if report_month not in ordered_months:
def resolve_month_values(report_month: str, month_mode: str) -> list[str]:
"""Resolve single-month or cumulative month filters."""
if month_mode == "single":
return [report_month]
index = ordered_months.index(report_month)
return ordered_months[: index + 1]
return build_cumulative_month_values(report_month)
def build_specs(report_month: str, report_year: int, compare_year: int) -> dict[str, Any]:
"""集中维护 S02/S03 的抓取口径、裁切参数与拼图布局。"""
s02_month_values = build_cumulative_month_values(report_month)
# S02 底部细条表需要展示「总和 + 累计月份」，按月份数动态放大裁切高度。
summary_strip_height = min(140, max(86, 68 + len(s02_month_values) * 18))
def build_specs(report_month: str, report_year: int, compare_year: int, month_mode: str) -> dict[str, Any]:
"""Build the Tableau capture and crop specs for monthly sales slides."""
# S02/S03 monthly sales always use cumulative months from January to the report month.
cumulative_month_value_candidates = build_month_value_variants(report_month, cumulative=True)
s03_month_values = build_cumulative_month_values(report_month)
summary_strip_crop = {"left": 0, "top": 528, "width": 1383, "height": 332}
common_monthly_filters = [
{
"label": "year",
"field_candidates": ["Year", "年(daily)"],
"value_candidates": [[str(report_year)]],
"apply_all_fields": True,
"target_scope": "active_sheet",
},
{
"label": "month",
"field_candidates": ["Month", "月(daily)"],
"value_candidates": cumulative_month_value_candidates,
"apply_all_fields": True,
"target_scope": "active_sheet",
},
{
"label": "store",
"field_candidates": ["store", "Storename (group)", "Storename"],
"value_candidates": [["CKC-VIP"], ["ckc-vip"]],
"apply_all_fields": True,
"target_scope": "active_sheet",
},
{"field": "Sales Type", "values": ["GMV"], "target_scope": "active_sheet"},
{"field": "Brand", "values": ["CK"], "target_scope": "active_sheet"},
]
# S02 overview needs both current-year and comparison-year measures, so it must not
# apply a hard Year filter that would exclude year2-backed values like Sales2/LFL.
s02_overview_filters = [item for item in common_monthly_filters if item.get("label") != "year"]
return {
"captures": [
{
"capture_id": "overview",
"session": SESSION_NAME,
"hash_url": "#/views/ECMonthlySalesReport_17250004228820/Overview?:iid=1",
"hash_url": "#/views/ECMonthlySalesReport_17250004228820/Overview?:iid=2",
"inner_frame_fragment": "/views/ECMonthlySalesReport_17250004228820/Overview?",
"activate_sheet": "Overview",
"filters": [
{"field": "Storename (group)", "values": ["CKC-VIP"]},
# 该字段决定 overall performance 是否收敛到单店，缺失会导致整体布局下沉。
{"field": "Storename", "values": ["CKC-VIP"]},
{"field": "月(daily)", "values": s02_month_values},
{"field": "Sales Type", "values": ["GMV"]},
{"field": "Brand", "values": ["CK"]},
],
"params": {"year1": report_year, "year2": compare_year},
"filters": [*s02_overview_filters],
"params": {},
"raw_screenshot_name": "monthly-sales-overview.png",
"note": "Overview 页面，负责 S02 主表与优先级最高的总览图块。",
"note": "S02 overview dashboard export.",
},
{
"capture_id": "overview_s03",
"session": SESSION_NAME,
"hash_url": "#/views/ECMonthlySalesReport_17250004228820/Overview?:iid=1",
"hash_url": "#/views/ECMonthlySalesReport_17250004228820/Overview?:iid=2",
"inner_frame_fragment": "/views/ECMonthlySalesReport_17250004228820/Overview?",
"activate_sheet": "Overview",
"filters": [
......@@ -117,12 +194,12 @@ def build_specs(report_month: str, report_year: int, compare_year: int) -> dict[
{"field": "Sales Type", "values": ["GMV"]},
{"field": "Storename (group)", "values": ["CKC-VIP"]},
{"field": "Storename", "values": ["CKC-VIP"]},
{"field": "Month", "values": s02_month_values},
{"field": "月(daily)", "values": s02_month_values},
{"field": "Month", "values": s03_month_values},
{"field": "月(daily)", "values": s03_month_values},
],
"params": {"year1": report_year, "year2": compare_year},
"raw_screenshot_name": "monthly-sales-overview-s03.png",
"note": "Overview 页面，负责 S03 四宫格的真实来源。",
"note": "S03 overview dashboard export.",
},
{
"capture_id": "store_sales_detail",
......@@ -130,20 +207,10 @@ def build_specs(report_month: str, report_year: int, compare_year: int) -> dict[
"hash_url": "#/views/ECMonthlySalesReport_17250004228820/StoreSalesinDetail?:iid=2",
"inner_frame_fragment": "/views/ECMonthlySalesReport_17250004228820/StoreSalesinDetail?",
"activate_sheet": "Store Sales in Detail",
"filters": [
{"field": "Storename (group)", "values": ["CKC-VIP"]},
{"field": "Storename", "values": ["CKC-VIP"]},
# 该页不同 worksheet 的字段命名不完全一致，双写保证月份/年份过滤可命中。
{"field": "Month", "values": s02_month_values},
{"field": "月(daily)", "values": s02_month_values},
{"field": "Sales Type", "values": ["GMV"]},
{"field": "Year", "values": [str(report_year)]},
{"field": "年(daily)", "values": [str(report_year)]},
{"field": "Brand", "values": ["CK"]},
],
"filters": [*common_monthly_filters],
"params": {},
"raw_screenshot_name": "monthly-sales-store-sales-detail.png",
"note": "Store Sales in Detail 页面，负责 S02 底部细条表。",
"note": "S02 detail dashboard export.",
},
{
"capture_id": "store_kpi_lfl",
......@@ -157,89 +224,87 @@ def build_specs(report_month: str, report_year: int, compare_year: int) -> dict[
{"field": "Storename", "values": ["CKC-VIP"]},
{"field": "Year", "values": [str(report_year)]},
{"field": "Month", "values": [report_month]},
{"field": "月(daily)", "values": [report_month]},
],
"params": {"year1": report_year, "year2": compare_year},
"raw_screenshot_name": "monthly-sales-store-kpi-lfl.png",
"note": "Store KPI LFL 页面，负责 S03 四宫格的真实来源。",
"note": "S03 store KPI export.",
},
],
"assets": [
{
"slide_code": "S02",
"slide": 2,
"shape_name": "图片 4",
"shape_name": "鍥剧墖 4",
"shape_id": 5,
"asset_name": "s02_monthly_sales_chart",
"capture_id": "overview",
# 顶图需要排除 year1/year2/month 筛选条，仅保留 Sales 图块与行标签。
"crop": {"left": 8, "top": 520, "width": 1224, "height": 360},
"crop": {"left": 12, "top": 510, "width": 1360, "height": 383},
"resize_to": {"width": 1106, "height": 319},
"source_view": "Overview",
"note": "S02 顶部图块，优先沿用 Overview 图形布局。",
"note": "S02 top chart.",
},
{
"slide_code": "S02",
"slide": 2,
"shape_name": "图片 6",
"shape_name": "鍥剧墖 6",
"shape_id": 7,
"asset_name": "s02_monthly_sales_overview_table",
"capture_id": "overview",
# 中部主表向下微调，去掉上方筛选控件边缘线。
"crop": {"left": 8, "top": 190, "width": 1224, "height": 300},
"crop": {"left": 55, "top": 140, "width": 1330, "height": 311},
"resize_to": {"width": 1097, "height": 278},
"source_view": "Overview",
"note": "S02 中部主表，来自 Overview 顶部 KPI 总览。",
"note": "S02 middle table.",
},
{
"slide_code": "S02",
"slide": 2,
"shape_name": "图片 7",
"shape_name": "鍥剧墖 7",
"shape_id": 8,
"asset_name": "s02_monthly_sales_summary_strip",
"capture_id": "store_sales_detail",
# 目标比例对齐模板 shape(1032.66 x 63.48)，尽量减少纵向压缩感。
"crop": {"left": 0, "top": 150, "width": 1400, "height": summary_strip_height},
"resize_to": {"width": 1360, "height": 84},
"crop": {"left": 0, "top": 460, "width": 1380, "height": 354},
"resize_to": {"width": 1155, "height": 71},
"source_view": "Store Sales in Detail",
"note": "S02 底部细条表，改为使用备注中的 Store Sales in Detail 真实页面。",
"note": "S02 summary strip.",
},
{
"slide_code": "S03",
"slide": 3,
"shape_name": "图片 1",
"shape_name": "鍥剧墖 1",
"shape_id": 2,
"asset_name": "s03_kpi_lfl_quad",
"capture_id": "overview_s03",
"source_view": "Overview",
"note": "S03 从 Overview 抽取 GP/Con/ATV/Returnqty 四块后重组，贴齐模板四宫格顺序。",
"note": "S03 four-panel composite.",
"composite": {
"canvas": {"width": 1224, "height": 685, "background": "#FFFFFF"},
"canvas": {"width": 1110, "height": 621, "background": "#FFFFFF"},
"panels": [
{
"panel_code": "gp",
"crop": {"left": 0, "top": 1540, "width": 620, "height": 370},
"dest": {"left": 0, "top": 0, "width": 598, "height": 328},
"crop": {"left": 0, "top": 1610, "width": 650, "height": 400},
"dest": {"left": 0, "top": 0, "width": 542, "height": 297},
},
{
"panel_code": "con",
"crop": {"left": 620, "top": 1540, "width": 620, "height": 370},
"dest": {"left": 626, "top": 0, "width": 598, "height": 328},
"crop": {"left": 650, "top": 1610, "width": 650, "height": 400},
"dest": {"left": 568, "top": 0, "width": 542, "height": 297},
},
{
"panel_code": "atv",
"crop": {"left": 0, "top": 1930, "width": 620, "height": 330},
"dest": {"left": 0, "top": 357, "width": 598, "height": 328},
"crop": {"left": 0, "top": 2000, "width": 650, "height": 360},
"dest": {"left": 0, "top": 324, "width": 542, "height": 297},
},
{
"panel_code": "returnqty",
"crop": {"left": 620, "top": 1930, "width": 620, "height": 330},
"dest": {"left": 626, "top": 357, "width": 598, "height": 328},
"crop": {"left": 650, "top": 2000, "width": 650, "height": 360},
"dest": {"left": 568, "top": 324, "width": 542, "height": 297},
},
],
},
},
],
}
def run_cmd(
args: list[str],
*,
......@@ -247,7 +312,7 @@ def run_cmd(
timeout: int = 120,
check: bool = True,
) -> subprocess.CompletedProcess[str]:
"""统一封装外部命令调用，保留 UTF-8 输出便于后续排查。"""
"""Helper."""
return subprocess.run(
args,
cwd=str(cwd),
......@@ -261,20 +326,20 @@ def run_cmd(
def run_playwright(args: list[str], *, cwd: Path, timeout: int = 120) -> str:
"""执行 Playwright CLI，并返回标准输出。"""
"""Helper."""
result = run_cmd(PLAYWRIGHT_CMD + args, cwd=cwd, timeout=timeout)
return result.stdout
def write_js(workdir: Path, name: str, content: str) -> Path:
"""把临时 JS 落盘，供 playwright-cli run-code 调用。"""
"""Helper."""
path = workdir / name
path.write_text(content, encoding="utf-8")
return path
def build_login_js(username: str, password: str) -> str:
"""构造 Tableau 登录脚本，只在落到 signin 页时触发输入。"""
"""Helper."""
payload = {
"username": username,
"password": password,
......@@ -304,7 +369,7 @@ def build_login_js(username: str, password: str) -> str:
def build_configure_view_js(spec: dict[str, Any]) -> str:
"""激活目标 sheet，并把同页所有 worksheet 都打上过滤条件。"""
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
......@@ -327,7 +392,7 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
if (activeSheet && typeof activeSheet.getWorksheets === 'function') {{
worksheets = activeSheet.getWorksheets();
}}
// 同时覆盖 dashboard 本体与内部 worksheets，避免部分字段只在其一上生效。
// 鍚屾椂瑕嗙洊 dashboard 鏈綋涓庡唴閮?worksheets锛岄伩鍏嶉儴鍒嗗瓧娈靛彧鍦ㄥ叾涓€涓婄敓鏁堛€?
const targets = [];
if (activeSheet) {{
targets.push(activeSheet);
......@@ -384,8 +449,256 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
"""
def build_configure_view_js_v2(spec: dict[str, Any]) -> str:
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
await page.waitForLoadState('domcontentloaded').catch(() => null);
await page.waitForTimeout(3000);
async function inspectFrame(frame, preferredFragment) {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : null;
try {{
const snapshot = await frame.evaluate(() => {{
const vizCount = window.tableau?.VizManager?.getVizs?.().length || 0;
return {{
url: location.href,
title: document.title,
readyState: document.readyState,
hasTableau: !!(window.tableau && window.tableau.VizManager),
vizCount,
}};
}});
return {{
...snapshot,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
}};
}} catch (error) {{
return {{
url: frameUrl,
title: null,
readyState: null,
hasTableau: false,
vizCount: 0,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
error: String(error && error.message || error),
}};
}}
}}
async function locateVizFrame(timeoutMs, preferredFragment) {{
const deadline = Date.now() + timeoutMs;
let lastSnapshots = [];
while (Date.now() < deadline) {{
const rankedFrames = page.frames()
.map((frame) => {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
const isPreferred = !!(preferredFragment && frameUrl.includes(preferredFragment));
return {{
frame,
rank: isPreferred ? 0 : (frame === page.mainFrame() ? 1 : 2),
}};
}})
.sort((left, right) => left.rank - right.rank);
lastSnapshots = [];
for (const item of rankedFrames) {{
const snapshot = await inspectFrame(item.frame, preferredFragment);
lastSnapshots.push(snapshot);
if (snapshot.vizCount > 0) {{
return {{ frame: item.frame, snapshots: lastSnapshots }};
}}
}}
const stillOnSignin =
(page.url() || '').includes('/#/signin') ||
lastSnapshots.some((snapshot) => (snapshot.url || '').includes('/#/signin'));
if (stillOnSignin) {{
throw new Error('Tableau page is still on sign-in: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
await page.waitForTimeout(2000);
}}
throw new Error('Timed out waiting for Tableau viz: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
const located = await locateVizFrame(Number(spec.tableau_wait_ms || 90000), spec.inner_frame_fragment || '');
const result = await located.frame.evaluate(async (config) => {{
const sleep = (ms) => new Promise((resolve) => window.setTimeout(resolve, ms));
const deadline = Date.now() + Number(config.workbook_wait_ms || 60000);
let viz = null;
let workbook = null;
while (Date.now() < deadline) {{
try {{
const vizs = window.tableau?.VizManager?.getVizs?.() || [];
viz = vizs[0] || null;
workbook = viz && typeof viz.getWorkbook === 'function' ? viz.getWorkbook() : null;
}} catch (error) {{
workbook = null;
}}
if (workbook) {{
break;
}}
await sleep(500);
}}
if (!workbook) {{
throw new Error('Tableau workbook is not ready: ' + JSON.stringify({{
url: location.href,
title: document.title,
readyState: document.readyState,
vizCount: window.tableau?.VizManager?.getVizs?.().length || 0,
}}));
}}
try {{
await workbook.revertAllAsync();
}} catch (error) {{
}}
await workbook.activateSheetAsync(config.activate_sheet);
const activeSheet = workbook.getActiveSheet();
let worksheets = [];
if (activeSheet && typeof activeSheet.getWorksheets === 'function') {{
worksheets = activeSheet.getWorksheets();
}}
const targets = [];
if (activeSheet) {{
targets.push(activeSheet);
}}
for (const worksheet of worksheets) {{
if (!targets.includes(worksheet)) {{
targets.push(worksheet);
}}
}}
const updateType = window.tableau.FilterUpdateType.REPLACE;
const filterApply = [];
function normalizeWorksheetName(name) {{
return String(name || '').trim().toLowerCase();
}}
function resolveSelectedTargets(filter) {{
let selectedTargets =
filter.target_scope === 'active_sheet' && activeSheet ? [activeSheet] : targets;
const worksheetNames = Array.isArray(filter.worksheet_names) ? filter.worksheet_names : [];
if (worksheetNames.length > 0) {{
const allowedWorksheetNames = new Set(
worksheetNames.map((name) => normalizeWorksheetName(name)).filter(Boolean)
);
selectedTargets = selectedTargets.filter((worksheet) => {{
const worksheetName = typeof worksheet.getName === 'function' ? worksheet.getName() : '';
return allowedWorksheetNames.has(normalizeWorksheetName(worksheetName));
}});
}}
return selectedTargets;
}}
async function applyFilterCandidate(field, values, selectedTargets) {{
let anySuccess = false;
for (const worksheet of targets) {{
const worksheetName = typeof worksheet.getName === 'function' ? worksheet.getName() : 'unknown';
try {{
if (!selectedTargets.includes(worksheet)) {{
continue;
}}
await worksheet.applyFilterAsync(field, values, updateType);
anySuccess = true;
filterApply.push({{
field,
values,
worksheet: worksheetName,
ok: true,
}});
}} catch (error) {{
filterApply.push({{
field,
values,
worksheet: worksheetName,
ok: false,
error: String(error && error.message || error),
}});
}}
}}
return anySuccess;
}}
for (const filter of config.filters || []) {{
const fieldCandidates = filter.field_candidates || (filter.field ? [filter.field] : []);
const valueCandidates = filter.value_candidates || (filter.values ? [filter.values] : []);
const applyAllFields = !!filter.apply_all_fields;
const selectedTargets = resolveSelectedTargets(filter);
let matched = false;
for (const field of fieldCandidates) {{
let fieldMatched = false;
for (const values of valueCandidates) {{
const result = await applyFilterCandidate(field, values, selectedTargets);
if (result) {{
fieldMatched = true;
matched = true;
break;
}}
}}
if (fieldMatched && !applyAllFields) {{
break;
}}
}}
if (!matched) {{
filterApply.push({{
field: filter.label || fieldCandidates.join('/'),
worksheet: null,
ok: false,
required: filter.required !== false,
}});
if (filter.required !== false) {{
throw new Error(`Required filter '${{filter.label || fieldCandidates.join('/')}}' did not match any worksheet.`);
}}
}}
}}
for (const [name, value] of Object.entries(config.params)) {{
try {{
await workbook.changeParameterValueAsync(name, value);
}} catch (error) {{
}}
}}
await sleep(Number(config.post_filter_wait_ms || 10000));
return {{
activeSheet: activeSheet.getName(),
targetCount: targets.length,
filterApply,
filters: config.filters,
params: config.params,
url: location.href,
}};
}}, {{
activate_sheet: spec.activate_sheet,
filters: spec.filters,
params: spec.params,
workbook_wait_ms: spec.workbook_wait_ms || 60000,
post_filter_wait_ms: spec.post_filter_wait_ms || 10000,
}});
return {{
...result,
frameSnapshots: located.snapshots,
}};
}}
"""
def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
"""优先截取 workbook 内层 frame；找不到时退回主页面 body。"""
"""Helper."""
payload = json.dumps(
{
"inner_frame_fragment": inner_frame_fragment,
......@@ -415,6 +728,7 @@ def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
def run_code(session: str, script_path: Path, *, cwd: Path, timeout: int = 120) -> str:
"""Helper."""
return run_playwright(
["--session", session, "run-code", "--filename", str(script_path)],
cwd=cwd,
......@@ -423,10 +737,12 @@ def run_code(session: str, script_path: Path, *, cwd: Path, timeout: int = 120)
def ensure_browser_session(session: str, *, cwd: Path) -> None:
"""Helper."""
run_playwright(["--session", session, "open", "about:blank"], cwd=cwd, timeout=60)
def save_state(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
state_path.parent.mkdir(parents=True, exist_ok=True)
run_playwright(
["--session", session, "state-save", str(state_path)],
......@@ -436,6 +752,7 @@ def save_state(session: str, state_path: Path, *, cwd: Path) -> None:
def load_state_if_present(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
if not state_path.exists():
return
run_playwright(
......@@ -446,7 +763,7 @@ def load_state_if_present(session: str, state_path: Path, *, cwd: Path) -> None:
def locate_session_file(root: Path, session: str, filename: str) -> Path:
"""Playwright 产物可能落在多个目录，这里按最近修改时间兜底定位。"""
"""Helper."""
direct = root / "output" / "playwright" / session / filename
if direct.exists():
return direct
......@@ -476,7 +793,7 @@ def crop_image(
*,
resize_to: dict[str, int] | None = None,
) -> None:
"""裁切图片；如指定 resize_to，则先裁后缩放到固定输出比例。"""
"""Helper."""
image = Image.open(source)
box = (
crop["left"],
......@@ -496,7 +813,7 @@ def compose_panels(
target: Path,
composite: dict[str, Any],
) -> list[dict[str, Any]]:
"""把 Store KPI LFL 中的多块 panel 重新排成模板所需的四宫格。"""
"""Helper."""
image = Image.open(source).convert("RGB")
canvas_spec = composite["canvas"]
canvas = Image.new("RGB", (canvas_spec["width"], canvas_spec["height"]), canvas_spec.get("background", "#FFFFFF"))
......@@ -528,6 +845,7 @@ def compose_panels(
def parse_args() -> argparse.Namespace:
"""Helper."""
parser = argparse.ArgumentParser(description="Capture monthly sales Tableau assets for VIP report slides S02/S03.")
parser.add_argument(
"--config",
......@@ -542,7 +860,7 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--report-month",
default="",
help="Report month label for Tableau filter, e.g. 二月 / 2 / 02",
help="Report month label for Tableau filter, e.g. 浜屾湀 / 2 / 02",
)
parser.add_argument(
"--report-year",
......@@ -556,14 +874,61 @@ def parse_args() -> argparse.Namespace:
default=0,
help="Comparison year for year2 parameter, e.g. 2025",
)
parser.add_argument(
"--month-mode",
choices=["cumulative", "single"],
default="cumulative",
help="Month filter mode: cumulative (default) or single",
)
parser.add_argument(
"--single-month",
action="store_true",
help="Shortcut for --month-mode single",
)
parser.add_argument(
"--output-dir",
default="",
help="Override the output/work directory used for generated assets and ops.",
)
return parser.parse_args()
def collect_required_capture_ids(filtered_assets: list[dict[str, Any]]) -> set[str]:
"""只抓当前请求页真正依赖的 view，避免无关跳页。"""
"""Helper."""
return {asset["capture_id"] for asset in filtered_assets}
def build_capture_execution_order(required_capture_ids: set[str]) -> list[str]:
"""Build deterministic capture order with S02 two-step download first."""
ordered = [capture_id for capture_id in CAPTURE_EXECUTION_ORDER if capture_id in required_capture_ids]
remaining = sorted(required_capture_ids - set(ordered))
return ordered + remaining
def build_tableau_target_url(base_url: str, hash_url: str) -> str:
"""Build Tableau hash-view URL in canonical /#/views/... style."""
normalized_hash_url = hash_url.strip()
if normalized_hash_url.startswith("#"):
return f"{base_url.rstrip('/')}/{normalized_hash_url}"
return f"{base_url.rstrip('/')}/{normalized_hash_url.lstrip('/')}"
def goto_with_retry(session: str, target_url: str, *, cwd: Path, attempts: int = 3) -> None:
"""Helper."""
last_error: subprocess.CalledProcessError | None = None
for attempt in range(1, attempts + 1):
try:
run_playwright(["--session", session, "goto", target_url], cwd=cwd, timeout=120)
return
except subprocess.CalledProcessError as error:
last_error = error
if attempt >= attempts:
raise
time.sleep(2)
if last_error is not None:
raise last_error
def capture_tableau_view(
capture_spec: dict[str, Any],
*,
......@@ -572,9 +937,9 @@ def capture_tableau_view(
workdir: Path,
workspace_root: Path,
) -> Path:
"""按单个 capture spec 完成跳转、过滤和截图。"""
target_url = f"{base_url}{capture_spec['hash_url']}"
run_playwright(["--session", session, "goto", target_url], cwd=workdir, timeout=120)
"""Helper."""
target_url = build_tableau_target_url(base_url, capture_spec["hash_url"])
goto_with_retry(session, target_url, cwd=workdir)
run_playwright(
["--session", session, "resize", str(VIEWPORT["width"]), str(VIEWPORT["height"])],
cwd=workdir,
......@@ -584,20 +949,22 @@ def capture_tableau_view(
configure_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-configure.js",
build_configure_view_js(
build_configure_view_js_v2(
{
"activate_sheet": capture_spec["activate_sheet"],
"filters": capture_spec["filters"],
"params": capture_spec["params"],
"inner_frame_fragment": capture_spec["inner_frame_fragment"],
}
),
)
capture_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-capture.js",
build_capture_js(
build_export_image_js(
capture_spec["inner_frame_fragment"],
capture_spec["raw_screenshot_name"],
str((workdir / capture_spec["raw_screenshot_name"]).resolve()),
viewport=VIEWPORT,
),
)
......@@ -609,7 +976,7 @@ def capture_tableau_view(
if "### Error" in configure_output:
last_error = configure_output
if attempt < max_attempts:
# Tableau 偶发 workbook 尚未就绪，回到同页后重试可恢复。
# Tableau 鍋跺彂 workbook 灏氭湭灏辩华锛屽洖鍒板悓椤靛悗閲嶈瘯鍙仮澶嶃€?
run_playwright(["--session", session, "goto", target_url], cwd=workdir, timeout=120)
run_playwright(
["--session", session, "resize", str(VIEWPORT["width"]), str(VIEWPORT["height"])],
......@@ -627,27 +994,32 @@ def capture_tableau_view(
configure_script.unlink(missing_ok=True)
capture_script.unlink(missing_ok=True)
# run-code 会把截图优先落在当前工作目录，直接取本次产物，避免误读旧 session 文件。
local_output = workdir / capture_spec["raw_screenshot_name"]
if local_output.exists():
normalize_image_size(local_output, {"width": 1400, "height": 3360})
return local_output
return locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
captured_path = locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
normalize_image_size(captured_path, {"width": 1400, "height": 3360})
return captured_path
def main() -> None:
"""Helper."""
script_started_at = time.monotonic()
args = parse_args()
config_path = Path(args.config)
config = yaml.safe_load(config_path.read_text(encoding="utf-8"))
report_month, report_year, compare_year = resolve_report_period(args, config)
specs = build_specs(report_month, report_year, compare_year)
month_mode = "single" if args.single_month else args.month_mode
specs = build_specs(report_month, report_year, compare_year, month_mode)
requested = {item.strip().upper() for item in args.slides.split(",") if item.strip()}
filtered_assets = [item for item in specs["assets"] if item["slide_code"] in requested]
if not filtered_assets:
raise SystemExit("No matching slides requested.")
vip_workdir = Path(config["paths"]["workdir"]).resolve()
workspace_root = vip_workdir.parents[1]
vip_workdir = Path(args.output_dir).resolve() if args.output_dir else Path(config["paths"]["workdir"]).resolve()
workspace_root = vip_workdir if args.output_dir else vip_workdir.parents[1]
asset_dir = vip_workdir / "assets" / "monthly-sales"
data_dir = vip_workdir / "data" / "monthly-sales"
asset_dir.mkdir(parents=True, exist_ok=True)
......@@ -655,17 +1027,25 @@ def main() -> None:
captures_by_id = {item["capture_id"]: item for item in specs["captures"]}
required_capture_ids = collect_required_capture_ids(filtered_assets)
capture_sequence = build_capture_execution_order(required_capture_ids)
capture_slide_codes = build_capture_slide_codes(filtered_assets)
session = SESSION_NAME
state_path = workspace_root / "output" / "playwright" / session / "state.json"
log_progress("stage", f"start monthly-sales slides={','.join(sorted(requested))}")
session = SESSION_NAME if not args.output_dir else f"{SESSION_NAME}-{vip_workdir.name}"
state_path = (
vip_workdir / ".playwright-cli" / f"{session}-state.json"
if args.output_dir
else workspace_root / "output" / "playwright" / session / "state.json"
)
ensure_browser_session(session, cwd=vip_workdir)
load_state_if_present(session, state_path, cwd=vip_workdir)
base_url = config["tableau"]["base_url"].rstrip("/")
first_capture = captures_by_id[next(iter(sorted(required_capture_ids)))]
first_target_url = f"{base_url}{first_capture['hash_url']}"
run_playwright(["--session", session, "goto", first_target_url], cwd=vip_workdir, timeout=120)
first_capture = captures_by_id[capture_sequence[0]]
first_target_url = build_tableau_target_url(base_url, first_capture["hash_url"])
goto_with_retry(session, first_target_url, cwd=vip_workdir)
login_script = write_js(
vip_workdir,
......@@ -678,21 +1058,31 @@ def main() -> None:
login_script.unlink(missing_ok=True)
raw_screenshots: dict[str, Path] = {}
# 固定抓取顺序，避免集合无序导致切页时触发 Tableau 临时异常。
for capture_id in sorted(required_capture_ids):
# Stage 1: download all required raw screenshots first (S02: overview -> store_sales_detail).
for capture_id in capture_sequence:
capture_spec = captures_by_id[capture_id]
page_label = describe_capture(capture_id, capture_spec, capture_slide_codes)
page_started_at = time.monotonic()
log_progress("page", f"start {page_label}")
raw_screenshots[capture_id] = capture_tableau_view(
captures_by_id[capture_id],
capture_spec,
base_url=base_url,
session=session,
workdir=vip_workdir,
workspace_root=workspace_root,
)
log_timing(f"page {page_label}", page_started_at)
save_state(session, state_path, cwd=vip_workdir)
operations = {"replace_text": [], "replace_images": []}
manifest_items: list[dict[str, Any]] = []
# Stage 2: crop/composite all assets after downloads are completed.
assets_started_at = time.monotonic()
for asset in filtered_assets:
asset_label = f"{asset['slide_code']} {asset['asset_name']}"
asset_started_at = time.monotonic()
log_progress("asset", f"start {asset_label}")
target = asset_dir / f"{asset['asset_name']}.png"
source_path = raw_screenshots[asset["capture_id"]]
......@@ -735,6 +1125,8 @@ def main() -> None:
manifest_item["panel_records"] = panel_records
manifest_item["composite_canvas"] = asset["composite"]["canvas"]
manifest_items.append(manifest_item)
log_timing(f"asset {asset_label}", asset_started_at)
log_timing(f"assets monthly-sales {','.join(sorted(requested))}", assets_started_at)
operations_path = vip_workdir / "render-ops.monthly-sales.live.json"
operations_path.write_text(json.dumps(operations, ensure_ascii=False, indent=2), encoding="utf-8")
......@@ -754,13 +1146,14 @@ def main() -> None:
"raw_screenshot": str(raw_screenshots[capture_id]),
"note": captures_by_id[capture_id]["note"],
}
for capture_id in sorted(raw_screenshots)
for capture_id in capture_sequence
],
"config_path": str(config_path),
"report_period": {
"month": report_month,
"year": report_year,
"compare_year": compare_year,
"month_mode": month_mode,
},
},
"assets": manifest_items,
......@@ -772,8 +1165,10 @@ def main() -> None:
encoding="utf-8",
)
log_timing(f"stage monthly-sales {','.join(sorted(requested))}", script_started_at)
print(json.dumps({"operations_path": str(operations_path), "manifest_path": str(manifest_path)}, ensure_ascii=False))
# 鍏抽敭琛屾敞閲婏細鑴氭湰鐩存帴杩愯鏃讹紝浠庤繖閲岃繘鍏ヤ富娴佺▼銆?
if __name__ == "__main__":
main()
vip-report/scripts/sync_top_products_assets.py
View file @ 1027cb42
from __future__ import annotations
from __future__ import annotations
import argparse
import calendar
import datetime
import json
import os
import re
import subprocess
import time
......@@ -12,9 +13,15 @@ from io import BytesIO
from pathlib import Path
from typing import Any
import matplotlib
matplotlib.use("Agg")
import pymysql
from matplotlib import pyplot as plt
from matplotlib.gridspec import GridSpec
from matplotlib.patches import Rectangle
from PIL import Image, ImageOps
import yaml
from tableau_export import build_export_image_js, normalize_image_size
NPX_EXECUTABLE = "npx.cmd"
......@@ -38,6 +45,18 @@ MONTH_CN_MAP = {
}
MONTH_INDEX_MAP = {label: index for index, label in MONTH_CN_MAP.items()}
PRODUCT_IMAGE_URL = "https://api.charleskeith.cn/img/{articlecode}.jpg"
QTY_BAR_COLORS = [
"#c01844",
"#d772a0",
"#8c6aac",
"#6b6fae",
"#6670ab",
"#5e70a9",
"#5670a6",
"#4e6ca0",
"#466898",
"#3f6794",
]
SLIDE_SPECS: dict[str, dict[str, Any]] = {
"S09": {
......@@ -50,7 +69,7 @@ SLIDE_SPECS: dict[str, dict[str, Any]] = {
"asset_name": "s09_top_products_chart",
"raw_screenshot_name": "top-products-bags.png",
"resize_to": {"width": 1034, "height": 587},
"crop": {"left": 0, "top": 180, "width": 1050, "height": 600},
"crop": {"left": 0, "top": 480, "width": 900, "height": 1900},
"note": "S09 main Tableau chart area.",
"product_slots": [
{"rank": 1, "shape_id": 4, "shape_name": "Picture 2", "canvas_width": 707, "canvas_height": 500},
......@@ -75,7 +94,7 @@ SLIDE_SPECS: dict[str, dict[str, Any]] = {
"asset_name": "s10_top_products_chart",
"raw_screenshot_name": "top-products-shoes.png",
"resize_to": {"width": 1039, "height": 585},
"crop": {"left": 0, "top": 120, "width": 1400, "height": 900},
"crop": {"left": 0, "top": 480, "width": 900, "height": 1900},
"note": "S10 main Tableau chart area.",
"product_slots": [
{"rank": 1, "shape_id": 3, "shape_name": "Picture 2", "canvas_width": 708, "canvas_height": 500},
......@@ -92,8 +111,37 @@ SLIDE_SPECS: dict[str, dict[str, Any]] = {
},
}
TOP_PRODUCTS_DASHBOARD = "Top Products"
TOP_PRODUCTS_TOP_WORKSHEET = "TOP"
TOP_PRODUCTS_CATEGORY_CONTRIBUTIONS_WORKSHEET = "category contributions"
TOP_PRODUCTS_TREND_WORKSHEET = "Soldqty Trend"
TOP_PRODUCTS_OVERALL_WORKSHEET = "OVERALL TOP"
TOP_PRODUCTS_DETAIL_WORKSHEET = "Soldqty and SOH details"
TOP_PRODUCTS_MONTH_FIELD_CANDIDATES = ["月(Daily)", "Month", "month"]
TOP_PRODUCTS_WEEK_FIELD_CANDIDATES = ["ISO周数(Daily)", "ISO Week(Daily)", "Week", "week"]
TOP_PRODUCTS_DAILY_FIELD_CANDIDATES = ["日(Daily)", "Daily", "daily"]
TOP_PRODUCTS_YEAR_FIELD_CANDIDATES = ["year", "Year"]
TOP_PRODUCTS_CATEGORY_FIELD_CANDIDATES = ["Category", "category"]
TOP_PRODUCTS_STORE_FIELD_CANDIDATES = [
"storename (group)",
"Storename (group)",
"storename",
"Storename",
]
MYSQL_CONNECT_TIMEOUT_SECONDS = 5
MYSQL_READ_TIMEOUT_SECONDS = 120
MYSQL_WRITE_TIMEOUT_SECONDS = 120
MYSQL_PRODUCTION_FALLBACK = {
"host": "erp.charleskeith.cn",
"port": 30004,
"database": "ckc_cep_db",
"username": "allen",
"password": "wangjun@123",
}
def normalize_month_label(raw_month: Any) -> str:
"""Helper."""
if raw_month is None:
return MONTH_CN_MAP[1]
if isinstance(raw_month, int):
......@@ -109,13 +157,34 @@ def normalize_month_label(raw_month: Any) -> str:
def resolve_month_index(report_month: str) -> int:
"""Helper."""
normalized = normalize_month_label(report_month)
if normalized not in MONTH_INDEX_MAP:
raise ValueError(f"Unsupported report month: {report_month}")
return MONTH_INDEX_MAP[normalized]
def resolve_previous_month(report_year: int, report_month: str) -> tuple[int, str]:
"""Helper."""
month_index = resolve_month_index(report_month)
if month_index == 1:
return report_year - 1, MONTH_CN_MAP[12]
return report_year, MONTH_CN_MAP[month_index - 1]
def build_report_month_values(report_year: int, report_month: str) -> tuple[list[str], list[str]]:
"""Helper."""
normalized_month = normalize_month_label(report_month)
previous_year, previous_month = resolve_previous_month(report_year, normalized_month)
month_values = [normalized_month, previous_month]
year_values = [str(report_year)]
if previous_year != report_year:
year_values.append(str(previous_year))
return month_values, year_values
def build_month_week_labels(report_year: int, report_month: str) -> list[str]:
"""Helper."""
month_index = resolve_month_index(report_month)
last_day = calendar.monthrange(report_year, month_index)[1]
week_numbers = {
......@@ -126,6 +195,7 @@ def build_month_week_labels(report_year: int, report_month: str) -> list[str]:
def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> tuple[str, int, int]:
"""Helper."""
report_cfg = config.get("report", {})
report_month = normalize_month_label(args.report_month or report_cfg.get("month_cn", MONTH_CN_MAP[1]))
report_year = int(args.report_year or report_cfg.get("year", 2026))
......@@ -133,37 +203,228 @@ def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> t
return report_month, report_year, report_compare_year
def log_progress(scope: str, message: str) -> None:
"""Print a compact progress line."""
print(f"[{scope}] {message}", flush=True)
def log_timing(label: str, started_at: float) -> None:
"""Print a compact timing line."""
elapsed_seconds = time.monotonic() - started_at
print(f"[timing] {label}: {elapsed_seconds:.1f}s", flush=True)
def resolve_configured_path(config_path: Path, raw_path: str) -> Path:
"""Helper."""
path = Path(raw_path)
if path.is_absolute():
return path
return (config_path.parent / path).resolve()
def normalize_mysql_config(raw_config: dict[str, Any]) -> dict[str, Any]:
"""Normalize a mysql config block to the keys expected by pymysql."""
return {
"host": str(raw_config["host"]),
"port": int(raw_config["port"]),
"database": str(raw_config["database"]),
"username": str(raw_config["username"]),
"password": str(raw_config["password"]),
}
def iter_mysql_connection_candidates(config: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]:
"""Return unique mysql targets in the order they should be tried."""
candidates: list[tuple[str, dict[str, Any]]] = []
seen: set[tuple[str, int, str, str]] = set()
for label, raw_candidate in (
("config.mysql", config.get("mysql")),
("config.mysql_production", config.get("mysql_production")),
("built-in production fallback", MYSQL_PRODUCTION_FALLBACK),
):
if not raw_candidate:
continue
candidate = normalize_mysql_config(raw_candidate)
signature = (
candidate["host"],
candidate["port"],
candidate["database"],
candidate["username"],
)
if signature in seen:
continue
seen.add(signature)
candidates.append((label, candidate))
return candidates
def connect_mysql(mysql_cfg: dict[str, Any]) -> pymysql.connections.Connection:
"""Open a mysql connection with short connect timeouts."""
return pymysql.connect(
host=mysql_cfg["host"],
port=int(mysql_cfg["port"]),
user=mysql_cfg["username"],
password=mysql_cfg["password"],
database=mysql_cfg["database"],
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
connect_timeout=MYSQL_CONNECT_TIMEOUT_SECONDS,
read_timeout=MYSQL_READ_TIMEOUT_SECONDS,
write_timeout=MYSQL_WRITE_TIMEOUT_SECONDS,
)
def open_mysql_connection(config: dict[str, Any]) -> tuple[pymysql.connections.Connection, dict[str, Any], str]:
"""Try configured mysql targets in order until one connects."""
errors: list[str] = []
for label, mysql_cfg in iter_mysql_connection_candidates(config):
try:
conn = connect_mysql(mysql_cfg)
return conn, mysql_cfg, label
except Exception as exc:
errors.append(
f"{label}={mysql_cfg['host']}:{mysql_cfg['port']}/{mysql_cfg['database']} -> {type(exc).__name__}: {exc}"
)
raise RuntimeError("Unable to connect to any mysql target: " + " | ".join(errors))
def parse_args() -> argparse.Namespace:
"""Helper."""
parser = argparse.ArgumentParser(description="Generate S09/S10 top-products assets for the VIP report.")
parser.add_argument("--config", required=True, help="Path to config.yaml")
parser.add_argument("--slides", default="S09,S10", help="Slide codes to generate.")
parser.add_argument("--report-month", help="Report month, for example 3 / 03 / 三月")
parser.add_argument("--report-month", help="Report month, for example 3 / 03 / 涓夋湀")
parser.add_argument("--report-year", type=int, help="Report year")
parser.add_argument("--compare-year", type=int, help="Compare year")
parser.add_argument(
"--output-dir",
default="",
help="Override the output/work directory used for generated assets and ops.",
)
return parser.parse_args()
def build_filter_actions(report_month: str, report_year: int, slide_code: str) -> list[dict[str, Any]]:
def build_api_pre_filter_actions(report_month: str, report_year: int, slide_code: str) -> list[dict[str, Any]]:
"""Helper."""
return [
{
"kind": "filter",
"label": "year",
"target_names": [TOP_PRODUCTS_TOP_WORKSHEET],
"field_candidates": TOP_PRODUCTS_YEAR_FIELD_CANDIDATES,
"values": [str(report_year)],
"required_successes": 1,
},
]
def build_ui_filter_actions(report_month: str, report_year: int, slide_code: str) -> list[dict[str, Any]]:
"""Helper."""
normalized_month = normalize_month_label(report_month)
return [
{
"kind": "quick-filter",
"label": "Month",
"values": [normalized_month],
},
{
"kind": "quick-filter",
"label": "Week",
"values": [ALL_FILTER_VALUE],
},
{
"kind": "quick-filter",
"label": "Daily",
"values": [ALL_FILTER_VALUE],
},
]
def build_api_post_filter_actions(report_month: str, report_year: int, slide_code: str) -> list[dict[str, Any]]:
"""Helper."""
level1 = SLIDE_SPECS[slide_code]["level1"]
week_labels = build_month_week_labels(report_year, report_month)
return [
{"kind": "filter", "worksheet": "TOP", "field": "year", "values": [str(report_year)]},
{"kind": "filter", "worksheet": "TOP", "field": "月(Daily)", "values": [report_month]},
{"kind": "filter", "worksheet": "TOP", "field": "ISO周数(Daily)", "values": week_labels},
{"kind": "filter", "worksheet": "category contributions", "field": "月(Daily)", "values": [report_month]},
{"kind": "filter", "worksheet": "Soldqty Trend", "field": "Category", "values": [level1]},
{"kind": "filter", "worksheet": "OVERALL TOP", "field": "Category", "values": [level1]},
{"kind": "filter", "worksheet": "TOP", "field": "Category", "values": [level1]},
{"kind": "filter", "worksheet": "Soldqty and SOH details", "field": "Category", "values": [level1]},
{"kind": "filter", "worksheet": "category contributions", "field": "storename (group)", "values": ["CKC-VIP"]},
{"kind": "filter", "worksheet": "category contributions", "field": "storename", "values": ["CKC-VIP"]},
{"kind": "filter", "worksheet": "Soldqty Trend", "field": "storename (group)", "values": ["CKC-VIP"]},
{"kind": "filter", "worksheet": "Soldqty Trend", "field": "storename", "values": ["CKC-VIP"]},
{
"kind": "filter",
"label": "category_trend",
"target_names": [TOP_PRODUCTS_TREND_WORKSHEET],
"field_candidates": TOP_PRODUCTS_CATEGORY_FIELD_CANDIDATES,
"values": [level1],
"required_successes": 1,
},
{
"kind": "filter",
"label": "category_overall",
"target_names": [TOP_PRODUCTS_OVERALL_WORKSHEET],
"field_candidates": TOP_PRODUCTS_CATEGORY_FIELD_CANDIDATES,
"values": [level1],
"required_successes": 1,
},
{
"kind": "filter",
"label": "category_top",
"target_names": [TOP_PRODUCTS_TOP_WORKSHEET],
"field_candidates": TOP_PRODUCTS_CATEGORY_FIELD_CANDIDATES,
"values": [level1],
"required_successes": 1,
},
{
"kind": "filter",
"label": "category_detail",
"target_names": [TOP_PRODUCTS_DETAIL_WORKSHEET],
"field_candidates": TOP_PRODUCTS_CATEGORY_FIELD_CANDIDATES,
"values": [level1],
"required_successes": 1,
},
{
"kind": "filter",
"label": "storename_group_category_contributions",
"target_names": [TOP_PRODUCTS_CATEGORY_CONTRIBUTIONS_WORKSHEET],
"field_candidates": ["storename (group)", "Storename (group)"],
"values": ["CKC-VIP"],
"required_successes": 1,
},
{
"kind": "filter",
"label": "storename_category_contributions",
"target_names": [TOP_PRODUCTS_CATEGORY_CONTRIBUTIONS_WORKSHEET],
"field_candidates": ["storename", "Storename"],
"values": ["CKC-VIP"],
"required_successes": 1,
},
{
"kind": "filter",
"label": "storename_group_trend",
"target_names": [TOP_PRODUCTS_TREND_WORKSHEET],
"field_candidates": ["storename (group)", "Storename (group)"],
"values": ["CKC-VIP"],
"required_successes": 1,
},
{
"kind": "filter",
"label": "storename_trend",
"target_names": [TOP_PRODUCTS_TREND_WORKSHEET],
"field_candidates": ["storename", "Storename"],
"values": ["CKC-VIP"],
"required_successes": 1,
},
{"kind": "parameter", "name": "Top Parameter", "value": 10},
]
def build_filter_actions(report_month: str, report_year: int, slide_code: str) -> list[dict[str, Any]]:
"""Helper."""
return (
build_api_pre_filter_actions(report_month, report_year, slide_code)
+ build_ui_filter_actions(report_month, report_year, slide_code)
+ build_api_post_filter_actions(report_month, report_year, slide_code)
)
def build_capture_spec(slide_code: str, report_month: str, report_year: int) -> dict[str, Any]:
"""Helper."""
slide_spec = SLIDE_SPECS[slide_code]
return {
"capture_id": slide_spec["capture_id"],
......@@ -171,16 +432,40 @@ def build_capture_spec(slide_code: str, report_month: str, report_year: int) ->
"hash_url": "#/views/CKTopProducts-General_16862068169500/TopProducts?:iid=1",
"inner_frame_fragment": "/views/CKTopProducts-General_16862068169500/TopProducts?",
"activate_sheet": "Top Products",
"api_pre_filter_actions": build_api_pre_filter_actions(report_month, report_year, slide_code),
"ui_filter_actions": build_ui_filter_actions(report_month, report_year, slide_code),
"api_post_filter_actions": build_api_post_filter_actions(report_month, report_year, slide_code),
"filter_actions": build_filter_actions(report_month, report_year, slide_code),
"expected_top_rows": len(slide_spec["product_slots"]),
"post_filter_wait_ms": 60000,
"post_filter_poll_interval_ms": 1000,
"post_filter_min_wait_ms": 0,
"post_filter_fallback_wait_ms": 0,
"normalize_quick_filters": False,
"raw_screenshot_name": slide_spec["raw_screenshot_name"],
"note": f"{slide_code} live Tableau screenshot with requested filters applied.",
}
def run_cmd(args: list[str], *, cwd: Path, timeout: int = 120, check: bool = True) -> subprocess.CompletedProcess[str]:
"""Helper."""
env = os.environ.copy()
local_tmp = cwd / ".tmp"
npm_cache = cwd / ".npm-cache"
browsers_path = cwd / ".ms-playwright"
daemon_dir = cwd / ".playwright-daemon"
for path in (local_tmp, npm_cache, browsers_path, daemon_dir):
path.mkdir(parents=True, exist_ok=True)
env["TEMP"] = str(local_tmp)
env["TMP"] = str(local_tmp)
env["npm_config_cache"] = str(npm_cache)
env["NPM_CONFIG_CACHE"] = str(npm_cache)
env["PLAYWRIGHT_BROWSERS_PATH"] = str(browsers_path)
env["PLAYWRIGHT_DAEMON_SESSION_DIR"] = str(daemon_dir)
return subprocess.run(
args,
cwd=str(cwd),
env=env,
text=True,
encoding="utf-8",
errors="replace",
......@@ -191,17 +476,20 @@ def run_cmd(args: list[str], *, cwd: Path, timeout: int = 120, check: bool = Tru
def run_playwright(args: list[str], *, cwd: Path, timeout: int = 120) -> str:
"""Helper."""
result = run_cmd(PLAYWRIGHT_CMD + args, cwd=cwd, timeout=timeout)
return result.stdout
def write_js(workdir: Path, name: str, content: str) -> Path:
"""Helper."""
path = workdir / name
path.write_text(content, encoding="utf-8")
return path
def build_login_js(username: str, password: str) -> str:
"""Helper."""
payload = {"username": username, "password": password}
spec = json.dumps(payload, ensure_ascii=False)
return f"""async function(page) {{
......@@ -221,25 +509,101 @@ def build_login_js(username: str, password: str) -> str:
page.waitForFunction(() => !location.href.includes('/#/signin'), null, {{ timeout: 30000 }}).catch(() => null),
button.click(),
]);
await page.waitForTimeout(3000);
await page.waitForLoadState('domcontentloaded').catch(() => null);
return {{ url: page.url(), title: await page.title() }};
}}
"""
def build_configure_view_js(spec: dict[str, Any]) -> str:
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
await page.waitForFunction(
() => !!(window.tableau && window.tableau.VizManager && window.tableau.VizManager.getVizs().length),
null,
{{ timeout: 30000 }}
);
await page.waitForLoadState('domcontentloaded').catch(() => null);
async function inspectFrame(frame, preferredFragment) {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : null;
try {{
const snapshot = await frame.evaluate(() => {{
const vizCount = window.tableau?.VizManager?.getVizs?.().length || 0;
return {{
url: location.href,
title: document.title,
readyState: document.readyState,
hasTableau: !!(window.tableau && window.tableau.VizManager),
vizCount,
}};
}});
return {{
...snapshot,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
}};
}} catch (error) {{
return {{
url: frameUrl,
title: null,
readyState: null,
hasTableau: false,
vizCount: 0,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
error: String(error && error.message || error),
}};
}}
}}
async function locateVizFrame(timeoutMs, preferredFragment) {{
const deadline = Date.now() + timeoutMs;
let lastSnapshots = [];
while (Date.now() < deadline) {{
const rankedFrames = page.frames()
.map((frame) => {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
const isPreferred = !!(preferredFragment && frameUrl.includes(preferredFragment));
return {{
frame,
rank: isPreferred ? 0 : (frame === page.mainFrame() ? 1 : 2),
}};
}})
.sort((left, right) => left.rank - right.rank);
lastSnapshots = [];
for (const item of rankedFrames) {{
const snapshot = await inspectFrame(item.frame, preferredFragment);
lastSnapshots.push(snapshot);
if (snapshot.vizCount > 0) {{
return {{ frame: item.frame, snapshots: lastSnapshots }};
}}
}}
const stillOnSignin =
(page.url() || '').includes('/#/signin') ||
lastSnapshots.some((snapshot) => (snapshot.url || '').includes('/#/signin'));
if (stillOnSignin) {{
throw new Error('Tableau page is still on sign-in: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
const hasChromeError =
(page.url() || '').startsWith('chrome-error://') ||
lastSnapshots.some((snapshot) => (snapshot.url || '').startsWith('chrome-error://'));
if (hasChromeError) {{
throw new Error('Tableau page failed to load: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
await page.waitForTimeout(1000);
}}
throw new Error('Timed out waiting for Tableau viz: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
const located = await locateVizFrame(Number(spec.tableau_wait_ms || 90000), spec.inner_frame_fragment || '');
return await page.evaluate(async (config) => {{
return await located.frame.evaluate(async (config) => {{
const sleep = (ms) => new Promise((resolve) => window.setTimeout(resolve, ms));
const deadline = Date.now() + 30000;
const deadline = Date.now() + Number(config.workbook_wait_ms || 60000);
let viz = null;
let workbook = null;
while (Date.now() < deadline) {{
......@@ -256,21 +620,54 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
await sleep(500);
}}
if (!workbook) {{
throw new Error('Tableau workbook is not ready');
throw new Error('Tableau workbook is not ready: ' + JSON.stringify({{
url: location.href,
title: document.title,
readyState: document.readyState,
vizCount: window.tableau?.VizManager?.getVizs?.().length || 0,
}}));
}}
const shouldRevert = config.revert_before_apply !== false;
if (shouldRevert) {{
try {{
await workbook.revertAllAsync();
}} catch (error) {{
}}
}}
await workbook.activateSheetAsync(config.activate_sheet);
const activeSheet = workbook.getActiveSheet();
const worksheets = activeSheet && typeof activeSheet.getWorksheets === 'function'
? Object.fromEntries(activeSheet.getWorksheets().map((worksheet) => [worksheet.getName(), worksheet]))
: {{}};
const worksheetList = activeSheet && typeof activeSheet.getWorksheets === 'function'
? activeSheet.getWorksheets()
: [];
const worksheets = Object.fromEntries(
worksheetList.map((worksheet) => [worksheet.getName(), worksheet])
);
const targetsByName = {{
...(activeSheet && typeof activeSheet.getName === 'function'
? {{ [activeSheet.getName()]: activeSheet }}
: {{}}),
...worksheets,
}};
const updateType = window.tableau.FilterUpdateType.REPLACE;
const applyLog = [];
const applySummary = {{}};
function ensureSummary(label, requiredSuccesses) {{
const resolvedLabel = label || 'unlabeled';
if (!applySummary[resolvedLabel]) {{
applySummary[resolvedLabel] = {{
label: resolvedLabel,
requiredSuccesses: Number(requiredSuccesses || 0),
successes: 0,
targetNames: [],
}};
}} else if (Number(requiredSuccesses || 0) > applySummary[resolvedLabel].requiredSuccesses) {{
applySummary[resolvedLabel].requiredSuccesses = Number(requiredSuccesses || 0);
}}
return applySummary[resolvedLabel];
}}
for (const action of config.filter_actions) {{
try {{
......@@ -279,32 +676,722 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
applyLog.push({{ kind: action.kind, name: action.name, ok: true }});
continue;
}}
if (action.kind !== 'filter') {{
applyLog.push({{
kind: action.kind || null,
label: action.label || null,
ok: true,
skipped: true,
}});
continue;
}}
const worksheet = worksheets[action.worksheet];
if (!worksheet) {{
applyLog.push({{ kind: action.kind, worksheet: action.worksheet, field: action.field, ok: false, error: 'worksheet not found' }});
const summary = ensureSummary(action.label, action.required_successes);
const targetNames = Array.isArray(action.target_names)
? action.target_names
: [action.worksheet].filter(Boolean);
const fieldCandidates = Array.isArray(action.field_candidates)
? action.field_candidates
: [action.field].filter(Boolean);
for (const targetName of targetNames) {{
if (!summary.targetNames.includes(targetName)) {{
summary.targetNames.push(targetName);
}}
const target = targetsByName[targetName];
if (!target) {{
applyLog.push({{
kind: action.kind,
label: action.label || null,
target: targetName,
values: action.values,
ok: false,
error: 'target not found',
}});
continue;
}}
await worksheet.applyFilterAsync(action.field, action.values, updateType);
applyLog.push({{ kind: action.kind, worksheet: action.worksheet, field: action.field, values: action.values, ok: true }});
let targetApplied = false;
let lastTargetError = null;
for (const fieldCandidate of fieldCandidates) {{
try {{
await target.applyFilterAsync(fieldCandidate, action.values, updateType);
summary.successes += 1;
targetApplied = true;
applyLog.push({{
kind: action.kind,
label: action.label || null,
target: targetName,
field: fieldCandidate,
values: action.values,
ok: true,
}});
break;
}} catch (error) {{
applyLog.push({{ kind: action.kind, worksheet: action.worksheet || null, field: action.field || null, name: action.name || null, ok: false, error: String(error) }});
lastTargetError = String(error && error.message || error);
}}
}}
await sleep(60000);
if (!targetApplied) {{
applyLog.push({{
kind: action.kind,
label: action.label || null,
target: targetName,
fields: fieldCandidates,
values: action.values,
ok: false,
error: lastTargetError || 'no matching field candidate succeeded',
}});
}}
}}
}} catch (error) {{
applyLog.push({{ kind: action.kind, label: action.label || null, name: action.name || null, ok: false, error: String(error) }});
}}
}}
const missingRequired = Object.values(applySummary).filter((entry) =>
Number(entry.requiredSuccesses || 0) > 0 && Number(entry.successes || 0) < Number(entry.requiredSuccesses || 0)
);
if (missingRequired.length) {{
throw new Error('Required Tableau filters did not reach enough targets: ' + JSON.stringify({{
missingRequired,
applyLog,
}}));
}}
return {{
frameUrl: location.href,
activeSheet: activeSheet.getName(),
filterActions: config.filter_actions,
applyLog,
applySummary,
url: location.href,
}};
}}, spec);
}}, {{
activate_sheet: spec.activate_sheet,
filter_actions: spec.filter_actions,
workbook_wait_ms: spec.workbook_wait_ms || 60000,
revert_before_apply: spec.revert_before_apply,
}});
}}
"""
def build_normalize_quick_filters_js(spec: dict[str, Any]) -> str:
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
const allFilterValue = String(spec.all_filter_value || '(全部)');
await page.waitForLoadState('domcontentloaded').catch(() => null);
async function readLocatorText(locator) {{
const count = await locator.count().catch(() => 0);
if (!count) {{
return '';
}}
const first = locator.first();
return String(
await first.innerText().catch(async () => await first.textContent().catch(() => '')) || ''
).trim();
}}
function escapeRegExp(value) {{
return String(value || '').replace(/[.*+?^${{}}()|[\\]\\\\]/g, '\\\\$&');
}}
function buildExactMatcher(value) {{
return new RegExp(`^\\\\s*${{escapeRegExp(value)}}\\\\s*$`, 'i');
}}
async function collectQuickFilterHeadings(frame) {{
const headings = [];
const locator = frame.getByRole('heading');
const count = await locator.count().catch(() => 0);
for (let index = 0; index < count; index += 1) {{
const heading = await readLocatorText(locator.nth(index));
if (heading) {{
headings.push(heading);
}}
}}
return headings;
}}
async function locateQuickFilterFrame(timeoutMs, preferredFragment) {{
const deadline = Date.now() + timeoutMs;
let lastFrames = [];
while (Date.now() < deadline) {{
lastFrames = [];
for (const frame of page.frames()) {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
if (preferredFragment && !frameUrl.includes(preferredFragment)) {{
continue;
}}
const headingCount = await frame.getByRole('heading').count().catch(() => 0);
const buttonCount = await frame.getByRole('button').count().catch(() => 0);
const monthHeadingCount = await frame.getByRole('heading', {{ name: /Month/i }}).count().catch(() => 0);
const weekHeadingCount = await frame.getByRole('heading', {{ name: /Week/i }}).count().catch(() => 0);
const dailyHeadingCount = await frame.getByRole('heading', {{ name: /Daily/i }}).count().catch(() => 0);
const snapshot = {{
frameUrl,
headingCount,
buttonCount,
monthHeadingCount,
weekHeadingCount,
dailyHeadingCount,
}};
lastFrames.push(snapshot);
if (buttonCount > 20 && (monthHeadingCount > 0 || weekHeadingCount > 0 || dailyHeadingCount > 0)) {{
return {{ frame, lastFrames }};
}}
}}
await page.waitForTimeout(1000);
}}
throw new Error('Timed out waiting for Tableau quick filters: ' + JSON.stringify(lastFrames));
}}
async function locateQuickFilterButton(frame, label, timeoutMs) {{
const deadline = Date.now() + timeoutMs;
const matcher = new RegExp(escapeRegExp(label), 'i');
while (Date.now() < deadline) {{
const button = frame.getByRole('button', {{ name: matcher }}).first();
const buttonCount = await button.count().catch(() => 0);
if (buttonCount > 0 && await button.isVisible().catch(() => true)) {{
return button;
}}
const heading = frame.getByRole('heading', {{ name: matcher }}).first();
const headingCount = await heading.count().catch(() => 0);
if (headingCount > 0 && await heading.isVisible().catch(() => true)) {{
const followingButton = heading.locator('xpath=following::button[1]').first();
const followingCount = await followingButton.count().catch(() => 0);
if (followingCount > 0 && await followingButton.isVisible().catch(() => true)) {{
return followingButton;
}}
}}
await page.waitForTimeout(1000);
}}
return null;
}}
async function waitForButtonText(button, expectedValue, timeoutMs) {{
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {{
const buttonText = await readLocatorText(button);
if (!expectedValue || buttonText.includes(expectedValue)) {{
return buttonText;
}}
await page.waitForTimeout(1000);
}}
return await readLocatorText(button);
}}
async function openQuickFilter(frame, button, label, timeoutMs) {{
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {{
await button.click({{ timeout: 5000 }}).catch(async () => {{
await button.click({{ timeout: 5000, force: true }}).catch(() => null);
}});
const listbox = frame.getByRole('listbox').last();
const listboxCount = await listbox.count().catch(() => 0);
if (listboxCount > 0 && await listbox.isVisible().catch(() => false)) {{
await page.waitForTimeout(500);
return listbox;
}}
await page.waitForTimeout(500);
}}
throw new Error(`Failed to open quick filter ${{label}}`);
}}
async function closeQuickFilter(button) {{
try {{
await button.click({{ timeout: 5000 }});
await page.waitForTimeout(1000);
return 'button-click';
}} catch (error) {{}}
try {{
await button.click({{ timeout: 5000, force: true }});
await page.waitForTimeout(1000);
return 'button-click-force';
}} catch (error) {{}}
await page.keyboard.press('Escape').catch(() => null);
await page.waitForTimeout(1000);
return 'escape';
}}
async function findRow(listbox, value) {{
const row = listbox.getByRole('checkbox', {{ name: buildExactMatcher(value) }}).first();
const count = await row.count().catch(() => 0);
return count > 0 ? row : null;
}}
async function readRowState(row) {{
const count = await row.count().catch(() => 0);
if (!count) {{
return {{
exists: false,
checked: false,
text: '',
}};
}}
return await row.evaluate((element) => {{
const target = element.closest('[role="checkbox"]') || element;
const input = target.querySelector('input.FICheckRadio');
const className = String(target.className || '');
const ariaChecked = String(target.getAttribute('aria-checked') || '');
const inputChecked = !!(input && input.checked);
return {{
exists: true,
text: String(target.textContent || '').trim(),
ariaChecked,
className,
inputChecked,
checked: inputChecked || ariaChecked.toLowerCase() === 'true' || className.includes('FIChecked'),
}};
}}).catch((error) => ({{
exists: true,
checked: false,
text: '',
error: String(error && error.message || error),
}}));
}}
async function clickFakeCheckbox(row) {{
const fake = row.locator('.fakeCheckBox').first();
const fakeCount = await fake.count().catch(() => 0);
if (fakeCount > 0) {{
try {{
await fake.click({{ timeout: 5000 }});
return 'fake-click';
}} catch (error) {{}}
try {{
await fake.click({{ timeout: 5000, force: true }});
return 'fake-click-force';
}} catch (error) {{}}
}}
try {{
await row.evaluate((element) => {{
const target = element.closest('[role="checkbox"]') || element;
const input = target.querySelector('input.FICheckRadio');
if (input) {{
input.click();
return;
}}
const fallback = target.querySelector('.fakeCheckBox') || target;
for (const type of ['pointerdown', 'mousedown', 'mouseup', 'click']) {{
fallback.dispatchEvent(new MouseEvent(type, {{ bubbles: true, cancelable: true, composed: true }}));
}}
}});
return 'dom-input-click';
}} catch (error) {{}}
await row.click({{ timeout: 5000, force: true }});
return 'row-click-force';
}}
async function listCheckedRows(listbox) {{
const rows = listbox.getByRole('checkbox');
const count = await rows.count().catch(() => 0);
const checkedRows = [];
for (let index = 0; index < count; index += 1) {{
const row = rows.nth(index);
const state = await readRowState(row);
if (state.checked) {{
checkedRows.push({{ index, ...state }});
}}
}}
return checkedRows;
}}
async function applyQuickFilter(frame, action) {{
const label = String(action?.label || '').trim();
const desiredValues = Array.isArray(action?.values)
? action.values.map((value) => String(value || '').trim()).filter(Boolean)
: [];
const expectedButtonText = desiredValues.length === 1 ? desiredValues[0] : desiredValues.join(', ');
const button = await locateQuickFilterButton(frame, label, Number(spec.quick_filter_wait_ms || 30000));
if (!button) {{
return {{
ok: false,
action: 'missing-filter',
detail: {{
label,
availableHeadings: await collectQuickFilterHeadings(frame).catch(() => []),
}},
}};
}}
const initialButtonText = await readLocatorText(button);
if (expectedButtonText && initialButtonText.includes(expectedButtonText)) {{
return {{
ok: true,
action: 'already-set',
detail: {{
initialButtonText,
finalButtonText: initialButtonText,
expectedButtonText,
desiredValues,
checkedBefore: [],
checkedAfter: [],
desiredStates: [],
actionsTaken: [],
missingValues: [],
}},
}};
}}
const listbox = await openQuickFilter(frame, button, label, Number(spec.quick_filter_wait_ms || 30000));
const checkedBefore = await listCheckedRows(listbox);
const actionsTaken = [];
const deselectLabels = [];
const currentValue = String(initialButtonText || '').trim();
if (
currentValue &&
currentValue !== expectedButtonText &&
currentValue !== allFilterValue &&
currentValue !== '(多个值)' &&
currentValue !== '(无)'
) {{
deselectLabels.push(currentValue);
}} else {{
const handledCheckedLabels = new Set();
for (const checkedEntry of checkedBefore) {{
const checkedLabel = String(checkedEntry.text || '').trim();
if (!checkedLabel || desiredValues.includes(checkedLabel) || handledCheckedLabels.has(checkedLabel)) {{
continue;
}}
handledCheckedLabels.add(checkedLabel);
deselectLabels.push(checkedLabel);
}}
}}
for (const deselectLabel of deselectLabels) {{
const checkedRow = await findRow(listbox, deselectLabel);
if (!checkedRow) {{
continue;
}}
const method = await clickFakeCheckbox(checkedRow);
actionsTaken.push({{
kind: 'deselect',
label: deselectLabel,
method,
}});
await page.waitForTimeout(300);
}}
const missingValues = [];
for (const desiredValue of desiredValues) {{
const desiredRow = await findRow(listbox, desiredValue);
if (!desiredRow) {{
missingValues.push(desiredValue);
continue;
}}
const desiredState = await readRowState(desiredRow);
if (!desiredState.checked) {{
const method = await clickFakeCheckbox(desiredRow);
actionsTaken.push({{
kind: 'select',
label: desiredValue,
method,
}});
await page.waitForTimeout(300);
}}
}}
const desiredStates = [];
for (const desiredValue of desiredValues) {{
const desiredRow = await findRow(listbox, desiredValue);
desiredStates.push({{
label: desiredValue,
...(desiredRow ? await readRowState(desiredRow) : {{ exists: false, checked: false, text: desiredValue }}),
}});
}}
const checkedAfter = await listCheckedRows(listbox);
const closeMethod = await closeQuickFilter(button);
const finalButtonText = await waitForButtonText(button, expectedButtonText, Number(spec.close_wait_ms || 15000));
const ok =
missingValues.length === 0 &&
desiredStates.every((entry) => entry.checked) &&
(!expectedButtonText || finalButtonText.includes(expectedButtonText));
return {{
ok,
action: 'applied',
detail: {{
initialButtonText,
finalButtonText,
expectedButtonText,
desiredValues,
checkedBefore,
checkedAfter,
desiredStates,
actionsTaken,
missingValues,
closeMethod,
}},
}};
}}
const located = await locateQuickFilterFrame(Number(spec.tableau_wait_ms || 90000), spec.inner_frame_fragment || '');
const frame = located.frame;
const logs = [];
const filters = {{}};
for (const action of (spec.filter_actions || [])) {{
if (action?.kind !== 'quick-filter' && action?.kind !== 'filter') {{
continue;
}}
try {{
const result = await applyQuickFilter(frame, action);
const label = String(action.label || '').trim();
logs.push({{
kind: 'quick-filter-ui',
label: label.toLowerCase(),
ok: result.ok,
action: result.action,
detail: result.detail,
}});
filters[label] = {{
buttonText: String(result.detail?.finalButtonText || result.detail?.initialButtonText || '').trim(),
value: String(result.detail?.finalButtonText || result.detail?.initialButtonText || '').trim(),
}};
}} catch (error) {{
logs.push({{
kind: 'quick-filter-ui',
label: String(action.label || '').toLowerCase(),
ok: false,
error: String(error && error.message || error),
}});
}}
}}
return {{
frameUrl: frame.url(),
frameCandidates: located.lastFrames,
quickFilterHeadings: await collectQuickFilterHeadings(frame).catch(() => []),
logs,
filters,
}};
}}
"""
def build_wait_for_top_products_js(spec: dict[str, Any]) -> str:
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
await page.waitForLoadState('domcontentloaded').catch(() => null);
async function inspectFrame(frame, preferredFragment) {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : null;
try {{
const snapshot = await frame.evaluate(() => {{
const vizCount = window.tableau?.VizManager?.getVizs?.().length || 0;
return {{
url: location.href,
title: document.title,
readyState: document.readyState,
hasTableau: !!(window.tableau && window.tableau.VizManager),
vizCount,
}};
}});
return {{
...snapshot,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
}};
}} catch (error) {{
return {{
url: frameUrl,
title: null,
readyState: null,
hasTableau: false,
vizCount: 0,
isMainFrame: frame === page.mainFrame(),
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
error: String(error && error.message || error),
}};
}}
}}
async function locateVizFrame(timeoutMs, preferredFragment) {{
const deadline = Date.now() + timeoutMs;
let lastSnapshots = [];
while (Date.now() < deadline) {{
const rankedFrames = page.frames()
.map((frame) => {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
const isPreferred = !!(preferredFragment && frameUrl.includes(preferredFragment));
return {{
frame,
rank: isPreferred ? 0 : (frame === page.mainFrame() ? 1 : 2),
}};
}})
.sort((left, right) => left.rank - right.rank);
lastSnapshots = [];
for (const item of rankedFrames) {{
const snapshot = await inspectFrame(item.frame, preferredFragment);
lastSnapshots.push(snapshot);
if (snapshot.vizCount > 0) {{
return {{ frame: item.frame, snapshots: lastSnapshots }};
}}
}}
await page.waitForTimeout(1000);
}}
throw new Error('Timed out waiting for Tableau viz: ' + JSON.stringify({{ pageUrl: page.url(), frames: lastSnapshots }}));
}}
const located = await locateVizFrame(Number(spec.tableau_wait_ms || 90000), spec.inner_frame_fragment || '');
return await located.frame.evaluate(async (config) => {{
const sleep = (ms) => new Promise((resolve) => window.setTimeout(resolve, ms));
const extractRowSignature = (table) => {{
const columns = table.getColumns().map((col) => col.getFieldName());
const rows = table.getData();
const articleIdx = columns.findIndex((name) => name === 'Articlecode');
const qtyIdx = columns.findIndex((name) => name === '鎬诲拰(Soldqty)');
const rankIdx = columns.findIndex((name) => name === '鑱氬悎(No.)');
const imageIdx = columns.findIndex((name) => String(name || '').includes('max url'));
const signatureRows = rows
.map((row) => {{
const article = articleIdx >= 0 ? String(row[articleIdx]?.formattedValue || row[articleIdx]?.value || '').trim() : '';
const qty = qtyIdx >= 0 ? String(row[qtyIdx]?.formattedValue || row[qtyIdx]?.value || '') : '';
const rank = rankIdx >= 0 ? String(row[rankIdx]?.formattedValue || row[rankIdx]?.value || '') : '';
const image = imageIdx >= 0 ? String(row[imageIdx]?.formattedValue || row[imageIdx]?.value || '').trim() : '';
return article || image ? [article, qty, rank, image].join('|') : null;
}})
.filter(Boolean);
return {{
columns,
rowCount: rows.length,
validRowCount: signatureRows.length,
signature: JSON.stringify(signatureRows),
}};
}};
const workbookDeadline = Date.now() + Number(config.workbook_wait_ms || 60000);
let viz = null;
let workbook = null;
while (Date.now() < workbookDeadline) {{
try {{
const vizs = window.tableau?.VizManager?.getVizs?.() || [];
viz = vizs[0] || null;
workbook = viz && typeof viz.getWorkbook === 'function' ? viz.getWorkbook() : null;
}} catch (error) {{
workbook = null;
}}
if (workbook) {{
break;
}}
await sleep(500);
}}
if (!workbook) {{
throw new Error('Tableau workbook is not ready for wait polling.');
}}
try {{
await workbook.activateSheetAsync(config.activate_sheet);
}} catch (error) {{
}}
const activeSheet = workbook.getActiveSheet();
const worksheetList = activeSheet && typeof activeSheet.getWorksheets === 'function'
? activeSheet.getWorksheets()
: [];
const worksheets = Object.fromEntries(
worksheetList.map((worksheet) => [worksheet.getName(), worksheet])
);
const worksheet = worksheets.TOP || null;
if (!worksheet || typeof worksheet.getSummaryDataAsync !== 'function') {{
return {{
frameUrl: location.href,
activeSheet: activeSheet && typeof activeSheet.getName === 'function' ? activeSheet.getName() : null,
waitResult: {{
ready: false,
skipped: true,
reason: 'TOP worksheet not available for polling',
}},
}};
}}
const pollIntervalMs = Number(config.post_filter_poll_interval_ms || 1000);
const timeoutMs = Number(config.post_filter_wait_ms || 180000);
const fallbackWaitMs = Number(config.post_filter_fallback_wait_ms || 0);
const stablePollsRequired = Number(config.post_filter_stable_polls || 2);
const expectedTopRows = Number(config.expected_top_rows || 10);
const deadline = Date.now() + timeoutMs;
const startedAt = Date.now();
let lastSignature = null;
let stablePolls = 0;
let lastSnapshot = null;
while (Date.now() < deadline) {{
try {{
const table = await worksheet.getSummaryDataAsync({{ maxRows: expectedTopRows }});
const snapshot = extractRowSignature(table);
lastSnapshot = snapshot;
if (snapshot.signature === lastSignature) {{
stablePolls += 1;
}} else {{
stablePolls = 1;
lastSignature = snapshot.signature;
}}
const elapsedMs = Date.now() - startedAt;
const hasExpectedRows = snapshot.validRowCount >= expectedTopRows;
const stableEnough = stablePolls >= stablePollsRequired;
if (hasExpectedRows && stableEnough) {{
return {{
frameUrl: location.href,
activeSheet: activeSheet && typeof activeSheet.getName === 'function' ? activeSheet.getName() : null,
waitResult: {{
ready: true,
skipped: false,
elapsedMs,
stablePolls,
...snapshot,
}},
}};
}}
}} catch (error) {{
lastSnapshot = {{
rowCount: 0,
validRowCount: 0,
signature: null,
error: String(error && error.message || error),
}};
}}
await sleep(pollIntervalMs);
}}
if (fallbackWaitMs > 0) {{
await sleep(fallbackWaitMs);
}}
return {{
frameUrl: location.href,
activeSheet: activeSheet && typeof activeSheet.getName === 'function' ? activeSheet.getName() : null,
waitResult: {{
ready: false,
skipped: false,
timedOut: true,
stablePolls,
...(lastSnapshot || {{ rowCount: 0, validRowCount: 0, signature: null }}),
}},
}};
}}, {{
activate_sheet: spec.activate_sheet,
expected_top_rows: spec.expected_top_rows || 10,
workbook_wait_ms: spec.workbook_wait_ms || 60000,
post_filter_wait_ms: spec.post_filter_wait_ms || 180000,
post_filter_poll_interval_ms: spec.post_filter_poll_interval_ms || 1000,
post_filter_fallback_wait_ms: spec.post_filter_fallback_wait_ms || 0,
post_filter_stable_polls: spec.post_filter_stable_polls || 2,
}});
}}
"""
def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
"""Helper."""
payload = json.dumps(
{
"inner_frame_fragment": inner_frame_fragment,
......@@ -314,7 +1401,7 @@ def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
)
return f"""async function(page) {{
const spec = {payload};
await page.waitForTimeout(3000);
await page.waitForLoadState('domcontentloaded').catch(() => null);
const frame = page.frames().find(
(candidate) => candidate !== page.mainFrame() && candidate.url().includes(spec.inner_frame_fragment)
);
......@@ -334,25 +1421,216 @@ def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
def run_code(session: str, script_path: Path, *, cwd: Path, timeout: int = 120) -> str:
return run_playwright(["--session", session, "run-code", "--filename", str(script_path)], cwd=cwd, timeout=timeout)
"""Helper."""
return run_playwright(["--session", session, "run-code", "--filename", str(script_path.resolve())], cwd=cwd, timeout=timeout)
def parse_playwright_result(output: str) -> Any:
"""Helper."""
if "### Error" in output:
raise RuntimeError(output.strip())
marker = "### Result"
start = output.find(marker)
if start == -1:
raise RuntimeError(f"Playwright output missing result marker: {output.strip()}")
payload = output[start + len(marker):].lstrip()
next_marker = payload.find("\n### ")
if next_marker != -1:
payload = payload[:next_marker]
return json.loads(payload.strip())
def extract_playwright_snapshot_yaml(output: str) -> str:
"""Extract the YAML snapshot block from playwright-cli snapshot output."""
marker = "### Snapshot"
start = output.find(marker)
if start == -1:
return output
code_start = output.find("```yaml", start)
if code_start == -1:
return output[start + len(marker) :]
code_start += len("```yaml")
code_end = output.find("```", code_start)
if code_end == -1:
code_end = len(output)
return output[code_start:code_end].strip()
def parse_top_products_quick_filter_snapshot(snapshot_yaml: str) -> dict[str, dict[str, Any]]:
"""Parse quick-filter refs and current values from a Tableau page snapshot."""
heading_pattern = re.compile(r'^\s*-\s+heading "筛选器 (?P<label>.+?) 包含".*$')
button_pattern = re.compile(r'^\s*-\s+button "筛选器 (?P<label>.+?) 包含".*?\[ref=(?P<ref>[^\]]+)\]:$')
value_pattern = re.compile(r'^\s*-\s+generic \[ref=[^\]]+\]: (?P<value>.+)$')
show_all_pattern = re.compile(
r'^\s*-\s+button "(?P<label>显示所有值|单击以显示所有值|單擊以顯示所有值)".*?\[ref=(?P<ref>[^\]]+)\]:$'
)
context_menu_pattern = re.compile(r'^\s*-\s+button "显示快速筛选器上下文菜单".*?\[ref=(?P<ref>[^\]]+)\].*$')
filters: dict[str, dict[str, Any]] = {}
current_filter: str | None = None
pending_value_filter: str | None = None
for line in snapshot_yaml.splitlines():
heading_match = heading_pattern.search(line)
if heading_match:
current_filter = heading_match.group("label").strip()
filters.setdefault(current_filter, {})
continue
button_match = button_pattern.search(line)
if button_match:
current_filter = button_match.group("label").strip()
filters.setdefault(current_filter, {})["button_ref"] = button_match.group("ref")
pending_value_filter = current_filter
continue
if pending_value_filter:
value_match = value_pattern.search(line)
if value_match:
filters.setdefault(pending_value_filter, {})["value"] = value_match.group("value").strip()
pending_value_filter = None
continue
if current_filter:
show_all_match = show_all_pattern.search(line)
if show_all_match:
entry = filters.setdefault(current_filter, {})
entry["show_all_ref"] = show_all_match.group("ref")
entry["show_all_label"] = show_all_match.group("label")
entry["show_all_disabled"] = "[disabled]" in line
continue
context_menu_match = context_menu_pattern.search(line)
if context_menu_match:
entry = filters.setdefault(current_filter, {})
entry["context_menu_ref"] = context_menu_match.group("ref")
entry["context_menu_disabled"] = "[disabled]" in line
return filters
def normalize_top_products_quick_filters(
session: str,
*,
cwd: Path,
timeout_seconds: int = 30,
) -> dict[str, Any]:
"""Normalize the hidden Week/Daily quick filters using snapshot refs."""
started_at = time.monotonic()
click_log: list[dict[str, Any]] = []
last_filters: dict[str, dict[str, Any]] = {}
last_snapshot_yaml = ""
targets = ("Week", "Daily")
while time.monotonic() - started_at < timeout_seconds:
snapshot_output = run_playwright(["--session", session, "snapshot"], cwd=cwd, timeout=60)
last_snapshot_yaml = extract_playwright_snapshot_yaml(snapshot_output)
last_filters = parse_top_products_quick_filter_snapshot(last_snapshot_yaml)
if all(str(last_filters.get(target, {}).get("value") or "").strip() == ALL_FILTER_VALUE for target in targets):
return {
"ok": True,
"filters": last_filters,
"click_log": click_log,
"elapsed_ms": int(round((time.monotonic() - started_at) * 1000)),
}
if "响应时间过长" in last_snapshot_yaml:
return {
"ok": False,
"filters": last_filters,
"click_log": click_log,
"elapsed_ms": int(round((time.monotonic() - started_at) * 1000)),
"reason": "backend-timeout",
"snapshot_excerpt": last_snapshot_yaml[:4000],
}
clicked = False
for target in targets:
state = last_filters.get(target, {})
current_value = str(state.get("value") or "").strip()
show_all_ref = str(state.get("show_all_ref") or "").strip()
if show_all_ref and not state.get("show_all_disabled"):
run_playwright(["--session", session, "click", show_all_ref], cwd=cwd, timeout=60)
click_log.append(
{
"kind": "quick-filter-snapshot",
"target": target,
"action": "click-show-all",
"ref": show_all_ref,
"label": state.get("show_all_label"),
"before": current_value,
}
)
clicked = True
break
context_menu_ref = str(state.get("context_menu_ref") or "").strip()
if context_menu_ref and not state.get("context_menu_disabled"):
run_playwright(["--session", session, "click", context_menu_ref], cwd=cwd, timeout=60)
click_log.append(
{
"kind": "quick-filter-snapshot",
"target": target,
"action": "open-context-menu",
"ref": context_menu_ref,
"before": current_value,
}
)
clicked = True
break
button_ref = str(state.get("button_ref") or "").strip()
if button_ref and current_value in {"", "(无)"}:
run_playwright(["--session", session, "click", button_ref], cwd=cwd, timeout=60)
click_log.append(
{
"kind": "quick-filter-snapshot",
"target": target,
"action": "open-filter",
"ref": button_ref,
"before": current_value,
}
)
clicked = True
break
if clicked:
time.sleep(1)
continue
time.sleep(1)
return {
"ok": False,
"filters": last_filters,
"click_log": click_log,
"elapsed_ms": int(round((time.monotonic() - started_at) * 1000)),
"snapshot_excerpt": last_snapshot_yaml[:4000],
}
def ensure_browser_session(session: str, *, cwd: Path) -> None:
"""Helper."""
run_playwright(["--session", session, "open", "about:blank"], cwd=cwd, timeout=60)
def save_state(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
state_path.parent.mkdir(parents=True, exist_ok=True)
run_playwright(["--session", session, "state-save", str(state_path)], cwd=cwd, timeout=60)
def load_state_if_present(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
if not state_path.exists():
return
run_playwright(["--session", session, "state-load", str(state_path)], cwd=cwd, timeout=60)
def locate_session_file(root: Path, session: str, filename: str) -> Path:
"""Helper."""
direct = root / "output" / "playwright" / session / filename
if direct.exists():
return direct
......@@ -370,6 +1648,7 @@ def locate_session_file(root: Path, session: str, filename: str) -> Path:
def crop_image(source: Path, target: Path, crop: dict[str, int], *, resize_to: dict[str, int] | None = None) -> None:
"""Helper."""
image = Image.open(source)
box = (
crop["left"],
......@@ -392,7 +1671,9 @@ def capture_tableau_view(
workdir: Path,
workspace_root: Path,
) -> Path:
"""Helper."""
target_url = f"{base_url}{capture_spec['hash_url']}"
def reset_target_page() -> None:
run_playwright(["--session", session, "goto", target_url], cwd=workdir, timeout=120)
run_playwright(
["--session", session, "resize", str(VIEWPORT["width"]), str(VIEWPORT["height"])],
......@@ -400,65 +1681,426 @@ def capture_tableau_view(
timeout=60,
)
configure_script = write_js(
reset_target_page()
pre_configure_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-configure-pre.js",
build_configure_view_js(
{
"activate_sheet": capture_spec["activate_sheet"],
"filter_actions": capture_spec.get("api_pre_filter_actions", []),
"inner_frame_fragment": capture_spec["inner_frame_fragment"],
"revert_before_apply": True,
}
),
)
ui_filter_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-quick-filters.js",
build_normalize_quick_filters_js(
{
"filter_actions": capture_spec.get("ui_filter_actions", []),
"inner_frame_fragment": capture_spec["inner_frame_fragment"],
}
),
)
post_configure_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-configure.js",
f"tmp-{capture_spec['capture_id']}-configure-post.js",
build_configure_view_js(
{
"activate_sheet": capture_spec["activate_sheet"],
"filter_actions": capture_spec["filter_actions"],
"filter_actions": capture_spec.get("api_post_filter_actions", []),
"inner_frame_fragment": capture_spec["inner_frame_fragment"],
"revert_before_apply": False,
}
),
)
wait_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-wait.js",
build_wait_for_top_products_js(
{
"activate_sheet": capture_spec["activate_sheet"],
"inner_frame_fragment": capture_spec["inner_frame_fragment"],
"expected_top_rows": capture_spec.get("expected_top_rows", 10),
"post_filter_wait_ms": capture_spec.get("post_filter_wait_ms", 180000),
"post_filter_poll_interval_ms": capture_spec.get("post_filter_poll_interval_ms", 1000),
"post_filter_fallback_wait_ms": capture_spec.get("post_filter_fallback_wait_ms", 0),
"post_filter_stable_polls": capture_spec.get("post_filter_stable_polls", 2),
}
),
)
capture_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-capture.js",
build_capture_js(capture_spec["inner_frame_fragment"], capture_spec["raw_screenshot_name"]),
build_export_image_js(
capture_spec["inner_frame_fragment"],
str((workdir / capture_spec["raw_screenshot_name"]).resolve()),
viewport=VIEWPORT,
),
)
try:
for attempt in range(1, 6):
configure_output = run_code(session, configure_script, cwd=workdir, timeout=420).strip()
if "### Error" in configure_output:
if attempt < 5:
run_playwright(["--session", session, "goto", target_url], cwd=workdir, timeout=120)
run_playwright(
["--session", session, "resize", str(VIEWPORT["width"]), str(VIEWPORT["height"])],
cwd=workdir,
timeout=60,
max_attempts = 5
last_error: str | None = None
for attempt in range(1, max_attempts + 1):
try:
pre_configure_output = run_code(session, pre_configure_script, cwd=workdir, timeout=420).strip()
except subprocess.TimeoutExpired as exc:
last_error = f"TimeoutExpired: {exc}"
pre_configure_output = f"### Error\nError: {last_error}"
if "### Error" in pre_configure_output:
last_error = pre_configure_output
if "chrome-error://chromewebdata/" in pre_configure_output or "HTTP ERROR 502" in pre_configure_output:
raise RuntimeError(
f"Failed to configure Tableau view for {capture_spec['capture_id']}: {last_error}"
)
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(
f"Failed to configure Tableau view for {capture_spec['capture_id']}: {last_error}"
)
pre_configure_result = parse_playwright_result(pre_configure_output)
configure_result: dict[str, Any] = {
"preApiConfigure": pre_configure_result,
"quickFilterUi": [],
"quickFilterState": {},
"postApiConfigure": {},
}
try:
ui_filter_output = run_code(session, ui_filter_script, cwd=workdir, timeout=180).strip()
except subprocess.TimeoutExpired as exc:
last_error = f"TimeoutExpired: {exc}"
ui_filter_output = f"### Error\nError: {last_error}"
if "### Error" in ui_filter_output:
last_error = ui_filter_output
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(
f"Failed to apply Tableau quick filters for {capture_spec['capture_id']}: {last_error}"
)
ui_filter_result = parse_playwright_result(ui_filter_output)
configure_result["quickFilterUi"] = ui_filter_result.get("logs", [])
configure_result["quickFilterState"] = ui_filter_result.get("filters", {})
failed_quick_filters = [item for item in configure_result["quickFilterUi"] if not item.get("ok")]
if failed_quick_filters:
last_error = json.dumps(
{
"configure": configure_result,
"uiFilterResult": ui_filter_result,
},
ensure_ascii=False,
)
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(
f"Failed to apply Tableau quick filters for {capture_spec['capture_id']}: {last_error}"
)
try:
post_configure_output = run_code(session, post_configure_script, cwd=workdir, timeout=420).strip()
except subprocess.TimeoutExpired as exc:
last_error = f"TimeoutExpired: {exc}"
post_configure_output = f"### Error\nError: {last_error}"
if "### Error" in post_configure_output:
last_error = post_configure_output
if "chrome-error://chromewebdata/" in post_configure_output or "HTTP ERROR 502" in post_configure_output:
raise RuntimeError(
f"Failed to configure Tableau view for {capture_spec['capture_id']}: {last_error}"
)
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(
f"Failed to configure Tableau view for {capture_spec['capture_id']}: {last_error}"
)
time.sleep(3)
configure_result["postApiConfigure"] = parse_playwright_result(post_configure_output)
try:
wait_output = run_code(session, wait_script, cwd=workdir, timeout=240).strip()
except subprocess.TimeoutExpired as exc:
last_error = f"TimeoutExpired: {exc}"
wait_output = f"### Error\nError: {last_error}"
if "### Error" in wait_output:
last_error = wait_output
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(
f"Failed while waiting for Tableau data for {capture_spec['capture_id']}: {last_error}"
)
wait_payload = parse_playwright_result(wait_output)
wait_result = wait_payload.get("waitResult", {})
configure_result["waitResult"] = wait_result
if not wait_result.get("ready"):
last_error = json.dumps(configure_result, ensure_ascii=False)
raise RuntimeError(
f"Failed to configure Tableau view for {capture_spec['capture_id']}: {last_error}"
)
capture_output = run_code(session, capture_script, cwd=workdir, timeout=240).strip()
if "### Error" in capture_output:
last_error = capture_output
if "chrome-error://chromewebdata/" in capture_output or "HTTP ERROR 502" in capture_output:
raise RuntimeError(
f"Failed to capture Tableau view for {capture_spec['capture_id']}: {last_error}"
)
if attempt < max_attempts:
reset_target_page()
time.sleep(1)
continue
raise RuntimeError(f"Failed to configure Tableau view: {configure_output}")
run_code(session, capture_script, cwd=workdir, timeout=180)
raise RuntimeError(
f"Failed to capture Tableau view for {capture_spec['capture_id']}: {last_error}"
)
break
finally:
configure_script.unlink(missing_ok=True)
pre_configure_script.unlink(missing_ok=True)
ui_filter_script.unlink(missing_ok=True)
post_configure_script.unlink(missing_ok=True)
wait_script.unlink(missing_ok=True)
capture_script.unlink(missing_ok=True)
local_output = workdir / capture_spec["raw_screenshot_name"]
if local_output.exists():
normalize_image_size(local_output, {"width": 1400, "height": 3760})
return local_output
return locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
captured_path = locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
normalize_image_size(captured_path, {"width": 1400, "height": 3760})
return captured_path
def build_dump_top_products_js(
limit: int,
*,
activate_sheet: str = TOP_PRODUCTS_DASHBOARD,
inner_frame_fragment: str = "/views/CKTopProducts-General_16862068169500/TopProducts?",
) -> str:
"""Helper."""
payload = json.dumps(
{
"limit": limit,
"activate_sheet": activate_sheet,
"inner_frame_fragment": inner_frame_fragment,
},
ensure_ascii=False,
)
return f"""async function(page) {{
const spec = {payload};
await page.waitForLoadState('domcontentloaded').catch(() => null);
async function inspectFrame(frame, preferredFragment) {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
try {{
const snapshot = await frame.evaluate(() => {{
const vizCount = window.tableau?.VizManager?.getVizs?.().length || 0;
return {{
url: location.href,
title: document.title,
readyState: document.readyState,
hasTableau: !!(window.tableau && window.tableau.VizManager),
vizCount,
}};
}});
return {{
...snapshot,
frameUrl,
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
}};
}} catch (error) {{
return {{
url: frameUrl,
title: null,
readyState: null,
hasTableau: false,
vizCount: 0,
frameUrl,
matchesTargetFrame: !!(preferredFragment && frameUrl && frameUrl.includes(preferredFragment)),
error: String(error && error.message || error),
}};
}}
}}
async function locateVizFrame(timeoutMs, preferredFragment) {{
const deadline = Date.now() + timeoutMs;
let lastSnapshots = [];
while (Date.now() < deadline) {{
const rankedFrames = page.frames()
.map((frame) => {{
const frameUrl = typeof frame.url === 'function' ? frame.url() : '';
const isPreferred = !!(preferredFragment && frameUrl.includes(preferredFragment));
return {{
frame,
rank: isPreferred ? 0 : (frame === page.mainFrame() ? 1 : 2),
}};
}})
.sort((left, right) => left.rank - right.rank);
lastSnapshots = [];
for (const item of rankedFrames) {{
const snapshot = await inspectFrame(item.frame, preferredFragment);
lastSnapshots.push(snapshot);
if (snapshot.vizCount > 0) {{
return {{ frame: item.frame, snapshots: lastSnapshots }};
}}
}}
await page.waitForTimeout(1000);
}}
throw new Error('No Tableau viz frame found for top-products dump: ' + JSON.stringify({{
pageUrl: page.url(),
frames: lastSnapshots,
}}));
}}
const located = await locateVizFrame(30000, spec.inner_frame_fragment || '');
const targetFrame = located.frame;
return await targetFrame.evaluate(async (config) => {{
const viz = window.tableau?.VizManager?.getVizs?.()?.[0];
if (!viz) {{
throw new Error('No Tableau viz found in target frame');
}}
const workbook = viz.getWorkbook();
try {{
await workbook.activateSheetAsync(config.activate_sheet);
}} catch (error) {{
}}
const activeSheet = workbook.getActiveSheet();
const worksheetList = activeSheet && typeof activeSheet.getWorksheets === 'function'
? activeSheet.getWorksheets()
: [];
const worksheet = worksheetList.find((item) => item.getName() === 'TOP')
|| (
activeSheet &&
typeof activeSheet.getName === 'function' &&
activeSheet.getName() === 'TOP'
? activeSheet
: null
);
if (!worksheet) {{
throw new Error('TOP worksheet not found');
}}
const table = await worksheet.getSummaryDataAsync({{ maxRows: config.limit }});
const columns = table.getColumns().map((col) => col.getFieldName());
const rows = table.getData().map((row) =>
Object.fromEntries(row.map((cell, idx) => [
columns[idx],
{{ value: cell.value, formattedValue: cell.formattedValue }},
]))
);
return {{
pageUrl: location.href,
activeSheet: activeSheet && activeSheet.getName ? activeSheet.getName() : null,
frameUrl: location.href,
columns,
rows,
}};
}}, spec);
}}
"""
def extract_top_products_from_current_view(session: str, *, workdir: Path, limit: int) -> dict[str, Any]:
"""Helper."""
script_path = write_js(
workdir,
"tmp-top-products-dump.js",
build_dump_top_products_js(limit, activate_sheet=TOP_PRODUCTS_DASHBOARD),
)
try:
result = parse_playwright_result(run_code(session, script_path, cwd=workdir, timeout=240))
finally:
script_path.unlink(missing_ok=True)
columns = [str(column or "").strip() for column in result.get("columns", [])]
def resolve_column(exact_candidates: list[str], token_candidates: list[str]) -> str | None:
lowered = {column.lower(): column for column in columns}
for candidate in exact_candidates:
resolved = lowered.get(candidate.lower())
if resolved:
return resolved
for column in columns:
normalized = column.lower()
if any(token in normalized for token in token_candidates):
return column
return None
article_column = resolve_column(["Articlecode"], ["articlecode"])
qty_column = resolve_column(["鎬诲拰(Soldqty)", "Sum(Soldqty)", "SUM(Soldqty)"], ["soldqty"])
image_column = resolve_column(["鑱氬悎(max url)", "AGG(max url)", "AGG(Max Url)", "max url"], ["url"])
rank_column = resolve_column(["鑱氬悎(No.)", "AGG(No.)", "No."], ["no."])
normalized_rows: list[dict[str, Any]] = []
for raw_row in result.get("rows", []):
article_cell = raw_row.get(article_column or "", {}) if article_column else {}
qty_cell = raw_row.get(qty_column or "", {}) if qty_column else {}
image_cell = raw_row.get(image_column or "", {}) if image_column else {}
rank_cell = raw_row.get(rank_column or "", {}) if rank_column else {}
articlecode = str(article_cell.get("formattedValue") or article_cell.get("value") or "").strip()
qty_value = qty_cell.get("value")
image_url = str(image_cell.get("formattedValue") or image_cell.get("value") or "").strip()
rank_value = rank_cell.get("value")
if not articlecode:
continue
try:
qty = int(float(qty_value or 0))
except (TypeError, ValueError):
qty = 0
try:
rank = int(float(rank_value or 0))
except (TypeError, ValueError):
rank = 0
normalized_rows.append(
{
"articlecode": articlecode,
"qty": qty,
"sales": 0.0,
"image_url": image_url,
"rank": rank,
}
)
normalized_rows.sort(key=lambda row: (row["rank"] or 9999, -row["qty"], row["articlecode"]))
if len(normalized_rows) < limit:
raise RuntimeError(
"Tableau top products rows are insufficient: "
f"expected {limit}, got {len(normalized_rows)}, columns={columns}"
)
return {
"source": "tableau",
"page_url": result.get("pageUrl"),
"active_sheet": result.get("activeSheet"),
"columns": columns,
"rows": normalized_rows[:limit],
}
def month_date_range(report_year: int, report_month: str) -> tuple[str, str]:
month_index = resolve_month_index(report_month)
"""Helper."""
normalized_month = normalize_month_label(report_month)
month_index = resolve_month_index(normalized_month)
last_day = calendar.monthrange(report_year, month_index)[1]
return f"{report_year}-{month_index:02d}-01", f"{report_year}-{month_index:02d}-{last_day:02d}"
def resolve_shop(config: dict[str, Any], store_name: str) -> tuple[int, str]:
mysql_cfg = config["mysql"]
conn = pymysql.connect(
host=mysql_cfg["host"],
port=int(mysql_cfg["port"]),
user=mysql_cfg["username"],
password=mysql_cfg["password"],
database=mysql_cfg["database"],
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
)
def resolve_shop(config: dict[str, Any], store_name: str) -> tuple[int, str, dict[str, Any], str]:
"""Helper."""
conn, mysql_cfg, mysql_label = open_mysql_connection(config)
try:
with conn.cursor() as cursor:
cursor.execute(
......@@ -476,7 +2118,7 @@ def resolve_shop(config: dict[str, Any], store_name: str) -> tuple[int, str]:
if not row or row.get("shopid") is None:
raise RuntimeError(f"Unable to resolve shop for storename={store_name!r}")
return int(row["shopid"]), str(row["shopname"])
return int(row["shopid"]), str(row["shopname"]), mysql_cfg, mysql_label
def fetch_top_products(
......@@ -488,18 +2130,15 @@ def fetch_top_products(
level1: str,
limit: int,
) -> dict[str, Any]:
shop_id, resolved_shop_name = resolve_shop(config, store_name)
start_date, end_date = month_date_range(report_year, report_month)
mysql_cfg = config["mysql"]
conn = pymysql.connect(
host=mysql_cfg["host"],
port=int(mysql_cfg["port"]),
user=mysql_cfg["username"],
password=mysql_cfg["password"],
database=mysql_cfg["database"],
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
"""Helper."""
shop_id, resolved_shop_name, mysql_cfg, mysql_label = resolve_shop(config, store_name)
if mysql_label != "config.mysql":
log_progress(
"sql",
f"use mysql fallback {mysql_label} -> {mysql_cfg['host']}:{mysql_cfg['port']}/{mysql_cfg['database']}",
)
start_date, end_date = month_date_range(report_year, report_month)
conn = connect_mysql(mysql_cfg)
try:
with conn.cursor() as cursor:
cursor.execute(
......@@ -507,7 +2146,8 @@ def fetch_top_products(
SELECT
articlecode,
SUM(CASE WHEN qty > 0 THEN qty ELSE 0 END) AS qty,
SUM(CASE WHEN sales > 0 THEN sales ELSE 0 END) AS sales
SUM(CASE WHEN sales > 0 THEN sales ELSE 0 END) AS sales,
SUM(CASE WHEN soh > 0 THEN soh ELSE 0 END) AS soh
FROM oms_daily_report_detail
WHERE shopid = %s
AND daily BETWEEN %s AND %s
......@@ -520,6 +2160,21 @@ def fetch_top_products(
(shop_id, start_date, end_date, level1, limit),
)
rows = cursor.fetchall()
cursor.execute(
"""
SELECT
SUM(CASE WHEN qty > 0 THEN qty ELSE 0 END) AS total_qty,
SUM(CASE WHEN sales > 0 THEN sales ELSE 0 END) AS total_sales,
SUM(CASE WHEN soh > 0 THEN soh ELSE 0 END) AS total_soh
FROM oms_daily_report_detail
WHERE shopid = %s
AND daily BETWEEN %s AND %s
AND level1 = %s
""",
(shop_id, start_date, end_date, level1),
)
totals = cursor.fetchone()
finally:
conn.close()
......@@ -528,6 +2183,7 @@ def fetch_top_products(
"articlecode": str(row["articlecode"]),
"qty": int(row["qty"] or 0),
"sales": float(row["sales"] or 0),
"soh": float(row.get("soh") or 0),
}
for row in rows
if row.get("articlecode")
......@@ -536,17 +2192,139 @@ def fetch_top_products(
raise RuntimeError(f"{level1} top products rows are insufficient: expected {limit}, got {len(normalized_rows)}")
return {
"source": "sql",
"store_name": store_name,
"resolved_shop_name": resolved_shop_name,
"shop_id": shop_id,
"mysql_target": mysql_label,
"mysql_host": mysql_cfg["host"],
"mysql_port": int(mysql_cfg["port"]),
"mysql_database": mysql_cfg["database"],
"start_date": start_date,
"end_date": end_date,
"level1": level1,
"total_qty": int((totals or {}).get("total_qty") or 0),
"total_sales": float((totals or {}).get("total_sales") or 0.0),
"total_soh": float((totals or {}).get("total_soh") or 0.0),
"rows": normalized_rows[:limit],
}
def style_chart_axis(axis: Any) -> None:
"""Helper."""
axis.set_xticks([])
axis.set_yticks([])
axis.tick_params(left=False, bottom=False, labelleft=False, labelbottom=False)
for spine in axis.spines.values():
spine.set_visible(False)
axis.set_facecolor("white")
def format_chart_number(value: float) -> str:
"""Helper."""
return f"{int(round(value)):,}"
def render_sql_top_products_chart(source_payload: dict[str, Any], target: Path) -> None:
"""Helper."""
rows = source_payload.get("rows", [])
total_qty = int(source_payload.get("total_qty") or 0)
total_sales = float(source_payload.get("total_sales") or 0.0)
total_soh = float(source_payload.get("total_soh") or 0.0)
if not rows or total_qty <= 0 or total_soh <= 0:
raise RuntimeError("SQL top-products chart source is incomplete.")
qty_values = [int(row["qty"]) for row in rows]
sales_values = [float(row["sales"]) for row in rows]
soh_values = [float(row.get("soh") or 0.0) for row in rows]
qty_percentages = [value / total_qty * 100 for value in qty_values]
sales_percentages = [value / total_sales * 100 for value in sales_values]
top_qty_percentage = sum(qty_values) / total_qty * 100
top_sales_percentage = sum(sales_values) / total_sales * 100
figure = plt.figure(figsize=(10.34, 5.87), dpi=100, facecolor="white")
grid = GridSpec(1, 4, figure=figure, width_ratios=[4.9, 1.8, 1.8, 2.2], wspace=0.08)
ax_qty = figure.add_subplot(grid[0, 0])
ax_qty_pct = figure.add_subplot(grid[0, 1], sharey=ax_qty)
ax_sales_pct = figure.add_subplot(grid[0, 2], sharey=ax_qty)
ax_sales = figure.add_subplot(grid[0, 3], sharey=ax_qty)
y_positions = list(range(len(rows)))
figure.subplots_adjust(left=0.03, right=0.99, top=0.83, bottom=0.05)
figure.text(0.02, 0.96, "Top Parameter", ha="left", va="bottom", fontsize=10)
figure.patches.append(
Rectangle((0.02, 0.915), 0.31, 0.034, transform=figure.transFigure, fill=False, linewidth=0.8, edgecolor="#666666")
)
figure.text(0.024, 0.918, str(len(rows)), ha="left", va="bottom", fontsize=10)
max_qty = max(qty_values) or 1
qty_left_margin = max_qty * 0.72
ax_qty.set_xlim(-qty_left_margin, max_qty * 1.28)
ax_qty.barh(y_positions, qty_values, height=0.75, color=QTY_BAR_COLORS[: len(rows)])
ax_qty.invert_yaxis()
ax_qty.axvline(0, color="#d6d6d6", linewidth=0.8)
ax_qty.text(-qty_left_margin * 0.88, -0.72, "No.", fontsize=10, fontweight="bold", ha="left", va="bottom")
ax_qty.text(-qty_left_margin * 0.47, -0.72, "Articlecode", fontsize=10, fontweight="bold", ha="left", va="bottom")
for index, row in enumerate(rows):
ax_qty.text(-qty_left_margin * 0.88, index, str(index + 1), fontsize=10, ha="left", va="center")
ax_qty.text(-qty_left_margin * 0.64, index, row["articlecode"], fontsize=10, ha="left", va="center")
ax_qty.text(
qty_values[index] + max_qty * 0.02,
index,
format_chart_number(qty_values[index]),
fontsize=9,
fontweight="bold",
ha="left",
va="center",
)
style_chart_axis(ax_qty)
pct_max = max(max(qty_percentages), max(sales_percentages), 1.0)
pct_limit = max(10.0, pct_max + 1.5)
ax_qty_pct.set_xlim(-pct_limit * 1.18, 0)
ax_qty_pct.barh(y_positions, [-value for value in qty_percentages], height=0.75, color="#a7a7a7")
ax_qty_pct.axvline(0, color="#d6d6d6", linewidth=0.8)
ax_qty_pct.set_title(f"Soldqty%\n{round(top_qty_percentage):.0f}%", fontsize=12, fontweight="bold", pad=24)
for index, value in enumerate(qty_percentages):
ax_qty_pct.text(-value - pct_limit * 0.05, index, f"{round(value):.0f}%", fontsize=9, fontweight="bold", ha="right", va="center")
style_chart_axis(ax_qty_pct)
ax_sales_pct.set_xlim(0, pct_limit * 1.18)
ax_sales_pct.barh(y_positions, sales_percentages, height=0.75, color="#8db9db")
ax_sales_pct.axvline(0, color="#d6d6d6", linewidth=0.8)
ax_sales_pct.set_title(f"Sales%\n{round(top_sales_percentage):.0f}%", fontsize=12, fontweight="bold", pad=24)
for index, value in enumerate(sales_percentages):
ax_sales_pct.text(value + pct_limit * 0.05, index, f"{round(value):.0f}%", fontsize=9, fontweight="bold", ha="left", va="center")
style_chart_axis(ax_sales_pct)
sales_title_x = (ax_qty_pct.get_position().x0 + ax_sales_pct.get_position().x1) / 2
figure.text(sales_title_x, 0.965, "TOP N%", ha="center", va="bottom", fontsize=13, fontweight="bold")
max_soh = max(soh_values) or 1.0
ax_sales.set_xlim(0, max_soh * 1.22)
ax_sales.barh(y_positions, soh_values, height=0.75, color="#a7a7a7")
ax_sales.axvline(0, color="#d6d6d6", linewidth=0.8)
for index, value in enumerate(soh_values):
ax_sales.text(
value + max_soh * 0.02,
index,
format_chart_number(value),
fontsize=9,
fontweight="bold",
ha="left",
va="center",
)
ax_sales.text(0, len(rows) - 0.08, "soh", fontsize=9, ha="left", va="bottom")
style_chart_axis(ax_sales)
target.parent.mkdir(parents=True, exist_ok=True)
figure.savefig(target, dpi=100, facecolor="white")
plt.close(figure)
def download_image(url: str, *, attempts: int = 4, timeout: int = 60) -> Image.Image:
"""Helper."""
last_error: Exception | None = None
for attempt in range(1, attempts + 1):
try:
......@@ -564,6 +2342,7 @@ def download_image(url: str, *, attempts: int = 4, timeout: int = 60) -> Image.I
def render_product_image(source: Image.Image, target: Path, *, canvas_width: int, canvas_height: int) -> None:
"""Helper."""
scale = min(canvas_width / source.width, canvas_height / source.height)
resized_width = max(1, int(round(source.width * scale)))
resized_height = max(1, int(round(source.height * scale)))
......@@ -577,6 +2356,7 @@ def render_product_image(source: Image.Image, target: Path, *, canvas_width: int
def sanitize_file_token(value: str) -> str:
"""Helper."""
return re.sub(r"[^A-Za-z0-9_-]+", "_", value).strip("_") or "item"
......@@ -586,13 +2366,14 @@ def build_product_assets(
asset_dir: Path,
source_payload: dict[str, Any],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""Helper."""
slide_spec = SLIDE_SPECS[slide_code]
operations: list[dict[str, Any]] = []
manifest_items: list[dict[str, Any]] = []
for slot, row in zip(slide_spec["product_slots"], source_payload["rows"]):
articlecode = row["articlecode"]
image_url = PRODUCT_IMAGE_URL.format(articlecode=articlecode)
image_url = row.get("image_url") or PRODUCT_IMAGE_URL.format(articlecode=articlecode)
asset_name = f"{slide_code.lower()}_top_product_{slot['rank']:02d}_{sanitize_file_token(articlecode)}"
asset_path = asset_dir / f"{asset_name}.png"
render_product_image(
......@@ -618,12 +2399,13 @@ def build_product_assets(
"asset_name": asset_name,
"asset_path": str(asset_path),
"source_capture_id": f"url-top-products-{slide_code.lower()}",
"source_view": "oms_daily_report_detail",
"source_view": "TopProducts" if source_payload.get("source") == "tableau" else "oms_daily_report_detail",
"note": f"{slide_code} product image for Top {slot['rank']}.",
"top_rank": slot["rank"],
"articlecode": articlecode,
"qty": row["qty"],
"sales": row["sales"],
"source": source_payload.get("source", "sql"),
"image_url": image_url,
"canvas_size": {"width": slot["canvas_width"], "height": slot["canvas_height"]},
}
......@@ -633,8 +2415,10 @@ def build_product_assets(
def main() -> None:
"""Helper."""
script_started_at = time.monotonic()
args = parse_args()
config_path = Path(args.config)
config_path = Path(args.config).resolve()
config = yaml.safe_load(config_path.read_text(encoding="utf-8"))
report_month, report_year, compare_year = resolve_report_period(args, config)
......@@ -644,16 +2428,21 @@ def main() -> None:
raise SystemExit("No supported slides requested. Use S09 and/or S10.")
capture_specs = {slide_code: build_capture_spec(slide_code, report_month, report_year) for slide_code in requested}
log_progress("stage", f"start top-products slides={','.join(requested)}")
vip_workdir = Path(config["paths"]["workdir"]).resolve()
workspace_root = vip_workdir.parents[1]
vip_workdir = Path(args.output_dir).resolve() if args.output_dir else resolve_configured_path(config_path, config["paths"]["workdir"])
workspace_root = vip_workdir if args.output_dir else config_path.parent
asset_dir = vip_workdir / "assets" / "top-products"
data_dir = vip_workdir / "data" / "top-products"
asset_dir.mkdir(parents=True, exist_ok=True)
data_dir.mkdir(parents=True, exist_ok=True)
session = SESSION_NAME
state_path = workspace_root / "output" / "playwright" / session / "state.json"
session = SESSION_NAME if not args.output_dir else f"{SESSION_NAME}-{vip_workdir.name}"
state_path = (
vip_workdir / ".playwright-cli" / f"{session}-state.json"
if args.output_dir
else workspace_root / "output" / "playwright" / session / "state.json"
)
ensure_browser_session(session, cwd=vip_workdir)
load_state_if_present(session, state_path, cwd=vip_workdir)
......@@ -667,12 +2456,21 @@ def main() -> None:
build_login_js(config["tableau"]["username"], config["tableau"]["password"]),
)
try:
run_code(session, login_script, cwd=vip_workdir, timeout=120)
login_output = run_code(session, login_script, cwd=vip_workdir, timeout=120).strip()
if "### Error" in login_output:
raise RuntimeError(f"Failed to run Tableau login script: {login_output}")
finally:
login_script.unlink(missing_ok=True)
raw_screenshots: dict[str, Path] = {}
tableau_top_sources: dict[str, dict[str, Any]] = {}
tableau_capture_errors: dict[str, str] = {}
for slide_code in requested:
slide_spec = SLIDE_SPECS[slide_code]
page_label = f"{slide_code} {slide_spec['capture_id']} ({capture_specs[slide_code]['activate_sheet']})"
page_started_at = time.monotonic()
log_progress("page", f"start {page_label}")
try:
raw_screenshots[slide_code] = capture_tableau_view(
capture_specs[slide_code],
base_url=base_url,
......@@ -680,7 +2478,19 @@ def main() -> None:
workdir=vip_workdir,
workspace_root=workspace_root,
)
tableau_top_sources[slide_code] = extract_top_products_from_current_view(
session,
workdir=vip_workdir,
limit=len(SLIDE_SPECS[slide_code]["product_slots"]),
)
log_timing(f"page {page_label}", page_started_at)
except Exception as exc:
tableau_capture_errors[slide_code] = str(exc)
log_progress("page", f"error {page_label}: {exc}")
try:
save_state(session, state_path, cwd=vip_workdir)
except Exception as exc:
log_progress("state", f"warn save-state failed: {exc}")
operations = {"replace_text": [], "replace_images": []}
manifest_items: list[dict[str, Any]] = []
......@@ -688,20 +2498,28 @@ def main() -> None:
for slide_code in requested:
slide_spec = SLIDE_SPECS[slide_code]
raw_screenshot = raw_screenshots[slide_code]
slide_started_at = time.monotonic()
log_progress("asset", f"start {slide_code} chart/products")
top_source = tableau_top_sources.get(slide_code)
if top_source:
sql_sources[slide_code] = top_source
log_progress("asset", f"use Tableau source for {slide_code} products")
else:
top_source = fetch_top_products(
config,
store_name="ckc-vip",
report_month=report_month,
report_year=report_year,
level1=slide_spec["level1"],
limit=len(slide_spec["product_slots"]),
)
sql_sources[slide_code] = top_source
chart_path = asset_dir / f"{slide_spec['asset_name']}.png"
raw_screenshot = raw_screenshots.get(slide_code)
if raw_screenshot:
crop_image(raw_screenshot, chart_path, slide_spec["crop"], resize_to=slide_spec["resize_to"])
operations["replace_images"].append(
{
"slide": slide_spec["slide"],
"shape_id": slide_spec["shape_id"],
"shape_name": slide_spec["shape_name"],
"image_path": str(chart_path),
}
)
manifest_items.append(
{
chart_manifest_item = {
"slide_code": slide_code,
"slide": slide_spec["slide"],
"shape_id": slide_spec["shape_id"],
......@@ -715,17 +2533,31 @@ def main() -> None:
"crop": slide_spec["crop"],
"resize_to": slide_spec["resize_to"],
}
)
else:
render_sql_top_products_chart(top_source, chart_path)
chart_manifest_item = {
"slide_code": slide_code,
"slide": slide_spec["slide"],
"shape_id": slide_spec["shape_id"],
"shape_name": slide_spec["shape_name"],
"asset_name": slide_spec["asset_name"],
"asset_path": str(chart_path),
"source_capture_id": f"sql-rendered-{slide_code.lower()}",
"source_view": "oms_daily_report_detail",
"note": f"{slide_code} live SQL-rendered top-products chart.",
"tableau_capture_error": tableau_capture_errors.get(slide_code),
"rendered_from_sql": True,
}
top_source = fetch_top_products(
config,
store_name="ckc-vip",
report_month=report_month,
report_year=report_year,
level1=slide_spec["level1"],
limit=len(slide_spec["product_slots"]),
operations["replace_images"].append(
{
"slide": slide_spec["slide"],
"shape_id": slide_spec["shape_id"],
"shape_name": slide_spec["shape_name"],
"image_path": str(chart_path),
}
)
sql_sources[slide_code] = top_source
manifest_items.append(chart_manifest_item)
product_operations, product_manifest_items = build_product_assets(
slide_code,
......@@ -734,6 +2566,7 @@ def main() -> None:
)
operations["replace_images"].extend(product_operations)
manifest_items.extend(product_manifest_items)
log_timing(f"assets {slide_code}", slide_started_at)
operations_path = vip_workdir / "render-ops.top-products.live.json"
operations_path.write_text(json.dumps(operations, ensure_ascii=False, indent=2), encoding="utf-8")
......@@ -747,8 +2580,9 @@ def main() -> None:
"hash_url": capture_specs[slide_code]["hash_url"],
"activate_sheet": capture_specs[slide_code]["activate_sheet"],
"filter_actions": capture_specs[slide_code]["filter_actions"],
"raw_screenshot": str(raw_screenshots[slide_code]),
"raw_screenshot": str(raw_screenshots[slide_code]) if slide_code in raw_screenshots else None,
"note": capture_specs[slide_code]["note"],
"error": tableau_capture_errors.get(slide_code),
}
for slide_code in requested
],
......@@ -765,8 +2599,10 @@ def main() -> None:
}
manifest_path.write_text(json.dumps(manifest_payload, ensure_ascii=False, indent=2), encoding="utf-8")
log_timing(f"stage top-products {','.join(requested)}", script_started_at)
print(json.dumps({"operations_path": str(operations_path), "manifest_path": str(manifest_path)}, ensure_ascii=False))
# 鍏抽敭琛屾敞閲婏細鑴氭湰鐩存帴杩愯鏃讹紝浠庤繖閲岃繘鍏ヤ富娴佺▼銆?
if __name__ == "__main__":
main()
vip-report/scripts/sync_warehouse_100060_assets.py
View file @ 1027cb42
......@@ -12,6 +12,7 @@ from typing import Any
from PIL import Image
import yaml
from tableau_export import build_export_image_js, normalize_image_size
NPX_EXECUTABLE = shutil.which("npx.cmd") or shutil.which("npx") or "npx"
......@@ -19,6 +20,8 @@ PLAYWRIGHT_CMD = [NPX_EXECUTABLE, "--yes", "--package", "@playwright/cli", "play
SESSION_NAME = "vip-report-warehouse-100060"
VIEWPORT = {"width": 1400, "height": 3400}
TEMPLATE_LOCK_DIR = Path(r"C:\workspace\cursor\output\vip-report\assets\template-lock-s04-s08-s13")
S13_HASH_URL = "#/views/WH100060SalesPerformance/60SalesSohbyDiscount?:iid=3"
S13_INNER_FRAME_FRAGMENT = "/views/WH100060SalesPerformance/60SalesSohbyDiscount?"
MONTH_LABELS = {
1: "一月",
......@@ -58,15 +61,12 @@ MONTH_LABEL_TO_NUMBER.update(MOJIBAKE_MONTH_LABELS)
def should_use_template_lock(report_month_label: str, report_year: int, compare_year: int) -> bool:
return (
report_year == 2026
and compare_year == 2025
and month_label_to_number(report_month_label) == 1
and TEMPLATE_LOCK_DIR.exists()
)
"""Helper."""
return False
def normalize_month_label(raw_month: Any) -> str:
"""Helper."""
if raw_month is None:
return MONTH_LABELS[1]
if isinstance(raw_month, int):
......@@ -113,10 +113,12 @@ def normalize_month_label(raw_month: Any) -> str:
def month_label_to_number(label: str) -> int:
"""Helper."""
return MONTH_LABEL_TO_NUMBER.get(label, 1)
def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> tuple[str, int, int, int]:
"""Helper."""
report_cfg = config.get("report", {})
raw_month = args.report_month or report_cfg.get("month_cn") or 1
month_label = normalize_month_label(raw_month)
......@@ -126,11 +128,42 @@ def resolve_report_period(args: argparse.Namespace, config: dict[str, Any]) -> t
return month_label, month_number, report_year, compare_year
def log_progress(scope: str, message: str) -> None:
"""Print a compact progress line."""
print(f"[{scope}] {message}", flush=True)
def log_timing(label: str, started_at: float) -> None:
"""Print a compact timing line."""
elapsed_seconds = time.monotonic() - started_at
print(f"[timing] {label}: {elapsed_seconds:.1f}s", flush=True)
def build_capture_slide_codes(filtered_assets: list[dict[str, Any]]) -> dict[str, list[str]]:
"""Map each capture_id to the slide codes that depend on it."""
capture_slide_codes: dict[str, set[str]] = {}
for asset in filtered_assets:
capture_slide_codes.setdefault(asset["capture_id"], set()).add(asset["slide_code"])
return {capture_id: sorted(slide_codes) for capture_id, slide_codes in capture_slide_codes.items()}
def describe_capture(
capture_id: str,
capture_spec: dict[str, Any],
capture_slide_codes: dict[str, list[str]],
) -> str:
"""Build a readable page label for logging."""
slide_codes = "/".join(capture_slide_codes.get(capture_id, [])) or "-"
return f"{slide_codes} {capture_id} ({capture_spec['activate_sheet']})"
def next_month_label(month_number: int) -> str:
"""Helper."""
return MONTH_LABELS[1 if month_number == 12 else month_number + 1]
def sales_day_labels(report_year: int, report_month_number: int) -> list[str]:
"""Helper."""
last_day = calendar.monthrange(report_year, report_month_number)[1]
start = date(report_year, report_month_number, 1)
return [
......@@ -140,6 +173,7 @@ def sales_day_labels(report_year: int, report_month_number: int) -> list[str]:
def format_cn_date(value: date) -> str:
"""Helper."""
return f"{value.year}年{value.month}月{value.day}日"
......@@ -151,6 +185,7 @@ def build_filter_config(
top_category: str | None = None,
top_store: str | None = None,
) -> list[dict[str, Any]]:
"""Helper."""
filters: list[dict[str, Any]] = [
{"caption": "Year", "action": "replace", "values": [str(report_year)]},
{
......@@ -180,6 +215,7 @@ def build_specs(
report_year: int,
compare_year: int,
) -> dict[str, Any]:
"""Helper."""
top_filters = build_filter_config(report_month_label, report_month_number, report_year)
bags_filters = build_filter_config(
report_month_label,
......@@ -200,8 +236,8 @@ def build_specs(
{
"capture_id": "warehouse_discount_top",
"session": SESSION_NAME,
"hash_url": "#/views/WH100060SalesPerformance/60SalesSohbyDiscount?:iid=1",
"inner_frame_fragment": "/views/WH100060SalesPerformance/60SalesSohbyDiscount?",
"hash_url": S13_HASH_URL,
"inner_frame_fragment": S13_INNER_FRAME_FRAGMENT,
"activate_sheet": "60 Sales& Soh by Discount",
"filter_config": top_filters,
"raw_screenshot_name": "warehouse-100060-top.png",
......@@ -210,8 +246,8 @@ def build_specs(
{
"capture_id": "warehouse_top10_bags",
"session": SESSION_NAME,
"hash_url": "#/views/WH100060SalesPerformance/60SalesSohbyDiscount?:iid=1",
"inner_frame_fragment": "/views/WH100060SalesPerformance/60SalesSohbyDiscount?",
"hash_url": S13_HASH_URL,
"inner_frame_fragment": S13_INNER_FRAME_FRAGMENT,
"activate_sheet": "60 Sales& Soh by Discount",
"filter_config": bags_filters,
"raw_screenshot_name": "warehouse-100060-top10-bags.png",
......@@ -220,8 +256,8 @@ def build_specs(
{
"capture_id": "warehouse_top10_shoes",
"session": SESSION_NAME,
"hash_url": "#/views/WH100060SalesPerformance/60SalesSohbyDiscount?:iid=1",
"inner_frame_fragment": "/views/WH100060SalesPerformance/60SalesSohbyDiscount?",
"hash_url": S13_HASH_URL,
"inner_frame_fragment": S13_INNER_FRAME_FRAGMENT,
"activate_sheet": "60 Sales& Soh by Discount",
"filter_config": shoes_filters,
"raw_screenshot_name": "warehouse-100060-top10-shoes.png",
......@@ -236,7 +272,7 @@ def build_specs(
"shape_id": 5,
"asset_name": "s13_top",
"capture_id": "warehouse_discount_top",
"crop": {"left": 0, "top": 600, "width": 980, "height": 350},
"crop": {"left": 10, "top": 150, "width": 840, "height": 1030},
"source_view": "Performance by Discount",
"note": "S13 第一行 Performance by Discount__年份-全部。",
},
......@@ -247,7 +283,7 @@ def build_specs(
"shape_id": 6,
"asset_name": "s13_left",
"capture_id": "warehouse_top10_bags",
"crop": {"left": 960, "top": 840, "width": 440, "height": 970},
"crop": {"left": 860, "top": 1640, "width": 530, "height": 1670},
"source_view": "TOP10 by Category",
"note": "S13 第二行左侧，TOP10 by Category___年份-全部（BAGS）。",
},
......@@ -258,7 +294,7 @@ def build_specs(
"shape_id": 10,
"asset_name": "s13_mid",
"capture_id": "warehouse_top10_shoes",
"crop": {"left": 960, "top": 840, "width": 440, "height": 970},
"crop": {"left": 860, "top": 1640, "width": 530, "height": 1670},
"source_view": "TOP10 by Category",
"note": "S13 第二行右侧，TOP10 by Category___年份-全部（SHOES）。",
},
......@@ -279,6 +315,7 @@ def run_cmd(
timeout: int = 120,
check: bool = True,
) -> subprocess.CompletedProcess[str]:
"""Helper."""
return subprocess.run(
args,
cwd=str(cwd),
......@@ -292,17 +329,20 @@ def run_cmd(
def run_playwright(args: list[str], *, cwd: Path, timeout: int = 120) -> str:
"""Helper."""
result = run_cmd(PLAYWRIGHT_CMD + args, cwd=cwd, timeout=timeout)
return result.stdout
def write_js(workdir: Path, name: str, content: str) -> Path:
"""Helper."""
path = workdir / name
path.write_text(content, encoding="utf-8")
return path
def build_login_js(username: str, password: str) -> str:
"""Helper."""
payload = {"username": username, "password": password}
spec = json.dumps(payload, ensure_ascii=False)
return f"""async function(page) {{
......@@ -329,6 +369,7 @@ def build_login_js(username: str, password: str) -> str:
def build_configure_view_js(spec: dict[str, Any]) -> str:
"""Helper."""
payload = json.dumps(spec, ensure_ascii=False)
return f"""async function(page) {{
const spec = {payload};
......@@ -405,11 +446,14 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
const applyByLabels = (widget, values) => {{
const tuples = widget.get_oFilter().table?.tuples || [];
const desired = new Set(values);
const desired = new Set(
(values || []).map((value) => String(value ?? '').trim().toLowerCase())
);
const indices = tuples
.map((tuple, index) => {{
const label = tuple.d ?? tuple.t?.map((item) => item.v).join('|') ?? '';
return desired.has(label) ? index : -1;
const normalizedLabel = String(label ?? '').trim().toLowerCase();
return desired.has(normalizedLabel) ? index : -1;
}})
.filter((index) => index >= 0);
if (!indices.length) {{
......@@ -473,6 +517,7 @@ def build_configure_view_js(spec: dict[str, Any]) -> str:
def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
"""Helper."""
payload = json.dumps(
{
"inner_frame_fragment": inner_frame_fragment,
......@@ -508,6 +553,7 @@ def build_capture_js(inner_frame_fragment: str, screenshot_name: str) -> str:
def run_code(session: str, script_path: Path, *, cwd: Path, timeout: int = 120) -> str:
"""Helper."""
return run_playwright(
["--session", session, "run-code", "--filename", str(script_path)],
cwd=cwd,
......@@ -516,10 +562,12 @@ def run_code(session: str, script_path: Path, *, cwd: Path, timeout: int = 120)
def ensure_browser_session(session: str, *, cwd: Path) -> None:
"""Helper."""
run_playwright(["--session", session, "open", "about:blank"], cwd=cwd, timeout=60)
def save_state(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
state_path.parent.mkdir(parents=True, exist_ok=True)
run_playwright(
["--session", session, "state-save", str(state_path)],
......@@ -529,6 +577,7 @@ def save_state(session: str, state_path: Path, *, cwd: Path) -> None:
def load_state_if_present(session: str, state_path: Path, *, cwd: Path) -> None:
"""Helper."""
if not state_path.exists():
return
try:
......@@ -543,6 +592,7 @@ def load_state_if_present(session: str, state_path: Path, *, cwd: Path) -> None:
def locate_session_file(root: Path, session: str, filename: str) -> Path:
"""Helper."""
search_roots = [
root / "output" / "playwright",
root / "output" / "vip-report",
......@@ -571,6 +621,7 @@ def crop_image(
*,
resize_to: dict[str, int] | None = None,
) -> None:
"""Helper."""
image = Image.open(source)
box = (
crop["left"],
......@@ -585,7 +636,26 @@ def crop_image(
result.save(target)
def wait_for_image_ready(path: Path, *, timeout_seconds: int = 20) -> Path:
"""Helper."""
deadline = time.monotonic() + timeout_seconds
last_error: Exception | None = None
while time.monotonic() < deadline:
if path.exists() and path.stat().st_size > 0:
try:
with Image.open(path) as image:
image.load()
return path
except OSError as exc:
last_error = exc
time.sleep(1)
if last_error is not None:
raise last_error
raise FileNotFoundError(f"Image file was not ready in time: {path}")
def parse_args() -> argparse.Namespace:
"""Helper."""
parser = argparse.ArgumentParser(description="Capture S13 Tableau assets for VIP report.")
parser.add_argument(
"--config",
......@@ -614,10 +684,16 @@ def parse_args() -> argparse.Namespace:
default=0,
help="Comparison year placeholder, default report year - 1",
)
parser.add_argument(
"--output-dir",
default="",
help="Override the output/work directory used for generated assets and ops.",
)
return parser.parse_args()
def collect_required_capture_ids(filtered_assets: list[dict[str, Any]]) -> set[str]:
"""Helper."""
return {asset["capture_id"] for asset in filtered_assets}
......@@ -629,7 +705,10 @@ def capture_tableau_view(
workdir: Path,
workspace_root: Path,
) -> Path:
"""Helper."""
target_url = f"{base_url}{capture_spec['hash_url']}"
local_output = workdir / capture_spec["raw_screenshot_name"]
local_output.unlink(missing_ok=True)
run_playwright(["--session", session, "goto", target_url], cwd=workdir, timeout=120)
run_playwright(
["--session", session, "resize", str(VIEWPORT["width"]), str(VIEWPORT["height"])],
......@@ -645,9 +724,10 @@ def capture_tableau_view(
capture_script = write_js(
workdir,
f"tmp-{capture_spec['capture_id']}-{time.time_ns()}-capture.js",
build_capture_js(
build_export_image_js(
capture_spec["inner_frame_fragment"],
capture_spec["raw_screenshot_name"],
str((workdir / capture_spec["raw_screenshot_name"]).resolve()),
viewport=VIEWPORT,
),
)
......@@ -690,13 +770,18 @@ def capture_tableau_view(
configure_script.unlink(missing_ok=True)
capture_script.unlink(missing_ok=True)
local_output = workdir / capture_spec["raw_screenshot_name"]
if local_output.exists():
wait_for_image_ready(local_output)
normalize_image_size(local_output, {"width": 1400, "height": 3360})
return local_output
return locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
captured_path = locate_session_file(workspace_root, session, capture_spec["raw_screenshot_name"])
wait_for_image_ready(captured_path)
normalize_image_size(captured_path, {"width": 1400, "height": 3360})
return captured_path
def resolve_config_path(requested_path: str) -> Path:
"""Helper."""
path = Path(requested_path)
if path.exists():
return path
......@@ -707,6 +792,8 @@ def resolve_config_path(requested_path: str) -> Path:
def main() -> None:
"""Helper."""
script_started_at = time.monotonic()
args = parse_args()
config_path = resolve_config_path(args.config)
config = yaml.safe_load(config_path.read_text(encoding="utf-8"))
......@@ -718,8 +805,8 @@ def main() -> None:
if not filtered_assets:
raise SystemExit("No matching slides requested.")
vip_workdir = Path(config["paths"]["workdir"]).resolve()
workspace_root = vip_workdir.parents[1]
vip_workdir = Path(args.output_dir).resolve() if args.output_dir else Path(config["paths"]["workdir"]).resolve()
workspace_root = vip_workdir if args.output_dir else vip_workdir.parents[1]
asset_dir = vip_workdir / "assets" / "warehouse-100060"
data_dir = vip_workdir / "data" / "warehouse-100060"
asset_dir.mkdir(parents=True, exist_ok=True)
......@@ -727,12 +814,18 @@ def main() -> None:
captures_by_id = {item["capture_id"]: item for item in specs["captures"]}
required_capture_ids = collect_required_capture_ids(filtered_assets)
capture_slide_codes = build_capture_slide_codes(filtered_assets)
use_template_lock = should_use_template_lock(report_month_label, report_year, compare_year)
log_progress("stage", f"start warehouse-100060 slides={','.join(sorted(requested))}")
raw_screenshots: dict[str, Path] = {}
if not use_template_lock:
session = SESSION_NAME
state_path = vip_workdir / ".playwright-cli" / f"{session}-state.json"
session = SESSION_NAME if not args.output_dir else f"{SESSION_NAME}-{vip_workdir.name}"
state_path = (
vip_workdir / ".playwright-cli" / f"{session}-state.json"
if args.output_dir
else vip_workdir / ".playwright-cli" / f"{session}-state.json"
)
ensure_browser_session(session, cwd=vip_workdir)
load_state_if_present(session, state_path, cwd=vip_workdir)
......@@ -753,19 +846,28 @@ def main() -> None:
login_script.unlink(missing_ok=True)
for capture_id in sorted(required_capture_ids):
capture_spec = captures_by_id[capture_id]
page_label = describe_capture(capture_id, capture_spec, capture_slide_codes)
page_started_at = time.monotonic()
log_progress("page", f"start {page_label}")
raw_screenshots[capture_id] = capture_tableau_view(
captures_by_id[capture_id],
capture_spec,
base_url=base_url,
session=session,
workdir=vip_workdir,
workspace_root=workspace_root,
)
log_timing(f"page {page_label}", page_started_at)
save_state(session, state_path, cwd=vip_workdir)
operations = {"replace_text": [], "replace_images": []}
manifest_items: list[dict[str, Any]] = []
assets_started_at = time.monotonic()
for asset in filtered_assets:
asset_label = f"{asset['slide_code']} {asset['asset_name']}"
asset_started_at = time.monotonic()
log_progress("asset", f"start {asset_label}")
target = asset_dir / f"{asset['asset_name']}.png"
if use_template_lock:
source_path = TEMPLATE_LOCK_DIR / f"s{asset['slide']}_shape{asset['shape_id']}.png"
......@@ -805,6 +907,8 @@ def main() -> None:
if asset.get("resize_to"):
manifest_item["resize_to"] = asset["resize_to"]
manifest_items.append(manifest_item)
log_timing(f"asset {asset_label}", asset_started_at)
log_timing(f"assets warehouse-100060 {','.join(sorted(requested))}", assets_started_at)
operations_path = vip_workdir / "render-ops.warehouse-100060.live.json"
operations_path.write_text(json.dumps(operations, ensure_ascii=False, indent=2), encoding="utf-8")
......@@ -837,6 +941,7 @@ def main() -> None:
encoding="utf-8",
)
log_timing(f"stage warehouse-100060 {','.join(sorted(requested))}", script_started_at)
print(
json.dumps(
{
......@@ -848,5 +953,6 @@ def main() -> None:
)
# 关键行注释：脚本直接运行时，从这里进入主流程。
if __name__ == "__main__":
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment