feat: P7 功能增强 — 通知器扩展(Slack/Discord/Telegram)、HTML 报告导出(to_html)、YAML 增强(matrix include-exclude/条件依赖/outputs)、缓存增强(invalidate 单键失效/MemoryBackend LRU 驱逐)
CI / Lint, Typecheck & Test (push) Has been cancelled
CI / Lint, Typecheck & Test (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# 迭代 19 — P7 功能增强
|
||||
|
||||
## 本轮目标
|
||||
|
||||
P7 阶段聚焦功能增强,涵盖四个子任务:
|
||||
- P7.1 通知器扩展(Slack/Discord/Telegram)
|
||||
- P7.2 HTML 报告导出(RunReport.to_html)
|
||||
- P7.3 YAML 增强特性(matrix include-exclude / 条件依赖 / outputs)
|
||||
- P7.4 任务缓存增强(invalidate 单键失效 / MemoryBackend LRU 驱逐)
|
||||
|
||||
## 改动文件清单
|
||||
|
||||
### P7.1 通知器扩展
|
||||
- `src/pyflowx/notification.py` — 新增 SlackNotifier/DiscordNotifier/TelegramNotifier
|
||||
- `src/pyflowx/__init__.py` — 导出新通知器类
|
||||
- `tests/test_notification.py` — 新增 14 个测试
|
||||
- `README.md` — 通知器章节扩展
|
||||
|
||||
### P7.2 HTML 报告导出
|
||||
- `src/pyflowx/report.py` — 新增 to_html 方法 + _render_summary_card 辅助
|
||||
- `tests/test_report.py` — 新增 16 个测试
|
||||
- `README.md` — 序列化章节扩展
|
||||
|
||||
### P7.3 YAML 增强特性
|
||||
- `src/pyflowx/yaml_loader.py` — _cartesian_product 支持 include/exclude;_expand_needs 支持条件依赖;_parse_optional_fields 解析 outputs
|
||||
- `src/pyflowx/task.py` — TaskSpec 新增 outputs 字段;task() 工厂支持 outputs 参数
|
||||
- `src/pyflowx/report.py` — to_dict/from_json 包含 outputs
|
||||
- `tests/test_yaml_loader.py` — 新增 24 个测试
|
||||
- `README.md` — YAML 章节扩展
|
||||
|
||||
### P7.4 任务缓存增强
|
||||
- `src/pyflowx/storage.py` — StateBackend 新增 invalidate 抽象方法;_TTLStateBackendMixin 新增 _delete_raw 原语与 invalidate 实现;MemoryBackend 新增 maxsize LRU 驱逐;JSONBackend/SQLiteBackend 实现 _delete_raw
|
||||
- `tests/test_storage.py` — 新增 17 个测试
|
||||
- `README.md` — 缓存章节扩展
|
||||
|
||||
## 关键决策与依据
|
||||
|
||||
### 通知器架构(P7.1)
|
||||
- 复用 WebhookNotifier 基类,仅覆盖 _build_payload 适配平台格式
|
||||
- TelegramNotifier 额外需要 chat_id 参数,故覆盖 __init__
|
||||
- 零新依赖(stdlib urllib.request)
|
||||
|
||||
### HTML 报告设计(P7.2)
|
||||
- 自包含 HTML5 文档,内联 CSS,无外部依赖
|
||||
- 使用 html.escape 转义所有用户内容(任务名/错误/原因/值)防 XSS
|
||||
- 状态徽章着色:success 绿 / failed 红 / skipped 黄 / running 蓝
|
||||
- _render_summary_card 静态方法避免重复代码
|
||||
|
||||
### YAML 矩阵 include/exclude 语义(P7.3)
|
||||
- exclude: 组合的所有 key-value 对都匹配则剔除(c.items() >= ex.items())
|
||||
- include: 追加额外组合,可引入基础矩阵之外的键
|
||||
- 无基础矩阵键时返回空列表而非 [{}],避免产生空名任务
|
||||
|
||||
### 条件依赖设计(P7.3)
|
||||
- needs 项支持 {job: name, if: expr} 格式
|
||||
- if 条件不满足时跳过该依赖(不加入 depends_on)
|
||||
- 复用 _parse_condition 解析 if 表达式
|
||||
- 条件求值时传空上下文 {}(env 条件直接读 os.environ)
|
||||
|
||||
### outputs 字段(P7.3)
|
||||
- TaskSpec 新增 outputs: Mapping[str, str] | None 作为元数据
|
||||
- 仅携带不参与执行,可通过 spec.outputs 查询
|
||||
- 支持 ${{ matrix.* }} 占位符替换
|
||||
- to_dict/from_json 往返保留
|
||||
|
||||
### 缓存 invalidate 设计(P7.4)
|
||||
- StateBackend 新增 invalidate 抽象方法(返回 bool 表示是否删除)
|
||||
- _TTLStateBackendMixin 通过 _delete_raw 原语统一委托
|
||||
- JSONBackend.invalidate 覆盖以在非 batch 模式下 flush 落盘
|
||||
- SQLiteBackend._delete_raw 用 cursor.rowcount > 0 判断是否存在
|
||||
|
||||
### MemoryBackend LRU 设计(P7.4)
|
||||
- 使用 OrderedDict 替代普通 dict
|
||||
- _get_raw 命中时 move_to_end(key) 更新访问顺序
|
||||
- _put_raw 超限时 popitem(last=False) 驱逐头部(最旧)
|
||||
- maxsize=None 时不驱逐(默认行为,向后兼容)
|
||||
|
||||
## 验证结果
|
||||
|
||||
- ruff check: All checks passed
|
||||
- ruff format: All files formatted
|
||||
- pyrefly: 0 errors (67 suppressed)
|
||||
- pytest: 1425 passed
|
||||
- coverage: 97.56%(分支覆盖率 ≥ 95%)
|
||||
|
||||
## 遗留事项
|
||||
|
||||
- P8(性能优化)尚未开始:自实现拓扑排序 / orjson 序列化 / 内存优化 / 执行器并发优化
|
||||
- outputs 字段当前仅作为元数据,未来可扩展为跨任务引用(${{ needs.job.outputs.name }})
|
||||
- TelegramNotifier 实际使用时需拼接 bot token URL,文档已说明
|
||||
@@ -23,8 +23,9 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
|
||||
- **软依赖** —— `soft_depends_on` 仅用于上下文注入,不参与拓扑分层
|
||||
- **并发限制** —— `concurrency_key` + `concurrency_limits` 按组限流
|
||||
- **任务钩子** —— `TaskHooks`(pre_run/post_run/on_failure)生命周期回调
|
||||
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘
|
||||
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘;`invalidate(key)` 单键失效
|
||||
- **缓存键** —— `cache_key` 函数基于输入计算稳定键,使不同输入产生独立缓存
|
||||
- **缓存驱逐** —— `MemoryBackend(maxsize=N)` LRU 驱逐最旧条目;TTL 过期自动忽略
|
||||
- **命令任务** —— `cmd` 参数直接执行外部命令,支持列表/shell/可调用对象
|
||||
- **条件执行** —— `conditions` 参数按平台、环境变量、应用安装等条件跳过任务
|
||||
- **图组合** —— `compose` / `GraphComposer` 编程式展开多图字符串引用
|
||||
@@ -33,11 +34,11 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
|
||||
- **CLI 运行器** —— `CliRunner` 把多个图映射为命令行子命令,替代 Makefile
|
||||
- **可观测** —— `on_event` 回调(RUNNING/SUCCESS/FAILED/SKIPPED)、`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
|
||||
- **进度监控** —— `run(progress=True)` 开箱即用的 rich 进度条;`ProgressCallback` 协议供程序化集成
|
||||
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信 webhook 与 Python 回调,全生命周期事件可配置
|
||||
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信/Slack/Discord/Telegram webhook 与 Python 回调,全生命周期事件可配置
|
||||
- **流式获取** —— `run_iter()` 生成器逐个 yield 任务结果,适用于大量任务逐个处理
|
||||
- **任务取消** —— `CancelToken` 线程安全取消令牌,支持外部取消与 `KeyboardInterrupt` 优雅停止
|
||||
- **子图执行** —— `run(only=[...], tags=[...])` 按名称或标签运行图的子集,自动包含传递依赖
|
||||
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` 多格式导出运行结果,`from_json()` 支持反序列化重建
|
||||
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` / `to_html()` 多格式导出运行结果,`from_json()` 支持反序列化重建;HTML 报告自包含内联 CSS,适合浏览器直接打开或邮件分享
|
||||
- **CLI 可视化** —— `pf graph <yaml>` 终端 DAG 渲染(ASCII 分层树按标签着色 / Mermaid / 依赖表),`pf yamlrun --progress` rich 进度条,`pf info [tool]` 工具详情,`pf completion bash|zsh|fish` shell 补全
|
||||
- **失败诊断** —— `RunReport.diagnose()` 自动分析根因、依赖链回溯、相似失败聚类与可操作建议;CLI 失败时自动打印诊断摘要
|
||||
- **可观测性** —— `run_id` 贯穿日志/序列化/诊断;结构化日志 `extra` 含 `task_name`/`status`/`attempts`/`error_type`;`profile(graph)` 离线性能分析(关键路径/并行度/瓶颈)
|
||||
@@ -159,7 +160,7 @@ resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
|
||||
|
||||
### YAML 任务编排
|
||||
|
||||
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`:
|
||||
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`,以及 `matrix.include`/`exclude` 过滤、条件依赖(`needs: [{job: ..., if: ...}]`)、`outputs` 声明:
|
||||
|
||||
```yaml
|
||||
# pipeline.yaml
|
||||
@@ -179,7 +180,22 @@ jobs:
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11", "3.12"]
|
||||
os: ["linux", "macos"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
os: "macos"
|
||||
include:
|
||||
- version: "3.13"
|
||||
os: "linux"
|
||||
if: "env.CI"
|
||||
outputs:
|
||||
version: "${{ matrix.version }}"
|
||||
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.BRANCH == 'main'"
|
||||
cmd: ["echo", "deploy"]
|
||||
|
||||
test:
|
||||
needs: [build]
|
||||
@@ -213,7 +229,11 @@ pf yamlrun pipeline.yaml --strategy sequential # 覆盖策略
|
||||
| `needs` | `depends_on` | 矩阵 job 依赖自动展开为所有变体 |
|
||||
| `if` | `conditions` | `success()`/`always()`/`env.VAR`/`env.VAR == 'x'`/`env.VAR != 'x'` |
|
||||
| `strategy.matrix` | 矩阵扇出 | 笛卡尔积,命名 `{job}_{key}-{value}` |
|
||||
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env` 中替换 |
|
||||
| `strategy.matrix.include` | 矩阵追加 | 额外组合(可引入新键) |
|
||||
| `strategy.matrix.exclude` | 矩阵剔除 | 匹配所有 key-value 的组合被移除 |
|
||||
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env`/`outputs` 中替换 |
|
||||
| `outputs` | `outputs` | 任务声明的输出键值映射(元数据,不参与执行) |
|
||||
| `needs: [{job: ..., if: ...}]` | 条件依赖 | `if` 不满足时跳过该依赖 |
|
||||
| `continue-on-error` | `continue_on_error` | 字段名支持 hyphen/underscore |
|
||||
| `runs-on` | `tags` | 追加到 tags 元组 |
|
||||
| `defaults` | `GraphDefaults` | 图级默认值(retry/timeout/env/cwd 等) |
|
||||
@@ -409,6 +429,22 @@ px.TaskSpec(
|
||||
)
|
||||
```
|
||||
|
||||
**LRU 驱逐**:`MemoryBackend(maxsize=N)` 限制最大条目数,超限时按 LRU(最近最少使用)
|
||||
策略驱逐最旧条目。`get`/`has` 命中会将条目移到最近使用位置:
|
||||
|
||||
```python
|
||||
from pyflowx import MemoryBackend
|
||||
|
||||
backend = MemoryBackend(maxsize=100) # 最多 100 条,超出驱逐最旧
|
||||
```
|
||||
|
||||
**手动失效**:`invalidate(key)` 删除单个键的缓存,`clear()` 清除全部:
|
||||
|
||||
```python
|
||||
backend.invalidate("fetch_user") # 删除单个键,返回是否已删除
|
||||
backend.clear() # 清除全部
|
||||
```
|
||||
|
||||
## 错误处理
|
||||
|
||||
所有错误都是 `PyFlowXError` 的子类:
|
||||
|
||||
@@ -80,9 +80,12 @@ from .notification import (
|
||||
ALL_LEVELS,
|
||||
CallbackNotifier,
|
||||
DingTalkNotifier,
|
||||
DiscordNotifier,
|
||||
FeishuNotifier,
|
||||
NotificationLevel,
|
||||
Notifier,
|
||||
SlackNotifier,
|
||||
TelegramNotifier,
|
||||
WebhookNotifier,
|
||||
WeChatNotifier,
|
||||
)
|
||||
@@ -128,6 +131,7 @@ __all__ = [
|
||||
"DependencyChain",
|
||||
"DiagnosticReport",
|
||||
"DingTalkNotifier",
|
||||
"DiscordNotifier",
|
||||
"DuplicateTaskError",
|
||||
"FailureCluster",
|
||||
"FeishuNotifier",
|
||||
@@ -147,6 +151,7 @@ __all__ = [
|
||||
"RichProgressMonitor",
|
||||
"RunReport",
|
||||
"SQLiteBackend",
|
||||
"SlackNotifier",
|
||||
"StateBackend",
|
||||
"StorageError",
|
||||
"Strategy",
|
||||
@@ -159,6 +164,7 @@ __all__ = [
|
||||
"TaskSpec",
|
||||
"TaskStatus",
|
||||
"TaskTimeoutError",
|
||||
"TelegramNotifier",
|
||||
"ToolSpec",
|
||||
"WeChatNotifier",
|
||||
"WebhookNotifier",
|
||||
|
||||
@@ -40,9 +40,12 @@ __all__ = [
|
||||
"ALL_LEVELS",
|
||||
"CallbackNotifier",
|
||||
"DingTalkNotifier",
|
||||
"DiscordNotifier",
|
||||
"FeishuNotifier",
|
||||
"NotificationLevel",
|
||||
"Notifier",
|
||||
"SlackNotifier",
|
||||
"TelegramNotifier",
|
||||
"WeChatNotifier",
|
||||
"WebhookNotifier",
|
||||
]
|
||||
@@ -258,3 +261,60 @@ class WeChatNotifier(WebhookNotifier):
|
||||
"msgtype": "text",
|
||||
"text": {"content": _format_event_message(event)},
|
||||
}
|
||||
|
||||
|
||||
class SlackNotifier(WebhookNotifier):
|
||||
"""Slack Incoming Webhook 通知器。
|
||||
|
||||
消息格式:``{"text": ...}``。Slack webhook 仅需 ``text`` 字段,
|
||||
与基类格式一致,独立成类便于类型区分与后续扩展(如 channel/username)。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {"text": _format_event_message(event)}
|
||||
|
||||
|
||||
class DiscordNotifier(WebhookNotifier):
|
||||
"""Discord Webhook 通知器。
|
||||
|
||||
消息格式:``{"content": ...}``。Discord webhook 使用 ``content`` 字段
|
||||
承载消息文本。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {"content": _format_event_message(event)}
|
||||
|
||||
|
||||
class TelegramNotifier(WebhookNotifier):
|
||||
"""Telegram Bot 通知器。
|
||||
|
||||
通过 ``sendMessage`` API 发送消息。需要在 ``__init__`` 中传入
|
||||
``chat_id``(聊天/群组 ID),并使用 bot token 拼接 webhook URL:
|
||||
``https://api.telegram.org/bot<TOKEN>/sendMessage``。
|
||||
|
||||
消息格式:``{"chat_id": ..., "text": ...}``。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
chat_id:
|
||||
接收消息的聊天或群组 ID(数字字符串或 @channelusername)。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
chat_id: str,
|
||||
levels: Iterable[NotificationLevel] = ALL_LEVELS,
|
||||
timeout: float = 10.0,
|
||||
) -> None:
|
||||
super().__init__(url=url, levels=levels, timeout=timeout)
|
||||
self._chat_id = chat_id
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"chat_id": self._chat_id,
|
||||
"text": _format_event_message(event),
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import html
|
||||
import io
|
||||
import json
|
||||
from collections.abc import Callable, Iterator
|
||||
@@ -215,6 +216,7 @@ class RunReport:
|
||||
"reason": r.reason,
|
||||
"tags": list(r.spec.tags),
|
||||
"depends_on": list(r.spec.depends_on),
|
||||
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
|
||||
}
|
||||
)
|
||||
return {
|
||||
@@ -287,6 +289,132 @@ class RunReport:
|
||||
writer.writerow(row)
|
||||
return buf.getvalue()
|
||||
|
||||
def to_html(
|
||||
self,
|
||||
include_value: bool = True,
|
||||
value_serializer: Callable[[Any], Any] | None = None,
|
||||
) -> str:
|
||||
"""导出为自包含的 HTML 报告字符串。
|
||||
|
||||
生成完整的 HTML5 文档,内联 CSS(零外部依赖),适合直接在浏览器
|
||||
打开或邮件附件分享。所有用户内容(任务名、错误、原因、值)均经
|
||||
:func:`html.escape` 转义,避免 XSS。
|
||||
|
||||
参数
|
||||
----
|
||||
include_value:
|
||||
是否在表格中展示任务返回值。值可能较大或含特殊字符,
|
||||
不需要时设为 ``False`` 得到更紧凑报告。
|
||||
value_serializer:
|
||||
自定义任务值序列化函数,透传给 :func:`_serialize_value`。
|
||||
``None`` 时尝试 JSON 兼容转换,失败回退到 ``repr()``。
|
||||
"""
|
||||
summary = self.summary()
|
||||
status_badge_class = {
|
||||
TaskStatus.SUCCESS: "status-success",
|
||||
TaskStatus.FAILED: "status-failed",
|
||||
TaskStatus.SKIPPED: "status-skipped",
|
||||
TaskStatus.RUNNING: "status-running",
|
||||
TaskStatus.PENDING: "status-pending",
|
||||
}
|
||||
parts: list[str] = [
|
||||
"<!DOCTYPE html>",
|
||||
'<html lang="zh-CN">',
|
||||
"<head>",
|
||||
'<meta charset="utf-8">',
|
||||
'<meta name="viewport" content="width=device-width, initial-scale=1">',
|
||||
f"<title>PyFlowX 运行报告 {html.escape(self.run_id)}</title>",
|
||||
"<style>",
|
||||
"body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;"
|
||||
"margin:0;padding:20px;background:#f5f5f5;color:#222;}",
|
||||
".container{max-width:1100px;margin:0 auto;}",
|
||||
".summary{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));"
|
||||
"gap:12px;margin-bottom:24px;}",
|
||||
".card{background:#fff;border-radius:8px;padding:16px;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
|
||||
".card h3{margin:0 0 8px;font-size:13px;color:#666;font-weight:600;text-transform:uppercase;}",
|
||||
".card .value{font-size:22px;font-weight:700;color:#222;}",
|
||||
".card.success .value{color:#16a34a;}",
|
||||
".card.failure .value{color:#dc2626;}",
|
||||
"table{width:100%;border-collapse:collapse;background:#fff;border-radius:8px;"
|
||||
"overflow:hidden;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
|
||||
"th,td{padding:10px 12px;text-align:left;border-bottom:1px solid #eee;font-size:13px;}",
|
||||
"th{background:#fafafa;font-weight:600;color:#555;}",
|
||||
"tr:last-child td{border-bottom:none;}",
|
||||
"td.name{font-family:monospace;font-weight:600;}",
|
||||
".badge{display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;}",
|
||||
".status-success{background:#dcfce7;color:#16a34a;}",
|
||||
".status-failed{background:#fee2e2;color:#dc2626;}",
|
||||
".status-skipped{background:#fef9c3;color:#ca8a04;}",
|
||||
".status-running{background:#dbeafe;color:#2563eb;}",
|
||||
".status-pending{background:#f3f4f6;color:#6b7280;}",
|
||||
".error{color:#dc2626;font-family:monospace;font-size:12px;}",
|
||||
".reason{color:#ca8a04;font-size:12px;}",
|
||||
".value-cell{font-family:monospace;font-size:12px;max-width:300px;"
|
||||
"overflow:hidden;text-overflow:ellipsis;white-space:nowrap;}",
|
||||
".empty{padding:32px;text-align:center;color:#999;}",
|
||||
"</style>",
|
||||
"</head>",
|
||||
"<body>",
|
||||
'<div class="container">',
|
||||
f"<h1>PyFlowX 运行报告 <code>{html.escape(self.run_id)}</code></h1>",
|
||||
'<div class="summary">',
|
||||
self._render_summary_card(
|
||||
"运行状态", "成功" if self.success else "失败", "success" if self.success else "failure"
|
||||
),
|
||||
self._render_summary_card("任务总数", str(summary["total_tasks"])),
|
||||
self._render_summary_card("总耗时", f"{summary['total_duration_seconds']:.3f}s"),
|
||||
self._render_summary_card("成功", str(summary["by_status"].get("success", 0))),
|
||||
self._render_summary_card("失败", str(summary["by_status"].get("failed", 0))),
|
||||
self._render_summary_card("跳过", str(summary["by_status"].get("skipped", 0))),
|
||||
"</div>", # .summary
|
||||
]
|
||||
|
||||
if not self.results:
|
||||
parts.append('<div class="empty">本次运行无任务结果。</div>')
|
||||
else:
|
||||
header_cells = [
|
||||
"<th>任务名</th>",
|
||||
"<th>状态</th>",
|
||||
"<th>尝试次数</th>",
|
||||
"<th>耗时(s)</th>",
|
||||
"<th>开始时间</th>",
|
||||
"<th>结束时间</th>",
|
||||
"<th>错误</th>",
|
||||
"<th>原因</th>",
|
||||
]
|
||||
if include_value:
|
||||
header_cells.append("<th>返回值</th>")
|
||||
parts.append("<table>")
|
||||
parts.append("<thead><tr>" + "".join(header_cells) + "</tr></thead>")
|
||||
parts.append("<tbody>")
|
||||
for name, r in self.results.items():
|
||||
badge_cls = status_badge_class.get(r.status, "status-pending")
|
||||
cells = [
|
||||
f'<td class="name">{html.escape(name)}</td>',
|
||||
f'<td><span class="badge {badge_cls}">{html.escape(r.status.value)}</span></td>',
|
||||
f"<td>{r.attempts}</td>",
|
||||
f"<td>{r.duration:.6f}</td>" if r.duration is not None else "<td>-</td>",
|
||||
f"<td>{html.escape(r.started_at.isoformat())}</td>" if r.started_at else "<td>-</td>",
|
||||
f"<td>{html.escape(r.finished_at.isoformat())}</td>" if r.finished_at else "<td>-</td>",
|
||||
f'<td class="error">{html.escape(repr(r.error))}</td>' if r.error else "<td>-</td>",
|
||||
f'<td class="reason">{html.escape(r.reason)}</td>' if r.reason else "<td>-</td>",
|
||||
]
|
||||
if include_value:
|
||||
val = _serialize_value(r.value, value_serializer)
|
||||
val_str = json.dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
|
||||
cells.append(f'<td class="value-cell">{html.escape(val_str)}</td>')
|
||||
parts.append("<tr>" + "".join(cells) + "</tr>")
|
||||
parts.append("</tbody></table>")
|
||||
|
||||
parts.extend(["</div>", "</body>", "</html>"])
|
||||
return "\n".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _render_summary_card(label: str, value: str, modifier: str = "") -> str:
|
||||
"""渲染单个汇总卡片 HTML 片段。"""
|
||||
cls = f"card {modifier}" if modifier else "card"
|
||||
return f'<div class="{cls}"><h3>{html.escape(label)}</h3><div class="value">{html.escape(value)}</div></div>'
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, text: str) -> RunReport:
|
||||
"""从 JSON 字符串重建 :class:`RunReport`。
|
||||
@@ -308,6 +436,7 @@ class RunReport:
|
||||
fn=_noop_fn,
|
||||
tags=tuple(entry.get("tags", ())),
|
||||
depends_on=tuple(entry.get("depends_on", ())),
|
||||
outputs=dict(entry["outputs"]) if entry.get("outputs") else None,
|
||||
)
|
||||
started = entry.get("started_at")
|
||||
finished = entry.get("finished_at")
|
||||
|
||||
+73
-3
@@ -19,6 +19,7 @@ import sys
|
||||
import threading
|
||||
import time
|
||||
from abc import ABC, abstractmethod
|
||||
from collections import OrderedDict
|
||||
from collections.abc import Iterator, Mapping
|
||||
from contextlib import AbstractContextManager, contextmanager, nullcontext
|
||||
from pathlib import Path
|
||||
@@ -58,6 +59,14 @@ class StateBackend(ABC):
|
||||
def clear(self) -> None:
|
||||
"""清除所有存储状态。"""
|
||||
|
||||
@abstractmethod
|
||||
def invalidate(self, key: str) -> bool:
|
||||
"""删除单个键的存储结果。
|
||||
|
||||
返回 ``True`` 表示键存在且已删除,``False`` 表示键不存在。
|
||||
与 :meth:`clear` 不同,仅删除指定键而非全部。
|
||||
"""
|
||||
|
||||
def flush(self) -> None: # noqa: B027
|
||||
"""将内存中暂存的状态持久化到外部介质。
|
||||
|
||||
@@ -106,6 +115,10 @@ class _TTLStateBackendMixin(StateBackend):
|
||||
def _clear_raw(self) -> None:
|
||||
"""清空所有记录。"""
|
||||
|
||||
@abstractmethod
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
"""删除单条记录。返回 ``True`` 表示存在且已删除,``False`` 表示不存在。"""
|
||||
|
||||
# ---- 共享实现 ---- #
|
||||
def _now(self) -> float:
|
||||
"""当前时间戳,默认为 wall-clock 秒。"""
|
||||
@@ -141,6 +154,10 @@ class _TTLStateBackendMixin(StateBackend):
|
||||
def clear(self) -> None:
|
||||
self._clear_raw()
|
||||
|
||||
@override
|
||||
def invalidate(self, key: str) -> bool:
|
||||
return self._delete_raw(key)
|
||||
|
||||
|
||||
class MemoryBackend(_TTLStateBackendMixin):
|
||||
"""进程内 dict 后端。进程退出即丢失。
|
||||
@@ -150,11 +167,20 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
ttl:
|
||||
条目存活秒数。``None`` 表示永不过期。``has`` 在条目超过 ttl 后
|
||||
返回 ``False``(但不主动删除,下次 ``save`` 覆盖)。
|
||||
maxsize:
|
||||
最大条目数。``None`` 表示不限。超出时按 LRU(最近最少使用)
|
||||
策略驱逐最旧条目。每次 ``get``/``has`` 命中会将条目移到末尾
|
||||
(最近使用),``save`` 新增条目时若超限则驱逐头部条目。
|
||||
"""
|
||||
|
||||
def __init__(self, ttl: float | None = None) -> None:
|
||||
self._store: dict[str, tuple[Any, float]] = {}
|
||||
def __init__(
|
||||
self,
|
||||
ttl: float | None = None,
|
||||
maxsize: int | None = None,
|
||||
) -> None:
|
||||
self._store: OrderedDict[str, tuple[Any, float]] = OrderedDict()
|
||||
self._ttl = ttl
|
||||
self._maxsize = maxsize
|
||||
|
||||
@override
|
||||
def _now(self) -> float:
|
||||
@@ -162,11 +188,23 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
|
||||
@override
|
||||
def _get_raw(self, key: str) -> tuple[Any, float] | None:
|
||||
return self._store.get(key)
|
||||
entry = self._store.get(key)
|
||||
if entry is None:
|
||||
return None
|
||||
# LRU:命中时移到末尾(最近使用)
|
||||
self._store.move_to_end(key)
|
||||
return entry
|
||||
|
||||
@override
|
||||
def _put_raw(self, key: str, value: Any, ts: float) -> None:
|
||||
if key in self._store:
|
||||
# 已存在:更新并移到末尾
|
||||
self._store.move_to_end(key)
|
||||
self._store[key] = (value, ts)
|
||||
# LRU 驱逐:超限时移除头部(最旧)
|
||||
if self._maxsize is not None:
|
||||
while len(self._store) > self._maxsize:
|
||||
self._store.popitem(last=False)
|
||||
|
||||
@override
|
||||
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
|
||||
@@ -177,6 +215,13 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
def _clear_raw(self) -> None:
|
||||
self._store.clear()
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
if key in self._store:
|
||||
del self._store[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class JSONBackend(_TTLStateBackendMixin):
|
||||
"""基于文件的 JSON 存储,用于跨进程续跑。
|
||||
@@ -245,11 +290,25 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
def _clear_raw(self) -> None:
|
||||
self._store.clear()
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
if key in self._store:
|
||||
del self._store[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
@override
|
||||
def clear(self) -> None:
|
||||
super().clear()
|
||||
self._flush()
|
||||
|
||||
@override
|
||||
def invalidate(self, key: str) -> bool:
|
||||
removed = super().invalidate(key)
|
||||
if removed and not self._defer_flush:
|
||||
self._flush()
|
||||
return removed
|
||||
|
||||
@override
|
||||
def save(self, key: str, value: Any) -> None:
|
||||
# batch 模式下延迟序列化验证到 flush 时(一次性 json.dump 整个 _store),
|
||||
@@ -382,6 +441,17 @@ class SQLiteBackend(_TTLStateBackendMixin):
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError("cannot clear sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
with self._lock:
|
||||
try:
|
||||
cursor = self._conn.execute("DELETE FROM state WHERE key = ?", (key,))
|
||||
if not self._in_batch:
|
||||
self._conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError(f"cannot delete key {key!r} from sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
def flush(self) -> None:
|
||||
with self._lock:
|
||||
|
||||
@@ -245,6 +245,9 @@ class TaskSpec(Generic[T]):
|
||||
同步任务的执行器:``"thread"``(默认,线程池)/ ``"process"``
|
||||
(进程池,绕过 GIL,适合 CPU 密集型;``fn`` 须可 pickle)/
|
||||
``"inline"``(直接在事件循环线程调用,最快但会阻塞循环)。
|
||||
outputs:
|
||||
任务声明的输出键值映射(YAML 编排场景使用)。仅作为元数据携带,
|
||||
不参与执行;可通过 ``report.result_of(name).spec.outputs`` 查询。
|
||||
"""
|
||||
|
||||
name: str
|
||||
@@ -271,6 +274,7 @@ class TaskSpec(Generic[T]):
|
||||
cache_key: CacheKeyFn | None = None
|
||||
hooks: TaskHooks = field(default_factory=TaskHooks)
|
||||
executor: str = "thread" # "thread" | "process" | "inline"
|
||||
outputs: Mapping[str, str] | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
@@ -468,6 +472,7 @@ def task(
|
||||
continue_on_error: bool = False,
|
||||
cache_key: CacheKeyFn | None = None,
|
||||
hooks: TaskHooks | None = None,
|
||||
outputs: Mapping[str, str] | None = None,
|
||||
name: str | None = None,
|
||||
) -> Any:
|
||||
"""装饰器:将函数转为 :class:`TaskSpec`。
|
||||
@@ -509,6 +514,7 @@ def task(
|
||||
continue_on_error=continue_on_error,
|
||||
cache_key=cache_key,
|
||||
hooks=hooks if hooks is not None else TaskHooks(),
|
||||
outputs=dict(outputs) if outputs else None,
|
||||
)
|
||||
|
||||
if fn is None and cmd is None:
|
||||
|
||||
@@ -224,11 +224,25 @@ def _extract_matrix(job_data: Mapping[str, Any]) -> Mapping[str, list[Any]] | No
|
||||
|
||||
|
||||
def _cartesian_product(matrix_data: Mapping[str, list[Any]]) -> list[dict[str, str]]:
|
||||
"""计算矩阵的笛卡尔积。"""
|
||||
keys = list(matrix_data.keys())
|
||||
result: list[dict[str, str]] = [{}]
|
||||
"""计算矩阵的笛卡尔积。
|
||||
|
||||
支持 ``include``/``exclude`` 特殊键(GitHub Actions 语义):
|
||||
``exclude`` 从笛卡尔积中剔除所有匹配的组合;
|
||||
``include`` 追加额外组合(可引入新键)。
|
||||
|
||||
无基础矩阵键且仅有 include 时,不产生空组合(``{}``),
|
||||
仅返回 include 项。
|
||||
"""
|
||||
# 分离 include/exclude 与普通矩阵键
|
||||
matrix_keys = {k: v for k, v in matrix_data.items() if k not in ("include", "exclude")}
|
||||
include_list = matrix_data.get("include")
|
||||
exclude_list = matrix_data.get("exclude")
|
||||
|
||||
# 基础笛卡尔积:无矩阵键时为空列表(不产生 [{}])
|
||||
keys = list(matrix_keys.keys())
|
||||
result: list[dict[str, str]] = [{}] if keys else []
|
||||
for key in keys:
|
||||
values = matrix_data[key]
|
||||
values = matrix_keys[key]
|
||||
if not isinstance(values, list):
|
||||
raise ValueError(f"matrix.{key} 必须是列表,收到: {type(values).__name__}")
|
||||
new_result: list[dict[str, str]] = []
|
||||
@@ -238,9 +252,32 @@ def _cartesian_product(matrix_data: Mapping[str, list[Any]]) -> list[dict[str, s
|
||||
combo[key] = str(value)
|
||||
new_result.append(combo)
|
||||
result = new_result
|
||||
|
||||
# exclude:剔除所有匹配的组合(组合的所有 key-value 对都匹配则剔除)
|
||||
if exclude_list is not None:
|
||||
if not isinstance(exclude_list, list):
|
||||
raise ValueError(f"matrix.exclude 必须是列表,收到: {type(exclude_list).__name__}")
|
||||
excludes = [_normalize_combo(item) for item in exclude_list]
|
||||
result = [c for c in result if not any(c.items() >= ex.items() for ex in excludes)]
|
||||
|
||||
# include:追加额外组合(可引入基础矩阵之外的键)
|
||||
if include_list is not None:
|
||||
if not isinstance(include_list, list):
|
||||
raise ValueError(f"matrix.include 必须是列表,收到: {type(include_list).__name__}")
|
||||
for item in include_list:
|
||||
combo = _normalize_combo(item)
|
||||
result.append(combo)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _normalize_combo(item: Any) -> dict[str, str]:
|
||||
"""将 include/exclude 项归一化为 {str: str} 字典。"""
|
||||
if not isinstance(item, Mapping):
|
||||
raise ValueError(f"matrix include/exclude 项必须是映射,收到: {type(item).__name__}")
|
||||
return {str(k): str(v) for k, v in item.items()}
|
||||
|
||||
|
||||
def _build_specs(
|
||||
jobs: Mapping[str, Any],
|
||||
expansions: dict[str, list[tuple[str, dict[str, str]]]],
|
||||
@@ -327,6 +364,12 @@ def _parse_optional_fields(job_data: Mapping[str, Any], matrix_values: dict[str,
|
||||
if tag_list:
|
||||
kwargs["tags"] = tuple(tag_list)
|
||||
|
||||
outputs = _get_field(job_data, "outputs")
|
||||
if outputs is not None:
|
||||
if not isinstance(outputs, Mapping):
|
||||
raise ValueError(f"outputs 必须是映射,收到: {type(outputs).__name__}")
|
||||
kwargs["outputs"] = {str(k): _substitute_matrix_str(str(v), matrix_values) for k, v in outputs.items()}
|
||||
|
||||
return kwargs
|
||||
|
||||
|
||||
@@ -334,17 +377,36 @@ def _expand_needs(
|
||||
needs: Any,
|
||||
expansions: dict[str, list[tuple[str, dict[str, str]]]],
|
||||
) -> list[str]:
|
||||
"""展开 needs 列表:矩阵 job 依赖展开为所有变体。"""
|
||||
"""展开 needs 列表:矩阵 job 依赖展开为所有变体。
|
||||
|
||||
支持两种格式:
|
||||
* 字符串:``needs: [build]``
|
||||
* 条件依赖对象:``needs: [{job: build, if: "env.CI == 'true'"}]``
|
||||
``if`` 条件不满足时跳过该依赖。
|
||||
"""
|
||||
if not needs:
|
||||
return []
|
||||
if isinstance(needs, str):
|
||||
needs = [needs]
|
||||
result: list[str] = []
|
||||
for dep in needs:
|
||||
if dep in expansions:
|
||||
result.extend(name for name, _ in expansions[dep])
|
||||
for entry in needs:
|
||||
if isinstance(entry, Mapping):
|
||||
job_name = entry.get("job")
|
||||
if not job_name or not isinstance(job_name, str):
|
||||
raise ValueError(f"needs 条件依赖项缺少 'job' 字段: {entry!r}")
|
||||
if_expr = entry.get("if")
|
||||
if if_expr is not None:
|
||||
conditions = _parse_condition(if_expr)
|
||||
# 任一条件不满足则跳过该依赖
|
||||
if not all(cond({}) for cond in conditions):
|
||||
continue
|
||||
dep_name = job_name
|
||||
else:
|
||||
result.append(dep)
|
||||
dep_name = entry
|
||||
if dep_name in expansions:
|
||||
result.extend(name for name, _ in expansions[dep_name])
|
||||
else:
|
||||
result.append(dep_name)
|
||||
return result
|
||||
|
||||
|
||||
|
||||
@@ -109,15 +109,17 @@ class TestPfYamlrunListFiltering:
|
||||
) -> None:
|
||||
"""--list-tag 与 --list-name 组合过滤(交集)。"""
|
||||
yaml_file = _write_yaml(tmp_path, _TAGGED_YAML)
|
||||
app = PfApp([
|
||||
"yamlrun",
|
||||
str(yaml_file),
|
||||
"--list",
|
||||
"--list-tag",
|
||||
"ingest",
|
||||
"--list-name",
|
||||
"b",
|
||||
])
|
||||
app = PfApp(
|
||||
[
|
||||
"yamlrun",
|
||||
str(yaml_file),
|
||||
"--list",
|
||||
"--list-tag",
|
||||
"ingest",
|
||||
"--list-name",
|
||||
"b",
|
||||
]
|
||||
)
|
||||
rc = app.run()
|
||||
assert rc == 0
|
||||
out = capsys.readouterr().out
|
||||
|
||||
@@ -507,10 +507,12 @@ class TestRunReportIntegration:
|
||||
return 1
|
||||
|
||||
# continue_on_error=True 使失败不抛异常而写入报告(success 保持 True)
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("boom", boom, continue_on_error=True),
|
||||
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
|
||||
])
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec("boom", boom, continue_on_error=True),
|
||||
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
|
||||
]
|
||||
)
|
||||
report = px.run(graph, strategy="sequential")
|
||||
# continue_on_error 不影响 success 标志,手动标记以触发诊断
|
||||
report.success = False
|
||||
|
||||
@@ -12,9 +12,12 @@ from pyflowx.notification import (
|
||||
ALL_LEVELS,
|
||||
CallbackNotifier,
|
||||
DingTalkNotifier,
|
||||
DiscordNotifier,
|
||||
FeishuNotifier,
|
||||
NotificationLevel,
|
||||
Notifier,
|
||||
SlackNotifier,
|
||||
TelegramNotifier,
|
||||
WebhookNotifier,
|
||||
WeChatNotifier,
|
||||
_format_event_message,
|
||||
@@ -146,6 +149,24 @@ def test_wechat_notifier_satisfies_protocol() -> None:
|
||||
assert isinstance(WeChatNotifier(url="http://example.com/hook"), Notifier)
|
||||
|
||||
|
||||
def test_slack_notifier_satisfies_protocol() -> None:
|
||||
"""SlackNotifier 满足 Notifier 协议。"""
|
||||
assert isinstance(SlackNotifier(url="http://example.com/hook"), Notifier)
|
||||
|
||||
|
||||
def test_discord_notifier_satisfies_protocol() -> None:
|
||||
"""DiscordNotifier 满足 Notifier 协议。"""
|
||||
assert isinstance(DiscordNotifier(url="http://example.com/hook"), Notifier)
|
||||
|
||||
|
||||
def test_telegram_notifier_satisfies_protocol() -> None:
|
||||
"""TelegramNotifier 满足 Notifier 协议。"""
|
||||
assert isinstance(
|
||||
TelegramNotifier(url="http://example.com/hook", chat_id="123456"),
|
||||
Notifier,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# CallbackNotifier
|
||||
# ---------------------------------------------------------------------- #
|
||||
@@ -380,6 +401,123 @@ def test_wechat_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None
|
||||
assert "跳过原因" in payload["text"]["content"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# SlackNotifier
|
||||
# ---------------------------------------------------------------------- #
|
||||
def test_slack_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Slack 消息格式:text 字段。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = SlackNotifier(url="http://example.com/hook")
|
||||
notifier.notify(_event(TaskStatus.FAILED, error="boom", duration=0.3))
|
||||
|
||||
fields = _get_req_fields(capture["req"])
|
||||
payload = json.loads(fields["data"]) # type: ignore[arg-type]
|
||||
assert "text" in payload
|
||||
assert "test_task" in payload["text"]
|
||||
assert "boom" in payload["text"]
|
||||
|
||||
|
||||
def test_slack_notifier_levels_filter(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Slack levels 过滤:SUCCESS 不在 levels 中时不发送。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = SlackNotifier(
|
||||
url="http://example.com/hook",
|
||||
levels=(NotificationLevel.FAILED,),
|
||||
)
|
||||
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.1))
|
||||
assert "req" not in capture
|
||||
|
||||
notifier.notify(_event(TaskStatus.FAILED, error="err"))
|
||||
assert "req" in capture
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# DiscordNotifier
|
||||
# ---------------------------------------------------------------------- #
|
||||
def test_discord_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Discord 消息格式:content 字段。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = DiscordNotifier(url="http://example.com/hook")
|
||||
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.5))
|
||||
|
||||
fields = _get_req_fields(capture["req"])
|
||||
payload = json.loads(fields["data"]) # type: ignore[arg-type]
|
||||
assert "content" in payload
|
||||
assert "test_task" in payload["content"]
|
||||
assert "success" in payload["content"]
|
||||
|
||||
|
||||
def test_discord_notifier_timeout_passed(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Discord timeout 参数透传到 urlopen。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = DiscordNotifier(url="http://example.com/hook", timeout=8.0)
|
||||
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.1))
|
||||
assert capture["timeout"] == 8.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# TelegramNotifier
|
||||
# ---------------------------------------------------------------------- #
|
||||
def test_telegram_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Telegram 消息格式:chat_id + text 字段。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="@mychannel")
|
||||
notifier.notify(_event(TaskStatus.FAILED, error="网络异常"))
|
||||
|
||||
fields = _get_req_fields(capture["req"])
|
||||
payload = json.loads(fields["data"]) # type: ignore[arg-type]
|
||||
assert payload["chat_id"] == "@mychannel"
|
||||
assert "text" in payload
|
||||
assert "test_task" in payload["text"]
|
||||
assert "网络异常" in payload["text"]
|
||||
|
||||
|
||||
def test_telegram_notifier_chat_id_numeric(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Telegram 数字 chat_id 正确传递。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="987654321")
|
||||
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.2))
|
||||
|
||||
fields = _get_req_fields(capture["req"])
|
||||
payload = json.loads(fields["data"]) # type: ignore[arg-type]
|
||||
assert payload["chat_id"] == "987654321"
|
||||
|
||||
|
||||
def test_telegram_notifier_levels_filter(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Telegram levels 过滤:SKIPPED 不在 levels 中时不发送。"""
|
||||
capture: dict[str, object] = {}
|
||||
_patch_urlopen(monkeypatch, capture)
|
||||
|
||||
notifier = TelegramNotifier(
|
||||
url="http://example.com/hook",
|
||||
chat_id="123",
|
||||
levels=(NotificationLevel.FAILED,),
|
||||
)
|
||||
notifier.notify(_event(TaskStatus.SKIPPED, reason="跳过"))
|
||||
assert "req" not in capture
|
||||
|
||||
notifier.notify(_event(TaskStatus.FAILED, error="err"))
|
||||
assert "req" in capture
|
||||
|
||||
|
||||
def test_telegram_notifier_close_is_noop() -> None:
|
||||
"""TelegramNotifier.close() 无副作用。"""
|
||||
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="123")
|
||||
notifier.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# run() 集成:notifiers 参数
|
||||
# ---------------------------------------------------------------------- #
|
||||
@@ -554,6 +692,9 @@ def test_notification_classes_exported() -> None:
|
||||
assert hasattr(px, "FeishuNotifier")
|
||||
assert hasattr(px, "DingTalkNotifier")
|
||||
assert hasattr(px, "WeChatNotifier")
|
||||
assert hasattr(px, "SlackNotifier")
|
||||
assert hasattr(px, "DiscordNotifier")
|
||||
assert hasattr(px, "TelegramNotifier")
|
||||
assert hasattr(px, "ALL_LEVELS")
|
||||
|
||||
|
||||
|
||||
+178
-8
@@ -395,6 +395,172 @@ class TestRunReportToCsv:
|
||||
assert "PosixPath('/tmp/x')" in s
|
||||
|
||||
|
||||
class TestRunReportToHtml:
|
||||
"""测试 RunReport.to_html."""
|
||||
|
||||
def test_to_html_basic_structure(self) -> None:
|
||||
"""应输出完整 HTML5 文档结构。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result("a", value=42)
|
||||
s = report.to_html()
|
||||
assert s.startswith("<!DOCTYPE html>")
|
||||
assert s.endswith("</html>")
|
||||
assert "<html" in s
|
||||
assert "<head>" in s
|
||||
assert "<body>" in s
|
||||
|
||||
def test_to_html_contains_run_id(self) -> None:
|
||||
"""HTML 应在标题与正文包含 run_id。"""
|
||||
report = px.RunReport(run_id="abcd1234")
|
||||
report.results["a"] = _make_result("a")
|
||||
s = report.to_html()
|
||||
assert "abcd1234" in s
|
||||
|
||||
def test_to_html_contains_task_name(self) -> None:
|
||||
"""HTML 表格应包含任务名。"""
|
||||
report = px.RunReport()
|
||||
report.results["extract"] = _make_result("extract", value=1)
|
||||
s = report.to_html()
|
||||
assert "extract" in s
|
||||
assert "<td" in s
|
||||
|
||||
def test_to_html_status_badge_class(self) -> None:
|
||||
"""不同状态应渲染对应 CSS 类。"""
|
||||
report = px.RunReport()
|
||||
report.results["ok"] = _make_result("ok", status=TaskStatus.SUCCESS)
|
||||
report.results["bad"] = _make_result("bad", status=TaskStatus.FAILED, error=ValueError("x"))
|
||||
report.results["skip"] = _make_result("skip", status=TaskStatus.SKIPPED, duration=0.0, reason="r")
|
||||
s = report.to_html()
|
||||
assert "status-success" in s
|
||||
assert "status-failed" in s
|
||||
assert "status-skipped" in s
|
||||
|
||||
def test_to_html_xss_escape_task_name(self) -> None:
|
||||
"""任务名含 HTML 特殊字符应被转义。"""
|
||||
report = px.RunReport()
|
||||
malicious = "<script>evil()</script>"
|
||||
spec: TaskSpec[Any] = TaskSpec[Any](malicious, _fn)
|
||||
result: TaskResult[Any] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SUCCESS,
|
||||
value=1,
|
||||
attempts=1,
|
||||
)
|
||||
report.results[malicious] = result
|
||||
s = report.to_html()
|
||||
assert "<script>" not in s
|
||||
assert "<script>" in s
|
||||
|
||||
def test_to_html_xss_escape_error(self) -> None:
|
||||
"""错误信息含 HTML 特殊字符应被转义。"""
|
||||
report = px.RunReport()
|
||||
err = ValueError("<img src=x onerror=alert(1)>")
|
||||
report.results["a"] = _make_result("a", status=TaskStatus.FAILED, error=err)
|
||||
s = report.to_html()
|
||||
assert "<img" not in s
|
||||
assert "<img" in s
|
||||
|
||||
def test_to_html_xss_escape_reason(self) -> None:
|
||||
"""跳过原因含 HTML 特殊字符应被转义。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result(
|
||||
"a",
|
||||
status=TaskStatus.SKIPPED,
|
||||
duration=0.0,
|
||||
reason="<b>bold</b>",
|
||||
)
|
||||
s = report.to_html()
|
||||
assert "<b>bold</b>" not in s
|
||||
assert "<b>" in s
|
||||
|
||||
def test_to_html_without_value(self) -> None:
|
||||
"""include_value=False 应省略返回值列。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result("a", value="secret")
|
||||
s = report.to_html(include_value=False)
|
||||
assert "secret" not in s
|
||||
assert "返回值" not in s
|
||||
|
||||
def test_to_html_with_value(self) -> None:
|
||||
"""include_value=True 应包含返回值(JSON 序列化形式)。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result("a", value=[1, 2, 3])
|
||||
s = report.to_html()
|
||||
assert "[1, 2, 3]" in s
|
||||
|
||||
def test_to_html_with_value_serializer(self) -> None:
|
||||
"""to_html 应透传 value_serializer。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
|
||||
s = report.to_html(value_serializer=str)
|
||||
assert "/tmp/x" in s
|
||||
assert "PosixPath" not in s
|
||||
|
||||
def test_to_html_empty_report(self) -> None:
|
||||
"""空报告应渲染提示文本而非空表格。"""
|
||||
report = px.RunReport()
|
||||
s = report.to_html()
|
||||
assert "无任务结果" in s
|
||||
assert "<table>" not in s
|
||||
|
||||
def test_to_html_failure_status_indicator(self) -> None:
|
||||
"""失败报告应在汇总卡片显示失败样式。"""
|
||||
report = px.RunReport(success=False)
|
||||
report.results["a"] = _make_result("a", status=TaskStatus.FAILED, error=RuntimeError("x"))
|
||||
s = report.to_html()
|
||||
assert "card failure" in s
|
||||
assert "失败" in s
|
||||
|
||||
def test_to_html_success_status_indicator(self) -> None:
|
||||
"""成功报告应在汇总卡片显示成功样式。"""
|
||||
report = px.RunReport(success=True)
|
||||
report.results["a"] = _make_result("a", status=TaskStatus.SUCCESS)
|
||||
s = report.to_html()
|
||||
assert "card success" in s
|
||||
assert "成功" in s
|
||||
|
||||
def test_to_html_summary_cards_content(self) -> None:
|
||||
"""汇总卡片应包含任务总数/耗时/各状态计数。"""
|
||||
report = px.RunReport()
|
||||
report.results["a"] = _make_result("a", status=TaskStatus.SUCCESS, duration=0.5)
|
||||
report.results["b"] = _make_result("b", status=TaskStatus.FAILED, error=ValueError("x"), duration=0.3)
|
||||
s = report.to_html()
|
||||
assert "任务总数" in s
|
||||
assert "总耗时" in s
|
||||
assert "成功" in s
|
||||
assert "失败" in s
|
||||
assert "跳过" in s
|
||||
assert "2" in s # 任务总数
|
||||
assert "0.800" in s # 0.5+0.3
|
||||
|
||||
def test_to_html_no_timestamps(self) -> None:
|
||||
"""无时间戳时应渲染占位符。"""
|
||||
report = px.RunReport()
|
||||
# 构造无时间戳的结果
|
||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn)
|
||||
result: TaskResult[Any] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SUCCESS,
|
||||
value=1,
|
||||
attempts=1,
|
||||
started_at=None,
|
||||
finished_at=None,
|
||||
)
|
||||
report.results["a"] = result
|
||||
s = report.to_html()
|
||||
# 无时间戳时表格单元格显示 "-"
|
||||
assert s.count("<td>-</td>") >= 2
|
||||
|
||||
def test_to_html_css_inlined(self) -> None:
|
||||
"""CSS 应内联在 <style> 标签中,无外部依赖。"""
|
||||
report = px.RunReport()
|
||||
s = report.to_html()
|
||||
assert "<style>" in s
|
||||
assert "</style>" in s
|
||||
assert "stylesheet" not in s.lower() # 无外部样式表引用
|
||||
assert "link rel" not in s.lower()
|
||||
|
||||
|
||||
class TestRunReportFromJson:
|
||||
"""测试 RunReport.from_json."""
|
||||
|
||||
@@ -485,10 +651,12 @@ class TestRunReportSerializationIntegration:
|
||||
def double(extract: list[int]) -> list[int]:
|
||||
return [x * 2 for x in extract]
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("extract", extract, tags=("ingest",)),
|
||||
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
|
||||
])
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec("extract", extract, tags=("ingest",)),
|
||||
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
|
||||
]
|
||||
)
|
||||
report = px.run(graph, strategy="sequential")
|
||||
|
||||
# 序列化 → 反序列化
|
||||
@@ -584,10 +752,12 @@ class TestRunId:
|
||||
def task_b(a: int) -> int:
|
||||
return a * 2
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("a", task_a),
|
||||
px.TaskSpec("b", task_b, depends_on=("a",)),
|
||||
])
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec("a", task_a),
|
||||
px.TaskSpec("b", task_b, depends_on=("a",)),
|
||||
]
|
||||
)
|
||||
report = px.run(graph, strategy="sequential")
|
||||
profile = report.profile(graph)
|
||||
assert profile.total_duration >= 0
|
||||
|
||||
@@ -591,3 +591,185 @@ def test_state_backend_is_abstract() -> None:
|
||||
"""StateBackend 是 ABC,不能直接实例化。"""
|
||||
with pytest.raises(TypeError):
|
||||
StateBackend() # type: ignore[abstract]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# invalidate(手动失效)
|
||||
# ---------------------------------------------------------------------- #
|
||||
def test_memory_backend_invalidate_existing_key() -> None:
|
||||
"""MemoryBackend.invalidate 删除已存在键返回 True。"""
|
||||
b = MemoryBackend()
|
||||
b.save("a", 1)
|
||||
assert b.invalidate("a") is True
|
||||
assert not b.has("a")
|
||||
|
||||
|
||||
def test_memory_backend_invalidate_missing_key() -> None:
|
||||
"""MemoryBackend.invalidate 删除不存在键返回 False。"""
|
||||
b = MemoryBackend()
|
||||
assert b.invalidate("missing") is False
|
||||
|
||||
|
||||
def test_memory_backend_invalidate_then_save() -> None:
|
||||
"""invalidate 后可重新 save。"""
|
||||
b = MemoryBackend()
|
||||
b.save("a", 1)
|
||||
b.invalidate("a")
|
||||
b.save("a", 2)
|
||||
assert b.get("a") == 2
|
||||
|
||||
|
||||
def test_json_backend_invalidate_existing_key(tmp_path: Path) -> None:
|
||||
"""JSONBackend.invalidate 删除并落盘。"""
|
||||
path = tmp_path / "state.json"
|
||||
b = JSONBackend(str(path))
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
assert b.invalidate("a") is True
|
||||
assert not b.has("a")
|
||||
assert b.has("b")
|
||||
# 重新加载验证落盘
|
||||
b2 = JSONBackend(str(path))
|
||||
assert not b2.has("a")
|
||||
assert b2.has("b")
|
||||
|
||||
|
||||
def test_json_backend_invalidate_missing_key(tmp_path: Path) -> None:
|
||||
"""JSONBackend.invalidate 不存在键返回 False。"""
|
||||
b = JSONBackend(str(tmp_path / "state.json"))
|
||||
assert b.invalidate("missing") is False
|
||||
|
||||
|
||||
def test_json_backend_invalidate_in_batch_no_flush(tmp_path: Path) -> None:
|
||||
"""batch 模式下 invalidate 不立即落盘。"""
|
||||
path = tmp_path / "state.json"
|
||||
b = JSONBackend(str(path))
|
||||
b.save("a", 1)
|
||||
with b.batch():
|
||||
b.invalidate("a")
|
||||
# batch 内不落盘,重新加载仍应有 a
|
||||
b2 = JSONBackend(str(path))
|
||||
assert b2.has("a")
|
||||
# batch 退出后落盘
|
||||
b2 = JSONBackend(str(path))
|
||||
assert not b2.has("a")
|
||||
|
||||
|
||||
def test_sqlite_backend_invalidate_existing_key(tmp_path: Path) -> None:
|
||||
"""SQLiteBackend.invalidate 删除已存在键。"""
|
||||
path = tmp_path / "state.db"
|
||||
b = SQLiteBackend(str(path))
|
||||
b.save("a", 1)
|
||||
assert b.invalidate("a") is True
|
||||
assert not b.has("a")
|
||||
|
||||
|
||||
def test_sqlite_backend_invalidate_missing_key(tmp_path: Path) -> None:
|
||||
"""SQLiteBackend.invalidate 不存在键返回 False。"""
|
||||
b = SQLiteBackend(str(tmp_path / "state.db"))
|
||||
assert b.invalidate("missing") is False
|
||||
|
||||
|
||||
def test_sqlite_backend_invalidate_in_batch(tmp_path: Path) -> None:
|
||||
"""SQLiteBackend batch 模式下 invalidate 延迟 commit。"""
|
||||
path = tmp_path / "state.db"
|
||||
b = SQLiteBackend(str(path))
|
||||
b.save("a", 1)
|
||||
with b.batch():
|
||||
b.invalidate("a")
|
||||
# batch 内不 commit,新连接仍可见 a
|
||||
b2 = SQLiteBackend(str(path))
|
||||
assert b2.has("a")
|
||||
# batch 退出后 commit,新连接不可见 a
|
||||
b3 = SQLiteBackend(str(path))
|
||||
assert not b3.has("a")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# MemoryBackend LRU 驱逐
|
||||
# ---------------------------------------------------------------------- #
|
||||
def test_memory_backend_maxsize_evicts_oldest() -> None:
|
||||
"""maxsize 超限时驱逐最旧条目。"""
|
||||
b = MemoryBackend(maxsize=2)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
b.save("c", 3) # 超限,驱逐 a
|
||||
assert not b.has("a")
|
||||
assert b.has("b")
|
||||
assert b.has("c")
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_get_updates_lru_order() -> None:
|
||||
"""get 命中更新 LRU 顺序,避免被驱逐。"""
|
||||
b = MemoryBackend(maxsize=2)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
_ = b.get("a") # a 移到末尾(最近使用)
|
||||
b.save("c", 3) # 超限,驱逐 b(最旧)
|
||||
assert b.has("a")
|
||||
assert not b.has("b")
|
||||
assert b.has("c")
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_has_updates_lru_order() -> None:
|
||||
"""has 命中也更新 LRU 顺序。"""
|
||||
b = MemoryBackend(maxsize=2)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
_ = b.has("a") # a 移到末尾
|
||||
b.save("c", 3) # 驱逐 b
|
||||
assert b.has("a")
|
||||
assert not b.has("b")
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_update_existing_no_evict() -> None:
|
||||
"""更新已存在键不触发驱逐。"""
|
||||
b = MemoryBackend(maxsize=2)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
b.save("a", 10) # 更新 a,不超限
|
||||
assert b.has("a")
|
||||
assert b.has("b")
|
||||
assert b.get("a") == 10
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_none_no_eviction() -> None:
|
||||
"""maxsize=None 不驱逐。"""
|
||||
b = MemoryBackend(maxsize=None)
|
||||
for i in range(100):
|
||||
b.save(f"key{i}", i)
|
||||
assert len(b.load()) == 100
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_one() -> None:
|
||||
"""maxsize=1 只保留最新。"""
|
||||
b = MemoryBackend(maxsize=1)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2) # 驱逐 a
|
||||
assert not b.has("a")
|
||||
assert b.has("b")
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_with_ttl() -> None:
|
||||
"""maxsize 与 TTL 组合工作。"""
|
||||
b = MemoryBackend(ttl=100, maxsize=2)
|
||||
b.save("a", 1)
|
||||
b.save("b", 2)
|
||||
b.save("c", 3) # 驱逐 a
|
||||
assert not b.has("a")
|
||||
assert b.has("b")
|
||||
assert b.has("c")
|
||||
|
||||
|
||||
def test_memory_backend_maxsize_load_count() -> None:
|
||||
"""load() 返回的条目数受 maxsize 限制。"""
|
||||
b = MemoryBackend(maxsize=3)
|
||||
for i in range(5):
|
||||
b.save(f"k{i}", i)
|
||||
data = b.load()
|
||||
assert len(data) == 3 # 只保留最近 3 个
|
||||
assert "k2" in data
|
||||
assert "k3" in data
|
||||
assert "k4" in data
|
||||
assert "k0" not in data
|
||||
assert "k1" not in data
|
||||
|
||||
@@ -302,6 +302,436 @@ jobs:
|
||||
assert "build_version-3.11" in deploy_spec.depends_on
|
||||
|
||||
|
||||
class TestMatrixIncludeExclude:
|
||||
"""矩阵 include/exclude 过滤测试。"""
|
||||
|
||||
def test_exclude_removes_matching_combo(self) -> None:
|
||||
"""exclude 剔除匹配的组合。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
os: ["linux", "macos"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
os: "macos"
|
||||
""")
|
||||
names = set(graph.names)
|
||||
assert "test_version-3.10_os-macos" not in names
|
||||
assert "test_version-3.10_os-linux" in names
|
||||
assert "test_version-3.11_os-linux" in names
|
||||
assert "test_version-3.11_os-macos" in names
|
||||
assert len(names) == 3
|
||||
|
||||
def test_exclude_partial_match_keeps_combo(self) -> None:
|
||||
"""exclude 仅匹配部分 key 时不剔除。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
os: ["linux", "macos"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
""")
|
||||
# exclude {version: 3.10} 匹配两个组合,都剔除
|
||||
names = set(graph.names)
|
||||
assert "test_version-3.10_os-linux" not in names
|
||||
assert "test_version-3.10_os-macos" not in names
|
||||
assert "test_version-3.11_os-linux" in names
|
||||
assert "test_version-3.11_os-macos" in names
|
||||
assert len(names) == 2
|
||||
|
||||
def test_include_adds_extra_combo(self) -> None:
|
||||
"""include 追加额外组合。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
include:
|
||||
- version: "3.12"
|
||||
extra: "new"
|
||||
""")
|
||||
names = set(graph.names)
|
||||
# 基础矩阵
|
||||
assert "test_version-3.10" in names
|
||||
assert "test_version-3.11" in names
|
||||
# include 追加(含新键 extra)
|
||||
assert "test_version-3.12_extra-new" in names
|
||||
assert len(names) == 3
|
||||
|
||||
def test_include_only_no_base_matrix(self) -> None:
|
||||
"""无基础矩阵时 include 仍可工作。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- version: "3.10"
|
||||
- version: "3.11"
|
||||
""")
|
||||
names = set(graph.names)
|
||||
assert "test_version-3.10" in names
|
||||
assert "test_version-3.11" in names
|
||||
assert len(names) == 2
|
||||
|
||||
def test_exclude_all_combos(self) -> None:
|
||||
"""exclude 剔除所有组合时,job 不产生任务。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
- version: "3.11"
|
||||
""")
|
||||
assert len(graph.names) == 0
|
||||
|
||||
def test_include_exclude_combined(self) -> None:
|
||||
"""include 与 exclude 组合使用。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
os: ["linux", "macos"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
os: "linux"
|
||||
include:
|
||||
- version: "3.12"
|
||||
os: "linux"
|
||||
""")
|
||||
names = set(graph.names)
|
||||
assert "test_version-3.10_os-linux" not in names # excluded
|
||||
assert "test_version-3.10_os-macos" in names
|
||||
assert "test_version-3.11_os-linux" in names
|
||||
assert "test_version-3.11_os-macos" in names
|
||||
assert "test_version-3.12_os-linux" in names # included
|
||||
assert len(names) == 4
|
||||
|
||||
def test_exclude_not_list_raises(self) -> None:
|
||||
"""exclude 非列表时抛 ValueError。"""
|
||||
with pytest.raises(ValueError, match=r"matrix\.exclude 必须是列表"):
|
||||
parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10"]
|
||||
exclude: "not a list"
|
||||
""")
|
||||
|
||||
def test_include_not_list_raises(self) -> None:
|
||||
"""include 非列表时抛 ValueError。"""
|
||||
with pytest.raises(ValueError, match=r"matrix\.include 必须是列表"):
|
||||
parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10"]
|
||||
include: "not a list"
|
||||
""")
|
||||
|
||||
def test_exclude_item_not_mapping_raises(self) -> None:
|
||||
"""exclude 项非映射时抛 ValueError。"""
|
||||
with pytest.raises(ValueError, match="matrix include/exclude 项必须是映射"):
|
||||
parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["echo", "test"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10"]
|
||||
exclude:
|
||||
- "not a mapping"
|
||||
""")
|
||||
|
||||
def test_include_with_matrix_placeholder_substitution(self) -> None:
|
||||
"""include 组合中的值参与占位符替换。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
test:
|
||||
cmd: ["python${{ matrix.version }}", "-m", "pytest"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10"]
|
||||
include:
|
||||
- version: "3.12"
|
||||
""")
|
||||
spec = graph.spec("test_version-3.12")
|
||||
assert spec.cmd == ["python3.12", "-m", "pytest"]
|
||||
|
||||
|
||||
class TestConditionalNeeds:
|
||||
"""条件依赖测试。"""
|
||||
|
||||
def test_conditional_need_with_env_var_exists(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""if env.CI 满足时依赖被加入。"""
|
||||
monkeypatch.setenv("CI", "true")
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.CI"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "build" in deploy_spec.depends_on
|
||||
|
||||
def test_conditional_need_skipped_when_env_missing(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""if env.CI 不满足时依赖被跳过。"""
|
||||
monkeypatch.delenv("MISSING_VAR", raising=False)
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.MISSING_VAR"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "build" not in deploy_spec.depends_on
|
||||
|
||||
def test_conditional_need_with_env_equals(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""if env.VAR == 'x' 精确匹配时依赖被加入。"""
|
||||
monkeypatch.setenv("BRANCH", "main")
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.BRANCH == 'main'"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "build" in deploy_spec.depends_on
|
||||
|
||||
def test_conditional_need_skipped_with_env_not_equals(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""if env.VAR == 'x' 不匹配时依赖被跳过。"""
|
||||
monkeypatch.setenv("BRANCH", "dev")
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.BRANCH == 'main'"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "build" not in deploy_spec.depends_on
|
||||
|
||||
def test_mixed_string_and_conditional_needs(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""字符串依赖与条件依赖混合使用。"""
|
||||
monkeypatch.setenv("CI", "true")
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
setup:
|
||||
cmd: ["echo", "setup"]
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- setup
|
||||
- job: build
|
||||
if: "env.CI"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "setup" in deploy_spec.depends_on
|
||||
assert "build" in deploy_spec.depends_on
|
||||
|
||||
def test_conditional_need_missing_job_field_raises(self) -> None:
|
||||
"""条件依赖项缺少 job 字段时抛 ValueError。"""
|
||||
with pytest.raises(ValueError, match="缺少 'job' 字段"):
|
||||
parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
deploy:
|
||||
needs:
|
||||
- if: "env.CI"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
|
||||
def test_conditional_need_with_matrix_job(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""条件依赖展开矩阵 job。"""
|
||||
monkeypatch.setenv("DEPLOY", "true")
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.DEPLOY"
|
||||
cmd: ["echo", "deploy"]
|
||||
""")
|
||||
deploy_spec = graph.spec("deploy")
|
||||
assert "build_version-3.10" in deploy_spec.depends_on
|
||||
assert "build_version-3.11" in deploy_spec.depends_on
|
||||
|
||||
|
||||
class TestOutputs:
|
||||
"""outputs 字段测试。"""
|
||||
|
||||
def test_outputs_parsed_as_metadata(self) -> None:
|
||||
"""outputs 作为元数据附加到 TaskSpec。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
outputs:
|
||||
version: "1.0.0"
|
||||
artifact: "dist.tar.gz"
|
||||
""")
|
||||
spec = graph.spec("build")
|
||||
assert spec.outputs is not None
|
||||
assert spec.outputs["version"] == "1.0.0"
|
||||
assert spec.outputs["artifact"] == "dist.tar.gz"
|
||||
|
||||
def test_outputs_none_when_not_declared(self) -> None:
|
||||
"""未声明 outputs 时为 None。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
""")
|
||||
spec = graph.spec("build")
|
||||
assert spec.outputs is None
|
||||
|
||||
def test_outputs_with_matrix_placeholder(self) -> None:
|
||||
"""outputs 值支持 ${{ matrix.* }} 占位符。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
outputs:
|
||||
version: "${{ matrix.version }}"
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11"]
|
||||
""")
|
||||
spec_310 = graph.spec("build_version-3.10")
|
||||
spec_311 = graph.spec("build_version-3.11")
|
||||
assert spec_310.outputs is not None
|
||||
assert spec_310.outputs["version"] == "3.10"
|
||||
assert spec_311.outputs is not None
|
||||
assert spec_311.outputs["version"] == "3.11"
|
||||
|
||||
def test_outputs_not_mapping_raises(self) -> None:
|
||||
"""outputs 非映射时抛 ValueError。"""
|
||||
with pytest.raises(ValueError, match="outputs 必须是映射"):
|
||||
parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
outputs: "not a mapping"
|
||||
""")
|
||||
|
||||
def test_outputs_round_trip_via_json(self) -> None:
|
||||
"""outputs 通过 to_json/from_json 往返保留。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
outputs:
|
||||
version: "1.0.0"
|
||||
""")
|
||||
spec = graph.spec("build")
|
||||
report = px.RunReport()
|
||||
from datetime import datetime
|
||||
|
||||
from pyflowx.task import TaskResult
|
||||
|
||||
report.results["build"] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SUCCESS,
|
||||
value=0,
|
||||
attempts=1,
|
||||
started_at=datetime.now(),
|
||||
finished_at=datetime.now(),
|
||||
)
|
||||
text = report.to_json()
|
||||
restored = px.RunReport.from_json(text)
|
||||
assert restored.result_of("build").spec.outputs == {"version": "1.0.0"}
|
||||
|
||||
def test_outputs_in_to_dict(self) -> None:
|
||||
"""to_dict 包含 outputs 字段。"""
|
||||
graph = parse_yaml_string("""
|
||||
jobs:
|
||||
build:
|
||||
cmd: ["echo", "build"]
|
||||
outputs:
|
||||
version: "1.0.0"
|
||||
""")
|
||||
spec = graph.spec("build")
|
||||
report = px.RunReport()
|
||||
from datetime import datetime
|
||||
|
||||
from pyflowx.task import TaskResult
|
||||
|
||||
report.results["build"] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SUCCESS,
|
||||
value=0,
|
||||
attempts=1,
|
||||
started_at=datetime.now(),
|
||||
finished_at=datetime.now(),
|
||||
)
|
||||
d = report.to_dict()
|
||||
assert d["results"][0]["outputs"] == {"version": "1.0.0"}
|
||||
|
||||
def test_outputs_none_in_to_dict(self) -> None:
|
||||
"""无 outputs 时 to_dict 中为 None。"""
|
||||
report = px.RunReport()
|
||||
from datetime import datetime
|
||||
|
||||
from pyflowx.task import TaskResult, TaskSpec
|
||||
|
||||
spec: TaskSpec[int] = TaskSpec("a", lambda: 1)
|
||||
report.results["a"] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SUCCESS,
|
||||
value=1,
|
||||
attempts=1,
|
||||
started_at=datetime.now(),
|
||||
finished_at=datetime.now(),
|
||||
)
|
||||
d = report.to_dict()
|
||||
assert d["results"][0]["outputs"] is None
|
||||
|
||||
|
||||
class TestConditions:
|
||||
"""条件解析测试。"""
|
||||
|
||||
|
||||
Reference in New Issue
Block a user