feat: P7 功能增强 — 通知器扩展(Slack/Discord/Telegram)、HTML 报告导出(to_html)、YAML 增强(matrix include-exclude/条件依赖/outputs)、缓存增强(invalidate 单键失效/MemoryBackend LRU 驱逐)
CI / Lint, Typecheck & Test (push) Has been cancelled

This commit is contained in:
2026-07-07 16:26:46 +08:00
parent 9573f1f2a2
commit a9f15e236d
14 changed files with 1424 additions and 38 deletions
+90
View File
@@ -0,0 +1,90 @@
# 迭代 19 — P7 功能增强
## 本轮目标
P7 阶段聚焦功能增强,涵盖四个子任务:
- P7.1 通知器扩展(Slack/Discord/Telegram
- P7.2 HTML 报告导出(RunReport.to_html
- P7.3 YAML 增强特性(matrix include-exclude / 条件依赖 / outputs
- P7.4 任务缓存增强(invalidate 单键失效 / MemoryBackend LRU 驱逐)
## 改动文件清单
### P7.1 通知器扩展
- `src/pyflowx/notification.py` — 新增 SlackNotifier/DiscordNotifier/TelegramNotifier
- `src/pyflowx/__init__.py` — 导出新通知器类
- `tests/test_notification.py` — 新增 14 个测试
- `README.md` — 通知器章节扩展
### P7.2 HTML 报告导出
- `src/pyflowx/report.py` — 新增 to_html 方法 + _render_summary_card 辅助
- `tests/test_report.py` — 新增 16 个测试
- `README.md` — 序列化章节扩展
### P7.3 YAML 增强特性
- `src/pyflowx/yaml_loader.py` — _cartesian_product 支持 include/exclude_expand_needs 支持条件依赖;_parse_optional_fields 解析 outputs
- `src/pyflowx/task.py` — TaskSpec 新增 outputs 字段;task() 工厂支持 outputs 参数
- `src/pyflowx/report.py` — to_dict/from_json 包含 outputs
- `tests/test_yaml_loader.py` — 新增 24 个测试
- `README.md` — YAML 章节扩展
### P7.4 任务缓存增强
- `src/pyflowx/storage.py` — StateBackend 新增 invalidate 抽象方法;_TTLStateBackendMixin 新增 _delete_raw 原语与 invalidate 实现;MemoryBackend 新增 maxsize LRU 驱逐;JSONBackend/SQLiteBackend 实现 _delete_raw
- `tests/test_storage.py` — 新增 17 个测试
- `README.md` — 缓存章节扩展
## 关键决策与依据
### 通知器架构(P7.1
- 复用 WebhookNotifier 基类,仅覆盖 _build_payload 适配平台格式
- TelegramNotifier 额外需要 chat_id 参数,故覆盖 __init__
- 零新依赖(stdlib urllib.request
### HTML 报告设计(P7.2
- 自包含 HTML5 文档,内联 CSS,无外部依赖
- 使用 html.escape 转义所有用户内容(任务名/错误/原因/值)防 XSS
- 状态徽章着色:success 绿 / failed 红 / skipped 黄 / running 蓝
- _render_summary_card 静态方法避免重复代码
### YAML 矩阵 include/exclude 语义(P7.3
- exclude: 组合的所有 key-value 对都匹配则剔除(c.items() >= ex.items()
- include: 追加额外组合,可引入基础矩阵之外的键
- 无基础矩阵键时返回空列表而非 [{}],避免产生空名任务
### 条件依赖设计(P7.3
- needs 项支持 {job: name, if: expr} 格式
- if 条件不满足时跳过该依赖(不加入 depends_on
- 复用 _parse_condition 解析 if 表达式
- 条件求值时传空上下文 {}(env 条件直接读 os.environ
### outputs 字段(P7.3
- TaskSpec 新增 outputs: Mapping[str, str] | None 作为元数据
- 仅携带不参与执行,可通过 spec.outputs 查询
- 支持 ${{ matrix.* }} 占位符替换
- to_dict/from_json 往返保留
### 缓存 invalidate 设计(P7.4
- StateBackend 新增 invalidate 抽象方法(返回 bool 表示是否删除)
- _TTLStateBackendMixin 通过 _delete_raw 原语统一委托
- JSONBackend.invalidate 覆盖以在非 batch 模式下 flush 落盘
- SQLiteBackend._delete_raw 用 cursor.rowcount > 0 判断是否存在
### MemoryBackend LRU 设计(P7.4
- 使用 OrderedDict 替代普通 dict
- _get_raw 命中时 move_to_end(key) 更新访问顺序
- _put_raw 超限时 popitem(last=False) 驱逐头部(最旧)
- maxsize=None 时不驱逐(默认行为,向后兼容)
## 验证结果
- ruff check: All checks passed
- ruff format: All files formatted
- pyrefly: 0 errors (67 suppressed)
- pytest: 1425 passed
- coverage: 97.56%(分支覆盖率 ≥ 95%
## 遗留事项
- P8(性能优化)尚未开始:自实现拓扑排序 / orjson 序列化 / 内存优化 / 执行器并发优化
- outputs 字段当前仅作为元数据,未来可扩展为跨任务引用(${{ needs.job.outputs.name }}
- TelegramNotifier 实际使用时需拼接 bot token URL,文档已说明
+41 -5
View File
@@ -23,8 +23,9 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **软依赖** —— `soft_depends_on` 仅用于上下文注入,不参与拓扑分层
- **并发限制** —— `concurrency_key` + `concurrency_limits` 按组限流
- **任务钩子** —— `TaskHooks`pre_run/post_run/on_failure)生命周期回调
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘`invalidate(key)` 单键失效
- **缓存键** —— `cache_key` 函数基于输入计算稳定键,使不同输入产生独立缓存
- **缓存驱逐** —— `MemoryBackend(maxsize=N)` LRU 驱逐最旧条目;TTL 过期自动忽略
- **命令任务** —— `cmd` 参数直接执行外部命令,支持列表/shell/可调用对象
- **条件执行** —— `conditions` 参数按平台、环境变量、应用安装等条件跳过任务
- **图组合** —— `compose` / `GraphComposer` 编程式展开多图字符串引用
@@ -33,11 +34,11 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **CLI 运行器** —— `CliRunner` 把多个图映射为命令行子命令,替代 Makefile
- **可观测** —— `on_event` 回调(RUNNING/SUCCESS/FAILED/SKIPPED)、`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
- **进度监控** —— `run(progress=True)` 开箱即用的 rich 进度条;`ProgressCallback` 协议供程序化集成
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信 webhook 与 Python 回调,全生命周期事件可配置
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信/Slack/Discord/Telegram webhook 与 Python 回调,全生命周期事件可配置
- **流式获取** —— `run_iter()` 生成器逐个 yield 任务结果,适用于大量任务逐个处理
- **任务取消** —— `CancelToken` 线程安全取消令牌,支持外部取消与 `KeyboardInterrupt` 优雅停止
- **子图执行** —— `run(only=[...], tags=[...])` 按名称或标签运行图的子集,自动包含传递依赖
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` 多格式导出运行结果,`from_json()` 支持反序列化重建
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` / `to_html()` 多格式导出运行结果,`from_json()` 支持反序列化重建;HTML 报告自包含内联 CSS,适合浏览器直接打开或邮件分享
- **CLI 可视化** —— `pf graph <yaml>` 终端 DAG 渲染(ASCII 分层树按标签着色 / Mermaid / 依赖表),`pf yamlrun --progress` rich 进度条,`pf info [tool]` 工具详情,`pf completion bash|zsh|fish` shell 补全
- **失败诊断** —— `RunReport.diagnose()` 自动分析根因、依赖链回溯、相似失败聚类与可操作建议;CLI 失败时自动打印诊断摘要
- **可观测性** —— `run_id` 贯穿日志/序列化/诊断;结构化日志 `extra``task_name`/`status`/`attempts`/`error_type``profile(graph)` 离线性能分析(关键路径/并行度/瓶颈)
@@ -159,7 +160,7 @@ resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
### YAML 任务编排
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`,以及 `matrix.include`/`exclude` 过滤、条件依赖(`needs: [{job: ..., if: ...}]`)、`outputs` 声明
```yaml
# pipeline.yaml
@@ -179,7 +180,22 @@ jobs:
strategy:
matrix:
version: ["3.10", "3.11", "3.12"]
os: ["linux", "macos"]
exclude:
- version: "3.10"
os: "macos"
include:
- version: "3.13"
os: "linux"
if: "env.CI"
outputs:
version: "${{ matrix.version }}"
deploy:
needs:
- job: build
if: "env.BRANCH == 'main'"
cmd: ["echo", "deploy"]
test:
needs: [build]
@@ -213,7 +229,11 @@ pf yamlrun pipeline.yaml --strategy sequential # 覆盖策略
| `needs` | `depends_on` | 矩阵 job 依赖自动展开为所有变体 |
| `if` | `conditions` | `success()`/`always()`/`env.VAR`/`env.VAR == 'x'`/`env.VAR != 'x'` |
| `strategy.matrix` | 矩阵扇出 | 笛卡尔积,命名 `{job}_{key}-{value}` |
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env` 中替换 |
| `strategy.matrix.include` | 矩阵追加 | 额外组合(可引入新键) |
| `strategy.matrix.exclude` | 矩阵剔除 | 匹配所有 key-value 的组合被移除 |
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env`/`outputs` 中替换 |
| `outputs` | `outputs` | 任务声明的输出键值映射(元数据,不参与执行) |
| `needs: [{job: ..., if: ...}]` | 条件依赖 | `if` 不满足时跳过该依赖 |
| `continue-on-error` | `continue_on_error` | 字段名支持 hyphen/underscore |
| `runs-on` | `tags` | 追加到 tags 元组 |
| `defaults` | `GraphDefaults` | 图级默认值(retry/timeout/env/cwd 等) |
@@ -409,6 +429,22 @@ px.TaskSpec(
)
```
**LRU 驱逐**`MemoryBackend(maxsize=N)` 限制最大条目数,超限时按 LRU(最近最少使用)
策略驱逐最旧条目。`get`/`has` 命中会将条目移到最近使用位置:
```python
from pyflowx import MemoryBackend
backend = MemoryBackend(maxsize=100) # 最多 100 条,超出驱逐最旧
```
**手动失效**`invalidate(key)` 删除单个键的缓存,`clear()` 清除全部:
```python
backend.invalidate("fetch_user") # 删除单个键,返回是否已删除
backend.clear() # 清除全部
```
## 错误处理
所有错误都是 `PyFlowXError` 的子类:
+6
View File
@@ -80,9 +80,12 @@ from .notification import (
ALL_LEVELS,
CallbackNotifier,
DingTalkNotifier,
DiscordNotifier,
FeishuNotifier,
NotificationLevel,
Notifier,
SlackNotifier,
TelegramNotifier,
WebhookNotifier,
WeChatNotifier,
)
@@ -128,6 +131,7 @@ __all__ = [
"DependencyChain",
"DiagnosticReport",
"DingTalkNotifier",
"DiscordNotifier",
"DuplicateTaskError",
"FailureCluster",
"FeishuNotifier",
@@ -147,6 +151,7 @@ __all__ = [
"RichProgressMonitor",
"RunReport",
"SQLiteBackend",
"SlackNotifier",
"StateBackend",
"StorageError",
"Strategy",
@@ -159,6 +164,7 @@ __all__ = [
"TaskSpec",
"TaskStatus",
"TaskTimeoutError",
"TelegramNotifier",
"ToolSpec",
"WeChatNotifier",
"WebhookNotifier",
+60
View File
@@ -40,9 +40,12 @@ __all__ = [
"ALL_LEVELS",
"CallbackNotifier",
"DingTalkNotifier",
"DiscordNotifier",
"FeishuNotifier",
"NotificationLevel",
"Notifier",
"SlackNotifier",
"TelegramNotifier",
"WeChatNotifier",
"WebhookNotifier",
]
@@ -258,3 +261,60 @@ class WeChatNotifier(WebhookNotifier):
"msgtype": "text",
"text": {"content": _format_event_message(event)},
}
class SlackNotifier(WebhookNotifier):
"""Slack Incoming Webhook 通知器。
消息格式:``{"text": ...}``。Slack webhook 仅需 ``text`` 字段,
与基类格式一致,独立成类便于类型区分与后续扩展(如 channel/username)。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {"text": _format_event_message(event)}
class DiscordNotifier(WebhookNotifier):
"""Discord Webhook 通知器。
消息格式:``{"content": ...}``。Discord webhook 使用 ``content`` 字段
承载消息文本。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {"content": _format_event_message(event)}
class TelegramNotifier(WebhookNotifier):
"""Telegram Bot 通知器。
通过 ``sendMessage`` API 发送消息。需要在 ``__init__`` 中传入
``chat_id``(聊天/群组 ID),并使用 bot token 拼接 webhook URL
``https://api.telegram.org/bot<TOKEN>/sendMessage``。
消息格式:``{"chat_id": ..., "text": ...}``。
Parameters
----------
chat_id:
接收消息的聊天或群组 ID(数字字符串或 @channelusername)。
"""
def __init__(
self,
url: str,
chat_id: str,
levels: Iterable[NotificationLevel] = ALL_LEVELS,
timeout: float = 10.0,
) -> None:
super().__init__(url=url, levels=levels, timeout=timeout)
self._chat_id = chat_id
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {
"chat_id": self._chat_id,
"text": _format_event_message(event),
}
+129
View File
@@ -7,6 +7,7 @@
from __future__ import annotations
import csv
import html
import io
import json
from collections.abc import Callable, Iterator
@@ -215,6 +216,7 @@ class RunReport:
"reason": r.reason,
"tags": list(r.spec.tags),
"depends_on": list(r.spec.depends_on),
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
}
)
return {
@@ -287,6 +289,132 @@ class RunReport:
writer.writerow(row)
return buf.getvalue()
def to_html(
self,
include_value: bool = True,
value_serializer: Callable[[Any], Any] | None = None,
) -> str:
"""导出为自包含的 HTML 报告字符串。
生成完整的 HTML5 文档,内联 CSS(零外部依赖),适合直接在浏览器
打开或邮件附件分享。所有用户内容(任务名、错误、原因、值)均经
:func:`html.escape` 转义,避免 XSS。
参数
----
include_value:
是否在表格中展示任务返回值。值可能较大或含特殊字符,
不需要时设为 ``False`` 得到更紧凑报告。
value_serializer:
自定义任务值序列化函数,透传给 :func:`_serialize_value`。
``None`` 时尝试 JSON 兼容转换,失败回退到 ``repr()``。
"""
summary = self.summary()
status_badge_class = {
TaskStatus.SUCCESS: "status-success",
TaskStatus.FAILED: "status-failed",
TaskStatus.SKIPPED: "status-skipped",
TaskStatus.RUNNING: "status-running",
TaskStatus.PENDING: "status-pending",
}
parts: list[str] = [
"<!DOCTYPE html>",
'<html lang="zh-CN">',
"<head>",
'<meta charset="utf-8">',
'<meta name="viewport" content="width=device-width, initial-scale=1">',
f"<title>PyFlowX 运行报告 {html.escape(self.run_id)}</title>",
"<style>",
"body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;"
"margin:0;padding:20px;background:#f5f5f5;color:#222;}",
".container{max-width:1100px;margin:0 auto;}",
".summary{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));"
"gap:12px;margin-bottom:24px;}",
".card{background:#fff;border-radius:8px;padding:16px;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
".card h3{margin:0 0 8px;font-size:13px;color:#666;font-weight:600;text-transform:uppercase;}",
".card .value{font-size:22px;font-weight:700;color:#222;}",
".card.success .value{color:#16a34a;}",
".card.failure .value{color:#dc2626;}",
"table{width:100%;border-collapse:collapse;background:#fff;border-radius:8px;"
"overflow:hidden;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
"th,td{padding:10px 12px;text-align:left;border-bottom:1px solid #eee;font-size:13px;}",
"th{background:#fafafa;font-weight:600;color:#555;}",
"tr:last-child td{border-bottom:none;}",
"td.name{font-family:monospace;font-weight:600;}",
".badge{display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;}",
".status-success{background:#dcfce7;color:#16a34a;}",
".status-failed{background:#fee2e2;color:#dc2626;}",
".status-skipped{background:#fef9c3;color:#ca8a04;}",
".status-running{background:#dbeafe;color:#2563eb;}",
".status-pending{background:#f3f4f6;color:#6b7280;}",
".error{color:#dc2626;font-family:monospace;font-size:12px;}",
".reason{color:#ca8a04;font-size:12px;}",
".value-cell{font-family:monospace;font-size:12px;max-width:300px;"
"overflow:hidden;text-overflow:ellipsis;white-space:nowrap;}",
".empty{padding:32px;text-align:center;color:#999;}",
"</style>",
"</head>",
"<body>",
'<div class="container">',
f"<h1>PyFlowX 运行报告 <code>{html.escape(self.run_id)}</code></h1>",
'<div class="summary">',
self._render_summary_card(
"运行状态", "成功" if self.success else "失败", "success" if self.success else "failure"
),
self._render_summary_card("任务总数", str(summary["total_tasks"])),
self._render_summary_card("总耗时", f"{summary['total_duration_seconds']:.3f}s"),
self._render_summary_card("成功", str(summary["by_status"].get("success", 0))),
self._render_summary_card("失败", str(summary["by_status"].get("failed", 0))),
self._render_summary_card("跳过", str(summary["by_status"].get("skipped", 0))),
"</div>", # .summary
]
if not self.results:
parts.append('<div class="empty">本次运行无任务结果。</div>')
else:
header_cells = [
"<th>任务名</th>",
"<th>状态</th>",
"<th>尝试次数</th>",
"<th>耗时(s)</th>",
"<th>开始时间</th>",
"<th>结束时间</th>",
"<th>错误</th>",
"<th>原因</th>",
]
if include_value:
header_cells.append("<th>返回值</th>")
parts.append("<table>")
parts.append("<thead><tr>" + "".join(header_cells) + "</tr></thead>")
parts.append("<tbody>")
for name, r in self.results.items():
badge_cls = status_badge_class.get(r.status, "status-pending")
cells = [
f'<td class="name">{html.escape(name)}</td>',
f'<td><span class="badge {badge_cls}">{html.escape(r.status.value)}</span></td>',
f"<td>{r.attempts}</td>",
f"<td>{r.duration:.6f}</td>" if r.duration is not None else "<td>-</td>",
f"<td>{html.escape(r.started_at.isoformat())}</td>" if r.started_at else "<td>-</td>",
f"<td>{html.escape(r.finished_at.isoformat())}</td>" if r.finished_at else "<td>-</td>",
f'<td class="error">{html.escape(repr(r.error))}</td>' if r.error else "<td>-</td>",
f'<td class="reason">{html.escape(r.reason)}</td>' if r.reason else "<td>-</td>",
]
if include_value:
val = _serialize_value(r.value, value_serializer)
val_str = json.dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
cells.append(f'<td class="value-cell">{html.escape(val_str)}</td>')
parts.append("<tr>" + "".join(cells) + "</tr>")
parts.append("</tbody></table>")
parts.extend(["</div>", "</body>", "</html>"])
return "\n".join(parts)
@staticmethod
def _render_summary_card(label: str, value: str, modifier: str = "") -> str:
"""渲染单个汇总卡片 HTML 片段。"""
cls = f"card {modifier}" if modifier else "card"
return f'<div class="{cls}"><h3>{html.escape(label)}</h3><div class="value">{html.escape(value)}</div></div>'
@classmethod
def from_json(cls, text: str) -> RunReport:
"""从 JSON 字符串重建 :class:`RunReport`。
@@ -308,6 +436,7 @@ class RunReport:
fn=_noop_fn,
tags=tuple(entry.get("tags", ())),
depends_on=tuple(entry.get("depends_on", ())),
outputs=dict(entry["outputs"]) if entry.get("outputs") else None,
)
started = entry.get("started_at")
finished = entry.get("finished_at")
+73 -3
View File
@@ -19,6 +19,7 @@ import sys
import threading
import time
from abc import ABC, abstractmethod
from collections import OrderedDict
from collections.abc import Iterator, Mapping
from contextlib import AbstractContextManager, contextmanager, nullcontext
from pathlib import Path
@@ -58,6 +59,14 @@ class StateBackend(ABC):
def clear(self) -> None:
"""清除所有存储状态。"""
@abstractmethod
def invalidate(self, key: str) -> bool:
"""删除单个键的存储结果。
返回 ``True`` 表示键存在且已删除,``False`` 表示键不存在。
与 :meth:`clear` 不同,仅删除指定键而非全部。
"""
def flush(self) -> None: # noqa: B027
"""将内存中暂存的状态持久化到外部介质。
@@ -106,6 +115,10 @@ class _TTLStateBackendMixin(StateBackend):
def _clear_raw(self) -> None:
"""清空所有记录。"""
@abstractmethod
def _delete_raw(self, key: str) -> bool:
"""删除单条记录。返回 ``True`` 表示存在且已删除,``False`` 表示不存在。"""
# ---- 共享实现 ---- #
def _now(self) -> float:
"""当前时间戳,默认为 wall-clock 秒。"""
@@ -141,6 +154,10 @@ class _TTLStateBackendMixin(StateBackend):
def clear(self) -> None:
self._clear_raw()
@override
def invalidate(self, key: str) -> bool:
return self._delete_raw(key)
class MemoryBackend(_TTLStateBackendMixin):
"""进程内 dict 后端。进程退出即丢失。
@@ -150,11 +167,20 @@ class MemoryBackend(_TTLStateBackendMixin):
ttl:
条目存活秒数。``None`` 表示永不过期。``has`` 在条目超过 ttl 后
返回 ``False``(但不主动删除,下次 ``save`` 覆盖)。
maxsize:
最大条目数。``None`` 表示不限。超出时按 LRU(最近最少使用)
策略驱逐最旧条目。每次 ``get``/``has`` 命中会将条目移到末尾
(最近使用),``save`` 新增条目时若超限则驱逐头部条目。
"""
def __init__(self, ttl: float | None = None) -> None:
self._store: dict[str, tuple[Any, float]] = {}
def __init__(
self,
ttl: float | None = None,
maxsize: int | None = None,
) -> None:
self._store: OrderedDict[str, tuple[Any, float]] = OrderedDict()
self._ttl = ttl
self._maxsize = maxsize
@override
def _now(self) -> float:
@@ -162,11 +188,23 @@ class MemoryBackend(_TTLStateBackendMixin):
@override
def _get_raw(self, key: str) -> tuple[Any, float] | None:
return self._store.get(key)
entry = self._store.get(key)
if entry is None:
return None
# LRU:命中时移到末尾(最近使用)
self._store.move_to_end(key)
return entry
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
if key in self._store:
# 已存在:更新并移到末尾
self._store.move_to_end(key)
self._store[key] = (value, ts)
# LRU 驱逐:超限时移除头部(最旧)
if self._maxsize is not None:
while len(self._store) > self._maxsize:
self._store.popitem(last=False)
@override
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
@@ -177,6 +215,13 @@ class MemoryBackend(_TTLStateBackendMixin):
def _clear_raw(self) -> None:
self._store.clear()
@override
def _delete_raw(self, key: str) -> bool:
if key in self._store:
del self._store[key]
return True
return False
class JSONBackend(_TTLStateBackendMixin):
"""基于文件的 JSON 存储,用于跨进程续跑。
@@ -245,11 +290,25 @@ class JSONBackend(_TTLStateBackendMixin):
def _clear_raw(self) -> None:
self._store.clear()
@override
def _delete_raw(self, key: str) -> bool:
if key in self._store:
del self._store[key]
return True
return False
@override
def clear(self) -> None:
super().clear()
self._flush()
@override
def invalidate(self, key: str) -> bool:
removed = super().invalidate(key)
if removed and not self._defer_flush:
self._flush()
return removed
@override
def save(self, key: str, value: Any) -> None:
# batch 模式下延迟序列化验证到 flush 时(一次性 json.dump 整个 _store),
@@ -382,6 +441,17 @@ class SQLiteBackend(_TTLStateBackendMixin):
except sqlite3.DatabaseError as exc:
raise StorageError("cannot clear sqlite state", exc) from exc
@override
def _delete_raw(self, key: str) -> bool:
with self._lock:
try:
cursor = self._conn.execute("DELETE FROM state WHERE key = ?", (key,))
if not self._in_batch:
self._conn.commit()
return cursor.rowcount > 0
except sqlite3.DatabaseError as exc:
raise StorageError(f"cannot delete key {key!r} from sqlite state", exc) from exc
@override
def flush(self) -> None:
with self._lock:
+6
View File
@@ -245,6 +245,9 @@ class TaskSpec(Generic[T]):
同步任务的执行器:``"thread"``(默认,线程池)/ ``"process"``
(进程池,绕过 GIL,适合 CPU 密集型;``fn`` 须可 pickle/
``"inline"``(直接在事件循环线程调用,最快但会阻塞循环)。
outputs:
任务声明的输出键值映射(YAML 编排场景使用)。仅作为元数据携带,
不参与执行;可通过 ``report.result_of(name).spec.outputs`` 查询。
"""
name: str
@@ -271,6 +274,7 @@ class TaskSpec(Generic[T]):
cache_key: CacheKeyFn | None = None
hooks: TaskHooks = field(default_factory=TaskHooks)
executor: str = "thread" # "thread" | "process" | "inline"
outputs: Mapping[str, str] | None = None
def __post_init__(self) -> None:
if not self.name:
@@ -468,6 +472,7 @@ def task(
continue_on_error: bool = False,
cache_key: CacheKeyFn | None = None,
hooks: TaskHooks | None = None,
outputs: Mapping[str, str] | None = None,
name: str | None = None,
) -> Any:
"""装饰器:将函数转为 :class:`TaskSpec`。
@@ -509,6 +514,7 @@ def task(
continue_on_error=continue_on_error,
cache_key=cache_key,
hooks=hooks if hooks is not None else TaskHooks(),
outputs=dict(outputs) if outputs else None,
)
if fn is None and cmd is None:
+71 -9
View File
@@ -224,11 +224,25 @@ def _extract_matrix(job_data: Mapping[str, Any]) -> Mapping[str, list[Any]] | No
def _cartesian_product(matrix_data: Mapping[str, list[Any]]) -> list[dict[str, str]]:
"""计算矩阵的笛卡尔积。"""
keys = list(matrix_data.keys())
result: list[dict[str, str]] = [{}]
"""计算矩阵的笛卡尔积。
支持 ``include``/``exclude`` 特殊键(GitHub Actions 语义):
``exclude`` 从笛卡尔积中剔除所有匹配的组合;
``include`` 追加额外组合(可引入新键)。
无基础矩阵键且仅有 include 时,不产生空组合(``{}``),
仅返回 include 项。
"""
# 分离 include/exclude 与普通矩阵键
matrix_keys = {k: v for k, v in matrix_data.items() if k not in ("include", "exclude")}
include_list = matrix_data.get("include")
exclude_list = matrix_data.get("exclude")
# 基础笛卡尔积:无矩阵键时为空列表(不产生 [{}])
keys = list(matrix_keys.keys())
result: list[dict[str, str]] = [{}] if keys else []
for key in keys:
values = matrix_data[key]
values = matrix_keys[key]
if not isinstance(values, list):
raise ValueError(f"matrix.{key} 必须是列表,收到: {type(values).__name__}")
new_result: list[dict[str, str]] = []
@@ -238,9 +252,32 @@ def _cartesian_product(matrix_data: Mapping[str, list[Any]]) -> list[dict[str, s
combo[key] = str(value)
new_result.append(combo)
result = new_result
# exclude:剔除所有匹配的组合(组合的所有 key-value 对都匹配则剔除)
if exclude_list is not None:
if not isinstance(exclude_list, list):
raise ValueError(f"matrix.exclude 必须是列表,收到: {type(exclude_list).__name__}")
excludes = [_normalize_combo(item) for item in exclude_list]
result = [c for c in result if not any(c.items() >= ex.items() for ex in excludes)]
# include:追加额外组合(可引入基础矩阵之外的键)
if include_list is not None:
if not isinstance(include_list, list):
raise ValueError(f"matrix.include 必须是列表,收到: {type(include_list).__name__}")
for item in include_list:
combo = _normalize_combo(item)
result.append(combo)
return result
def _normalize_combo(item: Any) -> dict[str, str]:
"""将 include/exclude 项归一化为 {str: str} 字典。"""
if not isinstance(item, Mapping):
raise ValueError(f"matrix include/exclude 项必须是映射,收到: {type(item).__name__}")
return {str(k): str(v) for k, v in item.items()}
def _build_specs(
jobs: Mapping[str, Any],
expansions: dict[str, list[tuple[str, dict[str, str]]]],
@@ -327,6 +364,12 @@ def _parse_optional_fields(job_data: Mapping[str, Any], matrix_values: dict[str,
if tag_list:
kwargs["tags"] = tuple(tag_list)
outputs = _get_field(job_data, "outputs")
if outputs is not None:
if not isinstance(outputs, Mapping):
raise ValueError(f"outputs 必须是映射,收到: {type(outputs).__name__}")
kwargs["outputs"] = {str(k): _substitute_matrix_str(str(v), matrix_values) for k, v in outputs.items()}
return kwargs
@@ -334,17 +377,36 @@ def _expand_needs(
needs: Any,
expansions: dict[str, list[tuple[str, dict[str, str]]]],
) -> list[str]:
"""展开 needs 列表:矩阵 job 依赖展开为所有变体。"""
"""展开 needs 列表:矩阵 job 依赖展开为所有变体。
支持两种格式:
* 字符串:``needs: [build]``
* 条件依赖对象:``needs: [{job: build, if: "env.CI == 'true'"}]``
``if`` 条件不满足时跳过该依赖。
"""
if not needs:
return []
if isinstance(needs, str):
needs = [needs]
result: list[str] = []
for dep in needs:
if dep in expansions:
result.extend(name for name, _ in expansions[dep])
for entry in needs:
if isinstance(entry, Mapping):
job_name = entry.get("job")
if not job_name or not isinstance(job_name, str):
raise ValueError(f"needs 条件依赖项缺少 'job' 字段: {entry!r}")
if_expr = entry.get("if")
if if_expr is not None:
conditions = _parse_condition(if_expr)
# 任一条件不满足则跳过该依赖
if not all(cond({}) for cond in conditions):
continue
dep_name = job_name
else:
result.append(dep)
dep_name = entry
if dep_name in expansions:
result.extend(name for name, _ in expansions[dep_name])
else:
result.append(dep_name)
return result
+11 -9
View File
@@ -109,15 +109,17 @@ class TestPfYamlrunListFiltering:
) -> None:
"""--list-tag 与 --list-name 组合过滤(交集)。"""
yaml_file = _write_yaml(tmp_path, _TAGGED_YAML)
app = PfApp([
"yamlrun",
str(yaml_file),
"--list",
"--list-tag",
"ingest",
"--list-name",
"b",
])
app = PfApp(
[
"yamlrun",
str(yaml_file),
"--list",
"--list-tag",
"ingest",
"--list-name",
"b",
]
)
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
+6 -4
View File
@@ -507,10 +507,12 @@ class TestRunReportIntegration:
return 1
# continue_on_error=True 使失败不抛异常而写入报告(success 保持 True
graph = px.Graph.from_specs([
px.TaskSpec("boom", boom, continue_on_error=True),
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
])
graph = px.Graph.from_specs(
[
px.TaskSpec("boom", boom, continue_on_error=True),
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
]
)
report = px.run(graph, strategy="sequential")
# continue_on_error 不影响 success 标志,手动标记以触发诊断
report.success = False
+141
View File
@@ -12,9 +12,12 @@ from pyflowx.notification import (
ALL_LEVELS,
CallbackNotifier,
DingTalkNotifier,
DiscordNotifier,
FeishuNotifier,
NotificationLevel,
Notifier,
SlackNotifier,
TelegramNotifier,
WebhookNotifier,
WeChatNotifier,
_format_event_message,
@@ -146,6 +149,24 @@ def test_wechat_notifier_satisfies_protocol() -> None:
assert isinstance(WeChatNotifier(url="http://example.com/hook"), Notifier)
def test_slack_notifier_satisfies_protocol() -> None:
"""SlackNotifier 满足 Notifier 协议。"""
assert isinstance(SlackNotifier(url="http://example.com/hook"), Notifier)
def test_discord_notifier_satisfies_protocol() -> None:
"""DiscordNotifier 满足 Notifier 协议。"""
assert isinstance(DiscordNotifier(url="http://example.com/hook"), Notifier)
def test_telegram_notifier_satisfies_protocol() -> None:
"""TelegramNotifier 满足 Notifier 协议。"""
assert isinstance(
TelegramNotifier(url="http://example.com/hook", chat_id="123456"),
Notifier,
)
# ---------------------------------------------------------------------- #
# CallbackNotifier
# ---------------------------------------------------------------------- #
@@ -380,6 +401,123 @@ def test_wechat_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None
assert "跳过原因" in payload["text"]["content"]
# ---------------------------------------------------------------------- #
# SlackNotifier
# ---------------------------------------------------------------------- #
def test_slack_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""Slack 消息格式:text 字段。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = SlackNotifier(url="http://example.com/hook")
notifier.notify(_event(TaskStatus.FAILED, error="boom", duration=0.3))
fields = _get_req_fields(capture["req"])
payload = json.loads(fields["data"]) # type: ignore[arg-type]
assert "text" in payload
assert "test_task" in payload["text"]
assert "boom" in payload["text"]
def test_slack_notifier_levels_filter(monkeypatch: pytest.MonkeyPatch) -> None:
"""Slack levels 过滤:SUCCESS 不在 levels 中时不发送。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = SlackNotifier(
url="http://example.com/hook",
levels=(NotificationLevel.FAILED,),
)
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.1))
assert "req" not in capture
notifier.notify(_event(TaskStatus.FAILED, error="err"))
assert "req" in capture
# ---------------------------------------------------------------------- #
# DiscordNotifier
# ---------------------------------------------------------------------- #
def test_discord_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""Discord 消息格式:content 字段。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = DiscordNotifier(url="http://example.com/hook")
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.5))
fields = _get_req_fields(capture["req"])
payload = json.loads(fields["data"]) # type: ignore[arg-type]
assert "content" in payload
assert "test_task" in payload["content"]
assert "success" in payload["content"]
def test_discord_notifier_timeout_passed(monkeypatch: pytest.MonkeyPatch) -> None:
"""Discord timeout 参数透传到 urlopen。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = DiscordNotifier(url="http://example.com/hook", timeout=8.0)
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.1))
assert capture["timeout"] == 8.0
# ---------------------------------------------------------------------- #
# TelegramNotifier
# ---------------------------------------------------------------------- #
def test_telegram_notifier_payload_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""Telegram 消息格式:chat_id + text 字段。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="@mychannel")
notifier.notify(_event(TaskStatus.FAILED, error="网络异常"))
fields = _get_req_fields(capture["req"])
payload = json.loads(fields["data"]) # type: ignore[arg-type]
assert payload["chat_id"] == "@mychannel"
assert "text" in payload
assert "test_task" in payload["text"]
assert "网络异常" in payload["text"]
def test_telegram_notifier_chat_id_numeric(monkeypatch: pytest.MonkeyPatch) -> None:
"""Telegram 数字 chat_id 正确传递。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="987654321")
notifier.notify(_event(TaskStatus.SUCCESS, duration=0.2))
fields = _get_req_fields(capture["req"])
payload = json.loads(fields["data"]) # type: ignore[arg-type]
assert payload["chat_id"] == "987654321"
def test_telegram_notifier_levels_filter(monkeypatch: pytest.MonkeyPatch) -> None:
"""Telegram levels 过滤:SKIPPED 不在 levels 中时不发送。"""
capture: dict[str, object] = {}
_patch_urlopen(monkeypatch, capture)
notifier = TelegramNotifier(
url="http://example.com/hook",
chat_id="123",
levels=(NotificationLevel.FAILED,),
)
notifier.notify(_event(TaskStatus.SKIPPED, reason="跳过"))
assert "req" not in capture
notifier.notify(_event(TaskStatus.FAILED, error="err"))
assert "req" in capture
def test_telegram_notifier_close_is_noop() -> None:
"""TelegramNotifier.close() 无副作用。"""
notifier = TelegramNotifier(url="http://example.com/hook", chat_id="123")
notifier.close()
# ---------------------------------------------------------------------- #
# run() 集成:notifiers 参数
# ---------------------------------------------------------------------- #
@@ -554,6 +692,9 @@ def test_notification_classes_exported() -> None:
assert hasattr(px, "FeishuNotifier")
assert hasattr(px, "DingTalkNotifier")
assert hasattr(px, "WeChatNotifier")
assert hasattr(px, "SlackNotifier")
assert hasattr(px, "DiscordNotifier")
assert hasattr(px, "TelegramNotifier")
assert hasattr(px, "ALL_LEVELS")
+178 -8
View File
@@ -395,6 +395,172 @@ class TestRunReportToCsv:
assert "PosixPath('/tmp/x')" in s
class TestRunReportToHtml:
"""测试 RunReport.to_html."""
def test_to_html_basic_structure(self) -> None:
"""应输出完整 HTML5 文档结构。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=42)
s = report.to_html()
assert s.startswith("<!DOCTYPE html>")
assert s.endswith("</html>")
assert "<html" in s
assert "<head>" in s
assert "<body>" in s
def test_to_html_contains_run_id(self) -> None:
"""HTML 应在标题与正文包含 run_id。"""
report = px.RunReport(run_id="abcd1234")
report.results["a"] = _make_result("a")
s = report.to_html()
assert "abcd1234" in s
def test_to_html_contains_task_name(self) -> None:
"""HTML 表格应包含任务名。"""
report = px.RunReport()
report.results["extract"] = _make_result("extract", value=1)
s = report.to_html()
assert "extract" in s
assert "<td" in s
def test_to_html_status_badge_class(self) -> None:
"""不同状态应渲染对应 CSS 类。"""
report = px.RunReport()
report.results["ok"] = _make_result("ok", status=TaskStatus.SUCCESS)
report.results["bad"] = _make_result("bad", status=TaskStatus.FAILED, error=ValueError("x"))
report.results["skip"] = _make_result("skip", status=TaskStatus.SKIPPED, duration=0.0, reason="r")
s = report.to_html()
assert "status-success" in s
assert "status-failed" in s
assert "status-skipped" in s
def test_to_html_xss_escape_task_name(self) -> None:
"""任务名含 HTML 特殊字符应被转义。"""
report = px.RunReport()
malicious = "<script>evil()</script>"
spec: TaskSpec[Any] = TaskSpec[Any](malicious, _fn)
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus.SUCCESS,
value=1,
attempts=1,
)
report.results[malicious] = result
s = report.to_html()
assert "<script>" not in s
assert "&lt;script&gt;" in s
def test_to_html_xss_escape_error(self) -> None:
"""错误信息含 HTML 特殊字符应被转义。"""
report = px.RunReport()
err = ValueError("<img src=x onerror=alert(1)>")
report.results["a"] = _make_result("a", status=TaskStatus.FAILED, error=err)
s = report.to_html()
assert "<img" not in s
assert "&lt;img" in s
def test_to_html_xss_escape_reason(self) -> None:
"""跳过原因含 HTML 特殊字符应被转义。"""
report = px.RunReport()
report.results["a"] = _make_result(
"a",
status=TaskStatus.SKIPPED,
duration=0.0,
reason="<b>bold</b>",
)
s = report.to_html()
assert "<b>bold</b>" not in s
assert "&lt;b&gt;" in s
def test_to_html_without_value(self) -> None:
"""include_value=False 应省略返回值列。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value="secret")
s = report.to_html(include_value=False)
assert "secret" not in s
assert "返回值" not in s
def test_to_html_with_value(self) -> None:
"""include_value=True 应包含返回值(JSON 序列化形式)。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=[1, 2, 3])
s = report.to_html()
assert "[1, 2, 3]" in s
def test_to_html_with_value_serializer(self) -> None:
"""to_html 应透传 value_serializer。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
s = report.to_html(value_serializer=str)
assert "/tmp/x" in s
assert "PosixPath" not in s
def test_to_html_empty_report(self) -> None:
"""空报告应渲染提示文本而非空表格。"""
report = px.RunReport()
s = report.to_html()
assert "无任务结果" in s
assert "<table>" not in s
def test_to_html_failure_status_indicator(self) -> None:
"""失败报告应在汇总卡片显示失败样式。"""
report = px.RunReport(success=False)
report.results["a"] = _make_result("a", status=TaskStatus.FAILED, error=RuntimeError("x"))
s = report.to_html()
assert "card failure" in s
assert "失败" in s
def test_to_html_success_status_indicator(self) -> None:
"""成功报告应在汇总卡片显示成功样式。"""
report = px.RunReport(success=True)
report.results["a"] = _make_result("a", status=TaskStatus.SUCCESS)
s = report.to_html()
assert "card success" in s
assert "成功" in s
def test_to_html_summary_cards_content(self) -> None:
"""汇总卡片应包含任务总数/耗时/各状态计数。"""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.SUCCESS, duration=0.5)
report.results["b"] = _make_result("b", status=TaskStatus.FAILED, error=ValueError("x"), duration=0.3)
s = report.to_html()
assert "任务总数" in s
assert "总耗时" in s
assert "成功" in s
assert "失败" in s
assert "跳过" in s
assert "2" in s # 任务总数
assert "0.800" in s # 0.5+0.3
def test_to_html_no_timestamps(self) -> None:
"""无时间戳时应渲染占位符。"""
report = px.RunReport()
# 构造无时间戳的结果
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn)
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus.SUCCESS,
value=1,
attempts=1,
started_at=None,
finished_at=None,
)
report.results["a"] = result
s = report.to_html()
# 无时间戳时表格单元格显示 "-"
assert s.count("<td>-</td>") >= 2
def test_to_html_css_inlined(self) -> None:
"""CSS 应内联在 <style> 标签中,无外部依赖。"""
report = px.RunReport()
s = report.to_html()
assert "<style>" in s
assert "</style>" in s
assert "stylesheet" not in s.lower() # 无外部样式表引用
assert "link rel" not in s.lower()
class TestRunReportFromJson:
"""测试 RunReport.from_json."""
@@ -485,10 +651,12 @@ class TestRunReportSerializationIntegration:
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
graph = px.Graph.from_specs([
px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
])
graph = px.Graph.from_specs(
[
px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
]
)
report = px.run(graph, strategy="sequential")
# 序列化 → 反序列化
@@ -584,10 +752,12 @@ class TestRunId:
def task_b(a: int) -> int:
return a * 2
graph = px.Graph.from_specs([
px.TaskSpec("a", task_a),
px.TaskSpec("b", task_b, depends_on=("a",)),
])
graph = px.Graph.from_specs(
[
px.TaskSpec("a", task_a),
px.TaskSpec("b", task_b, depends_on=("a",)),
]
)
report = px.run(graph, strategy="sequential")
profile = report.profile(graph)
assert profile.total_duration >= 0
+182
View File
@@ -591,3 +591,185 @@ def test_state_backend_is_abstract() -> None:
"""StateBackend 是 ABC,不能直接实例化。"""
with pytest.raises(TypeError):
StateBackend() # type: ignore[abstract]
# ---------------------------------------------------------------------- #
# invalidate(手动失效)
# ---------------------------------------------------------------------- #
def test_memory_backend_invalidate_existing_key() -> None:
"""MemoryBackend.invalidate 删除已存在键返回 True。"""
b = MemoryBackend()
b.save("a", 1)
assert b.invalidate("a") is True
assert not b.has("a")
def test_memory_backend_invalidate_missing_key() -> None:
"""MemoryBackend.invalidate 删除不存在键返回 False。"""
b = MemoryBackend()
assert b.invalidate("missing") is False
def test_memory_backend_invalidate_then_save() -> None:
"""invalidate 后可重新 save。"""
b = MemoryBackend()
b.save("a", 1)
b.invalidate("a")
b.save("a", 2)
assert b.get("a") == 2
def test_json_backend_invalidate_existing_key(tmp_path: Path) -> None:
"""JSONBackend.invalidate 删除并落盘。"""
path = tmp_path / "state.json"
b = JSONBackend(str(path))
b.save("a", 1)
b.save("b", 2)
assert b.invalidate("a") is True
assert not b.has("a")
assert b.has("b")
# 重新加载验证落盘
b2 = JSONBackend(str(path))
assert not b2.has("a")
assert b2.has("b")
def test_json_backend_invalidate_missing_key(tmp_path: Path) -> None:
"""JSONBackend.invalidate 不存在键返回 False。"""
b = JSONBackend(str(tmp_path / "state.json"))
assert b.invalidate("missing") is False
def test_json_backend_invalidate_in_batch_no_flush(tmp_path: Path) -> None:
"""batch 模式下 invalidate 不立即落盘。"""
path = tmp_path / "state.json"
b = JSONBackend(str(path))
b.save("a", 1)
with b.batch():
b.invalidate("a")
# batch 内不落盘,重新加载仍应有 a
b2 = JSONBackend(str(path))
assert b2.has("a")
# batch 退出后落盘
b2 = JSONBackend(str(path))
assert not b2.has("a")
def test_sqlite_backend_invalidate_existing_key(tmp_path: Path) -> None:
"""SQLiteBackend.invalidate 删除已存在键。"""
path = tmp_path / "state.db"
b = SQLiteBackend(str(path))
b.save("a", 1)
assert b.invalidate("a") is True
assert not b.has("a")
def test_sqlite_backend_invalidate_missing_key(tmp_path: Path) -> None:
"""SQLiteBackend.invalidate 不存在键返回 False。"""
b = SQLiteBackend(str(tmp_path / "state.db"))
assert b.invalidate("missing") is False
def test_sqlite_backend_invalidate_in_batch(tmp_path: Path) -> None:
"""SQLiteBackend batch 模式下 invalidate 延迟 commit。"""
path = tmp_path / "state.db"
b = SQLiteBackend(str(path))
b.save("a", 1)
with b.batch():
b.invalidate("a")
# batch 内不 commit,新连接仍可见 a
b2 = SQLiteBackend(str(path))
assert b2.has("a")
# batch 退出后 commit,新连接不可见 a
b3 = SQLiteBackend(str(path))
assert not b3.has("a")
# ---------------------------------------------------------------------- #
# MemoryBackend LRU 驱逐
# ---------------------------------------------------------------------- #
def test_memory_backend_maxsize_evicts_oldest() -> None:
"""maxsize 超限时驱逐最旧条目。"""
b = MemoryBackend(maxsize=2)
b.save("a", 1)
b.save("b", 2)
b.save("c", 3) # 超限,驱逐 a
assert not b.has("a")
assert b.has("b")
assert b.has("c")
def test_memory_backend_maxsize_get_updates_lru_order() -> None:
"""get 命中更新 LRU 顺序,避免被驱逐。"""
b = MemoryBackend(maxsize=2)
b.save("a", 1)
b.save("b", 2)
_ = b.get("a") # a 移到末尾(最近使用)
b.save("c", 3) # 超限,驱逐 b(最旧)
assert b.has("a")
assert not b.has("b")
assert b.has("c")
def test_memory_backend_maxsize_has_updates_lru_order() -> None:
"""has 命中也更新 LRU 顺序。"""
b = MemoryBackend(maxsize=2)
b.save("a", 1)
b.save("b", 2)
_ = b.has("a") # a 移到末尾
b.save("c", 3) # 驱逐 b
assert b.has("a")
assert not b.has("b")
def test_memory_backend_maxsize_update_existing_no_evict() -> None:
"""更新已存在键不触发驱逐。"""
b = MemoryBackend(maxsize=2)
b.save("a", 1)
b.save("b", 2)
b.save("a", 10) # 更新 a,不超限
assert b.has("a")
assert b.has("b")
assert b.get("a") == 10
def test_memory_backend_maxsize_none_no_eviction() -> None:
"""maxsize=None 不驱逐。"""
b = MemoryBackend(maxsize=None)
for i in range(100):
b.save(f"key{i}", i)
assert len(b.load()) == 100
def test_memory_backend_maxsize_one() -> None:
"""maxsize=1 只保留最新。"""
b = MemoryBackend(maxsize=1)
b.save("a", 1)
b.save("b", 2) # 驱逐 a
assert not b.has("a")
assert b.has("b")
def test_memory_backend_maxsize_with_ttl() -> None:
"""maxsize 与 TTL 组合工作。"""
b = MemoryBackend(ttl=100, maxsize=2)
b.save("a", 1)
b.save("b", 2)
b.save("c", 3) # 驱逐 a
assert not b.has("a")
assert b.has("b")
assert b.has("c")
def test_memory_backend_maxsize_load_count() -> None:
"""load() 返回的条目数受 maxsize 限制。"""
b = MemoryBackend(maxsize=3)
for i in range(5):
b.save(f"k{i}", i)
data = b.load()
assert len(data) == 3 # 只保留最近 3 个
assert "k2" in data
assert "k3" in data
assert "k4" in data
assert "k0" not in data
assert "k1" not in data
+430
View File
@@ -302,6 +302,436 @@ jobs:
assert "build_version-3.11" in deploy_spec.depends_on
class TestMatrixIncludeExclude:
"""矩阵 include/exclude 过滤测试。"""
def test_exclude_removes_matching_combo(self) -> None:
"""exclude 剔除匹配的组合。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10", "3.11"]
os: ["linux", "macos"]
exclude:
- version: "3.10"
os: "macos"
""")
names = set(graph.names)
assert "test_version-3.10_os-macos" not in names
assert "test_version-3.10_os-linux" in names
assert "test_version-3.11_os-linux" in names
assert "test_version-3.11_os-macos" in names
assert len(names) == 3
def test_exclude_partial_match_keeps_combo(self) -> None:
"""exclude 仅匹配部分 key 时不剔除。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10", "3.11"]
os: ["linux", "macos"]
exclude:
- version: "3.10"
""")
# exclude {version: 3.10} 匹配两个组合,都剔除
names = set(graph.names)
assert "test_version-3.10_os-linux" not in names
assert "test_version-3.10_os-macos" not in names
assert "test_version-3.11_os-linux" in names
assert "test_version-3.11_os-macos" in names
assert len(names) == 2
def test_include_adds_extra_combo(self) -> None:
"""include 追加额外组合。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10", "3.11"]
include:
- version: "3.12"
extra: "new"
""")
names = set(graph.names)
# 基础矩阵
assert "test_version-3.10" in names
assert "test_version-3.11" in names
# include 追加(含新键 extra
assert "test_version-3.12_extra-new" in names
assert len(names) == 3
def test_include_only_no_base_matrix(self) -> None:
"""无基础矩阵时 include 仍可工作。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
include:
- version: "3.10"
- version: "3.11"
""")
names = set(graph.names)
assert "test_version-3.10" in names
assert "test_version-3.11" in names
assert len(names) == 2
def test_exclude_all_combos(self) -> None:
"""exclude 剔除所有组合时,job 不产生任务。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10", "3.11"]
exclude:
- version: "3.10"
- version: "3.11"
""")
assert len(graph.names) == 0
def test_include_exclude_combined(self) -> None:
"""include 与 exclude 组合使用。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10", "3.11"]
os: ["linux", "macos"]
exclude:
- version: "3.10"
os: "linux"
include:
- version: "3.12"
os: "linux"
""")
names = set(graph.names)
assert "test_version-3.10_os-linux" not in names # excluded
assert "test_version-3.10_os-macos" in names
assert "test_version-3.11_os-linux" in names
assert "test_version-3.11_os-macos" in names
assert "test_version-3.12_os-linux" in names # included
assert len(names) == 4
def test_exclude_not_list_raises(self) -> None:
"""exclude 非列表时抛 ValueError。"""
with pytest.raises(ValueError, match=r"matrix\.exclude 必须是列表"):
parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10"]
exclude: "not a list"
""")
def test_include_not_list_raises(self) -> None:
"""include 非列表时抛 ValueError。"""
with pytest.raises(ValueError, match=r"matrix\.include 必须是列表"):
parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10"]
include: "not a list"
""")
def test_exclude_item_not_mapping_raises(self) -> None:
"""exclude 项非映射时抛 ValueError。"""
with pytest.raises(ValueError, match="matrix include/exclude 项必须是映射"):
parse_yaml_string("""
jobs:
test:
cmd: ["echo", "test"]
strategy:
matrix:
version: ["3.10"]
exclude:
- "not a mapping"
""")
def test_include_with_matrix_placeholder_substitution(self) -> None:
"""include 组合中的值参与占位符替换。"""
graph = parse_yaml_string("""
jobs:
test:
cmd: ["python${{ matrix.version }}", "-m", "pytest"]
strategy:
matrix:
version: ["3.10"]
include:
- version: "3.12"
""")
spec = graph.spec("test_version-3.12")
assert spec.cmd == ["python3.12", "-m", "pytest"]
class TestConditionalNeeds:
"""条件依赖测试。"""
def test_conditional_need_with_env_var_exists(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""if env.CI 满足时依赖被加入。"""
monkeypatch.setenv("CI", "true")
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
deploy:
needs:
- job: build
if: "env.CI"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "build" in deploy_spec.depends_on
def test_conditional_need_skipped_when_env_missing(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""if env.CI 不满足时依赖被跳过。"""
monkeypatch.delenv("MISSING_VAR", raising=False)
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
deploy:
needs:
- job: build
if: "env.MISSING_VAR"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "build" not in deploy_spec.depends_on
def test_conditional_need_with_env_equals(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""if env.VAR == 'x' 精确匹配时依赖被加入。"""
monkeypatch.setenv("BRANCH", "main")
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
deploy:
needs:
- job: build
if: "env.BRANCH == 'main'"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "build" in deploy_spec.depends_on
def test_conditional_need_skipped_with_env_not_equals(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""if env.VAR == 'x' 不匹配时依赖被跳过。"""
monkeypatch.setenv("BRANCH", "dev")
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
deploy:
needs:
- job: build
if: "env.BRANCH == 'main'"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "build" not in deploy_spec.depends_on
def test_mixed_string_and_conditional_needs(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""字符串依赖与条件依赖混合使用。"""
monkeypatch.setenv("CI", "true")
graph = parse_yaml_string("""
jobs:
setup:
cmd: ["echo", "setup"]
build:
cmd: ["echo", "build"]
deploy:
needs:
- setup
- job: build
if: "env.CI"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "setup" in deploy_spec.depends_on
assert "build" in deploy_spec.depends_on
def test_conditional_need_missing_job_field_raises(self) -> None:
"""条件依赖项缺少 job 字段时抛 ValueError。"""
with pytest.raises(ValueError, match="缺少 'job' 字段"):
parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
deploy:
needs:
- if: "env.CI"
cmd: ["echo", "deploy"]
""")
def test_conditional_need_with_matrix_job(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""条件依赖展开矩阵 job。"""
monkeypatch.setenv("DEPLOY", "true")
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
strategy:
matrix:
version: ["3.10", "3.11"]
deploy:
needs:
- job: build
if: "env.DEPLOY"
cmd: ["echo", "deploy"]
""")
deploy_spec = graph.spec("deploy")
assert "build_version-3.10" in deploy_spec.depends_on
assert "build_version-3.11" in deploy_spec.depends_on
class TestOutputs:
"""outputs 字段测试。"""
def test_outputs_parsed_as_metadata(self) -> None:
"""outputs 作为元数据附加到 TaskSpec。"""
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
outputs:
version: "1.0.0"
artifact: "dist.tar.gz"
""")
spec = graph.spec("build")
assert spec.outputs is not None
assert spec.outputs["version"] == "1.0.0"
assert spec.outputs["artifact"] == "dist.tar.gz"
def test_outputs_none_when_not_declared(self) -> None:
"""未声明 outputs 时为 None。"""
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
""")
spec = graph.spec("build")
assert spec.outputs is None
def test_outputs_with_matrix_placeholder(self) -> None:
"""outputs 值支持 ${{ matrix.* }} 占位符。"""
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
outputs:
version: "${{ matrix.version }}"
strategy:
matrix:
version: ["3.10", "3.11"]
""")
spec_310 = graph.spec("build_version-3.10")
spec_311 = graph.spec("build_version-3.11")
assert spec_310.outputs is not None
assert spec_310.outputs["version"] == "3.10"
assert spec_311.outputs is not None
assert spec_311.outputs["version"] == "3.11"
def test_outputs_not_mapping_raises(self) -> None:
"""outputs 非映射时抛 ValueError。"""
with pytest.raises(ValueError, match="outputs 必须是映射"):
parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
outputs: "not a mapping"
""")
def test_outputs_round_trip_via_json(self) -> None:
"""outputs 通过 to_json/from_json 往返保留。"""
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
outputs:
version: "1.0.0"
""")
spec = graph.spec("build")
report = px.RunReport()
from datetime import datetime
from pyflowx.task import TaskResult
report.results["build"] = TaskResult(
spec=spec,
status=TaskStatus.SUCCESS,
value=0,
attempts=1,
started_at=datetime.now(),
finished_at=datetime.now(),
)
text = report.to_json()
restored = px.RunReport.from_json(text)
assert restored.result_of("build").spec.outputs == {"version": "1.0.0"}
def test_outputs_in_to_dict(self) -> None:
"""to_dict 包含 outputs 字段。"""
graph = parse_yaml_string("""
jobs:
build:
cmd: ["echo", "build"]
outputs:
version: "1.0.0"
""")
spec = graph.spec("build")
report = px.RunReport()
from datetime import datetime
from pyflowx.task import TaskResult
report.results["build"] = TaskResult(
spec=spec,
status=TaskStatus.SUCCESS,
value=0,
attempts=1,
started_at=datetime.now(),
finished_at=datetime.now(),
)
d = report.to_dict()
assert d["results"][0]["outputs"] == {"version": "1.0.0"}
def test_outputs_none_in_to_dict(self) -> None:
"""无 outputs 时 to_dict 中为 None。"""
report = px.RunReport()
from datetime import datetime
from pyflowx.task import TaskResult, TaskSpec
spec: TaskSpec[int] = TaskSpec("a", lambda: 1)
report.results["a"] = TaskResult(
spec=spec,
status=TaskStatus.SUCCESS,
value=1,
attempts=1,
started_at=datetime.now(),
finished_at=datetime.now(),
)
d = report.to_dict()
assert d["results"][0]["outputs"] is None
class TestConditions:
"""条件解析测试。"""