feat: RunReport 结果序列化支持 to_json/to_dict/to_csv/from_json
CI / Lint, Typecheck & Test (push) Has been cancelled

新增 RunReport 序列化方法,支持 JSON/dict/CSV 多格式导出与反序列化重建;
非 JSON 可序列化值默认回退到 repr,可通过 value_serializer 自定义。
This commit is contained in:
2026-07-07 12:51:06 +08:00
parent 7b5e4864c2
commit 6499210dc9
4 changed files with 630 additions and 3 deletions
@@ -0,0 +1,85 @@
# 迭代 11:结果序列化
## 本轮目标
1. `RunReport.to_dict(value_serializer=)` —— 转为 JSON 兼容字典
2. `RunReport.to_json(indent=, value_serializer=)` —— 序列化为 JSON 字符串
3. `RunReport.to_csv(include_value=)` —— 表格 CSV 导出
4. `RunReport.from_json(text)` —— 反序列化(仅供查询,不重新执行)
## 改动文件清单
- `src/pyflowx/report.py` —— 新增 `to_dict`/`to_json`/`to_csv`/`from_json` 方法
- `tests/test_report.py` —— 新增序列化测试
- `README.md` —— 更新文档
## 关键设计
### 1. 值序列化策略
任务返回值可能是任意 Python 对象(不可 JSON 序列化)。默认策略:
- `None` 直接返回 `None`
- 可 JSON 序列化的原样返回
- 不可序列化的回退到 `repr(value)`
- 调用方可通过 `value_serializer` 自定义
### 2. to_dict 结构
```python
{
"success": bool,
"summary": {...},
"results": [
{
"name": str,
"status": str, # TaskStatus.value
"value": Any,
"error": str | None, # repr(error)
"attempts": int,
"started_at": str | None, # ISO 8601
"finished_at": str | None,
"duration_seconds": float | None,
"reason": str | None,
"tags": list[str],
"depends_on": list[str],
},
...,
],
}
```
`results` 为列表(保序),便于 CSV/JSON 数组导出。
### 3. to_csv
CSV 列:`name, status, attempts, duration_seconds, started_at, finished_at, error, reason[, value]`
`include_value=False` 时省略 value 列(值可能含逗号/换行符,干扰解析)。
### 4. from_json
- 反序列化只重建可序列化字段
- `TaskSpec``fn`/`cmd` 等不可恢复,使用 noop 占位
- 重建后的 report **仅供查询/分析**,不能用于重新执行
- `datetime` 通过 `fromisoformat` 还原
## 验收标准
- to_dict/to_json/to_csv/from_json 全部实现
- 非 JSON 序列化值有合理回退(repr)
- 自定义 value_serializer 可用
- from_json 可 round-trip 已序列化的 report
- 覆盖率 ≥ 95%
- ruff / pyrefly / pytest 全部通过
## 验证结果
- **ruff**All checks passed(含 format
- **pyrefly**0 errors
- **pytest**1257 passed(含 27 个新增序列化测试)
- **覆盖率**97.29%report.py 99%,仅 `_noop_fn` 占位函数体未覆盖,与 `task.py:_task_noop` 同模式)
- **修复**:扩展 `_make_result` 测试辅助以支持 tags/depends_on/reason 参数
## 遗留事项
-
+36
View File
@@ -37,6 +37,7 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **流式获取** —— `run_iter()` 生成器逐个 yield 任务结果,适用于大量任务逐个处理
- **任务取消** —— `CancelToken` 线程安全取消令牌,支持外部取消与 `KeyboardInterrupt` 优雅停止
- **子图执行** —— `run(only=[...], tags=[...])` 按名称或标签运行图的子集,自动包含传递依赖
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` 多格式导出运行结果,`from_json()` 支持反序列化重建
- **状态后端** —— `MemoryBackend` / `JSONBackend` / `SQLiteBackend`WAL 模式,适合大规模任务)
- **YAML 任务编排** —— GitHub Actions 风格 `jobs`/`needs`/`strategy.matrix`/`if` 条件,`pf yamlrun pipeline.yaml` 一键执行
- **最小依赖** —— `rich` + `typer` + `typing-extensions`3.13 以下)+ `pyyaml`
@@ -537,6 +538,41 @@ pf yamlrun pipeline.yaml --only report,deploy
pf yamlrun pipeline.yaml --tags ingest,report
```
### 结果序列化
`RunReport` 支持将运行结果导出为 JSON / dict / CSV,便于持久化、跨进程传输或对接外部工具:
```python
report = px.run(graph, strategy="sequential")
# JSON 字符串(含 success/summary/results
text = report.to_json(indent=2)
# JSON 兼容字典
data = report.to_dict()
# CSV 表格导出(不含 value 列以避免特殊字符干扰)
csv_text = report.to_csv(include_value=False)
```
任务返回值可能是任意 Python 对象(不一定 JSON 可序列化)。默认策略:
可序列化原样保留,不可序列化回退到 `repr(value)`;调用方可通过
`value_serializer` 自定义:
```python
# 自定义值序列化(如把 Path 转为字符串)
text = report.to_json(value_serializer=str)
```
`from_json()` 可从 JSON 字符串重建 `RunReport`**仅供查询/分析**,不能重新
执行(`TaskSpec``fn`/`cmd` 等不可序列化字段无法恢复):
```python
restored = px.RunReport.from_json(text)
print(restored["double"]) # 任务返回值
print(restored.result_of("a").status) # TaskStatus
```
## 开发
```bash
+168 -2
View File
@@ -6,11 +6,40 @@
from __future__ import annotations
from collections.abc import Iterator
import csv
import io
import json
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any
from .task import TaskResult, TaskStatus
from .task import TaskResult, TaskSpec, TaskStatus
def _noop_fn() -> None:
"""反序列化重建报告时的占位 ``fn``。"""
return None
def _serialize_value(
value: Any,
value_serializer: Callable[[Any], Any] | None = None,
) -> Any:
"""序列化任务返回值为 JSON 兼容形式。
``None`` 直接返回;提供 ``value_serializer`` 时由其转换;否则尝试
JSON 序列化,失败则回退到 ``repr(value)``,保证任意值都能落盘。
"""
if value is None:
return None
if value_serializer is not None:
return value_serializer(value)
try:
_ = json.dumps(value)
except (TypeError, ValueError):
return repr(value)
return value
@dataclass
@@ -94,3 +123,140 @@ class RunReport:
err = f" error={r.error!r}" if r.error else ""
lines.append(f" {name}: {r.status.value} ({dur} attempts={r.attempts}){err}")
return "\n".join(lines)
# ---- 序列化 ------------------------------------------------------- #
def to_dict(
self,
value_serializer: Callable[[Any], Any] | None = None,
) -> dict[str, Any]:
"""转换为 JSON 兼容的字典。
参数
----
value_serializer:
自定义任务值序列化函数。``None`` 时尝试 JSON 兼容转换,
失败回退到 ``repr()``。
返回结构包含顶层 ``success``、``summary`` 与 ``results`` 列表
(按完成顺序保序),每个结果项含 name/status/value/error/
attempts/started_at/finished_at/duration_seconds/reason/tags/
depends_on 字段。
"""
results_list: list[dict[str, Any]] = []
for name, r in self.results.items():
results_list.append(
{
"name": name,
"status": r.status.value,
"value": _serialize_value(r.value, value_serializer),
"error": repr(r.error) if r.error else None,
"attempts": r.attempts,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": r.duration,
"reason": r.reason,
"tags": list(r.spec.tags),
"depends_on": list(r.spec.depends_on),
}
)
return {
"success": self.success,
"summary": self.summary(),
"results": results_list,
}
def to_json(
self,
indent: int = 2,
value_serializer: Callable[[Any], Any] | None = None,
) -> str:
"""序列化为 JSON 字符串。
参数
----
indent:
JSON 缩进空格数,``0`` 表示紧凑单行。
value_serializer:
自定义任务值序列化函数,透传给 :meth:`to_dict`。
"""
return json.dumps(
self.to_dict(value_serializer),
ensure_ascii=False,
indent=indent if indent > 0 else None,
)
def to_csv(self, include_value: bool = True) -> str:
"""导出为 CSV 字符串。
列顺序:``name, status, attempts, duration_seconds, started_at,
finished_at, error, reason`` + 可选 ``value``。
参数
----
include_value:
是否包含 value 列。任务返回值可能含逗号/换行符干扰解析,
不需要值时设为 ``False`` 得到更干净输出。
"""
buf = io.StringIO()
writer = csv.writer(buf)
header = [
"name",
"status",
"attempts",
"duration_seconds",
"started_at",
"finished_at",
"error",
"reason",
]
if include_value:
header.append("value")
writer.writerow(header)
for name, r in self.results.items():
row = [
name,
r.status.value,
r.attempts,
f"{r.duration:.6f}" if r.duration is not None else "",
r.started_at.isoformat() if r.started_at else "",
r.finished_at.isoformat() if r.finished_at else "",
repr(r.error) if r.error else "",
r.reason or "",
]
if include_value:
row.append(_serialize_value(r.value))
writer.writerow(row)
return buf.getvalue()
@classmethod
def from_json(cls, text: str) -> RunReport:
"""从 JSON 字符串重建 :class:`RunReport`。
注意
----
只能重建可序列化字段;:class:`TaskSpec` 的 ``fn``/``cmd`` 等
不可序列化字段无法恢复,重建的 spec 仅含 ``name``/``tags``/
``depends_on``。重建后的 report 仅供查询/分析,**不能用于重新执行**。
"""
data = json.loads(text)
report = cls(success=data.get("success", True))
for entry in data.get("results", []):
spec: TaskSpec[Any] = TaskSpec(
name=entry["name"],
fn=_noop_fn,
tags=tuple(entry.get("tags", ())),
depends_on=tuple(entry.get("depends_on", ())),
)
started = entry.get("started_at")
finished = entry.get("finished_at")
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus(entry["status"]),
value=entry.get("value"),
attempts=entry.get("attempts", 0),
started_at=datetime.fromisoformat(started) if started else None,
finished_at=datetime.fromisoformat(finished) if finished else None,
reason=entry.get("reason"),
)
report.results[spec.name] = result
return report
+341 -1
View File
@@ -2,9 +2,15 @@
from __future__ import annotations
import csv
import io
import json
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any
import pytest
import pyflowx as px
from pyflowx.task import TaskResult, TaskSpec, TaskStatus
@@ -20,9 +26,12 @@ def _make_result(
error: BaseException | None = None,
duration: float = 0.5,
attempts: int = 1,
tags: tuple[str, ...] = (),
depends_on: tuple[str, ...] = (),
reason: str | None = None,
) -> TaskResult[Any]:
"""构造测试用 TaskResult 实例."""
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn)
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn, tags=tags, depends_on=depends_on)
start = datetime(2024, 1, 1, 0, 0, 0)
# 用 timedelta 精确表达秒数,避免 int() 截断小数
end = start + timedelta(seconds=duration) if duration else None
@@ -34,6 +43,7 @@ def _make_result(
attempts=attempts,
started_at=start,
finished_at=end,
reason=reason,
)
@@ -172,3 +182,333 @@ class TestRunReportQueries:
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
durs = report.durations()
assert durs["a"] == 0.0
# ---------------------------------------------------------------------- #
# 序列化:to_dict / to_json / to_csv / from_json
# ---------------------------------------------------------------------- #
class TestRunReportToDict:
"""测试 RunReport.to_dict."""
def test_to_dict_basic(self) -> None:
"""应输出包含 success/summary/results 的字典。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=42, duration=0.5, tags=("ingest",), depends_on=("x",))
d = report.to_dict()
assert d["success"] is True
assert d["summary"]["total_tasks"] == 1
assert len(d["results"]) == 1
entry = d["results"][0]
assert entry["name"] == "a"
assert entry["status"] == "success"
assert entry["value"] == 42
assert entry["error"] is None
assert entry["attempts"] == 1
assert entry["started_at"] == "2024-01-01T00:00:00"
assert entry["finished_at"] == "2024-01-01T00:00:00.500000"
assert entry["duration_seconds"] == 0.5
assert entry["reason"] is None
assert entry["tags"] == ["ingest"]
assert entry["depends_on"] == ["x"]
def test_to_dict_preserves_order(self) -> None:
"""results 列表应按插入顺序保留。"""
report = px.RunReport()
report.results["a"] = _make_result("a")
report.results["b"] = _make_result("b")
report.results["c"] = _make_result("c")
d = report.to_dict()
assert [r["name"] for r in d["results"]] == ["a", "b", "c"]
def test_to_dict_with_error(self) -> None:
"""error 应转为 repr 字符串。"""
report = px.RunReport(success=False)
report.results["a"] = _make_result("a", status=TaskStatus.FAILED, error=ValueError("boom"))
d = report.to_dict()
entry = d["results"][0]
assert entry["error"] == "ValueError('boom')"
assert entry["status"] == "failed"
def test_to_dict_none_value(self) -> None:
"""None 值应保持为 None。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=None)
d = report.to_dict()
assert d["results"][0]["value"] is None
def test_to_dict_non_serializable_value_falls_back_to_repr(self) -> None:
"""非 JSON 可序列化的值应回退到 repr。"""
report = px.RunReport()
# Path 对象不可 JSON 序列化
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
d = report.to_dict()
assert d["results"][0]["value"] == "PosixPath('/tmp/x')"
def test_to_dict_with_custom_serializer(self) -> None:
"""自定义 value_serializer 应被调用。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
d = report.to_dict(value_serializer=str)
assert d["results"][0]["value"] == "/tmp/x"
def test_to_dict_no_timestamps(self) -> None:
"""未开始/未结束的任务 started_at/finished_at/duration 应为 None。"""
report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
d = report.to_dict()
entry = d["results"][0]
assert entry["started_at"] is None
assert entry["finished_at"] is None
assert entry["duration_seconds"] is None
def test_to_dict_with_reason(self) -> None:
"""reason 字段应被保留。"""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.SKIPPED, duration=0.0, reason="条件不满足")
d = report.to_dict()
assert d["results"][0]["reason"] == "条件不满足"
def test_to_dict_empty_report(self) -> None:
"""空报告也应可序列化。"""
report = px.RunReport()
d = report.to_dict()
assert d["success"] is True
assert d["results"] == []
assert d["summary"]["total_tasks"] == 0
class TestRunReportToJson:
"""测试 RunReport.to_json."""
def test_to_json_basic(self) -> None:
"""应输出合法 JSON 字符串。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=42)
s = report.to_json()
data = json.loads(s)
assert data["success"] is True
assert data["results"][0]["value"] == 42
def test_to_json_indent_zero_compact(self) -> None:
"""indent=0 应输出紧凑单行(无换行)。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=1)
s = report.to_json(indent=0)
assert "\n" not in s
def test_to_json_indent_default(self) -> None:
"""默认 indent=2 应输出多行格式。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=1)
s = report.to_json()
assert "\n" in s
def test_to_json_ensure_chinese(self) -> None:
"""中文字符不应被转义(ensure_ascii=False)。"""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.SKIPPED, duration=0.0, reason="条件不满足")
s = report.to_json()
assert "条件不满足" in s
assert "\\u" not in s
def test_to_json_with_value_serializer(self) -> None:
"""to_json 应透传 value_serializer。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
s = report.to_json(value_serializer=str)
data = json.loads(s)
assert data["results"][0]["value"] == "/tmp/x"
class TestRunReportToCsv:
"""测试 RunReport.to_csv."""
def test_to_csv_basic(self) -> None:
"""应输出带表头与数据行的 CSV。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=42, duration=0.5)
s = report.to_csv()
reader = csv.reader(io.StringIO(s))
rows = list(reader)
assert rows[0] == [
"name",
"status",
"attempts",
"duration_seconds",
"started_at",
"finished_at",
"error",
"reason",
"value",
]
assert rows[1][0] == "a"
assert rows[1][1] == "success"
assert rows[1][2] == "1"
assert rows[1][3] == "0.500000"
assert rows[1][8] == "42"
def test_to_csv_without_value(self) -> None:
"""include_value=False 应省略 value 列。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=42)
s = report.to_csv(include_value=False)
reader = csv.reader(io.StringIO(s))
rows = list(reader)
assert rows[0] == [
"name",
"status",
"attempts",
"duration_seconds",
"started_at",
"finished_at",
"error",
"reason",
]
assert len(rows[1]) == 8
def test_to_csv_empty_value_quoted(self) -> None:
"""含逗号的值应被引号包裹(csv 模块自动处理)。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value="a,b,c")
s = report.to_csv()
# csv 模块会自动给含特殊字符的值加引号
assert '"a,b,c"' in s
def test_to_csv_empty_report(self) -> None:
"""空报告应只输出表头。"""
report = px.RunReport()
s = report.to_csv()
reader = csv.reader(io.StringIO(s))
rows = list(reader)
assert len(rows) == 1
assert rows[0][0] == "name"
def test_to_csv_non_serializable_value(self) -> None:
"""非 JSON 可序列化值应回退到 repr。"""
report = px.RunReport()
report.results["a"] = _make_result("a", value=Path("/tmp/x"))
s = report.to_csv()
assert "PosixPath('/tmp/x')" in s
class TestRunReportFromJson:
"""测试 RunReport.from_json."""
def test_from_json_round_trip(self) -> None:
"""to_json → from_json 应能恢复所有可序列化字段。"""
report = px.RunReport()
report.results["a"] = _make_result(
"a",
value=42,
duration=0.5,
tags=("ingest",),
depends_on=("x",),
)
report.results["b"] = _make_result(
"b",
status=TaskStatus.SKIPPED,
duration=0.0,
reason="条件不满足",
)
text = report.to_json()
restored = px.RunReport.from_json(text)
assert restored.success == report.success
assert len(restored) == 2
assert restored["a"] == 42
assert restored.result_of("a").status == TaskStatus.SUCCESS
assert restored.result_of("a").attempts == 1
assert restored.result_of("a").duration == 0.5
assert restored.result_of("a").started_at == datetime(2024, 1, 1, 0, 0, 0)
assert restored.result_of("a").finished_at == datetime(2024, 1, 1, 0, 0, 0, 500000)
assert restored.result_of("a").spec.tags == ("ingest",)
assert restored.result_of("a").spec.depends_on == ("x",)
assert restored.result_of("b").status == TaskStatus.SKIPPED
assert restored.result_of("b").reason == "条件不满足"
def test_from_json_preserves_order(self) -> None:
"""from_json 应保持 results 顺序。"""
report = px.RunReport()
report.results["a"] = _make_result("a")
report.results["b"] = _make_result("b")
report.results["c"] = _make_result("c")
restored = px.RunReport.from_json(report.to_json())
assert list(restored) == ["a", "b", "c"]
def test_from_json_empty_report(self) -> None:
"""空报告应可正确反序列化。"""
report = px.RunReport()
restored = px.RunReport.from_json(report.to_json())
assert len(restored) == 0
assert restored.success is True
def test_from_json_invalid_status_raises(self) -> None:
"""非法 status 值应抛 ValueErrorTaskStatus 枚举校验)。"""
data = {
"success": True,
"summary": {},
"results": [{"name": "a", "status": "unknown_status"}],
}
with pytest.raises(ValueError):
_ = px.RunReport.from_json(json.dumps(data))
def test_from_json_missing_optional_fields(self) -> None:
"""缺少可选字段(tags/depends_on/attempts)应使用默认值。"""
data = {
"success": False,
"results": [{"name": "a", "status": "success"}],
}
restored = px.RunReport.from_json(json.dumps(data))
assert restored.success is False
assert restored.result_of("a").attempts == 0
assert restored.result_of("a").spec.tags == ()
assert restored.result_of("a").spec.depends_on == ()
def test_from_json_invalid_json_raises(self) -> None:
"""非法 JSON 字符串应抛 JSONDecodeError。"""
with pytest.raises(json.JSONDecodeError):
_ = px.RunReport.from_json("not json{")
class TestRunReportSerializationIntegration:
"""序列化与实际执行流程的集成测试。"""
def test_real_run_serialization_round_trip(self) -> None:
"""实际 run() 产出的报告应可序列化/反序列化。"""
def extract() -> list[int]:
return [1, 2, 3]
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
graph = px.Graph.from_specs(
[
px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
]
)
report = px.run(graph, strategy="sequential")
# 序列化 → 反序列化
text = report.to_json()
restored = px.RunReport.from_json(text)
assert restored.success is True
assert restored["double"] == [2, 4, 6]
assert restored.result_of("extract").spec.tags == ("ingest",)
assert restored.result_of("double").spec.depends_on == ("extract",)
def test_csv_export_from_real_run(self) -> None:
"""实际 run() 报告应可导出 CSV。"""
def extract() -> list[int]:
return [1, 2, 3]
graph = px.Graph.from_specs([px.TaskSpec("extract", extract)])
report = px.run(graph, strategy="sequential")
csv_text = report.to_csv()
assert "extract" in csv_text
assert "success" in csv_text
# 值应被 repr 化(列表用 repr)
assert "[1, 2, 3]" in csv_text