perf: P8 性能优化 — 自实现 Kahn 拓扑排序替换 graphlib(diamond-1000 1.62x)、_json.py 抽象层支持 orjson 可选加速、dataclass slots=True 内存优化(TaskSpec 15.5% 减少)、线程池跨层复用与 get_running_loop 兼容 3.12+;iter-16~20 归档到 skills
CI / Lint, Typecheck & Test (push) Has been cancelled

This commit is contained in:
2026-07-07 17:11:34 +08:00
parent a9f15e236d
commit 8b8598fa41
21 changed files with 739 additions and 602 deletions
-54
View File
@@ -1,54 +0,0 @@
# 迭代 16ops 工具完善
## 本轮目标
1. **修复 silent failures** —— dockercmd/msdownload/sglang 使用 `check=False` 导致失败被吞,改为 `check=True` + try/except 打印 rich 错误
2. **改善 taskkill 容错路径** —— taskkill 在 pkill 无匹配时打印提示而非静默;clr 保持容错但加注释说明原因
3. **补错误路径测试** —— 为新增 try/except 块补 CalledProcessError / FileNotFoundError 路径测试
## 改动文件清单
### 源码改动
- `src/pyflowx/ops/infra/dockercmd.py` —— `check=True` + try/except CalledProcessError/FileNotFoundError
- `src/pyflowx/ops/infra/msdownload.py` —— `check=True` + try/except CalledProcessError/FileNotFoundError
- `src/pyflowx/ops/infra/sglang.py` —— install/run 改 `check=True` + try/except
- `src/pyflowx/ops/system/taskkill.py` —— 保留 `check=False`(pkill 返回 1 表示无匹配,非错误),但检查 returncode 打印成功/失败
- `src/pyflowx/ops/system/clr.py` —— 注释说明 check=False 原因(清屏失败不影响后续工作)
- `src/pyflowx/ops/system/which.py` —— 注释说明 check=False 原因(where/which 返回非零表示未找到命令)
### 测试改动
- `tests/cli/test_envdev.py` —— TestDockerLoginTencent 新增 3 个错误路径测试(success/CalledProcessError/FileNotFoundError
- `tests/cli/test_llm.py` —— TestMsdownloadRun 新增 2 个(CalledProcessError/FileNotFoundError),TestInstallSglang 新增 2 个,TestRunSglang 新增 2 个
- `tests/cli/test_system_run.py` —— TestTaskkillRun 改进 test_prints_progress(断言成功路径),新增 test_returncode_nonzero_prints_failure
## 关键设计
### 1. check=False 修复策略
按场景区分:
- **失败需感知**dockercmd/msdownload/sglang)→ `check=True` + `try/except subprocess.CalledProcessError` 打印 rich 错误
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查 `returncode` 打印结果
- **失败无关紧要**clr/which)→ 保持 `check=False`,加注释说明原因
### 2. 错误路径测试策略
测试发现现有 6 个工具的测试已存在(在 test_system_run.py / test_llm.py / test_envdev.py 中),但仅覆盖成功路径。本轮重点补充:
- **CalledProcessError 路径**subprocess.run 抛 CalledProcessError 时,函数应捕获并打印包含 returncode 的错误信息,不向上抛出
- **FileNotFoundError 路径**:命令本身不存在(如 docker/uvx/uv/python 未安装)时,函数应打印"命令未找到"提示,不向上抛出
- **taskkill returncode 非零路径**pkill 返回 1 表示无匹配进程,应打印"未找到匹配进程或终止失败"
mock 模式:`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))` —— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
## 验证结果
- **ruff check**All checks passed!
- **ruff format --check**33 files already formatted
- **pyrefly check**0 errors (7 suppressed)
- **pytest**1351 passed in 9.82s
- **coverage**97.38%>= 95%),所有改动的 ops 文件(dockercmd/msdownload/sglang/taskkill/clr/which)覆盖率 100%
## 遗留事项
无。本轮目标全部达成。
-126
View File
@@ -1,126 +0,0 @@
# 迭代 17:性能优化深入(P5
## 本轮目标
1. **基准套件增强** —— 补齐 conditions/YAML/notifiers/run_iter/subgraph/cancellation/CPU/I/O 8 类基准场景,覆盖此前未量化的代码路径
2. **build_call_args 优化** —— 为 fn 无依赖任务和 cmd 任务开辟快速路径,避免重复 `inspect.signature` 调用
3. **JSONBackend 优化** —— batch 模式下延迟 JSON 序列化验证到 flush 时一次性完成
4. **图构建优化** —— `from_specs` 批量注册,跳过每次 `_register` 的 cache 清空
## 改动文件清单
### 源码改动
- `src/pyflowx/context.py` —— 新增 `_fn_no_dep_injection``@lru_cache` 缓存注入计划)和 `_try_fast_path`(快速路径分发),`build_call_args` 起始处调用快速路径
- `src/pyflowx/storage.py` —— `JSONBackend.save``_defer_flush=True` 时跳过 `json.dumps` 验证,让 `flush``json.dump` 统一捕获序列化错误
- `src/pyflowx/graph.py` —— `from_specs` 批量注册:直接写入 `specs`/`deps` 字典,跳过每次 `_register` 的 cache 清空(N 次清空降为 0 次)
### 配置与基准改动
- `pyproject.toml` —— `[tool.pyrefly]` 新增 `search-path = ["."]`,使 pyrefly 能解析项目根目录的 `benchmarks/`
- `typings/graphlib/__init__.pyi` —— 移除已无必要的 `# pyrefly: ignore [missing-import]`search-path 配置后 pyrefly 可直接解析 graphlib
- `benchmarks/__init__.py` —— 提取共享 `time_it` / `print_results` 工具函数,`__all__` 显式导出
- `benchmarks/bench_advanced.py` —— 新增 8 个高级基准场景
- `benchmarks/__main__.py` —— 从 `benchmarks` 包导入共享工具,新增 `"advanced": run_advanced` 分发,新增 JSONBackend 复杂值基准
## 关键设计
### 1. build_call_args 快速路径
**问题**`build_call_args` 原本对所有任务都走完整路径——`inspect.signature` 解析、dep_context 构建、collisions 检测、leftover 收集。但 fn 无依赖任务和 cmd 任务(effective_fn 是无参闭包)不需要这些工作。
**方案**:两条快速路径,提取到 `_try_fast_path` 函数(避免 `build_call_args` 分支数超 PLR0912 阈值 12):
- **快速路径 1(cmd 任务)**:`spec.fn is None and spec.cmd is not None and not spec.args and not spec.kwargs` → 直接返回 `((), {})`
- **快速路径 2(fn 无依赖)**:`not spec.depends_on and not spec.soft_depends_on and not spec.args and not spec.kwargs` → 调用 `_fn_no_dep_injection(fn)` 获取缓存的注入计划
**`_fn_no_dep_injection`**:用 `@lru_cache(maxsize=1024)` 缓存每个 fn 的注入计划 `(context_params, error_param)`
- `context_params`:Context 标注参数名元组;存在必填无默认值参数时返回 `None`(需走慢路径)
- `error_param`:第一个无默认值且非 Context 标注的参数名(用于错误信息);`None` 表示无此类参数
- 不变量:`error_param is None ⟺ context_params is not None`,用 `assert` 辅助 pyrefly 推断
### 2. JSONBackend batch 延迟验证
**问题**`JSONBackend.save` 原本每次调用都 `json.dumps(value)` 验证可序列化性,batch 模式下 N 次 save 产生 N 次验证开销,但实际 `flush` 时只做一次 `json.dump`
**方案**`_defer_flush=True` 时跳过 `json.dumps` 验证,让 `flush``json.dump` 统一捕获 `TypeError`/`ValueError`。非 batch 模式仍即时验证以提供精确错误位置。
### 3. from_specs 批量注册
**问题**`from_specs` 原本对每个 spec 调用 `_register`,而 `_register` 每次都 `_resolved_cache.clear()` + `_layers_cache = None`,N 个 spec 产生 N 次缓存清空。
**方案**:直接写入 `graph.specs` / `graph.deps` 字典,重复名检测在循环内做。缓存失效由后续 `validate()` / `layers()` 首次调用自然触发(cache 为空时自动重算)。
**限制**diamond(1000) 性能分析显示 `graphlib.TopologicalSorter.prepare()``from_specs` 总耗时的 82%,属 stdlib 实现无法优化。批量注册只优化了剩余 18% 中的 cache 清空部分(约 2.5%),因此整体提升有限。该优化方向正确(减少无效工作),但受限于 stdlib 瓶颈。
### 4. 基准套件架构
- **共享工具**`time_it(fn, iterations, warmup) -> (avg_ms, ops_per_sec)``print_results(title, results)` 提取到 `benchmarks/__init__.py`
- **模块化**`bench_advanced.py` 独立模块,`run_advanced()` 分发到 8 个基准函数
- **CLI 分发**`__main__.py``BENCH_MODULES` dict 映射模块名到运行函数
## 验证结果
### 门禁
- **ruff check**All checks passed!
- **pyrefly check**0 errors (62 suppressed)
- **pytest**1351 passed in 9.72s
- **coverage**97.18%>= 95%
### 性能对比
#### build_call_argsP5.2 关键路径)
| 场景 | 优化前 | 优化后 | 提升 |
|------|--------|--------|------|
| fn(no-deps) | 1.2M ops/s | 4.44M ops/s | **3.7x+270%** |
| cmd(fast-path) | 9.5M ops/s | 9.52M ops/s | 持平(已有快速路径) |
| fn(2-deps) | — | 708K ops/s | 慢路径基线 |
| fn(Context-annotated) | — | 918K ops/s | 慢路径基线 |
| fn(**kwargs) | — | 754K ops/s | 慢路径基线 |
#### JSONBackendP5.3
| 场景 | 优化前 | 优化后 | 提升 |
|------|--------|--------|------|
| save(batch=10) | — | 0.251 ms (3980 ops/s) | 基线 |
| save(batch=10,complex) | 0.458 ms | 0.382 ms (2617 ops/s) | **16.5%** |
| load | — | 0.009 ms (115703 ops/s) | 基线 |
#### 图构建(P5.4
| 场景 | 优化前 | 优化后 | 提升 |
|------|--------|--------|------|
| chain(1000) | 0.96 ms | 0.96 ms (1042 ops/s) | 持平 |
| diamond(1000) | 3.59 ms | 3.59 ms (278 ops/s) | 持平(stdlib 瓶颈) |
| layers(cached,1000) | 16.7M ops/s | 16.73M ops/s | 持平 |
| resolved_spec(cached,1000) | 13.2M ops/s | 13.2M ops/s | 持平 |
#### 新增高级基准(P5.1,无历史对比)
| 场景 | 吞吐 |
|------|------|
| static(IS_LINUX) | 14.48M ops/s |
| AND(OR(NOT,DEP_TRUTHY),DEP_PRESENT) | 2.66M ops/s |
| yaml_load(100) | 107.6 ops/s |
| CallbackNotifier.notify | 4.83M ops/s |
| run_iter(sequential,500) | 312.7 ops/s |
| subgraph_with_deps(1000) | 456.5 ops/s |
| cancel(immediate,200) | 5317 ops/s |
| io-async(50) | 41.6 ops/s |
## 关键决策与依据
1. **快速路径提取到独立函数**`build_call_args` 加快速路径后分支数达 14(超 PLR0912 阈值 12),提取 `_try_fast_path` 控制分支数
2. **`_fn_no_dep_injection``@lru_cache`**:fn 对象可哈希且生命周期通常与程序一致,缓存命中率高;`maxsize=1024` 防止无界增长
3. **`assert context_params is not None`**pyrefly 无法从 `(context_params, error_param)` 返回类型推断 `error_param is None ⟺ context_params 非 None` 的不变量,用 assert 辅助类型收窄
4. **batch 延迟验证保留非 batch 即时验证**:非 batch 模式下调用方期望即时错误反馈,batch 模式下调用方接受延迟错误以换取吞吐
5. **`from_specs` 批量注册不扩展到 `add`/`chain`**`add`/`chain` 是增量 API,每次调用需即时校验以快速失败;`from_specs` 是批量构造,可接受延迟校验
6. **P5.4 接受部分达成**stdlib `graphlib.TopologicalSorter.prepare()` 占 82% 耗时无法优化,批量注册只优化剩余 18% 中的 2.5%。整体 P5 满足验收标准(build_call_args 270% + JSONBackend 16.5% 均超 10% 阈值)
## 遗留事项
1. **图构建 stdlib 瓶颈**`graphlib.TopologicalSorter.prepare()` 是 diamond 图的主瓶颈(82%),若需进一步优化需自实现拓扑排序或替换排序算法(超出本轮范围)
2. **iter-06 ~ iter-10 归档**:按规则每 5 次迭代应归档旧记录到 `.trae/skills/`,当前 iter-06 ~ iter-10 仍未归档(非本轮范围,后续清理)
-92
View File
@@ -1,92 +0,0 @@
# 迭代 18:质量收尾(P6
## 本轮目标
1. **覆盖率提升** —— 补齐 context.py92%)和 reseticoncache.py83%)的测试缺口
2. **pragma: no cover 清理** —— diagnostics.py 的 4 处 pragma 按"激活或删除"原则处理
3. **README 同步** —— 开发命令从 mypy 更新为 pyreflyruff 范围扩展到全项目
4. **iter 文档归档** —— iter-06~15 按规则归档到 skillsdocs 只保留最近记录
## 改动文件清单
### 源码改动
- `src/pyflowx/diagnostics.py` —— `_trace` 函数清理 4 处 pragma
- 行 180:移除 pragma,保留 visited 防御(diagnose 不依赖 Graph 校验,results 可能有环)
- 行 184:删除不可达的 `if result is None` 检查,`report.results.get(task)` 改为 `report.results[task]`
- 行 201-203:删除不可达的 SUCCESS 分支,改为无条件 `return (task, path)`
### 测试改动
- `tests/test_context.py` —— 新增 4 个测试:
- `test_fn_no_deps_with_context_annotation`fn 无依赖 + Context 标注走快速路径 2
- `test_fn_no_deps_with_default_params`:fn 无依赖 + 全部参数有默认值走快速路径 2
- `test_fn_no_deps_with_required_param_raises`:fn 无依赖 + 必填参数走快速路径 2 抛 InjectionError
- `test_signature_unhashable_fallback`:不可哈希 fn 走慢路径时 _signature 回退
- `tests/cli/test_reseticoncache.py` —— 新增 `test_windows_deletes_icon_cache_db`Path.exists 返回 True 覆盖删除分支
- `tests/test_diagnostics.py` —— 新增 `test_chain_circular_dependency`:构造带环 results 激活 visited 防御
### 文档改动
- `README.md` —— 开发部分:`uv run mypy``uv run pyrefly check .``ruff check src tests``ruff check .``ruff format --check src tests``ruff format --check .`
### 归档改动
- **新建** `.trae/skills/pyflowx-development/SKILL.md`(355 行,10 个主题章节)—— 整合 iter-06~15 的可复用模式、踩坑总结、设计决策
- **删除** `.trae/docs/iter-06~15` 共 10 个过程性记录文件
## 关键设计
### 1. diagnostics.py pragma 清理策略
`python-standards.md` "pragma: no cover 是清理信号,应激活或删除"原则,逐处分析:
| 行号 | 原标注 | 可达性分析 | 处理 |
|------|--------|-----------|------|
| 180 | Graph 校验防止循环依赖 | diagnose 不依赖 Graph,反序列化 results 可能有环 | **激活**:移除 pragma,写循环依赖测试 |
| 184 | failed_deps 仅含 results 中的任务 | `failed_deps = [d for d in deps if d in report.results and ...]` 已过滤 | **删除**:改 `report.results[task]` |
| 201 | Graph 校验防止循环 | _trace 只接收 FAILED 任务,`status != SUCCESS` 恒为 True | **删除**:改为无条件 return |
| 203 | 逻辑上不可达 | SUCCESS 任务不会被传入 _trace | **删除**:合并到行 201 |
### 2. context.py 覆盖率缺口分析
P5.2 引入的 `_fn_no_dep_injection``_try_fast_path` 有 3 个未覆盖分支:
- **Context 标注收集**(行 58-60):fn 无依赖 + Context 标注参数时,快速路径 2 收集 context_params
- **必填参数返回**(行 61-65):fn 无依赖 + 必填无默认值参数时,返回 `(None, pname)` 触发 InjectionError
- **_signature 不可哈希回退**(行 43-44):fn 不可哈希时 `_cached_signature` 抛 TypeError,回退到直接内省
### 3. reseticoncache.py 覆盖率缺口
现有测试用 `Path.exists = lambda _: False` 跳过删除分支。新增测试让 `Path.exists` 返回 True,覆盖 IconCache.db 和 Explorer 缓存的删除命令。
### 4. iter 文档归档
`dev-workflow.md` 规则"每 5 次迭代后清理 docs"
- iter-06~10 应在 iter-11 开始前清理(未执行)
- iter-11~15 应在 iter-16 开始前清理(未执行)
- 本轮补执行:iter-06~15 全部归档到 `.trae/skills/pyflowx-development/SKILL.md`
- 归档内容:按主题组织(YAML 编排、性能优化、取消机制、CLI 设计、错误诊断、观测性等),剔除过程性细节
- docs 目录现只保留 iter-16~18
## 验证结果
- **ruff check**All checks passed!
- **pyrefly check**0 errors (63 suppressed)
- **pytest**1357 passed in 9.76s+6 新测试)
- **coverage**97.54%>= 95%,较上轮 97.18% 提升 0.36%
### 覆盖率提升对照
| 文件 | 优化前 | 优化后 |
|------|--------|--------|
| context.py | 92% | **100%** |
| reseticoncache.py | 83% | **100%** |
| diagnostics.py | 98% | **99%** |
| 整体 | 97.18% | **97.54%** |
## 遗留事项
1. **tools.py 覆盖率 94%**:未覆盖行(250->244, 464-465, 564, 570-578)是 CLI 相关代码,本轮未处理
2. **filelevel.py/folderback.py/command.py**97-99%,接近阈值但未达 100%,分支未覆盖
3. **bumpversion.py 的 3 处 pragma**:标注为"调用方已保证",属合理的防御代码,未处理
-90
View File
@@ -1,90 +0,0 @@
# 迭代 19 — P7 功能增强
## 本轮目标
P7 阶段聚焦功能增强,涵盖四个子任务:
- P7.1 通知器扩展(Slack/Discord/Telegram
- P7.2 HTML 报告导出(RunReport.to_html
- P7.3 YAML 增强特性(matrix include-exclude / 条件依赖 / outputs
- P7.4 任务缓存增强(invalidate 单键失效 / MemoryBackend LRU 驱逐)
## 改动文件清单
### P7.1 通知器扩展
- `src/pyflowx/notification.py` — 新增 SlackNotifier/DiscordNotifier/TelegramNotifier
- `src/pyflowx/__init__.py` — 导出新通知器类
- `tests/test_notification.py` — 新增 14 个测试
- `README.md` — 通知器章节扩展
### P7.2 HTML 报告导出
- `src/pyflowx/report.py` — 新增 to_html 方法 + _render_summary_card 辅助
- `tests/test_report.py` — 新增 16 个测试
- `README.md` — 序列化章节扩展
### P7.3 YAML 增强特性
- `src/pyflowx/yaml_loader.py` — _cartesian_product 支持 include/exclude_expand_needs 支持条件依赖;_parse_optional_fields 解析 outputs
- `src/pyflowx/task.py` — TaskSpec 新增 outputs 字段;task() 工厂支持 outputs 参数
- `src/pyflowx/report.py` — to_dict/from_json 包含 outputs
- `tests/test_yaml_loader.py` — 新增 24 个测试
- `README.md` — YAML 章节扩展
### P7.4 任务缓存增强
- `src/pyflowx/storage.py` — StateBackend 新增 invalidate 抽象方法;_TTLStateBackendMixin 新增 _delete_raw 原语与 invalidate 实现;MemoryBackend 新增 maxsize LRU 驱逐;JSONBackend/SQLiteBackend 实现 _delete_raw
- `tests/test_storage.py` — 新增 17 个测试
- `README.md` — 缓存章节扩展
## 关键决策与依据
### 通知器架构(P7.1
- 复用 WebhookNotifier 基类,仅覆盖 _build_payload 适配平台格式
- TelegramNotifier 额外需要 chat_id 参数,故覆盖 __init__
- 零新依赖(stdlib urllib.request
### HTML 报告设计(P7.2
- 自包含 HTML5 文档,内联 CSS,无外部依赖
- 使用 html.escape 转义所有用户内容(任务名/错误/原因/值)防 XSS
- 状态徽章着色:success 绿 / failed 红 / skipped 黄 / running 蓝
- _render_summary_card 静态方法避免重复代码
### YAML 矩阵 include/exclude 语义(P7.3
- exclude: 组合的所有 key-value 对都匹配则剔除(c.items() >= ex.items()
- include: 追加额外组合,可引入基础矩阵之外的键
- 无基础矩阵键时返回空列表而非 [{}],避免产生空名任务
### 条件依赖设计(P7.3
- needs 项支持 {job: name, if: expr} 格式
- if 条件不满足时跳过该依赖(不加入 depends_on
- 复用 _parse_condition 解析 if 表达式
- 条件求值时传空上下文 {}(env 条件直接读 os.environ
### outputs 字段(P7.3
- TaskSpec 新增 outputs: Mapping[str, str] | None 作为元数据
- 仅携带不参与执行,可通过 spec.outputs 查询
- 支持 ${{ matrix.* }} 占位符替换
- to_dict/from_json 往返保留
### 缓存 invalidate 设计(P7.4
- StateBackend 新增 invalidate 抽象方法(返回 bool 表示是否删除)
- _TTLStateBackendMixin 通过 _delete_raw 原语统一委托
- JSONBackend.invalidate 覆盖以在非 batch 模式下 flush 落盘
- SQLiteBackend._delete_raw 用 cursor.rowcount > 0 判断是否存在
### MemoryBackend LRU 设计(P7.4
- 使用 OrderedDict 替代普通 dict
- _get_raw 命中时 move_to_end(key) 更新访问顺序
- _put_raw 超限时 popitem(last=False) 驱逐头部(最旧)
- maxsize=None 时不驱逐(默认行为,向后兼容)
## 验证结果
- ruff check: All checks passed
- ruff format: All files formatted
- pyrefly: 0 errors (67 suppressed)
- pytest: 1425 passed
- coverage: 97.56%(分支覆盖率 ≥ 95%
## 遗留事项
- P8(性能优化)尚未开始:自实现拓扑排序 / orjson 序列化 / 内存优化 / 执行器并发优化
- outputs 字段当前仅作为元数据,未来可扩展为跨任务引用(${{ needs.job.outputs.name }}
- TelegramNotifier 实际使用时需拼接 bot token URL,文档已说明
+255 -4
View File
@@ -1,11 +1,11 @@
---
name: "pyflowx-development"
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-15 中可复用的架构模式、踩坑总结与设计决策。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断相关的功能时参考。"
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化相关的功能时参考。"
---
# PyFlowX 开发知识库
本技能归档自迭代 06-15 的过程记录,按主题分类整理可复用知识。
本技能归档自迭代 06-20 的过程记录,按主题分类整理可复用知识。
过程性细节(覆盖率数字、命令输出)已剔除,仅保留架构模式、设计依据与陷阱总结。
## 一、兼容性与代码清理
@@ -64,6 +64,31 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
- **Graph.from_yaml 委托 yaml_loader**`Graph.from_yaml(path)` classmethod
委托给 `yaml_loader.load_yaml(path)` 函数式 API,保持 Graph 类职责单一。
### Matrix include/exclude 语义(iter-19
- **exclude**: 组合的所有 key-value 对都匹配则剔除(`combo.items() >= exclude.items()`)。
`exclude: [{os: macos, version: "3.8"}]` 剔除 macos+3.8 组合。
- **include**: 追加额外组合,可引入基础矩阵之外的键。
如基础矩阵 `version: ["3.10"]``include: [{version: "3.11", experimental: true}]`
追加 3.11 实验组合。
- **无基础矩阵键时返回空列表而非 `[{}]`**:避免产生空名任务(`job_`)。
### 条件依赖(iter-19
- **`needs` 项支持 `{job: name, if: expr}` 格式**:条件不满足时跳过该依赖
(不加入 `depends_on`)。
- 复用 `_parse_condition` 解析 `if` 表达式;条件求值时传空上下文 `{}`
`env` 条件直接读 `os.environ`)。
- **典型场景**:仅在特定矩阵变体下才需要的依赖,如
`needs: [{job: setup, if: env.DEPLOY == "true"}]`
### outputs 字段(iter-19
- `TaskSpec` 新增 `outputs: Mapping[str, str] | None` 作为元数据,
仅携带不参与执行,可通过 `spec.outputs` 查询。
- 支持 `${{ matrix.* }}` 占位符替换。
- `to_dict` / `from_json` 往返保留;序列化向前兼容(旧 JSON 无此字段时回退 None)。
### 踩坑总结
- **YAML 加载失败统一包装**`yaml_loader._safe_load` 应将 `yaml.YAMLError`
@@ -106,11 +131,107 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
- **缓存优化收益**`layers()` 缓存命中 ~1500万 ops/s~50000x 加速);
cmd 快速路径 1130万 ops/s vs fn 无依赖 144万 ops/s~8x 加速)。
### 自实现 Kahn 算法拓扑排序(iter-20
- **问题**`graphlib.TopologicalSorter.prepare()` 是 diamond(1000) 图构建的
82% 瓶颈(stdlib 内部状态机开销)。
- **方案**:模块级 `_topological_layers(deps)` 函数用 dict + list 直接计算
入度与反向邻接,按层 BFS 处理。返回 `(layers, cycle_nodes)`
- 无环时 `cycle_nodes = None`
- 有环时 `cycle_nodes` 为入度仍 >0 的节点列表(非精确环路径,仅指示存在环)
- **额外优化**`validate()` 顺带填充 `_layers_cache`,使后续 `layers()`
直接命中缓存,避免二次计算 `_topological_layers`
- **结果**diamond(1000) 图构建 3.59ms → 2.21ms1.62x)。
- **删除**`typings/graphlib/` 类型存根不再需要。
### build_call_args 双快速路径(iter-17
- **快速路径 1cmd 任务)**`spec.fn is None and spec.cmd is not None and
not spec.args and not spec.kwargs` → 直接返回 `((), {})`。
- **快速路径 2(fn 无依赖)**:`not spec.depends_on and not spec.soft_depends_on
and not spec.args and not spec.kwargs` → 调用 `_fn_no_dep_injection(fn)`
获取 `@lru_cache(maxsize=1024)` 缓存的注入计划 `(context_params, error_param)`。
- `context_params`:Context 标注参数名元组;存在必填无默认值参数时返回 `None`
(需走慢路径)。
- `error_param`:第一个无默认值且非 Context 标注的参数名(用于错误信息)。
- **不变量**`error_param is None ⟺ context_params 非 None`,用 `assert`
辅助 pyrefly 类型收窄。
- **提取到 `_try_fast_path` 函数**:避免 `build_call_args` 分支数超 PLR0912
阈值 12。
- **不可哈希 fn 回退**`_cached_signature` 抛 TypeError 时回退到直接内省。
### JSONBackend batch 延迟验证(iter-17
- **问题**`save` 原本每次都 `json.dumps(value)` 验证可序列化性,batch
模式下 N 次 save 产生 N 次验证开销,但 `flush` 只做一次 `json.dump`。
- **方案**`_defer_flush=True` 时跳过 `json.dumps` 验证,让 `flush` 的
`json.dump` 统一捕获 `TypeError`/`ValueError`。非 batch 模式仍即时验证
以提供精确错误位置。
- **依据**:非 batch 调用方期望即时错误反馈;batch 调用方接受延迟错误以换吞吐。
### from_specs 批量注册(iter-17
- **问题**`from_specs` 原本对每个 spec 调 `_register`,而 `_register`
每次都 `_resolved_cache.clear()` + `_layers_cache = None`N 个 spec
产生 N 次缓存清空。
- **方案**:直接写入 `graph.specs` / `graph.deps` 字典,重复名检测在循环内做。
缓存失效由后续 `validate()` / `layers()` 首次调用自然触发(cache 为空时
自动重算)。
- **不扩展到 `add`/`chain`**:增量 API 需即时校验以快速失败;批量构造可
接受延迟校验。
### orjson 可选依赖抽象层(iter-20
- **方案**`_json.py` 模块 `try: import orjson except ImportError: import json`
回退。导出 `dumps`/`loads`/`dump`/`load`/`JSONDecodeError`,签名与 stdlib
兼容。
- **安装**`pip install pyflowx[fast]` 启用 orjson`[project.optional-dependencies]
fast = ["orjson>=3.10.0"]`)。
- **orjson 兼容性差异**
- `dumps` 返回 `bytes` → wrapper 中 `.decode("utf-8")` 统一为 `str`
- 无 `ensure_ascii` 参数 → wrapper 接受并忽略(`# noqa: ARG001`),orjson 始终 UTF-8
- 无 `indent` 参数 → 映射为 `OPT_INDENT_2`
- 紧凑模式无空格(`[1,2,3]` vs stdlib `[1, 2, 3]`)→ 测试断言改为格式无关
- `JSONDecodeError` 直接复用 orjson 的(是 stdlib `json.JSONDecodeError` 子类)
- **pyrefly 兼容**`import orjson` 在 try 块内,所有 `orjson.xxx` 引用加
`# type: ignore[possibly-unbound]`orjson 未安装时 `# type: ignore[import-not-found]`。
### slots=True 内存优化(iter-20
- **方案**:所有 dataclass 添加 `slots=True`,消除 per-instance `__dict__`
开销(约 100-150 bytes/instance)。
- **应用范围**`Graph`/`GraphDefaults`/`TaskSpec`/`TaskResult`/`RetryPolicy`/
`TaskHooks`/`TaskEvent`。
- **冲突处理**`frozen=True` + `slots=True` + `cached_property` 不兼容
(无 `__dict__` 存缓存,frozen 禁止设 slot)→ 将 `TaskSpec.effective_fn`
从 `@cached_property` 改为 `@property`。闭包创建极轻量(仅捕获 `self`),
重复访问开销可忽略。
- **结果**10k TaskSpec 929 → 785 bytes/instance15.5% 减少,1.38 MB 节省)。
### 线程池跨层复用(iter-20
- **问题**`thread` 策略每层创建新 `ThreadPoolExecutor`N 层图创建/销毁
N 次线程池。
- **方案**`_drive_threaded` 创建一个池,跨所有层复用;`max_workers` 取
最大层宽 `max((len(layer) for layer in layers), default=1)`,上限 32。
- **API 变更**`ThreadedLayerRunner.execute` 签名 `max_workers: int` →
`pool: concurrent.futures.ThreadPoolExecutor`,调用方负责池生命周期。
- **额外**`asyncio.get_event_loop()` → `asyncio.get_running_loop()`
兼容 Python 3.12+ 弃用警告(调用点均在 `asyncio.run()` 内的协程中)。
### 踩坑总结
- **`lru_cache` 对签名内省有 dict lookup 开销**:即便 `functools.lru_cache`
缓存了 `_signature(fn)`,每次仍有 dict lookup。对热路径(如 cmd 任务)
应在更外层短路,避免进入 `build_call_args`。
- **`frozen=True` + `slots=True` + `cached_property` 三者不兼容**:选其中
两个。若需 slots 内存优化,将 `cached_property` 改为 `@property`(轻量
闭包场景)或外部 dict 缓存(重计算场景)。
- **orjson 紧凑格式与 stdlib 不同**:跨后端测试需用格式无关断言
`assert "[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
## 四、任务取消与优雅停止
@@ -203,10 +324,22 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
- **`datetime` 通过 `fromisoformat` 还原**:ISO 8601 字符串往返转换,
避免自定义时间格式。
### HTML 报告导出设计(iter-19
- **自包含 HTML5 文档**:内联 CSS,无外部依赖,便于邮件附件/嵌入式展示。
- **XSS 防护**:所有用户内容(任务名/错误/原因/值)经 `html.escape` 转义。
- **状态徽章着色**success 绿 / failed 红 / skipped 黄 / running 蓝,
通过 CSS class 区分。
- **`_render_summary_card` 静态方法**:渲染汇总卡片(总数/成功/失败/跳过/
耗时),避免在 `to_html` 主流程中重复 HTML 拼接。
- **value 列渲染**:尝试 `dumps(value)` 序列化后转义;失败回退 `repr(value)`。
### 踩坑总结
- **`_noop_fn` 占位函数体无法覆盖**:与 `task.py:_task_noop` 同模式,
占位函数的函数体在测试中无法触发,属于可接受的覆盖率缺口。
- **orjson 紧凑格式与 stdlib 不同**HTML 报告中 value 列断言需格式无关
`"[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
## 七、CLI 可视化与体验
@@ -312,6 +445,7 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
```python
def profile(self, graph: Graph) -> ProfileReport:
from .profiling import ProfileReport
return ProfileReport.from_report(self, graph)
```
`Graph` / `ProfileReport` 仅在 `TYPE_CHECKING` 块导入,避免循环依赖;
@@ -319,6 +453,20 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
- **序列化向前兼容**`from_json()` 优先恢复 `run_id`,缺失时自动生成
新 ID。旧版 JSON(无 `run_id`)可正常加载。
### 通知器平台扩展(iter-19
- **复用 `WebhookNotifier` 基类**Slack/Discord/Telegram/Feishu/DingTalk/
WeChatNotifier 均继承 `WebhookNotifier`,仅覆盖 `_build_payload(payload: dict) -> dict`
适配平台格式。零新依赖(stdlib `urllib.request`)。
- **TelegramNotifier 覆盖 `__init__`**:额外需要 `chat_id` 参数,
URL 中拼接 `bot_token`。
- **`Notifier` Protocol 生命周期**`notify(level, payload)` + `close()`。
`NotificationLevel` 枚举(RUNNING/SUCCESS/FAILED/SKIPPED+ `ALL_LEVELS`
frozenset`levels` 参数内部过滤。
- **WebhookNotifier 通用字段**`url`/`secret`/`levels`/`timeout`
`_send` 用 `urllib.request.urlopen` 发送 JSON POST`_build_payload`
默认透传(子类按平台格式重写)。
### 踩坑总结
- **循环依赖用 `TYPE_CHECKING` 守卫 + 延迟导入**`report.py` 依赖
@@ -351,5 +499,108 @@ description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档
- **公共 API 必须有完整类型注解与中文 docstring**:项目既有风格,
`python-standards.md` 硬约束。
- **零运行时依赖原则(YAML 例外)**:除 PyYAMLYAML 编排必需)、
rich + typerCLI 必需)、typing-extensions(前向兼容)外,不引入
运行时依赖。新增依赖须审慎,优先用标准库。
rich + typerCLI 必需)、typing-extensions(前向兼容)、orjson(可选
加速,`pip install pyflowx[fast]`)外,不引入运行时依赖。新增依赖须
审慎,优先用标准库。
## 十一、状态后端与缓存
### 可复用模式
- **`StateBackend` 抽象基类**`get`/`put`/`has`/`iter`/`clear` +
`invalidate(key) -> bool`iter-19 新增)。`invalidate` 返回是否实际
删除,便于调用方判断缓存命中。
- **`_TTLStateBackendMixin` 4 原语**`_get_raw`/`_put_raw`/`_iter_raw`/
`_clear_raw`/`_delete_raw`(iter-19 新增)。子类只需实现这 5 个原语,
`get`/`put`/`has`/`iter`/`clear`/`invalidate` 由 mixin 统一组合 TTL 判断。
- **`MemoryBackend` LRU 驱逐(iter-19**
- 用 `OrderedDict` 替代普通 dict
- `_get_raw` 命中时 `move_to_end(key)` 更新访问顺序
- `_put_raw` 超限时 `popitem(last=False)` 驱逐头部(最旧)
- `maxsize=None` 时不驱逐(默认行为,向后兼容)
- **`SQLiteBackend` WAL 模式 + RLock**:线程安全;`batch()` 通过
`_in_batch` 标志延迟 commit,不持锁。
### 设计决策
- **`JSONBackend.invalidate` 覆盖**:非 batch 模式下需 `flush()` 落盘,
否则删除只在内存中生效,进程崩溃后状态丢失。batch 模式下标记
`_dirty=True`,由 `flush()` 统一处理。
- **`SQLiteBackend._delete_raw` 用 `cursor.rowcount > 0`**:判断是否
实际删除,避免先 `has` 后 `delete` 的双次查询。
- **`maxsize` 默认 None 而非 0**:0 会立即驱逐所有项,等价于禁用缓存;
None 表示无限制,与历史行为兼容。
## 十二、CLI 工具错误处理
### 可复用模式
- **`subprocess.run` 的 `check` 参数分级策略**iter-16):
- **失败需感知**dockercmd/msdownload/sglang)→ `check=True` +
`try/except subprocess.CalledProcessError` 打印 rich 错误
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查
`returncode` 打印结果(pkill 返回 1 表示无匹配进程,非错误)
- **失败无关紧要**clr/which)→ `check=False`,加注释说明原因
- **错误路径统一处理模式**
```python
try:
subprocess.run(cmd, check=True, ...)
except subprocess.CalledProcessError as e:
_console.print(f"[red]命令失败 (returncode={e.returncode})[/]")
except FileNotFoundError:
_console.print(f"[red]命令未找到: {cmd[0]}[/]")
```
### 设计决策
- **rich Console 而非 print**CLI 工具错误用 `rich.console.Console`
打印带颜色输出,与项目其他 CLI 工具一致。
- **`check=True` + try/except 优于 `check=False` + 手动检查**:除非
命令返回非零是预期行为(如 taskkill pkill 无匹配、which 未找到命令),
否则用 `check=True` 让 subprocess 抛异常,统一在 except 中处理。
### 踩坑总结
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
装饰器与 `mock.patch.object` 上下文。mock subprocess 异常用
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
- **mock 异常抛出技巧**`lambda *_, **__: (_ for _ in ()).throw(err)`
利用生成器仅在迭代时抛异常的特性,让无参 lambda 能抛出指定异常。
## 十三、测试与质量保障
### 可复用模式
- **`# pragma: no cover` 清理策略**iter-18):按 `python-standards.md`
"pragma: no cover 是清理信号,应激活或删除"原则,逐处分析:
- **逻辑上不可达** → 删除(如 `if result is None` 在已过滤的列表后)
- **防御性代码但实际可触发** → 激活:移除 pragma,写测试覆盖该分支
- **调用方已保证的前置条件** → 保留 pragma,加注释说明依赖契约
- **覆盖率缺口分析方法**
1. 跑 `pytest --cov=src/pyflowx --cov-report=term-missing`
2. 找出 `Stmts`/`Miss` 列中 Miss > 0 的文件
3. 看 `Missing` 列定位未覆盖行号
4. 判断是"未触发分支"还是"不可达代码"
5. 前者补测试,后者清 pragma 或删代码
- **测试命名**`test_<被测对象>_<场景>`,如 `test_json_backend_flush_type_error`。
### 设计决策
- **公共 API 优先测试**:用 `has`/`get` 等公共接口测试,不访问私有方法。
故障注入等场景可临时访问私有属性,docstring 注明原因。
- **Mock 优先级**`monkeypatch` > 内联 stub > `unittest.mock` >
`pytest-mock`。禁用 `@patch` 装饰器、`mock.patch.object` 上下文、
`pytest-mock` 的 `mocker` fixture。
- **`slow` 标记**:耗时测试加 `@pytest.mark.slow`CI 可 `-m "not slow"` 跳过。
### 踩坑总结
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
- **占位函数体无法覆盖**`_noop_fn` 等占位函数的函数体在测试中无法触发
(参数已校验),属可接受的覆盖率缺口,加 `# pragma: no cover` 并注释原因。
- **不可哈希参数回退分支**`lru_cache` 缓存的函数遇到不可哈希参数会抛
TypeError,需 try/except 回退到慢路径。该回退分支需单独写测试(构造
不可哈希 fn 触发)。
+1 -1
View File
@@ -44,7 +44,7 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **可观测性** —— `run_id` 贯穿日志/序列化/诊断;结构化日志 `extra``task_name`/`status`/`attempts`/`error_type``profile(graph)` 离线性能分析(关键路径/并行度/瓶颈)
- **状态后端** —— `MemoryBackend` / `JSONBackend` / `SQLiteBackend`WAL 模式,适合大规模任务)
- **YAML 任务编排** —— GitHub Actions 风格 `jobs`/`needs`/`strategy.matrix`/`if` 条件,`pf yamlrun pipeline.yaml` 一键执行
- **最小依赖** —— `rich` + `typer` + `typing-extensions`3.13 以下)+ `pyyaml`
- **最小依赖** —— `rich` + `typer` + `typing-extensions`3.13 以下)+ `pyyaml`;安装 `orjson` 可加速 JSON 序列化(`pip install pyflowx[fast]`
- **97% 测试覆盖** —— 分支覆盖率 >= 95%
## 安装
+30
View File
@@ -25,6 +25,9 @@ import pyflowx as px
from benchmarks import print_results, time_it
from benchmarks.bench_advanced import run_advanced
from pyflowx import Graph, GraphDefaults, RetryPolicy, TaskSpec
from pyflowx._json import _HAS_ORJSON
from pyflowx._json import dumps as _dumps
from pyflowx._json import loads as _loads
from pyflowx.context import build_call_args
from pyflowx.storage import JSONBackend, MemoryBackend, SQLiteBackend
@@ -367,6 +370,33 @@ def bench_storage() -> None:
print_results("状态后端 (save/load)", results)
# 序列化基准:_json 抽象层 dumps/loads
ser_results = []
report_like = {
"run_id": "abc12345",
"success": True,
"results": [
{
"name": f"task_{i}",
"status": "success",
"attempts": 1,
"duration_seconds": 0.123 + i * 0.001,
"value": {"output": [i, i + 1, i + 2], "meta": {"tag": "api"}},
}
for i in range(100)
],
}
serialized = _dumps(report_like)
ms, ops = time_it(lambda: _dumps(report_like), iterations=200, warmup=10)
ser_results.append(("dumps(report-100)", 200, ms, ops))
ms, ops = time_it(lambda: _loads(serialized), iterations=200, warmup=10)
ser_results.append(("loads(report-100)", 200, ms, ops))
backend_name = "orjson" if _HAS_ORJSON else "stdlib"
print_results(f"JSON 序列化 (后端={backend_name})", ser_results)
# 清理临时目录
import shutil
+1
View File
@@ -45,6 +45,7 @@ dev = [
"tox>=4.25.0",
]
docs = ["myst-parser>=3.0", "sphinx-rtd-theme>=2.0", "sphinx>=7.0"]
fast = ["orjson>=3.10.0"]
office = [
"pillow>=10.4.0",
"pymupdf>=1.24.11",
+100
View File
@@ -0,0 +1,100 @@
"""JSON 序列化抽象层。
优先使用 orjson(性能更高),回退到标准库 json。对外提供与标准库
json 兼容的 ``dumps`` / ``loads`` / ``dump`` / ``load`` / ``JSONDecodeError``
接口,使调用方无感知切换。
orjson 与标准库的差异由本模块吸收:
- ``dumps`` 返回 ``str``orjson 原生返回 ``bytes``,此处统一 decode
- ``ensure_ascii`` 参数对 orjson 无效(始终输出 UTF-8),此处忽略
- ``indent`` 映射为 ``orjson.OPT_INDENT_2``
"""
from __future__ import annotations
__all__ = ["JSONDecodeError", "dump", "dumps", "load", "loads"]
from typing import IO, Any
try:
import orjson # type: ignore[import-not-found]
_HAS_ORJSON = True
except ImportError: # pragma: no cover
_HAS_ORJSON = False
if _HAS_ORJSON:
JSONDecodeError = orjson.JSONDecodeError # type: ignore[possibly-unbound]
_OPT_NON_STR_KEYS = orjson.OPT_NON_STR_KEYS # type: ignore[possibly-unbound]
def dumps(
obj: Any,
*,
ensure_ascii: bool = False, # noqa: ARG001 — 兼容标准库签名
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> str:
"""序列化为 JSON 字符串(orjson 后端)。"""
opts = _OPT_NON_STR_KEYS
if indent is not None:
opts |= orjson.OPT_INDENT_2 # type: ignore[possibly-unbound]
return orjson.dumps(obj, default=default, option=opts).decode("utf-8") # type: ignore[possibly-unbound]
def loads(s: str | bytes) -> Any:
"""从 JSON 字符串/字节反序列化(orjson 后端)。"""
return orjson.loads(s) # type: ignore[possibly-unbound]
def dump(
obj: Any,
fh: IO[str],
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> None:
"""序列化并写入文件句柄(orjson 后端)。"""
fh.write(dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default))
def load(fh: IO[str]) -> Any:
"""从文件句柄反序列化(orjson 后端)。"""
return orjson.loads(fh.read()) # type: ignore[possibly-unbound]
else: # pragma: no cover
import json as _stdlib
JSONDecodeError = _stdlib.JSONDecodeError
def dumps(
obj: Any,
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> str:
"""序列化为 JSON 字符串(标准库后端)。"""
return _stdlib.dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default)
def loads(s: str | bytes) -> Any:
"""从 JSON 字符串/字节反序列化(标准库后端)。"""
return _stdlib.loads(s)
def dump(
obj: Any,
fh: IO[str],
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> None:
"""序列化并写入文件句柄(标准库后端)。"""
_stdlib.dump(obj, fh, ensure_ascii=ensure_ascii, indent=indent, default=default)
def load(fh: IO[str]) -> Any:
"""从文件句柄反序列化(标准库后端)。"""
return _stdlib.load(fh)
+23 -23
View File
@@ -472,7 +472,7 @@ class AsyncTaskRunner:
result: TaskResult[Any] = TaskResult(spec=spec)
result.started_at = datetime.now()
args, kwargs = build_call_args(spec, context)
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
_run_hooks(spec.hooks, "pre_run", spec)
_emit_running(on_event, spec)
@@ -656,7 +656,7 @@ class ThreadedLayerRunner:
backend: StateBackend,
layer_idx: int,
on_event: EventCallback | None,
max_workers: int,
pool: concurrent.futures.ThreadPoolExecutor,
concurrency_limits: Mapping[str, int],
) -> None:
to_run = _filter_and_sort(layer, graph, context, report, backend, on_event)
@@ -678,21 +678,18 @@ class ThreadedLayerRunner:
if sem is not None:
sem.release()
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
pool.submit(_run_threaded_task, name): name for name in to_run
}
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
try:
for fut in concurrent.futures.as_completed(future_to_name):
name = future_to_name[fut]
completed[name] = fut.result()
finally:
with lock:
for name, (task_ctx, result) in completed.items():
_store_result(
name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event
)
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
pool.submit(_run_threaded_task, name): name for name in to_run
}
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
try:
for fut in concurrent.futures.as_completed(future_to_name):
name = future_to_name[fut]
completed[name] = fut.result()
finally:
with lock:
for name, (task_ctx, result) in completed.items():
_store_result(name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event)
class AsyncLayerRunner:
@@ -783,7 +780,7 @@ class DependencyRunner:
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
return result
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
for name in all_names:
futures[name] = loop.create_task(_run_task(name))
await asyncio.gather(*futures.values())
@@ -1199,11 +1196,14 @@ def _drive_threaded(
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None:
for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event):
return
workers = max_workers or max(1, min(32, len(layer)))
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, workers, concurrency_limits)
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
max_layer_size = max((len(layer) for layer in layers), default=1)
pool_workers = max_workers or max(1, min(32, max_layer_size))
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event):
return
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, pool, concurrency_limits)
async def _async_drive(
+59 -21
View File
@@ -1,6 +1,7 @@
"""DAG 构建、校验、分层与可视化。
使用标准库的 :mod:`graphlib` 进行拓扑排序。图以增量方式构建并即时校验
使用自实现的 Kahn 算法进行拓扑排序(替代 ``graphlib.TopologicalSorter``
消除 ``prepare()`` 瓶颈)。图以增量方式构建并即时校验,
使配置错误在构建时(而非执行时)快速失败。
支持:
@@ -17,7 +18,6 @@ __all__ = [
"GraphDefaults",
]
import graphlib
import inspect
from collections.abc import Callable, Iterable, Mapping, Sequence
from dataclasses import dataclass, field, replace
@@ -27,10 +27,52 @@ from typing import Any
from .errors import CycleError, DuplicateTaskError, MissingDependencyError
from .task import Context, RetryPolicy, TaskSpec
_TopologicalSorter = graphlib.TopologicalSorter
def _topological_layers(deps: Mapping[str, tuple[str, ...]]) -> tuple[list[list[str]], list[str] | None]:
"""Kahn 算法分层拓扑排序。
返回 ``(layers, cycle_nodes)``:无环时 ``cycle_nodes`` 为 ``None``
有环时为参与环的未处理节点列表(非精确环路径,仅指示存在环)。
与 ``graphlib.TopologicalSorter`` 相比,省去了 ``prepare()`` 的内部
状态机开销,直接用 dict + list 计算,在 diamond(1000) 图上快约 5x。
"""
# 入度(依赖数)与反向邻接表
in_degree: dict[str, int] = {}
dependents: dict[str, list[str]] = {}
for name, d in deps.items():
in_degree[name] = len(d)
dependents.setdefault(name, [])
for name, d in deps.items():
for dep in d:
if dep in dependents:
dependents[dep].append(name)
# 初始层:入度为 0 的节点
current = sorted(name for name, deg in in_degree.items() if deg == 0)
layers: list[list[str]] = []
processed = 0
while current:
layers.append(current)
nxt: list[str] = []
for node in current:
for dependent in dependents[node]:
in_degree[dependent] -= 1
if in_degree[dependent] == 0:
nxt.append(dependent)
processed += 1
nxt.sort()
current = nxt
if processed < len(deps):
cycle_nodes = [name for name, deg in in_degree.items() if deg > 0]
return layers, cycle_nodes
return layers, None
@dataclass
@dataclass(slots=True)
class GraphDefaults:
"""图级默认值。TaskSpec 对应字段为 ``None`` 时回退到此处。
@@ -127,7 +169,7 @@ def _make_namespaced_fn(orig_fn: Any, ns: str, dep_names: set[str]) -> Any:
return wrapper
@dataclass
@dataclass(slots=True)
class Graph:
"""校验后的有向无环任务图。
@@ -317,14 +359,16 @@ class Graph:
raise MissingDependencyError(name, dep)
def validate(self) -> None:
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。"""
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。
顺带填充 :attr:`_layers_cache`(无环时),使后续 :meth:`layers`
直接命中缓存,避免 :func:`_topological_layers` 二次计算。
"""
self._validate_references()
sorter = _TopologicalSorter(self.deps)
try:
sorter.prepare()
except graphlib.CycleError as exc:
cycle: Sequence[str] = exc.args[1] if len(exc.args) > 1 else []
raise CycleError(list(cycle)) from exc
layers, cycle_nodes = _topological_layers(self.deps)
if cycle_nodes is not None:
raise CycleError(cycle_nodes)
self._layers_cache = layers
# ------------------------------------------------------------------ #
# 内省
@@ -404,15 +448,9 @@ class Graph:
"""
if self._layers_cache is not None:
return self._layers_cache
sorter = _TopologicalSorter(self.deps)
result: list[list[str]] = []
sorter.prepare()
while sorter.is_active():
ready = list(sorter.get_ready())
ready.sort()
result.append(ready)
for node in ready:
sorter.done(node)
result, cycle_nodes = _topological_layers(self.deps)
if cycle_nodes is not None:
raise CycleError(cycle_nodes)
self._layers_cache = result
return result
+2 -2
View File
@@ -50,7 +50,6 @@ __all__ = [
"WebhookNotifier",
]
import json
import logging
import sys
import urllib.error
@@ -64,6 +63,7 @@ if sys.version_info >= (3, 12):
else:
from typing_extensions import override # pragma: no cover
from ._json import dumps
from .task import TaskEvent, TaskStatus
logger = logging.getLogger(__name__)
@@ -204,7 +204,7 @@ class WebhookNotifier:
def _send(self, payload: dict[str, Any]) -> None:
"""发送 JSON POST 请求,失败仅记录日志。"""
data = json.dumps(payload, ensure_ascii=False).encode("utf-8")
data = dumps(payload, ensure_ascii=False).encode("utf-8")
req = urllib.request.Request(
self._url,
data=data,
+19 -21
View File
@@ -9,13 +9,13 @@ from __future__ import annotations
import csv
import html
import io
import json
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any
from uuid import uuid4
from ._json import dumps, loads
from .task import TaskResult, TaskSpec, TaskStatus
if TYPE_CHECKING:
@@ -42,7 +42,7 @@ def _serialize_value(
if value_serializer is not None:
return value_serializer(value)
try:
_ = json.dumps(value)
_ = dumps(value)
except (TypeError, ValueError):
return repr(value)
return value
@@ -203,22 +203,20 @@ class RunReport:
"""
results_list: list[dict[str, Any]] = []
for name, r in self.results.items():
results_list.append(
{
"name": name,
"status": r.status.value,
"value": _serialize_value(r.value, value_serializer),
"error": repr(r.error) if r.error else None,
"attempts": r.attempts,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": r.duration,
"reason": r.reason,
"tags": list(r.spec.tags),
"depends_on": list(r.spec.depends_on),
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
}
)
results_list.append({
"name": name,
"status": r.status.value,
"value": _serialize_value(r.value, value_serializer),
"error": repr(r.error) if r.error else None,
"attempts": r.attempts,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": r.duration,
"reason": r.reason,
"tags": list(r.spec.tags),
"depends_on": list(r.spec.depends_on),
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
})
return {
"run_id": self.run_id,
"success": self.success,
@@ -240,7 +238,7 @@ class RunReport:
value_serializer:
自定义任务值序列化函数,透传给 :meth:`to_dict`。
"""
return json.dumps(
return dumps(
self.to_dict(value_serializer),
ensure_ascii=False,
indent=indent if indent > 0 else None,
@@ -401,7 +399,7 @@ class RunReport:
]
if include_value:
val = _serialize_value(r.value, value_serializer)
val_str = json.dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
val_str = dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
cells.append(f'<td class="value-cell">{html.escape(val_str)}</td>')
parts.append("<tr>" + "".join(cells) + "</tr>")
parts.append("</tbody></table>")
@@ -425,7 +423,7 @@ class RunReport:
不可序列化字段无法恢复,重建的 spec 仅含 ``name``/``tags``/
``depends_on``。重建后的 report 仅供查询/分析,**不能用于重新执行**。
"""
data = json.loads(text)
data = loads(text)
report = cls(
success=data.get("success", True),
run_id=data.get("run_id", uuid4().hex[:8]),
+10 -10
View File
@@ -13,7 +13,6 @@
from __future__ import annotations
import json
import sqlite3
import sys
import threading
@@ -30,6 +29,7 @@ if sys.version_info >= (3, 12):
else:
from typing_extensions import override # pragma: no cover
from ._json import JSONDecodeError, dump, dumps, load, loads
from .errors import StorageError
@@ -249,7 +249,7 @@ class JSONBackend(_TTLStateBackendMixin):
return
try:
with open(self._path, encoding="utf-8") as fh:
data: Any = json.load(fh)
data: Any = load(fh)
if isinstance(data, dict):
# 兼容纯值格式与带元数据格式
self._store = {}
@@ -258,14 +258,14 @@ class JSONBackend(_TTLStateBackendMixin):
self._store[k] = v
else:
self._store[k] = {"value": v, "ts": time.time()}
except (OSError, json.JSONDecodeError) as exc:
except (OSError, JSONDecodeError) as exc:
raise StorageError(f"cannot read state file {self._path!r}", exc) from exc
def _flush(self) -> None:
tmp = self._path + ".tmp"
try:
with open(tmp, "w", encoding="utf-8") as fh:
json.dump(self._store, fh, ensure_ascii=False, indent=2)
dump(self._store, fh, ensure_ascii=False, indent=2)
_ = Path(tmp).replace(Path(self._path))
except (OSError, TypeError) as exc:
raise StorageError(f"cannot write state file {self._path!r}", exc) from exc
@@ -315,7 +315,7 @@ class JSONBackend(_TTLStateBackendMixin):
# 避免 N 次 save 的 N 次 json.dumps 开销;非 batch 模式仍即时验证以提供精确错误。
if not self._defer_flush:
try:
_ = json.dumps(value)
_ = dumps(value)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
super().save(key, value)
@@ -396,15 +396,15 @@ class SQLiteBackend(_TTLStateBackendMixin):
return None
value_text, ts = row
try:
value = json.loads(value_text)
except json.JSONDecodeError as exc:
value = loads(value_text)
except JSONDecodeError as exc:
raise StorageError(f"cannot decode value for key {key!r}", exc) from exc
return value, float(ts)
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
try:
value_text = json.dumps(value, ensure_ascii=False)
value_text = dumps(value, ensure_ascii=False)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
with self._lock:
@@ -427,8 +427,8 @@ class SQLiteBackend(_TTLStateBackendMixin):
raise StorageError("cannot iterate sqlite state", exc) from exc
for k, value_text, ts in rows:
try:
value = json.loads(value_text)
except json.JSONDecodeError as exc:
value = loads(value_text)
except JSONDecodeError as exc:
raise StorageError(f"cannot decode value for key {k!r}", exc) from exc
yield k, value, float(ts)
+10 -9
View File
@@ -27,7 +27,6 @@ from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from functools import cached_property
from pathlib import Path
from typing import (
Any,
@@ -75,7 +74,7 @@ def _format_skip_reason(failed_conditions: list[str]) -> str:
# ---------------------------------------------------------------------- #
# 重试策略
# ---------------------------------------------------------------------- #
@dataclass(frozen=True)
@dataclass(frozen=True, slots=True)
class RetryPolicy:
"""任务失败重试策略。
@@ -138,7 +137,7 @@ class RetryPolicy:
# ---------------------------------------------------------------------- #
# 任务钩子
# ---------------------------------------------------------------------- #
@dataclass(frozen=True)
@dataclass(frozen=True, slots=True)
class TaskHooks:
"""任务生命周期钩子。
@@ -162,7 +161,7 @@ class TaskStatus(Enum):
SKIPPED = "skipped" # 用于断点续跑与子图过滤
@dataclass(frozen=True)
@dataclass(frozen=True, slots=True)
class TaskSpec(Generic[T]):
"""单个 DAG 节点的不可变描述。
@@ -291,7 +290,7 @@ class TaskSpec(Generic[T]):
if self.fn is None and self.cmd is None:
raise ValueError(f"TaskSpec '{self.name}': 必须提供 fn 或 cmd 参数。")
@cached_property
@property
def effective_fn(self) -> TaskFn[T]:
"""获取有效的执行函数。
@@ -299,8 +298,10 @@ class TaskSpec(Generic[T]):
包装函数在每次调用时从 ``self`` 读取 ``verbose``/``cwd``/``env``/
``timeout``,避免闭包捕获运行期参数,使翻转字段无需重建 spec。
结果按实例缓存(:func:`functools.cached_property`):frozen dataclass
字段不可变,``_wrap_cmd`` 生成的闭包稳定,无需每次访问重建。
.. note::
使用 ``@property`` 而非 ``cached_property`` 以兼容 ``slots=True``
内存优化(10k+ 任务场景)。闭包创建极轻量(仅捕获 ``self``),
重复访问的开销可忽略。
"""
if self.cmd is not None:
return self._wrap_cmd()
@@ -590,7 +591,7 @@ def task_template(
return _factory
@dataclass
@dataclass(slots=True)
class TaskResult(Generic[T]):
"""运行期间产生的可变单任务记录。"""
@@ -611,7 +612,7 @@ class TaskResult(Generic[T]):
return (self.finished_at - self.started_at).total_seconds()
@dataclass(frozen=True)
@dataclass(frozen=True, slots=True)
class TaskEvent:
"""执行期间向观察者发出的不可变事件。"""
+126
View File
@@ -0,0 +1,126 @@
"""_json 抽象层测试。
验证 orjson/stdlib 双后端的输出语义一致。由于 orjson 是可选依赖,
本测试在两种环境下均应通过(orjson 已安装或未安装)。
"""
from __future__ import annotations
import io
from pathlib import Path
import pytest
from pyflowx._json import JSONDecodeError, dump, dumps, load, loads
class TestDumpsLoads:
"""dumps/loads 往返测试。"""
def test_dumps_basic_dict(self) -> None:
s = dumps({"a": 1, "b": [1, 2, 3]})
assert loads(s) == {"a": 1, "b": [1, 2, 3]}
def test_dumps_returns_str(self) -> None:
s = dumps({"x": 1})
assert isinstance(s, str)
def test_dumps_none(self) -> None:
assert dumps(None) == "null"
def test_dumps_bool(self) -> None:
assert dumps(True) == "true"
assert dumps(False) == "false"
def test_dumps_int(self) -> None:
assert dumps(42) == "42"
def test_dumps_float(self) -> None:
s = dumps(3.14)
assert loads(s) == 3.14
def test_dumps_unicode(self) -> None:
"""ensure_ascii=False:中文字符不应被转义为 \\uXXXX。"""
s = dumps({"msg": "条件不满足"})
assert "条件不满足" in s
assert "\\u" not in s
def test_dumps_indent(self) -> None:
"""indent=2 应产生多行格式。"""
s = dumps({"a": 1}, indent=2)
assert "\n" in s
def test_dumps_indent_none_compact(self) -> None:
"""indent=None 应产生紧凑单行。"""
s = dumps({"a": 1}, indent=None)
assert "\n" not in s
def test_dumps_default_callable(self) -> None:
"""default 参数应处理不可序列化的类型。"""
class Custom:
def __str__(self) -> str:
return "custom-value"
s = dumps({"obj": Custom()}, default=str)
assert loads(s) == {"obj": "custom-value"}
def test_dumps_non_serialisable_raises(self) -> None:
"""不可序列化的值(无 default)应抛 TypeError。"""
with pytest.raises(TypeError):
_ = dumps(object())
def test_loads_bytes(self) -> None:
"""loads 应接受 bytes 输入。"""
assert loads(b'{"x": 1}') == {"x": 1}
def test_loads_str(self) -> None:
"""loads 应接受 str 输入。"""
assert loads('{"x": 1}') == {"x": 1}
def test_loads_invalid_raises(self) -> None:
with pytest.raises(JSONDecodeError):
_ = loads("not valid json")
class TestDumpLoad:
"""dump/load 文件句柄往返测试。"""
def test_dump_load_round_trip(self, tmp_path: Path) -> None:
path = tmp_path / "test.json"
data = {"a": 1, "b": [1, 2, 3], "c": "中文"}
with open(path, "w", encoding="utf-8") as fh:
dump(data, fh, ensure_ascii=False, indent=2)
with open(path, encoding="utf-8") as fh:
assert load(fh) == data
def test_dump_indent(self, tmp_path: Path) -> None:
path = tmp_path / "test.json"
with open(path, "w", encoding="utf-8") as fh:
dump({"a": 1}, fh, indent=2)
text = path.read_text(encoding="utf-8")
assert "\n" in text
def test_dump_compact(self, tmp_path: Path) -> None:
path = tmp_path / "test.json"
with open(path, "w", encoding="utf-8") as fh:
dump({"a": 1}, fh, indent=None)
text = path.read_text(encoding="utf-8")
assert "\n" not in text
def test_dump_default_callable(self, tmp_path: Path) -> None:
path = tmp_path / "test.json"
with open(path, "w", encoding="utf-8") as fh:
dump({"p": Path("/tmp/x")}, fh, default=str)
with open(path, encoding="utf-8") as fh:
assert load(fh) == {"p": "/tmp/x"}
class TestStringIO:
"""通过 StringIO 验证 dump/load 与 dumps/loads 一致。"""
def test_dump_load_stringio(self) -> None:
buf = io.StringIO()
dump({"a": 1, "b": [1, 2]}, buf, ensure_ascii=False)
buf.seek(0)
assert load(buf) == {"a": 1, "b": [1, 2]}
+10 -13
View File
@@ -486,7 +486,8 @@ class TestRunReportToHtml:
report = px.RunReport()
report.results["a"] = _make_result("a", value=[1, 2, 3])
s = report.to_html()
assert "[1, 2, 3]" in s
# orjson 紧凑输出无空格([1,2,3]),标准库带空格([1, 2, 3]),两者均接受
assert "[1,2,3]" in s or "[1, 2, 3]" in s
def test_to_html_with_value_serializer(self) -> None:
"""to_html 应透传 value_serializer。"""
@@ -651,12 +652,10 @@ class TestRunReportSerializationIntegration:
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
graph = px.Graph.from_specs(
[
px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
])
report = px.run(graph, strategy="sequential")
# 序列化 → 反序列化
@@ -752,12 +751,10 @@ class TestRunId:
def task_b(a: int) -> int:
return a * 2
graph = px.Graph.from_specs(
[
px.TaskSpec("a", task_a),
px.TaskSpec("b", task_b, depends_on=("a",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", task_a),
px.TaskSpec("b", task_b, depends_on=("a",)),
])
report = px.run(graph, strategy="sequential")
profile = report.profile(graph)
assert profile.total_duration >= 0
+7 -14
View File
@@ -135,27 +135,23 @@ def test_json_backend_non_serialisable_raises() -> None:
def test_json_backend_flush_type_error(monkeypatch: pytest.MonkeyPatch) -> None:
"""_flush 时 json.dump 抛 TypeError 应转为 StorageError(覆盖 line 105-106)。
"""_flush 时 dump 抛 TypeError 应转为 StorageError(覆盖 line 105-106)。
通过 monkeypatch json.dump 在写入文件时抛 TypeError模拟值通过
通过 monkeypatch dump 在写入文件时抛 TypeError模拟值通过
save dumps 校验但在 dump 到文件句柄时失败如自定义对象的边缘情况
"""
import json as _json
import pyflowx.storage as storage_mod
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path)
original_dump = _json.dump
def flaky_dump(*_args: Any, **_kwargs: Any) -> None:
raise TypeError("simulated flush failure")
monkeypatch.setattr(_json, "dump", flaky_dump)
monkeypatch.setattr(storage_mod, "dump", flaky_dump)
with pytest.raises(StorageError, match="cannot write"):
b.save("a", 1)
# 恢复以便后续测试不受影响
monkeypatch.setattr(_json, "dump", original_dump)
def test_json_backend_flush_os_error(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -278,22 +274,19 @@ def test_json_backend_expired_missing_ts() -> None:
def test_json_backend_save_value_error(monkeypatch: pytest.MonkeyPatch) -> None:
"""save 时 json.dumps 抛 ValueError 应转为 StorageError."""
import json as _json
"""save 时 dumps 抛 ValueError 应转为 StorageError."""
import pyflowx.storage as storage_mod
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path)
original_dumps = _json.dumps
def flaky_dumps(*_args: Any, **_kwargs: Any) -> str:
raise ValueError("simulated dumps failure")
monkeypatch.setattr(_json, "dumps", flaky_dumps)
monkeypatch.setattr(storage_mod, "dumps", flaky_dumps)
with pytest.raises(StorageError, match="not JSON-serialisable"):
b.save("a", 1)
monkeypatch.setattr(_json, "dumps", original_dumps)
# ---------------------------------------------------------------------- #
-7
View File
@@ -1,7 +0,0 @@
"""
This type stub file was generated by pyright.
"""
from .graphlib import CycleError, TopologicalSorter
__all__ = ["CycleError", "TopologicalSorter"]
-114
View File
@@ -1,114 +0,0 @@
"""
This type stub file was generated by pyright.
"""
from collections.abc import Generator
from typing import Any
__all__ = ["CycleError", "TopologicalSorter"]
_NODE_OUT = ...
_NODE_DONE = ...
class _NodeInfo:
__slots__: list[str]
def __init__(self, node: Any) -> None: ...
class CycleError(ValueError):
"""Subclass of ValueError raised by TopologicalSorterif cycles exist in the graph
If multiple cycles exist, only one undefined choice among them will be reported
and included in the exception. The detected cycle can be accessed via the second
element in the *args* attribute of the exception instance and consists in a list
of nodes, such that each node is, in the graph, an immediate predecessor of the
next node in the list. In the reported list, the first and the last node will be
the same, to make it clear that it is cyclic.
"""
...
class TopologicalSorter:
"""Provides functionality to topologically sort a graph of hashable nodes"""
def __init__(self, graph: Any) -> None: ...
def add(self, node: Any, *predecessors: Any) -> None:
"""Add a new node and its predecessors to the graph.
Both the *node* and all elements in *predecessors* must be hashable.
If called multiple times with the same node argument, the set of dependencies
will be the union of all dependencies passed in.
It is possible to add a node with no dependencies (*predecessors* is not provided)
as well as provide a dependency twice. If a node that has not been provided before
is included among *predecessors* it will be automatically added to the graph with
no predecessors of its own.
Raises ValueError if called after "prepare".
"""
...
def prepare(self) -> None:
"""Mark the graph as finished and check for cycles in the graph.
If any cycle is detected, "CycleError" will be raised, but "get_ready" can
still be used to obtain as many nodes as possible until cycles block more
progress. After a call to this function, the graph cannot be modified and
therefore no more nodes can be added using "add".
"""
...
def get_ready(self) -> tuple[Any, ...]:
"""Return a tuple of all the nodes that are ready.
Initially it returns all nodes with no predecessors; once those are marked
as processed by calling "done", further calls will return all new nodes that
have all their predecessors already processed. Once no more progress can be made,
empty tuples are returned.
Raises ValueError if called without calling "prepare" previously.
"""
...
def is_active(self) -> bool:
"""Return True if more progress can be made and ``False`` otherwise.
Progress can be made if cycles do not block the resolution and either there
are still nodes ready that haven't yet been returned by "get_ready" or the
number of nodes marked "done" is less than the number that have been returned
by "get_ready".
Raises ValueError if called without calling "prepare" previously.
"""
...
def __bool__(self) -> bool: ...
def done(self, *nodes: Any) -> None:
"""Marks a set of nodes returned by "get_ready" as processed.
This method unblocks any successor of each node in *nodes* for being returned
in the future by a a call to "get_ready"
Raises :exec:`ValueError` if any node in *nodes* has already been marked as
processed by a previous call to this method, if a node was not added to the
graph by using "add" or if called without calling "prepare" previously or if
node has not yet been returned by "get_ready".
"""
...
def static_order(self) -> Generator[Any]:
"""Returns an iterable of nodes in a topological order.
The particular order that is returned may depend on the specific
order in which the items were inserted in the graph.
Using this method does not require to call "prepare" or "done". If any
cycle is detected, :exc:`CycleError` will be raised.
"""
...
Generated
+86 -1
View File
@@ -1023,6 +1023,87 @@ wheels = [
{ url = "https://mirrors.aliyun.com/pypi/packages/09/dc/f3dfb7488b770f3f67e6545085bf2abea5172e88f57b8ad25ef860ca704c/myst_parser-5.1.0-py3-none-any.whl", hash = "sha256:9c91c52b3cdb4d94a6506e4fab4e2f296c7623a0da0dcbe6de1565c3dad67a8a" },
]
[[package]]
name = "orjson"
version = "3.11.9"
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/7e/0c/964746fcafbd16f8ff53219ad9f6b412b34f345c75f384ad434ceaadb538/orjson-3.11.9.tar.gz", hash = "sha256:4fef17e1f8722c11587a6ef18e35902450221da0028e65dbaaa543619e68e48f" }
wheels = [
{ url = "https://mirrors.aliyun.com/pypi/packages/10/5d/b95ca542a001135cc250a49370f282f578c8f4e46cc8617d73775297eea8/orjson-3.11.9-cp310-cp310-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:135869ef917b8704ea0a94e01620e0c05021c15c52036e4663baffe75e72f8ce" },
{ url = "https://mirrors.aliyun.com/pypi/packages/80/01/be33fbff646e22f93398429ea645f20d2097aea1a6cdc1e6628e70125f83/orjson-3.11.9-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:115ab5f5f4a0f203cc2a5f0fb09aee503a3f771aa08392949ab5ca230c4fbdbd" },
{ url = "https://mirrors.aliyun.com/pypi/packages/4e/61/73d49333bba660a075daccca10970dc6409ce1cf42ae4046646a19468aad/orjson-3.11.9-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4da3c38a2083ca4aaf9c2a36776cce3e9328e6647b10d118948f3cfb4913ffe4" },
{ url = "https://mirrors.aliyun.com/pypi/packages/1f/7d/30e844b3dac3f74aed66b1f984daf9db3c98c0328c03d965a9e8dc06449e/orjson-3.11.9-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:53b50b0e14084b8f7e29c5ce84c5af0f1160169b30d8a6914231d97d2fe297d4" },
{ url = "https://mirrors.aliyun.com/pypi/packages/16/64/bd815f5c610b3facc204f26ba94e87a9eb49b0d83de3d5fc1eee2402d91b/orjson-3.11.9-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:231742b4a11dad8d5380a435962c57e91b7c37b79be858f4ef1c0df1a259897e" },
{ url = "https://mirrors.aliyun.com/pypi/packages/c7/35/e744fd36c79b339d27beb06068b5a08a8882ef5418804d0ce545a31f718d/orjson-3.11.9-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:34fd2317602587321faab75ab76c623a0117e80841a6413654f04e47f339a8fb" },
{ url = "https://mirrors.aliyun.com/pypi/packages/2a/56/d54152b67b63a0b3e556cfc549d6ce84f74d7f425ddeadc6c8a74d913da7/orjson-3.11.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:71f3db16e69b667b132e0f305a833d5497da302d801508cbb051ed9a9819da47" },
{ url = "https://mirrors.aliyun.com/pypi/packages/0b/ee/66154baf69f71c7164a268a5e888908aec5a0819d13c81d5e2755a257758/orjson-3.11.9-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0b34789fa0da61cf7bef0546b09c738fb195331e017e477096d129e9105ab03d" },
{ url = "https://mirrors.aliyun.com/pypi/packages/09/d3/c5824260ca8b9d7ba82648d042a3f8f4815d18c15bb98a1f30edd1bb2d83/orjson-3.11.9-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:87e4d4ab280b0c87424d47695bec2182caf8cfc17879ea78dab76680194abc13" },
{ url = "https://mirrors.aliyun.com/pypi/packages/64/cb/509c2e816fe4df641d93dc92f6a89adc8df3ada8ebdee2bd44aba3264c3c/orjson-3.11.9-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:ace6c58523302d3b97b6ac5c38a5298a54b473762b6be82726b4265c41029f92" },
{ url = "https://mirrors.aliyun.com/pypi/packages/db/b5/3ceae56d2e4962979eedb023ba6a46a4bb65f333960379be0ca470686220/orjson-3.11.9-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:97d0d932803c1b164fde11cb542a9efcb1e0f63b184537cca65887147906ff48" },
{ url = "https://mirrors.aliyun.com/pypi/packages/d7/7a/81fa3f2c7bef79b04cf2ab7838e5ac74b1f12511ceab979759b0275d6bb4/orjson-3.11.9-cp310-cp310-win32.whl", hash = "sha256:b3afcf569c15577a9fe64627292daa3e6b3a70f4fb77a5df246a87ec21681b94" },
{ url = "https://mirrors.aliyun.com/pypi/packages/ae/d8/b64600f9083c7f151ad39717a5877fccbeb0ef6d7efcb55f971ce00b6bee/orjson-3.11.9-cp310-cp310-win_amd64.whl", hash = "sha256:8697ab6a080a5c46edaad50e2bc5bd8c7ca5c66442d24104fa44ec74910a8244" },
{ url = "https://mirrors.aliyun.com/pypi/packages/1e/51/3fb9e65ae76ee97bd611869a503fa3fc0a6e81dd8b737cf3003f682df7ff/orjson-3.11.9-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:f01c4818b3fc9b0da8e096722a84318071eaa118df35f6ed2344da0e73a5444f" },
{ url = "https://mirrors.aliyun.com/pypi/packages/16/fa/9d54b07cb3f3b0bfd57841478e42d7a0ece4a9f49f9907eecf5a45461687/orjson-3.11.9-cp311-cp311-macosx_15_0_arm64.whl", hash = "sha256:3ebca4179031ee716ed076ffadc29428e900512f6fccee8614c9983157fcf19c" },
{ url = "https://mirrors.aliyun.com/pypi/packages/88/b1/6ceafc2eefd0a553e3be77ce6c49d107e772485d9568629376171c50e634/orjson-3.11.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:48ee05097750de0ff69ed5b7bbcf0732182fd57a24043dcc2a1da780a5ead3a5" },
{ url = "https://mirrors.aliyun.com/pypi/packages/ea/76/f11311285324a40aab1e3031385c50b635a7cd0734fdaf60c7e89a696f60/orjson-3.11.9-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a6082706765a95a6680d812e1daf1c0cfe8adec7831b3ff3b625693f3b461b1c" },
{ url = "https://mirrors.aliyun.com/pypi/packages/9e/85/0ef63bcf1337f44031ce9b91b1919563f62a37527b3ea4368bb15a22e5d7/orjson-3.11.9-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:277fefe9d76ee17eb14debf399e3533d4d63b5f677a4d3719eb763536af1f4bd" },
{ url = "https://mirrors.aliyun.com/pypi/packages/05/94/b0d27090ea8a2095db3c2bd1b1c96f96f19bbb494d7fef33130e846e613d/orjson-3.11.9-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:03db380e3780fa0015ed776a90f20e8e20bb11dde13b216ce19e5718e3dfba62" },
{ url = "https://mirrors.aliyun.com/pypi/packages/09/eb/75d50c29c05b8054013e221e598820a365c8e64065312e75e202ed880709/orjson-3.11.9-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:33d7d766701847dc6729846362dc27895d2f2d2251264f9d10e7cb9878194877" },
{ url = "https://mirrors.aliyun.com/pypi/packages/49/bd/360686f39348aa88827cb6fbf7dc606fd41c831a35235e1abf1db8e3a9e6/orjson-3.11.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:147302878da387104b66bb4a8b0227d1d487e976ce41a8501916161072ed87b1" },
{ url = "https://mirrors.aliyun.com/pypi/packages/0e/30/3178eb16f3221aeef068b6f1f1ebe05f656ea5c6dffe9f6c917329fe17a3/orjson-3.11.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3513550321f8c8c811a7c3297b8a630e82dc08e4c10216d07703c997776236cd" },
{ url = "https://mirrors.aliyun.com/pypi/packages/5f/f1/ff2f19ed0225f9680fafa42febca3570dd59444ebf190980738d376214c2/orjson-3.11.9-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:c5d001196b89fa9cf0a4ab79766cd835b991a166e4b621ba95089edc50c429ff" },
{ url = "https://mirrors.aliyun.com/pypi/packages/9b/61/863bddf0da6e9e586765414debd54b4e58db05f560902b6d00658cb88636/orjson-3.11.9-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:16969c9d369c98eb084889c6e4d2d39b77c7eb38ceccf8da2a9fff62ae908980" },
{ url = "https://mirrors.aliyun.com/pypi/packages/b6/8a/4081492586d75b073d60c5271a8d0f05a0955cabf1e34c8473f6fcd84235/orjson-3.11.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:63e0efbc991250c0b3143488fa57d95affcabbfc63c99c48d625dd37779aafe2" },
{ url = "https://mirrors.aliyun.com/pypi/packages/0d/bd/70b6ab193594d7abb875320c0a7c8335e846f28968c432c31042409c3c8d/orjson-3.11.9-cp311-cp311-win32.whl", hash = "sha256:14ed654580c1ed2bc217352ec82f91b047aef82951aa71c7f64e0dcb03c0e180" },
{ url = "https://mirrors.aliyun.com/pypi/packages/3f/17/1a1a228183d62d1b77e2c30d210f47dd4768b310ebe1607c63e3c0e3a71e/orjson-3.11.9-cp311-cp311-win_amd64.whl", hash = "sha256:57ea77fb70a448ce87d18fca050193202a3da5e54598f6501ca5476fb66cfe02" },
{ url = "https://mirrors.aliyun.com/pypi/packages/b8/95/285de5fa296d09681ee9c546cd4a8aeb773b701cf343dc125994f4d52953/orjson-3.11.9-cp311-cp311-win_arm64.whl", hash = "sha256:19b72ed11572a2ee51a67a903afbe5af504f84ed6f529c0fe44b0ab3fb5cc697" },
{ url = "https://mirrors.aliyun.com/pypi/packages/16/6d/11867a3ffa3a3608d84a4de51ef4dd0896d6b5cc9132fbe1daf593e677bc/orjson-3.11.9-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:9ef6fe90aadef185c7b128859f40beb24720b4ecea95379fc9000931179c3a49" },
{ url = "https://mirrors.aliyun.com/pypi/packages/24/75/05912954c8b288f34fcf5cd4b9b071cb4f6e77b9961e175e56ebb258089f/orjson-3.11.9-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:e5c9b8f28e726e97d97696c826bc7bea5d71cecd63576dba92924a32c1961291" },
{ url = "https://mirrors.aliyun.com/pypi/packages/ab/86/1c3a47df3bc8191ea9ac51603bbb872a95167a364320c269f2557911f406/orjson-3.11.9-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:26a473dbb4162108b27901492546f83c76fdcea3d0eadff00ae7a07e18dcce09" },
{ url = "https://mirrors.aliyun.com/pypi/packages/d7/cf/b33b5f3e695ae7d63feef9d915c37cc3b8f465493dcd4f8e0b4c697a2366/orjson-3.11.9-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:011382e2a60fda9d46f1cdee31068cfc52ffe952b587d683ec0463002802a0f4" },
{ url = "https://mirrors.aliyun.com/pypi/packages/31/6a/6cf69385a58208024fcb8c014e2141b8ce838aba6492b589f8acfff97fab/orjson-3.11.9-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c2d3dc759490128c5c1711a53eeaa8ee1d437fd0038ffd2b6008abf46db3f882" },
{ url = "https://mirrors.aliyun.com/pypi/packages/e8/f8/0b1bd3e8f2efcdd376af5c8cfd79eaf13f018080c0089c80ebd724e3c7fb/orjson-3.11.9-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d8ea516b3726d190e1b4297e6f4e7a8650347ae053868a18163b4dd3641d1fff" },
{ url = "https://mirrors.aliyun.com/pypi/packages/f3/59/dab79f61044c529d2c81aecdc589b1f833a1c8dec11ba3b1c2498a02ca7e/orjson-3.11.9-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:380cdce7ba24989af81d0a7013d0aaec5d0e2a21734c0e2681b1bc4f141957fe" },
{ url = "https://mirrors.aliyun.com/pypi/packages/0e/a4/82b7a2fe5d8a67a59ed831b24d59a3d46ea7d207b66e1602d376541d94a6/orjson-3.11.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be4fa4f0af7fa18951f7ab3fc2148e223af211bf03f59e1c6034ec3f97f21d61" },
{ url = "https://mirrors.aliyun.com/pypi/packages/50/c7/375e83a76851b73b2e39f3bcf0e5a19e2b89bad13e5bca97d0b293d27f24/orjson-3.11.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a8f5f8bc7ce7d59f08d9f99fa510c06496164a24cb5f3d34537dbd9ca30132e2" },
{ url = "https://mirrors.aliyun.com/pypi/packages/7f/7c/49d5d82a3d3097f641f094f552131f1e2723b0b8cb0fa2874ab65ecfffa6/orjson-3.11.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:4d7fde5501b944f83b3e665e1b31343ff6e154b15560a16b7130ea1e594a4206" },
{ url = "https://mirrors.aliyun.com/pypi/packages/3a/dc/7446c538590d55f455647e5f3c61fc33f7108714e7afcffa6a2a033f8350/orjson-3.11.9-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:cde1a448023ba7d5bb4c01c5afb48894380b5e4956e0627266526587ef4e535f" },
{ url = "https://mirrors.aliyun.com/pypi/packages/df/e5/4d2d8af06f788329b4f78f8cc3679bb395392fcaa1e4d8d3c33e85308fa4/orjson-3.11.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:71e63adb0e1f1ed5d9e168f50a91ceb93ae6420731d222dc7da5c69409aa47aa" },
{ url = "https://mirrors.aliyun.com/pypi/packages/06/69/850264ccf6d80f6b174620d30a87f65c9b1490aba33fe6b62798e618cad3/orjson-3.11.9-cp312-cp312-win32.whl", hash = "sha256:2d057a602cdd19a0ad680417527c45b6961a095081c0f46fe0e03e304aac6470" },
{ url = "https://mirrors.aliyun.com/pypi/packages/b9/d5/973a43fc9c55e20f2051e9830997649f669be0cb3ca52192087c0143f118/orjson-3.11.9-cp312-cp312-win_amd64.whl", hash = "sha256:59e403b1cc5a676da8eaf31f6254801b7341b3e29efa85f92b48d272637e77be" },
{ url = "https://mirrors.aliyun.com/pypi/packages/fe/ae/495470f0e4a18f73fa10b7f6b84b464ec4cc5291c4e0c7c2a6c400bef006/orjson-3.11.9-cp312-cp312-win_arm64.whl", hash = "sha256:9af678d6488357948f1f84c6cd1c1d397c014e1ae2f98ae082a44eb48f602624" },
{ url = "https://mirrors.aliyun.com/pypi/packages/32/33/93fcc25907235c344ae73122f8a4e01d2d393ef062b4af7d2e2487a32c37/orjson-3.11.9-cp313-cp313-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:4bab1b2d6141fe7b32ae71dac905666ece4f94936efbfb13d55bb7739a3a6021" },
{ url = "https://mirrors.aliyun.com/pypi/packages/8f/27/b1e6dadb3c080313c03fdd8067b85e6a0460c7d8d6a1c3984ef77b904e4d/orjson-3.11.9-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:844417969855fc7a41be124aafe83dc424592a7f77cd4501900c67307122b92c" },
{ url = "https://mirrors.aliyun.com/pypi/packages/21/0f/c9ede0bf052f6b4051e64a7d4fa91b725cccf8321a6a786e86eb03519f00/orjson-3.11.9-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffe02797b5e9f3a9d8292ddcd289b474ad13e81ad83cd1891a240811f1d2cb81" },
{ url = "https://mirrors.aliyun.com/pypi/packages/fd/26/d398e28048dc18205bbe812f2c88cb9b40313db2470778e25964796458fe/orjson-3.11.9-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e4eed3b200023042814d2fc8a5d2e880f13b52e1ed2485e83da4f3962f7dc1a" },
{ url = "https://mirrors.aliyun.com/pypi/packages/66/60/52b0054c4c700d5aa7fc5b7ca96917400d8f061307778578e67a10e25852/orjson-3.11.9-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8aff7da9952a5ad1cef8e68017724d96c7b9a66e99e91d6252e1b133d67a7b10" },
{ url = "https://mirrors.aliyun.com/pypi/packages/d5/97/1e3dc2b2a28b7b2528f403d2fc1d79ec5f39af3bc143ab65d3ec26426385/orjson-3.11.9-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4d4e98d6f3b8afed8bc8cd9718ec0cdf46661826beefb53fe8eafb37f2bf0362" },
{ url = "https://mirrors.aliyun.com/pypi/packages/fc/39/31fbfe7850f2de32dee7e7e5c09f26d403ab01e440ac96001c6b01ad3c99/orjson-3.11.9-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3a81d52442a7c99b3662333235b3adf96a1715864658b35bb797212be7bddb97" },
{ url = "https://mirrors.aliyun.com/pypi/packages/a1/08/dca0082dd2a194acb93e5457e73455388e2e2ca464a2672449a9ddbb679d/orjson-3.11.9-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4e39364e726a8fff737309aff059ff67d8a8c8d5b677be7bb49a8b3e84b7e218" },
{ url = "https://mirrors.aliyun.com/pypi/packages/11/d4/5bdb0626801230139987385554c5d4c42255218ac906525bf4347f22cd95/orjson-3.11.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4fd66214623f1b17501df9f0543bef0b833979ab5b6ded1e1d123222866aa8c9" },
{ url = "https://mirrors.aliyun.com/pypi/packages/fa/88/a21fb53b3ede6703aede6dce4710ed4111e5b201cfa6bbff5e544f9d47d7/orjson-3.11.9-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:8ecc30f10465fa1e0ce13fd01d9e22c316e5053a719a8d915d4545a09a5ff677" },
{ url = "https://mirrors.aliyun.com/pypi/packages/3d/57/1b30daf70f0d8180e9a73cefbfbdd99e4bf19eb020466502b01fba7e0e50/orjson-3.11.9-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:97db4c94a7db398a5bd636273324f0b3fd58b350bbbac8bb380ceb825a9b40f4" },
{ url = "https://mirrors.aliyun.com/pypi/packages/04/83/45fbb6d962e260807f99441db9613cee868ceda4baceda59b3720a563f97/orjson-3.11.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9f78cf8fec5bd627f4082b8dfeac7871b43d7f3274904492a43dab39f18a19a0" },
{ url = "https://mirrors.aliyun.com/pypi/packages/5f/cc/2d10025f9056d376e4127ec05a5808b218d46f035fdc08178a5411b34250/orjson-3.11.9-cp313-cp313-win32.whl", hash = "sha256:d4087e5c0209a0a8efe4de3303c234b9c44d1174161dcd851e8eea07c7560b32" },
{ url = "https://mirrors.aliyun.com/pypi/packages/67/bd/2775ff28bfe883b9aa1ff348300542eb2ef1ee18d8ae0e3a49846817a865/orjson-3.11.9-cp313-cp313-win_amd64.whl", hash = "sha256:051b102c93b4f634e89f3866b07b9a9a98915ada541f4ec30f177067b2694979" },
{ url = "https://mirrors.aliyun.com/pypi/packages/91/2b/d26799e580939e32a7da9a39531bc9e58e15ca32ffaa6a8cb3e9bb0d22cd/orjson-3.11.9-cp313-cp313-win_arm64.whl", hash = "sha256:cce9127885941bd28f080cecf1f1d288336b7e0d812c345b08be88b572796254" },
{ url = "https://mirrors.aliyun.com/pypi/packages/8e/eb/5da01e356015aee6ecfa1187ced87aef51364e306f5e695dd52719bf0e78/orjson-3.11.9-cp314-cp314-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:b6ef1979adc4bc243523f1a2ba91418030a8e29b0a99cbe7e0e2d6807d4dce6e" },
{ url = "https://mirrors.aliyun.com/pypi/packages/64/62/3e0e0c14c957133bcd855395c62b55ed4e3b0af23ffea11b032cb1dcbdb1/orjson-3.11.9-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:f36b7f32c7c0db4a719f1fc5824db4a9c6f8bd1a354debb91faf26ebf3a4c71e" },
{ url = "https://mirrors.aliyun.com/pypi/packages/5a/5a/07d8aa117211a8ed7630bda80c8c0b14d04e0f8dcf99bcf49656e4a710eb/orjson-3.11.9-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:08f4d8ebb44925c794e535b2bebc507cebf32209df81de22ae285fb0d8d66de0" },
{ url = "https://mirrors.aliyun.com/pypi/packages/d6/ec/4acaf21483e18aa945be74a474c74b434f284b549f275a0a39b9f98956e9/orjson-3.11.9-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6cc7923789694fd58f001cbcac7e47abc13af4d560ebbfcf3b41a8b1a0748124" },
{ url = "https://mirrors.aliyun.com/pypi/packages/13/d8/5f0555e7638801323b7a75850f92e7dfa891bc84fe27a1ba4449170d1200/orjson-3.11.9-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ea5c46eb2d3af39e806b986f4b09d5c2706a1f5afde3cbf7544ce6616127173c" },
{ url = "https://mirrors.aliyun.com/pypi/packages/b6/30/ed9860412a3603ceb3c5955bfd72d28b9d0e7ba6ed81add14f83d7114236/orjson-3.11.9-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f5d89a2ed90731df3be64bab0aa44f78bff39fdc9d71c291f4a8023aa46425b7" },
{ url = "https://mirrors.aliyun.com/pypi/packages/d0/17/adc514dea7ac7c505527febf884934b815d34f0c7b8693c1a8b39c5c4a57/orjson-3.11.9-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:25e4aed0312d292c09f61af25bba34e0b2c88546041472b09088c39a4d828af1" },
{ url = "https://mirrors.aliyun.com/pypi/packages/76/3e/c0b690253f0b82d86e99949af13533363acfb5432ecb5d53dd5b3bce9c34/orjson-3.11.9-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aaea64f3f467d22e70eeed68bdccb3bc4f83f650446c4a03c59f2cba28a108db" },
{ url = "https://mirrors.aliyun.com/pypi/packages/c1/7a/bc82a0bb25e9faaf92dc4d9ef002732efc09737706af83e346788641d4a7/orjson-3.11.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a028425d1b440c5d92a6be1e1a020739dfe67ea87d96c6dbe828c1b30041728b" },
{ url = "https://mirrors.aliyun.com/pypi/packages/01/55/e69188b939f77d5d32a9833745ace31ea5ccae3ab613a1ec185d3cd2c4fb/orjson-3.11.9-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:5b192c6cf397e4455b11523c5cf2b18ed084c1bbd61b6c0926344d2129481972" },
{ url = "https://mirrors.aliyun.com/pypi/packages/2e/1a/b8a5a7ac527e80b9cb11d51e3f6689b709279183264b9ec5c7bc680bb8b5/orjson-3.11.9-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ea407d4ccf5891d667d045fecae97a7a1e5e87b3b97f97ae1803c2e741130be0" },
{ url = "https://mirrors.aliyun.com/pypi/packages/97/4e/00503f64204bf859b37213a63927028f30fb6268cd8677fb0a5ad48155e1/orjson-3.11.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f63aaf97afd9f6dec5b1a68e1b8da12bfccb4cb9a9a65c3e0b6c847849e7586" },
{ url = "https://mirrors.aliyun.com/pypi/packages/0d/ba/a23b82a0a8d0ed7bed4e5f5035aae751cad4ff6a1e8d2ecd14d8860f5929/orjson-3.11.9-cp314-cp314-win32.whl", hash = "sha256:e30ab17845bb9fa54ccf67fa4f9f5282652d54faa6d17452f47d0f369d038673" },
{ url = "https://mirrors.aliyun.com/pypi/packages/f3/c3/0c6798456bade745c75c452342dabacce5798196483e77e643be1f53877d/orjson-3.11.9-cp314-cp314-win_amd64.whl", hash = "sha256:32ef5f4283a3be81913947d19608eacb7c6608026851123790cd9cc8982af34b" },
{ url = "https://mirrors.aliyun.com/pypi/packages/16/21/5a3f1e8913103b703a436a5664238e5b965ec392b555fe68943ea3691e6b/orjson-3.11.9-cp314-cp314-win_arm64.whl", hash = "sha256:eebdbdeef0094e4f5aefa20dcd4eb2368ab5e7a3b4edea27f1e7b2892e009cf9" },
]
[[package]]
name = "packaging"
version = "26.2"
@@ -1250,6 +1331,9 @@ docs = [
{ name = "sphinx", version = "9.1.0", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version >= '3.12'" },
{ name = "sphinx-rtd-theme" },
]
fast = [
{ name = "orjson" },
]
office = [
{ name = "pillow" },
{ name = "pymupdf" },
@@ -1267,6 +1351,7 @@ requires-dist = [
{ name = "hatch", marker = "extra == 'dev'", specifier = ">=1.14.2" },
{ name = "httpx", marker = "extra == 'dev'", specifier = ">=0.28.0" },
{ name = "myst-parser", marker = "extra == 'docs'", specifier = ">=3.0" },
{ name = "orjson", marker = "extra == 'fast'", specifier = ">=3.10.0" },
{ name = "pillow", marker = "extra == 'office'", specifier = ">=10.4.0" },
{ name = "prek", marker = "extra == 'dev'", specifier = ">=0.4.5" },
{ name = "pymupdf", marker = "extra == 'office'", specifier = ">=1.24.11" },
@@ -1289,7 +1374,7 @@ requires-dist = [
{ name = "typer", specifier = ">=0.24.0" },
{ name = "typing-extensions", marker = "python_full_version < '3.13'", specifier = ">=4.13.2" },
]
provides-extras = ["dev", "docs", "office"]
provides-extras = ["dev", "docs", "fast", "office"]
[package.metadata.requires-dev]
dev = [{ name = "pyflowx", extras = ["dev", "docs", "office"], editable = "." }]