Compare commits

..

1 Commits

Author SHA1 Message Date
zhou 02539d60fd bump version to 0.3.2
Release / Build, Publish & Release (push) Failing after 1m40s
2026-07-04 10:50:37 +08:00
169 changed files with 8974 additions and 22326 deletions
+2 -1
View File
@@ -38,8 +38,9 @@ env
*.swp
*.swo
# 文档(按需保留)
# 文档与示例(按需保留)
docs
examples
# 系统文件
.DS_Store
+2 -2
View File
@@ -27,14 +27,14 @@ jobs:
run: uv build
- name: Publish to pypi
run: uv publish --token '${{ secrets.PYPI_TOKEN }}'
run: uv publish --token '${{ secrets.PYPI_API_TOKEN }}'
- name: Create Gitea Release & Upload Assets
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG_NAME: ${{ github.ref_name }}
REPO: ${{ github.repository }}
GITEA_URL: http://172.17.0.1:3000
GITEA_URL: http://gitea:3000
run: |
set -e
# 1. 创建 Release
-4
View File
@@ -11,7 +11,3 @@ wheels/
.coverage
.idea
*_profile.html
# Sphinx 文档构建输出
docs/_build/
.trae/refs
-23
View File
@@ -1,23 +0,0 @@
# ReadTheDocs 配置
# https://docs.readthedocs.io/en/stable/config-file/v2.html
version: 2
# 构建配置
build:
os: ubuntu-24.04
tools:
python: "3.11"
# Python 依赖与构建命令
python:
install:
- method: pip
path: .
extra_requirements:
- docs
# Sphinx 构建
sphinx:
configuration: docs/conf.py
builder: html
fail_on_warning: false
@@ -1,93 +0,0 @@
# 迭代 21 — P9 功能扩展(任务编排/数据流/持久化/监控)
## 本轮目标
延续 P8 性能优化后,扩展 PyFlowX 在四个方向的核心能力:
1. **P9.1 任务编排增强**:任务分组(`Graph.group`+ 动态任务生成(`TaskSpec(dynamic=True)`)。
2. **P9.2 数据流增强**`RunReport.output_of` 命名输出提取 + `Graph.pipeline` 语义别名。
3. **P9.3 持久化与恢复**`run(resume_from=...)` 检查点恢复 + `RunHistory` 历史管理。
4. **P9.4 监控导出**`MetricsCollector`Prometheus 文本)+ `health_check` + `start_metrics_server`(零依赖 HTTP)。
## 改动文件清单
### 新增
- `src/pyflowx/monitoring.py``MetricsCollector` / `health_check` / `start_metrics_server`
- `src/pyflowx/history.py``RunHistory` 文件系统历史管理。
- `tests/test_group.py` — 13 用例(P9.1.2 任务分组)。
- `tests/test_dynamic.py` — 11 用例(P9.1.3 动态任务)。
- `tests/test_dataflow.py` — 12 用例(P9.2 数据流)。
- `tests/test_persistence.py` — 15 用例(P9.3 检查点/历史)。
- `tests/test_monitoring.py` — 16 用例(P9.4 监控导出)。
### 修改
- `src/pyflowx/task.py``TaskSpec` 新增 `dynamic: bool = False` 字段;`task()` 工厂新增 `dynamic` 参数。
- `src/pyflowx/graph.py` — 新增 `group()` 方法;`_rewrite_deps_for_loops` 改名 `_rewrite_deps`,统一处理 `_loop_groups` + `_groups`;新增 `pipeline()` 语义别名。
- `src/pyflowx/report.py` — 新增 `_resolve_output_path` 辅助 + `RunReport.output_of(name, output)` 方法。
- `src/pyflowx/executors.py`
- `DependencyRunner.execute` 增加 while 循环 + `f.exception()` 手动传播(替代 `asyncio.gather`)。
- 新增 `_extract_dynamic_specs` 辅助;`_run_task` 检测动态生成、注册派生 spec、启动派生 task。
- `_dispatch_strategy` 增加 `has_dynamic` 校验(仅 `dependency` 策略支持)。
- 新增 `_load_resume_report` + `_apply_resume` 辅助;`run()` 接受 `resume_from` 参数。
- `_filter_and_sort` + `_run_task` 检查点跳过逻辑。
- `src/pyflowx/__init__.py` — 导出 `RunHistory` / `MetricsCollector` / `health_check` / `start_metrics_server`
## 关键决策与依据
### 1. 动态任务仅支持 `dependency` 策略
`dynamic=True` 允许任务 `fn` 运行时返回 `TaskSpec | list[TaskSpec]`DependencyRunner 中途注册并调度派生任务。层模型(sequential/thread/async)的层屏障与动态扩展语义冲突,因此 `_dispatch_strategy` 显式拒绝其他策略。
依据:层模型按拓扑分层执行,动态任务可能跨层;依赖驱动模型无层屏障,可自然容纳运行时新增节点。
### 2. `asyncio.wait` 替代 `asyncio.gather` + 手动异常传播
`asyncio.gather` 在 first-failure 时取消其他任务并 re-raise,但所有 task 必须在调用前确定。动态任务场景下 `futures` 字典会随执行增长,必须用 while 循环 + `asyncio.wait(return_when=FIRST_COMPLETED)` 增量调度。
陷阱:`asyncio.wait` **不会**自动 re-raise task 异常。需手动 `f.exception()` 检查并显式 `raise`,同时取消未完成任务,才能与 `gather` 的 fail-fast 行为一致。该陷阱导致 7 个测试失败后定位修复。
### 3. 检查点恢复:仅恢复 SUCCESS 任务
`resume_from` 接受 `RunReport | str | Path`,仅恢复 `status == SUCCESS` 且名称在当前图中的任务。FAILED/SKIPPED 任务会重新执行。`_apply_resume` 抽取为独立函数以控制 `run()` 的 PLR0912 分支数(≤12)。
### 4. `metrics_text` 按 section 条件化输出
最初实现总是输出 `# HELP` / `# TYPE` 行(即便无数据)。这导致 `reset()` 后文本仍包含 `pyflowx_task_total` 字样,违反测试预期。改为所有 section(含 task_total/duration/duration_sum)都按数据存在性条件化,与 retries/run_total 一致。
### 5. `start_metrics_server` 用 `http.server` 零依赖
未引入 `prometheus_client` 等三方库,仅用标准库 `http.server.HTTPServer` + `BaseHTTPRequestHandler`。生产环境可叠加反向代理或替换为 `prometheus_client``@override` 装饰器按 Python 版本条件导入(`typing` 3.12+ / `typing_extensions` <3.12)。
### 6. `RunHistory` 文件命名与 `__contains__`/`__len__`
`{run_id}.json` 命名存储;`__contains__` / `__len__` 委托文件存在性检查,避免维护内存索引。`latest()` 按 mtime 排序取最新。
## 验证结果
```
ruff check . → All checks passed!
ruff format --check → 126 files already formatted
pyrefly check . → 0 errors (68 suppressed)
pytest --cov=pyflowx → 1525 passed in 11.56s
coverage → 97.58%branch;门槛 95%
```
各模块覆盖率(P9 新增部分):
- `monitoring.py` 99%(仅 `typing_extensions` 回退分支未覆盖)
- `history.py` 95%
- `executors.py` 97%dynamic/resume 分支已覆盖)
- `report.py` 99%`output_of` 路径解析已覆盖)
- `graph.py` 98%`group`/`pipeline`/`_rewrite_deps` 已覆盖)
## 遗留事项
- 监控导出未实现 OpenTelemetry 协议(当前仅 Prometheus 文本格式)。如需 OTLP,可在后续迭代引入 `opentelemetry-sdk`(违反零依赖原则,需用户确认)。
- `RunHistory` 未实现压缩/归档;大量历史运行可能占用较多磁盘。后续可加 `max_entries` 或滚动归档。
- 动态任务当前仅在 DependencyRunner 中实现;如层模型用户也需要,需额外设计跨层调度策略。
- `_apply_resume` 仅恢复 `report.results`,未恢复 `backend` 缓存;如需检查点+缓存联合恢复,需额外设计。
## 归档说明
iter-21 暂不归档(前 5 次 iter-16~20 已归档至 `.trae/skills/pyflowx-development/SKILL.md`)。下次清理窗口为 iter-26 前夕。
-23
View File
@@ -1,23 +0,0 @@
# 开发流程约束
本规则补充 `self-driven.md`,定义执行过程记录与规则变更的两条硬约束。
## 执行过程记录
- **每次迭代**(一个完整的"计划 → 实现 → 测试 → 文档 → 验证"闭环)必须记录到
`.trae/docs/` 目录,文件名形如 `iter-NN-<主题>.md``NN` 为零填充序号。
- 记录内容至少包括:本轮目标、改动文件清单、关键决策与依据、验证结果、遗留事项。
- **每 5 次迭代后**清理 `.trae/docs/`
- 把需要长期保留的整合性内容(可复用模式、踩坑总结、设计决策)
归档到 `.trae/skills/` 对应技能文档中;
- 其余过程性记录删除;
- docs 目录只保留最近未归档的迭代记录(即第 6 次迭代开始前,清理 1–5 号记录)。
- 归档动作需在迭代记录中标注"已归档至 skills/<文件>"。
## 规则变更约束
- 任何对 `.trae/rules/` 下文件的修改(新增、编辑、删除)**必须先询问用户**,
获得明确授权后方可变动;**未授权前不得擅自修改 rules**。
- 用户在对话中显式要求写入的规则内容(如"把 X 写入 rules")视为已授权,
可直接写入,写入后在回复中说明改了哪个文件、加了什么。
- 规则变更后需同步更新 `project_memory.md` 中的"开发约定"小节。
+2 -2
View File
@@ -25,7 +25,7 @@ uvx --from pyflowx pymake cov
- **最低 Python 3.8**:用 `from __future__ import annotations` 延迟注解求值;
按版本用 `typing.List`(3.8) → 内置泛型(3.9) → `X | Y`(3.10) → `typing.override`(3.12)。
- **版本守卫**`if sys.version_info >= (3, X):` 引入高版本 API;低版本回退分支加 `# pragma: no cover`
- **零运行时依赖**:仅依赖标准库(`typing-extensions` 用于 `override`/`TypeVar` 前向兼容)。
- **零运行时依赖**:仅依赖标准库(3.8 需 `graphlib_backport``typing-extensions`)。
新增依赖须审慎,优先用标准库。
## 类型注解
@@ -150,7 +150,7 @@ uvx --from pyflowx pymake cov
## Git 与提交
- **自动提交**:任务完成后自动 `git add`(按文件名)+ `git commit` + `git push`(仅当分支已跟踪远程时执行 push;新分支跳过 push 并在总结中说明)
- **自动提交/push**:除非用户明确要求
- **不修改 git config**。
- **不运行破坏性命令**`push --force`/`reset --hard`/`clean -f`)除非用户明确要求。
- **staging**:按文件名添加,不用 `git add -A`/`git add .`,避免误加敏感文件。
-151
View File
@@ -1,151 +0,0 @@
---
alwaysApply: true
---
# 自驱动开发规则
本规则定义一种"目标驱动、闭环执行"的工作模式:仅在任务开始时与用户确认一次目标与边界,后续由 Agent 自主完成"计划 → 编码 → 测试 → 文档 → 验证"的迭代循环,直到用户目标达成。
## 核心原则
- **目标导向**:始终以用户最终目标为准绳,所有阶段产出都应服务于该目标。
- **闭环执行**:每个子任务必须走完"计划 → 实现 → 测试 → 文档 → 验证"五步;禁止跳步留半成品。
- **自主决策**:初始确认之后,实现路径、API 形态、重构范围、文件命名、测试组织、错误修复策略等由 Agent 自行决断,不再逐项请示。**可逆操作(编辑文件、运行测试、修复 lint、调整实现)直接执行,不询问**;只有不可逆/高风险操作才暂停。
- **透明沟通**:每个阶段开始前用一句话说明意图;关键节点(完成、阻塞、转向)给简短更新;不复述内部思考,**不在收尾时停下询问"是否继续"或"是否提交"**——直接输出总结并结束。
- **安全边界**:仅在高风险、不可逆操作或真正阻塞时才暂停找用户。
## 初始确认(一次性,仅在最开始)
任务启动时,用 `AskUserQuestion` 一次性确认以下信息(已由项目规范覆盖的不必重复确认):
1. **目标与范围**:要解决什么问题?交付物是什么?显式列出不在范围内的内容。
2. **验收标准**:怎样算"完成"?可观测的判定条件(功能、性能、覆盖率阈值)。
3. **特殊约束**:除 `python-standards.md` 之外的约束(兼容性、依赖限制、API 兼容策略等)。
4. **测试要求**:覆盖率门槛(项目默认 ≥95%,branch);是否需要新增 `slow` 标记。
**git commit/push 不在确认范围内**:任务完成后自动 commit + push(仅当分支已跟踪远程时执行 push;新分支跳过 push 并在总结中说明),遵循 `.trae/rules/git-commit-message.md` 风格。仅 force-push、reset --hard、clean -f、修改 git config 等真正破坏性操作才需暂停确认。
确认后,将目标与验收标准固化进 `TaskCreate` 任务列表,后续不再就同一信息反复询问。
## 迭代循环
下列五个阶段构成一个完整闭环。未达验收标准时,回到「计划」开启下一轮;达标准时,进入「收尾」。
### 1. 计划(Plan
- 用 Explore/Glob/Grep 研究相关代码与既有模式,避免凭空设计。
-`TaskCreate` 把目标拆为可独立验证的子任务;每完成一项立即 `TaskUpdate` 为 completed。
- 优先复用现有抽象;不为本轮假想需求设计接口。
- 不过早抽象:三处相似才考虑提取,否则就地写。
### 2. 实现(Code
- 严格遵守 `.trae/rules/python-standards.md` 与既有代码风格。
- 优先 Edit 现有文件;新增文件需有明确职责边界。
- 不引入运行时依赖(项目零依赖原则);确需引入须在计划阶段说明。
- 公共 API 必须有完整类型注解与中文 docstring。
- 不写未被要求的功能、不为未来场景预留扩展点。
### 3. 测试(Test
- 新增/修改的公共 API 必须配套测试;优先通过公共接口测试,故障注入可访问私有属性并在 docstring 注明。
- Mock 优先级:`monkeypatch` > 内联 stub > `unittest.mock` > `pytest-mock`;禁用 `@patch` 装饰器。
- 必跑校验(每次修改后):
```bash
uvx --from pyflowx pymake tc
uvx --from pyflowx pymake cov
```
- 测试失败时定位根因再修复,不通过放宽断言或 `# pragma: no cover` 绕过。
- 覆盖率不得低于上一次的值(项目门槛 95%,branch)。
### 4. 文档(Docs
- 同步更新 docstring、README、模块结构说明。
- 行为变更须同步更新 `.agents/skills/pyflowx-development/SKILL.md` 中的对应章节。
- 跨会话有价值的设计决策、约束、陷阱,追加到 memory(`project_memory.md` 或对应 `topics.md`)。
- 不主动新建 `*.md` 文档;除非用户明确要求。
### 5. 验证(Verify
- 逐条对照初始确认的「验收标准」核验;未满足则回到「计划」继续下一轮。
- 全套门禁通过:ruff、pyrefly、pytest、coverage。
- 给出本轮变更清单(改了哪些文件、为什么)。
## 多阶段项目
当项目按阶段(P0/P1/P2/...)划分时,初始确认的目标是**整个项目**,而非单个阶段。
- **阶段是里程碑,不是边界**:确认"推进 P1"只是指定下一步重点,不缩小整体目标范围;
也不意味着 P1 完成后就可以停下。整体目标 = 所有阶段交付完毕。
- **阶段完成不是停止条件**:某阶段验收标准全部满足后,**自动进入下一阶段的「计划」步骤**,
不停下来输出"是否继续"或"是否扩展范围"的询问。仅在阶段切换时用一句话说明
"进入 Pn,重点:…",然后继续自驱动迭代。
- **「收尾」仅适用于整个项目完成**:只有所有阶段(含最后一个)都交付后,才执行
「收尾」输出最终总结。单阶段完成 → 进入下一阶段的「计划」,不触发收尾。
- **跨阶段依赖**:若下一阶段依赖外部资源(如新依赖审批、环境配置),属"不可恢复的失败"
暂停条件;否则一律连续推进。
- **阶段范围超出预期**:执行中发现某阶段需要显著扩大范围或改变方向(如 P2 本来只做
表格抽取,发现必须先重写 IR),属"超出初始确认范围"暂停条件,需找用户确认。
## 暂停条件(仅在以下情况中断自驱动找用户)
1. **歧义无法自决**:需求存在多种合理解读且无既有约定可循。
2. **高风险/不可逆操作**:删除非临时文件、`git push --force`、`reset --hard`、删表、修改 CI 配置、修改 git config、卸载依赖等。**普通 `git commit`/`push` 不属于此类**(任务完成后自动执行)。
3. **不可恢复的失败**:根因不在本仓库、需外部环境/权限配合、或经两轮尝试仍无法定位。
4. **超出初始确认范围**:用户目标在执行中发现需要显著扩大范围或改变方向。
5. **用户主动询问**:用户在对话中提出新问题或要求澄清。
**注意**"目标已达成"**不是**暂停条件——但"目标已达成"指**整个项目目标**(所有阶段交付完毕),而非单个阶段验收满足。单阶段验收满足后应自动进入下一阶段(见「多阶段项目」),不触发收尾。
非以上情况,一律继续自驱动,不要为"求确认"而暂停。
## 决策判据:该问还是自决
遇到不确定时,按以下顺序判断:
1. **是否不可逆/高风险?** 是 → 暂停确认(如删除文件、`push --force`、修改 CI 配置、卸载依赖)。否 → 继续。
2. **是否在初始确认范围内?** 是 → 按确认执行,不询问。否 → 视为"超出初始确认范围",暂停。
3. **是否有既有约定可循?** 是 → 按约定执行(参考 `python-standards.md`、`project_memory.md`)。否 → 视为"歧义无法自决",暂停。
4. **是否可逆?** 是 → 直接执行,即使结果可能不完美(可在后续迭代修正)。否 → 暂停。
**可直接自决(不询问)的典型情况**:
- 测试失败、覆盖率不达标、lint/类型检查报错 → 定位根因并修复。
- 代码风格选择(命名、模块划分、参数顺序)→ 自决。
- 文件编辑、运行测试、运行校验命令 → 直接执行。
- 任务完成后输出收尾总结 → 直接输出,不询问下一步。
- 显式指定 `name` 参数以保持测试兼容性 → 自决。
- 重命名局部变量以避免遮蔽 → 自决。
**必须暂停询问的典型情况**
- 删除非临时文件、重命名公共模块/包。
- `git push --force`、`reset --hard`、`clean -f`、修改 git config(普通 commit/push 自动执行,无需询问)。
- 引入新的运行时依赖(违反项目零依赖原则)。
- 修改 CI 配置、pre-commit 钩子、pyproject.toml 的工具链配置。
- 卸载或降级既有依赖。
## 沟通风格
- 阶段切换时一句话说明即可;不要把内部推理写给用户看。
- 完成子任务后用一两句总结改了什么、下一步做什么。
- 遇到阻塞时直接说明:卡在哪、试了什么、需要用户做什么。
- **不在收尾时询问"是否需要提交"或"是否扩展范围"**——直接输出总结并结束。用户后续若有新需求,由用户主动提出。
- 不使用 emoji,除非用户明确要求。
## 工具使用
- 独立操作尽量并行调用(多个 Read/Grep/Glob 一批发出)。
- 用 `TaskCreate`/`TaskUpdate` 维护进度,不批量推迟标记。
- 长命令用后台运行(`run_in_background`),完成会自动通知。
- 文件操作一律用专用工具:Read/Edit/Write/Glob/Grep,不用 `cat`/`sed`/`grep`/`find`。
## 收尾
- **仅当整个项目所有阶段都交付后**,才执行收尾:直接输出最终总结并结束任务(交付物、关键决策、遗留事项)。
- 单阶段完成**不属于收尾**——见「多阶段项目」,应自动进入下一阶段的「计划」。
- **自动提交**:收尾时自动 `git add`(按文件名)+ `git commit`(遵循 `.trae/rules/git-commit-message.md` 风格)+ `git push`(仅当分支已跟踪远程时执行;新分支跳过 push 并在总结中说明);**不询问**"是否需要提交"或"是否扩展范围"。
- 若验收标准未全部满足,回到「计划」继续下一轮,不停下询问。
- 将本次会话的关键产出与决策更新到 memory,便于后续会话续接。
-603
View File
@@ -1,603 +0,0 @@
---
name: "pyflowx-development"
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化相关的功能时参考。"
---
# PyFlowX 开发知识库
本技能归档自迭代 06-20 的过程记录,按主题分类整理可复用知识。
过程性细节(覆盖率数字、命令输出)已剔除,仅保留架构模式、设计依据与陷阱总结。
## 一、兼容性与代码清理
### 设计决策
- **`typing_extensions` 版本守卫不可移除**`override`3.12+)与
`TypeVar(default=)`3.13+)在 3.10/3.11/3.12 上仍需 `typing_extensions`
回退。守卫是前向兼容代码,不是 3.8 死代码。
`pyproject.toml` 声明 `typing-extensions>=4.13.2; python_version < '3.13'`
- **`from __future__ import annotations` 保留**PEP 563 在 3.10+ 仍提供
前向引用与延迟求值价值。统一移除需逐文件审计前向引用、自引用类型,
风险高收益低。
- **`rules/` 修改必须先获用户授权**`.trae/rules/` 下文件修改受
`dev-workflow.md` 约束,未授权前不得擅自变动,即便描述已过时
(如 "最低 Python 3.8" 实际已升至 >=3.10)。
### 踩坑总结
- **不要凭"3.8 兼容代码"假设做清理**:调研后才发现 `typing_extensions`
守卫实际是 3.12/3.13 前向兼容,删除会破坏 3.10-3.12 上的 `override`
`TypeVar(default=)`。清理前必须验证"过期代码"的真实用途。
- **README/文档与代码状态必须同步**:未实现的特性声明(如未落地的
YAML 编排章节)、过期的覆盖率徽章、错误的依赖描述都会误导用户。
清理时优先对齐 README 与 `pyproject.toml`
## 二、YAML 任务编排(GitHub Actions 风格)
### 可复用模式
- **Matrix 笛卡尔积展开**:多键矩阵用 `itertools.product` 展开为笛卡尔积,
命名 `{job_id}_{key}-{value}`(多键用 `_` 连接)。
`version: ["3.10","3.11"]` × `os: ["linux","macos"]` 展开 4 个任务
`test_version-3.10_os-linux` 等。
- **矩阵依赖自动展开**:若 job A `needs: [B]`,B 是矩阵 job,A 依赖 B 的
所有变体(无需手动列出每个变体)。
- **`${{ matrix.key }}` 占位符替换**:在 cmd/run/cwd/env 的字符串值中
统一替换矩阵占位符。
- **字段名 hyphen/underscore 兼容**YAML 中 `continue-on-error`
`continue_on_error` 等价,读取时统一规范化。
- **cmd vs run 区分**
- `cmd: [...]` → list 形式(无 shell,直接 exec
- `run: "..."` → shell 字符串形式
### 设计决策
- **条件解析映射到内置 Conditions**
- `success()` / `always()` → 无条件(空 conditions
- `env.VAR``ENV_VAR_EXISTS("VAR")`
- `env.VAR == 'x'``ENV_VAR_EQUALS("VAR", "x")`
- `env.VAR != 'x'``NOT(ENV_VAR_EQUALS("VAR", "x"))`
- `failure()` → 不支持,抛 `ValueError`PyFlowX 不支持 failure 触发)
- **yamlrun 作为特殊命令而非 `@px.tool`**CLI 路由
`pf yamlrun <file> [--strategy S] [--dry-run] [--list] [--quiet] [--progress]`
作为 `PfApp` 的 builtin 特殊命令注册,与 `pf graph` 同级,不通过 `@px.tool` 装饰器。
- **Graph.from_yaml 委托 yaml_loader**`Graph.from_yaml(path)` classmethod
委托给 `yaml_loader.load_yaml(path)` 函数式 API,保持 Graph 类职责单一。
### Matrix include/exclude 语义(iter-19
- **exclude**: 组合的所有 key-value 对都匹配则剔除(`combo.items() >= exclude.items()`)。
`exclude: [{os: macos, version: "3.8"}]` 剔除 macos+3.8 组合。
- **include**: 追加额外组合,可引入基础矩阵之外的键。
如基础矩阵 `version: ["3.10"]``include: [{version: "3.11", experimental: true}]`
追加 3.11 实验组合。
- **无基础矩阵键时返回空列表而非 `[{}]`**:避免产生空名任务(`job_`)。
### 条件依赖(iter-19
- **`needs` 项支持 `{job: name, if: expr}` 格式**:条件不满足时跳过该依赖
(不加入 `depends_on`)。
- 复用 `_parse_condition` 解析 `if` 表达式;条件求值时传空上下文 `{}`
`env` 条件直接读 `os.environ`)。
- **典型场景**:仅在特定矩阵变体下才需要的依赖,如
`needs: [{job: setup, if: env.DEPLOY == "true"}]`
### outputs 字段(iter-19
- `TaskSpec` 新增 `outputs: Mapping[str, str] | None` 作为元数据,
仅携带不参与执行,可通过 `spec.outputs` 查询。
- 支持 `${{ matrix.* }}` 占位符替换。
- `to_dict` / `from_json` 往返保留;序列化向前兼容(旧 JSON 无此字段时回退 None)。
### 踩坑总结
- **YAML 加载失败统一包装**`yaml_loader._safe_load` 应将 `yaml.YAMLError`
包装为 `ValueError` 抛出,CLI 层统一捕获处理。否则 YAML 语法错误会
以底层异常形式泄漏到用户面前。
- **不可达防御代码应清理**`_cartesian_product``_parse_condition`
的防御性分支若逻辑上不可达,应删除而非保留 `# pragma: no cover`
## 三、性能优化与基准套件
### 可复用模式
- **独立 `benchmarks/` 目录**(非 pytest 测试),用 `time.perf_counter` 计时,
通过 `python -m benchmarks` CLI 入口运行。覆盖:
- 图构建:10/100/500/1000 节点 DAG 的构建 + validate + layers
- 任务执行:空 fn 任务 × 100/500,四种策略对比
- 上下文注入:有/无依赖、有/无 Context 标注
- 状态后端:MemoryBackend vs JSONBackend vs SQLiteBackend
- **`layers()` 缓存模式**Graph.layers() 每次 run() 都重算拓扑排序,
应缓存。模式:
- 添加 `_layers_cache` 私有字段
- 首次调用计算并缓存
- `add()` / `clear()` 等变更方法中失效缓存
- 参照 `resolved_spec` 已有缓存模式
- **cmd 任务快速路径**:cmd 任务的 `effective_fn` 是无参闭包 `_run()`
无需上下文注入。检测 `spec.fn is None and spec.cmd is not None`
直接返回 `((), {})`,跳过 `build_call_args` 的签名内省与 dep_context 构建。
### 设计决策
- **性能特征(500 空任务)**:
- `sequential`:514 ops/s(最快,无并发开销)—— 小任务/无 I/O 场景首选
- `thread`:93 ops/s(线程池开销)—— I/O 密集型场景
- `async`:44 ops/s(事件循环开销)—— 高并发 I/O 场景
- `dependency`:42 ops/s(最大并行度但调度开销高)—— 复杂依赖图
- **状态后端性能特征**
- `MemoryBackend`save/load 600万/447万 ops/s(首选,无持久化需求时)
- `JSONBackend`save(batch=10)/load 3913/11.9万 ops/s
- `SQLiteBackend`save(batch=10)/load 9657/1.5万 ops/ssave 更快,load 较慢)
- **缓存优化收益**`layers()` 缓存命中 ~1500万 ops/s~50000x 加速);
cmd 快速路径 1130万 ops/s vs fn 无依赖 144万 ops/s~8x 加速)。
### 自实现 Kahn 算法拓扑排序(iter-20
- **方案**:模块级 `_topological_layers(deps)` 函数用 dict + list 直接计算
入度与反向邻接,按层 BFS 处理。返回 `(layers, cycle_nodes)`
- 无环时 `cycle_nodes = None`
- 有环时 `cycle_nodes` 为入度仍 >0 的节点列表(非精确环路径,仅指示存在环)
- **额外优化**`validate()` 顺带填充 `_layers_cache`,使后续 `layers()`
直接命中缓存,避免二次计算 `_topological_layers`
- **结果**diamond(1000) 图构建约 2.21ms~452 ops/s)。
### build_call_args 双快速路径(iter-17
- **快速路径 1cmd 任务)**`spec.fn is None and spec.cmd is not None and
not spec.args and not spec.kwargs` → 直接返回 `((), {})`。
- **快速路径 2(fn 无依赖)**:`not spec.depends_on and not spec.soft_depends_on
and not spec.args and not spec.kwargs` → 调用 `_fn_no_dep_injection(fn)`
获取 `@lru_cache(maxsize=1024)` 缓存的注入计划 `(context_params, error_param)`。
- `context_params`:Context 标注参数名元组;存在必填无默认值参数时返回 `None`
(需走慢路径)。
- `error_param`:第一个无默认值且非 Context 标注的参数名(用于错误信息)。
- **不变量**`error_param is None ⟺ context_params 非 None`,用 `assert`
辅助 pyrefly 类型收窄。
- **提取到 `_try_fast_path` 函数**:避免 `build_call_args` 分支数超 PLR0912
阈值 12。
- **不可哈希 fn 回退**`_cached_signature` 抛 TypeError 时回退到直接内省。
### JSONBackend batch 延迟验证(iter-17
- **问题**`save` 原本每次都 `json.dumps(value)` 验证可序列化性,batch
模式下 N 次 save 产生 N 次验证开销,但 `flush` 只做一次 `json.dump`。
- **方案**`_defer_flush=True` 时跳过 `json.dumps` 验证,让 `flush` 的
`json.dump` 统一捕获 `TypeError`/`ValueError`。非 batch 模式仍即时验证
以提供精确错误位置。
- **依据**:非 batch 调用方期望即时错误反馈;batch 调用方接受延迟错误以换吞吐。
### from_specs 批量注册(iter-17
- **问题**`from_specs` 原本对每个 spec 调 `_register`,而 `_register`
每次都 `_resolved_cache.clear()` + `_layers_cache = None`N 个 spec
产生 N 次缓存清空。
- **方案**:直接写入 `graph.specs` / `graph.deps` 字典,重复名检测在循环内做。
缓存失效由后续 `validate()` / `layers()` 首次调用自然触发(cache 为空时
自动重算)。
- **不扩展到 `add`/`chain`**:增量 API 需即时校验以快速失败;批量构造可
接受延迟校验。
### orjson 可选依赖抽象层(iter-20
- **方案**`_json.py` 模块 `try: import orjson except ImportError: import json`
回退。导出 `dumps`/`loads`/`dump`/`load`/`JSONDecodeError`,签名与 stdlib
兼容。
- **安装**`pip install pyflowx[fast]` 启用 orjson`[project.optional-dependencies]
fast = ["orjson>=3.10.0"]`)。
- **orjson 兼容性差异**
- `dumps` 返回 `bytes` → wrapper 中 `.decode("utf-8")` 统一为 `str`
- 无 `ensure_ascii` 参数 → wrapper 接受并忽略(`# noqa: ARG001`),orjson 始终 UTF-8
- 无 `indent` 参数 → 映射为 `OPT_INDENT_2`
- 紧凑模式无空格(`[1,2,3]` vs stdlib `[1, 2, 3]`)→ 测试断言改为格式无关
- `JSONDecodeError` 直接复用 orjson 的(是 stdlib `json.JSONDecodeError` 子类)
- **pyrefly 兼容**`import orjson` 在 try 块内,所有 `orjson.xxx` 引用加
`# type: ignore[possibly-unbound]`orjson 未安装时 `# type: ignore[import-not-found]`。
### slots=True 内存优化(iter-20
- **方案**:所有 dataclass 添加 `slots=True`,消除 per-instance `__dict__`
开销(约 100-150 bytes/instance)。
- **应用范围**`Graph`/`GraphDefaults`/`TaskSpec`/`TaskResult`/`RetryPolicy`/
`TaskHooks`/`TaskEvent`。
- **冲突处理**`frozen=True` + `slots=True` + `cached_property` 不兼容
(无 `__dict__` 存缓存,frozen 禁止设 slot)→ 将 `TaskSpec.effective_fn`
从 `@cached_property` 改为 `@property`。闭包创建极轻量(仅捕获 `self`),
重复访问开销可忽略。
- **结果**10k TaskSpec 929 → 785 bytes/instance15.5% 减少,1.38 MB 节省)。
### 线程池跨层复用(iter-20
- **问题**`thread` 策略每层创建新 `ThreadPoolExecutor`N 层图创建/销毁
N 次线程池。
- **方案**`_drive_threaded` 创建一个池,跨所有层复用;`max_workers` 取
最大层宽 `max((len(layer) for layer in layers), default=1)`,上限 32。
- **API 变更**`ThreadedLayerRunner.execute` 签名 `max_workers: int` →
`pool: concurrent.futures.ThreadPoolExecutor`,调用方负责池生命周期。
- **额外**`asyncio.get_event_loop()` → `asyncio.get_running_loop()`
兼容 Python 3.12+ 弃用警告(调用点均在 `asyncio.run()` 内的协程中)。
### 踩坑总结
- **`lru_cache` 对签名内省有 dict lookup 开销**:即便 `functools.lru_cache`
缓存了 `_signature(fn)`,每次仍有 dict lookup。对热路径(如 cmd 任务)
应在更外层短路,避免进入 `build_call_args`。
- **`frozen=True` + `slots=True` + `cached_property` 三者不兼容**:选其中
两个。若需 slots 内存优化,将 `cached_property` 改为 `@property`(轻量
闭包场景)或外部 dict 缓存(重计算场景)。
- **orjson 紧凑格式与 stdlib 不同**:跨后端测试需用格式无关断言
`assert "[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
## 四、任务取消与优雅停止
### 可复用模式
- **`CancelToken` 基于 `threading.Event`**:线程安全取消令牌,可跨线程/协程使用。
接口:
- `cancel()` 设置取消状态
- `is_cancelled` 属性检查
- `wait(timeout)` 等待取消信号
- **统一取消检查**`_is_cancelled(effective_cancel) or internal_cancel.is_cancelled`
统一外部取消与 KeyboardInterrupt 的内部取消,避免各处写两段检查逻辑。
- **各策略取消点设计**
- `sequential`:每个任务执行前检查
- `thread`:每层提交前检查;已提交任务等待完成(不强制中断)
- `async`:每个 task 创建前检查;用 `asyncio.CancelledError`
- `dependency`:每个 `_run_task` 创建前检查
- **KeyInterrupt 优雅处理模式**
```python
try:
execute(...)
except KeyboardInterrupt:
cancel_event.set()
# 等待运行中任务完成(最多 5 秒)
# 标记未完成任务为 SKIPPED
report.success = False
finally:
cleanup(...) # 进程池、监控器、通知器
```
### 设计决策
- **取消后待运行任务标记 SKIPPED,运行中任务等待完成**:不强制中断运行中
的任务,避免资源泄漏与状态不一致。这是"优雅停止"的核心定义。
- **`run()` 接受可选 `cancel_event`**:调用方未提供时内部创建。这让
外部代码可以预先持有 cancel_event 引用,在任意时刻触发取消。
- **提取 `_dispatch_strategy` 辅助函数控制分支数**`run()` 函数中
四种策略的分发若用 if/elif 链会导致 PLR0912(分支过多)。提取辅助
函数将 `run()` 的 return 数量控制在 12 以内。
### 踩坑总结
- **资源清理必须在 finally 块**:进程池、监控器、通知器若在 try 块内
清理,KeyboardInterrupt 会跳过清理导致泄漏。
- **`PLR0912`/`PLR0911` 分支/返回过多**:复杂调度逻辑容易触发。优先
提取辅助函数(如 `_dispatch_strategy`、`_print_diagnostics`),
而非用 `# noqa` 抑制。合并语义相同的分支(如 no-args 与 --help
合并)也是有效手段。
## 五、子图执行
### 可复用模式
- **`Graph.subgraph_with_deps(names)` 传递闭包**:从 `names` 出发,
沿 `depends_on` + `soft_depends_on` 向上遍历,收集所有必须先完成的任务。
返回新的 Graph 或任务名集合。
- **`run(only, tags)` 参数组合**
- `only`:任务名列表,运行这些任务及其传递依赖
- `tags`:标签列表,运行匹配任意标签的任务及其传递依赖
- 两者可组合:并集后计算传递闭包
- 都为 None 时运行全图(保持现有行为不变)
- **CLI 选项设计**`pf yamlrun pipeline.yaml --only task1,task2 --tags ingest,report`
逗号分隔,与执行过滤解耦。
### 设计决策
- **传递闭包同时包含软依赖**`subgraph_with_deps` 沿 `depends_on` 与
`soft_depends_on` 双向遍历。软依赖也是必须先完成的任务,不可遗漏。
## 六、结果序列化
### 可复用模式
- **值序列化默认策略**(处理任意 Python 对象):
1. `None` 直接返回 `None`
2. 可 JSON 序列化的原样返回
3. 不可序列化的回退到 `repr(value)`
4. 调用方可通过 `value_serializer` 自定义
- **`to_dict` 结构用列表保序**`results` 为列表(非 dict),便于
CSV/JSON 数组导出,保留执行顺序。
字段:`name/status/value/error/attempts/started_at/finished_at/duration_seconds/reason/tags/depends_on`。
- **CSV 导出省略 value 列**`include_value=False` 时省略 value 列,
因为值可能含逗号/换行符干扰 CSV 解析。
### 设计决策
- **`from_json` 仅供查询,不能重新执行**:反序列化只重建可序列化字段,
`TaskSpec` 的 `fn`/`cmd` 等不可恢复,使用 noop 占位。重建后的 report
仅供查询/分析,不能用于重新执行。这是明确的契约边界。
- **`datetime` 通过 `fromisoformat` 还原**:ISO 8601 字符串往返转换,
避免自定义时间格式。
### HTML 报告导出设计(iter-19
- **自包含 HTML5 文档**:内联 CSS,无外部依赖,便于邮件附件/嵌入式展示。
- **XSS 防护**:所有用户内容(任务名/错误/原因/值)经 `html.escape` 转义。
- **状态徽章着色**success 绿 / failed 红 / skipped 黄 / running 蓝,
通过 CSS class 区分。
- **`_render_summary_card` 静态方法**:渲染汇总卡片(总数/成功/失败/跳过/
耗时),避免在 `to_html` 主流程中重复 HTML 拼接。
- **value 列渲染**:尝试 `dumps(value)` 序列化后转义;失败回退 `repr(value)`。
### 踩坑总结
- **`_noop_fn` 占位函数体无法覆盖**:与 `task.py:_task_noop` 同模式,
占位函数的函数体在测试中无法触发,属于可接受的覆盖率缺口。
- **orjson 紧凑格式与 stdlib 不同**HTML 报告中 value 列断言需格式无关
`"[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
## 七、CLI 可视化与体验
### 可复用模式
- **`pf graph` 多格式输出**
- `--format ascii`(默认):`rich.tree.Tree` 按层分组 + `rich.table.Table` 显示依赖/标签
- `--format mermaid`:复用 `Graph.to_mermaid()` 直接打印(不加颜色,便于复制到 mermaid.live
- `--format list`:仅列任务名(与 `yamlrun --list` 一致)
- `--format deps`:纯依赖关系表
- `--orientation`Mermaid 方向(TD/TB/BT/LR/RL
- **`--color-by tag` 着色模式**:维护 `tag_colors` 字典(标签 → rich 颜色名),
首次见到标签时从调色板循环分配。无标签任务回退默认色。`color_by="none"`
时统一颜色,不显示图例。
- **`--list` 过滤与执行过滤解耦**
- `--only`/`--tags` 控制执行范围
- `--list-tag`/`--list-name` 仅控制 `--list` 模式的显示
- 名称匹配大小写不敏感(子串),标签匹配精确,两者可组合(交集)
- **`pf info [tool]` 工具详情**
- 无参数 → 所有工具简表(工具/别名/子命令数/描述)
- 带参数 → 工具详情(别名、描述、子命令表)
- 支持别名解析(`pf info clearscreen` 等价于 `pf info clr`
### 设计决策
- **shell 补全不依赖第三方库**:直接生成静态脚本
- bash: `complete -F _pf_complete pf` + `compgen -W "..."`
- zsh: `#compdef pf` + `_describe 'command' cmds`
- fish: 每个命令一行 `complete -c pf -n "__fish_use_subcommand" -a "..."`
工具名来自 `_TOOL_ALIASES` 规范名集合,二级补全交给子命令的 `--help`。
- **`--progress` 与 `--quiet` 互斥**:quiet 时无进度条,避免输出冲突。
### 踩坑总结
- **闭包内可变计数器用 `counter[0]` 单元素 list**pyrefly 在闭包内对
父作用域变量的读-写支持不佳,`nonlocal` 会触发类型检查告警。改用
`counter = [0]` 单元素 list,闭包内 `counter[0] += 1` 修改的是 list
内容(非绑定),pyrefly 接受。
- **`_TOOL_REGISTRY` 的 key 类型为 `str | None`**:排序时必须过滤 `None`
否则 `sorted()` 会因类型比较失败抛错。
## 八、错误诊断
### 可复用模式
- **诊断数据结构(frozen dataclass**
- `FailureCluster``pattern`(异常类型+消息前缀)/`tasks`/`sample_error`
- `DependencyChain``failed_task`/`root_cause`/`chain`/`broken_at`
- `DiagnosticReport``failed_tasks`/`skipped_due_to_failure`/`root_causes`/`dependency_chains`/`failure_clusters`/`hints`
- **根因识别算法**
- **根因任务**:FAILED 状态且其所有硬依赖都是 SUCCESS(或无依赖)
- **依赖链**:从根因到失败任务的路径,用 BFS 反向遍历依赖图
- **broken_at**:链中第一个非 SUCCESS 的任务(通常是根因本身)
- **相似失败聚类**:按 `(异常类型名, 消息前 50 字符)` 聚类,
便于发现批量失败模式。
- **根因提示表**:识别常见异常类型并给出建议
- `FileNotFoundError` → 检查路径配置
- `ImportError`/`ModuleNotFoundError` → 检查依赖安装
- `TimeoutError`/`TaskTimeoutError` → 考虑增加 timeout 或优化性能
- `ConnectionError` → 检查网络连通性
- `PermissionError` → 检查权限
- `KeyError` → 检查上下文注入参数
### 设计决策
- **`TaskFailedError` 携带 `report` 字段**:失败时 `_finalize_failure`
传入 report,CLI 层可从异常对象直接获取诊断信息,无需重复构造。
`report` 字段可选(`Optional[RunReport]`),向后兼容。
- **`report.diagnose()` 成功时返回 `None`**:明确区分"无诊断"与"空诊断"
避免调用方误判。
- **CLI 失败时自动打印诊断摘要到 stderr**:`pf yamlrun` 失败时
`report.success == False`)自动调用 `_print_diagnostics` 辅助方法,
无需用户额外参数。
### 踩坑总结
- **CLI 异常处理合并同类**:`TaskFailedError` 与 `PyFlowXError` 的
处理逻辑相似,应合并到同一 except 块,提取 `_print_diagnostics`
辅助方法。重复的 except 分支会增加 PLR0911 风险。
## 九、观测性
### 可复用模式
- **`run_id` 短码生成**`uuid4().hex[:8]` 生成 8 字符十六进制短码,
足以区分并发运行,便于人工阅读。`RunReport.run_id` 字段默认自动生成,
显式指定时尊重调用方。
- **结构化日志 extra 字段标准化**(不用 `LoggerAdapter`):
- `run_id` — 运行 ID(所有日志)
- `task_name` — 任务名(任务级日志)
- `status` — 任务状态(success/failed/skipped/cancelled
- `attempts` — 尝试次数(失败日志)
- `error_type` — 异常类型名(失败日志)
- `strategy` / `total_tasks` — run() 入口/结束日志
- `run_id` 从 `report` 参数读取,无 report 上下文时填 `"-"`
### 设计决策
- **放弃 `LoggerAdapter` 方案**:会强制改写所有 `logger.xxx` 调用点,
且对模块级函数(`_apply_cached`/`_handle_failure`)签名侵入大。
改用直接在每条日志显式 `extra={...}`,散点 extra 已足够,无需过度抽象。
- **`profile()` 便捷方法委托 ProfileReport**
```python
def profile(self, graph: Graph) -> ProfileReport:
from .profiling import ProfileReport
return ProfileReport.from_report(self, graph)
```
`Graph` / `ProfileReport` 仅在 `TYPE_CHECKING` 块导入,避免循环依赖;
`from __future__ import annotations` 使注解延迟求值。
- **序列化向前兼容**`from_json()` 优先恢复 `run_id`,缺失时自动生成
新 ID。旧版 JSON(无 `run_id`)可正常加载。
### 通知器平台扩展(iter-19
- **复用 `WebhookNotifier` 基类**Slack/Discord/Telegram/Feishu/DingTalk/
WeChatNotifier 均继承 `WebhookNotifier`,仅覆盖 `_build_payload(payload: dict) -> dict`
适配平台格式。零新依赖(stdlib `urllib.request`)。
- **TelegramNotifier 覆盖 `__init__`**:额外需要 `chat_id` 参数,
URL 中拼接 `bot_token`。
- **`Notifier` Protocol 生命周期**`notify(level, payload)` + `close()`。
`NotificationLevel` 枚举(RUNNING/SUCCESS/FAILED/SKIPPED+ `ALL_LEVELS`
frozenset`levels` 参数内部过滤。
- **WebhookNotifier 通用字段**`url`/`secret`/`levels`/`timeout`
`_send` 用 `urllib.request.urlopen` 发送 JSON POST`_build_payload`
默认透传(子类按平台格式重写)。
### 踩坑总结
- **循环依赖用 `TYPE_CHECKING` 守卫 + 延迟导入**`report.py` 依赖
`profiling.py` 的 `ProfileReport` 与 `graph.py` 的 `Graph`,但运行时
不需要。`TYPE_CHECKING` 块导入 + `from __future__ import annotations`
注解延迟求值,是打破循环依赖的标准模式。
- **不要为未来场景预留扩展点**:`LoggerAdapter` + `extra` 的统一封装
属于"未来可考虑"项,当前散点 extra 已足够,不应预先抽象。
## 十、API 设计通用模式
### 可复用模式
- **classmethod 委托函数式 API**:如 `Graph.from_yaml(path)` 委托
`yaml_loader.load_yaml(path)`,保持类职责单一,函数式 API 可独立复用。
- **frozen dataclass 表达不可变结果**`FailureCluster`/
`DependencyChain`/`DiagnosticReport` 等只读结果用 `@dataclass(frozen=True)`
自动获得 `__hash__` 与 `__eq__`,适合作为字典 key 或集合元素。
- **列表保序而非 dict**:当顺序有意义(如执行顺序、依赖链路径)时,
用 list 而非 dict 存储结果,便于序列化数组导出。
- **可选参数 None 哨兵**`run(only=None, tags=None)`None 表示
"不限制",与空列表 `[]` 语义区分(空列表通常表示"无匹配")。
- **提取辅助函数控制分支数**`run()`、CLI 命令分发等复杂函数容易
触发 PLR0911/PLR0912。优先提取 `_dispatch_strategy`/
`_print_diagnostics`/`_print_task_list` 等辅助函数,而非用 noqa 抑制
或合并语义不同的分支。
### 设计决策
- **公共 API 必须有完整类型注解与中文 docstring**:项目既有风格,
`python-standards.md` 硬约束。
- **零运行时依赖原则(YAML 例外)**:除 PyYAMLYAML 编排必需)、
rich + typerCLI 必需)、typing-extensions(前向兼容)、orjson(可选
加速,`pip install pyflowx[fast]`)外,不引入运行时依赖。新增依赖须
审慎,优先用标准库。
## 十一、状态后端与缓存
### 可复用模式
- **`StateBackend` 抽象基类**`get`/`put`/`has`/`iter`/`clear` +
`invalidate(key) -> bool`iter-19 新增)。`invalidate` 返回是否实际
删除,便于调用方判断缓存命中。
- **`_TTLStateBackendMixin` 4 原语**`_get_raw`/`_put_raw`/`_iter_raw`/
`_clear_raw`/`_delete_raw`(iter-19 新增)。子类只需实现这 5 个原语,
`get`/`put`/`has`/`iter`/`clear`/`invalidate` 由 mixin 统一组合 TTL 判断。
- **`MemoryBackend` LRU 驱逐(iter-19**
- 用 `OrderedDict` 替代普通 dict
- `_get_raw` 命中时 `move_to_end(key)` 更新访问顺序
- `_put_raw` 超限时 `popitem(last=False)` 驱逐头部(最旧)
- `maxsize=None` 时不驱逐(默认行为,向后兼容)
- **`SQLiteBackend` WAL 模式 + RLock**:线程安全;`batch()` 通过
`_in_batch` 标志延迟 commit,不持锁。
### 设计决策
- **`JSONBackend.invalidate` 覆盖**:非 batch 模式下需 `flush()` 落盘,
否则删除只在内存中生效,进程崩溃后状态丢失。batch 模式下标记
`_dirty=True`,由 `flush()` 统一处理。
- **`SQLiteBackend._delete_raw` 用 `cursor.rowcount > 0`**:判断是否
实际删除,避免先 `has` 后 `delete` 的双次查询。
- **`maxsize` 默认 None 而非 0**:0 会立即驱逐所有项,等价于禁用缓存;
None 表示无限制,与历史行为兼容。
## 十二、CLI 工具错误处理
### 可复用模式
- **`subprocess.run` 的 `check` 参数分级策略**iter-16):
- **失败需感知**dockercmd/msdownload/sglang)→ `check=True` +
`try/except subprocess.CalledProcessError` 打印 rich 错误
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查
`returncode` 打印结果(pkill 返回 1 表示无匹配进程,非错误)
- **失败无关紧要**clr/which)→ `check=False`,加注释说明原因
- **错误路径统一处理模式**
```python
try:
subprocess.run(cmd, check=True, ...)
except subprocess.CalledProcessError as e:
_console.print(f"[red]命令失败 (returncode={e.returncode})[/]")
except FileNotFoundError:
_console.print(f"[red]命令未找到: {cmd[0]}[/]")
```
### 设计决策
- **rich Console 而非 print**CLI 工具错误用 `rich.console.Console`
打印带颜色输出,与项目其他 CLI 工具一致。
- **`check=True` + try/except 优于 `check=False` + 手动检查**:除非
命令返回非零是预期行为(如 taskkill pkill 无匹配、which 未找到命令),
否则用 `check=True` 让 subprocess 抛异常,统一在 except 中处理。
### 踩坑总结
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
装饰器与 `mock.patch.object` 上下文。mock subprocess 异常用
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
- **mock 异常抛出技巧**`lambda *_, **__: (_ for _ in ()).throw(err)`
利用生成器仅在迭代时抛异常的特性,让无参 lambda 能抛出指定异常。
## 十三、测试与质量保障
### 可复用模式
- **`# pragma: no cover` 清理策略**iter-18):按 `python-standards.md`
"pragma: no cover 是清理信号,应激活或删除"原则,逐处分析:
- **逻辑上不可达** → 删除(如 `if result is None` 在已过滤的列表后)
- **防御性代码但实际可触发** → 激活:移除 pragma,写测试覆盖该分支
- **调用方已保证的前置条件** → 保留 pragma,加注释说明依赖契约
- **覆盖率缺口分析方法**
1. 跑 `pytest --cov=src/pyflowx --cov-report=term-missing`
2. 找出 `Stmts`/`Miss` 列中 Miss > 0 的文件
3. 看 `Missing` 列定位未覆盖行号
4. 判断是"未触发分支"还是"不可达代码"
5. 前者补测试,后者清 pragma 或删代码
- **测试命名**`test_<被测对象>_<场景>`,如 `test_json_backend_flush_type_error`。
### 设计决策
- **公共 API 优先测试**:用 `has`/`get` 等公共接口测试,不访问私有方法。
故障注入等场景可临时访问私有属性,docstring 注明原因。
- **Mock 优先级**`monkeypatch` > 内联 stub > `unittest.mock` >
`pytest-mock`。禁用 `@patch` 装饰器、`mock.patch.object` 上下文、
`pytest-mock` 的 `mocker` fixture。
- **`slow` 标记**:耗时测试加 `@pytest.mark.slow`CI 可 `-m "not slow"` 跳过。
### 踩坑总结
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
- **占位函数体无法覆盖**`_noop_fn` 等占位函数的函数体在测试中无法触发
(参数已校验),属可接受的覆盖率缺口,加 `# pragma: no cover` 并注释原因。
- **不可哈希参数回退分支**`lru_cache` 缓存的函数遇到不可哈希参数会抛
TypeError,需 try/except 回退到慢路径。该回退分支需单独写测试(构造
不可哈希 fn 触发)。
+29 -384
View File
@@ -5,8 +5,7 @@
[![CI](https://github.com/gookeryoung/pyflowx/actions/workflows/ci.yml/badge.svg)](https://github.com/gookeryoung/pyflowx/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/pyflowx.svg)](https://pypi.org/project/pyflowx/)
[![Python](https://img.shields.io/pypi/pyversions/pyflowx.svg)](https://pypi.org/project/pyflowx/)
[![Documentation Status](https://readthedocs.org/projects/pyflowx/badge/?version=latest)](https://pyflowx.readthedocs.io/zh/latest/)
[![Coverage](https://img.shields.io/badge/coverage-97%25-brightgreen.svg)](https://github.com/gookeryoung/pyflowx)
[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen.svg)](https://github.com/gookeryoung/pyflowx)
[![License](https://img.shields.io/pypi/l/pyflowx.svg)](https://github.com/gookeryoung/pyflowx/blob/main/LICENSE)
PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖声明**。无需装饰器、
@@ -23,9 +22,8 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **软依赖** —— `soft_depends_on` 仅用于上下文注入,不参与拓扑分层
- **并发限制** —— `concurrency_key` + `concurrency_limits` 按组限流
- **任务钩子** —— `TaskHooks`pre_run/post_run/on_failure)生命周期回调
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘`invalidate(key)` 单键失效
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘
- **缓存键** —— `cache_key` 函数基于输入计算稳定键,使不同输入产生独立缓存
- **缓存驱逐** —— `MemoryBackend(maxsize=N)` LRU 驱逐最旧条目;TTL 过期自动忽略
- **命令任务** —— `cmd` 参数直接执行外部命令,支持列表/shell/可调用对象
- **条件执行** —— `conditions` 参数按平台、环境变量、应用安装等条件跳过任务
- **图组合** —— `compose` / `GraphComposer` 编程式展开多图字符串引用
@@ -33,18 +31,7 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
- **图级默认值** —— `GraphDefaults` 统一配置 retry/timeout/concurrency 等
- **CLI 运行器** —— `CliRunner` 把多个图映射为命令行子命令,替代 Makefile
- **可观测** —— `on_event` 回调(RUNNING/SUCCESS/FAILED/SKIPPED)、`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
- **进度监控** —— `run(progress=True)` 开箱即用的 rich 进度条;`ProgressCallback` 协议供程序化集成
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信/Slack/Discord/Telegram webhook 与 Python 回调,全生命周期事件可配置
- **流式获取** —— `run_iter()` 生成器逐个 yield 任务结果,适用于大量任务逐个处理
- **任务取消** —— `CancelToken` 线程安全取消令牌,支持外部取消与 `KeyboardInterrupt` 优雅停止
- **子图执行** —— `run(only=[...], tags=[...])` 按名称或标签运行图的子集,自动包含传递依赖
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` / `to_html()` 多格式导出运行结果,`from_json()` 支持反序列化重建;HTML 报告自包含内联 CSS,适合浏览器直接打开或邮件分享
- **CLI 可视化** —— `pf graph <yaml>` 终端 DAG 渲染(ASCII 分层树按标签着色 / Mermaid / 依赖表),`pf yamlrun --progress` rich 进度条,`pf info [tool]` 工具详情,`pf completion bash|zsh|fish` shell 补全
- **失败诊断** —— `RunReport.diagnose()` 自动分析根因、依赖链回溯、相似失败聚类与可操作建议;CLI 失败时自动打印诊断摘要
- **可观测性** —— `run_id` 贯穿日志/序列化/诊断;结构化日志 `extra``task_name`/`status`/`attempts`/`error_type``profile(graph)` 离线性能分析(关键路径/并行度/瓶颈)
- **状态后端** —— `MemoryBackend` / `JSONBackend` / `SQLiteBackend`WAL 模式,适合大规模任务)
- **YAML 任务编排** —— GitHub Actions 风格 `jobs`/`needs`/`strategy.matrix`/`if` 条件,`pf yamlrun pipeline.yaml` 一键执行
- **最小依赖** —— `rich` + `typer` + `typing-extensions`3.13 以下)+ `pyyaml`;安装 `orjson` 可加速 JSON 序列化(`pip install pyflowx[fast]`
- **零运行时依赖** —— 仅依赖标准库(3.8 需 `graphlib_backport`
- **97% 测试覆盖** —— 分支覆盖率 >= 95%
## 安装
@@ -158,86 +145,6 @@ resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
引用格式:`"command_name"`(整个图)或 `"command_name.task_name"`(特定任务)。
`CliRunner` 内部自动调用 `compose`
### YAML 任务编排
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`,以及 `matrix.include`/`exclude` 过滤、条件依赖(`needs: [{job: ..., if: ...}]`)、`outputs` 声明:
```yaml
# pipeline.yaml
strategy: thread
defaults:
retry: {max_attempts: 3}
env: {CI: "true"}
jobs:
setup:
cmd: ["echo", "setup"]
runs-on: linux
build:
needs: [setup]
cmd: ["echo", "build-${{ matrix.version }}"]
strategy:
matrix:
version: ["3.10", "3.11", "3.12"]
os: ["linux", "macos"]
exclude:
- version: "3.10"
os: "macos"
include:
- version: "3.13"
os: "linux"
if: "env.CI"
outputs:
version: "${{ matrix.version }}"
deploy:
needs:
- job: build
if: "env.BRANCH == 'main'"
cmd: ["echo", "deploy"]
test:
needs: [build]
cmd: ["echo", "test"]
continue-on-error: true
```
```python
import pyflowx as px
graph = px.Graph.from_yaml("pipeline.yaml")
# 或从字符串解析:graph = px.parse_yaml_string("...")
report = px.run(graph, strategy="thread")
```
CLI 一键执行:
```bash
pf yamlrun pipeline.yaml # 执行
pf yamlrun pipeline.yaml --dry-run # 仅预览,不执行
pf yamlrun pipeline.yaml --list # 列出所有任务
pf yamlrun pipeline.yaml --quiet # 静默模式
pf yamlrun pipeline.yaml --strategy sequential # 覆盖策略
```
**字段映射**
| YAML 字段 | TaskSpec 字段 | 说明 |
|-----------|---------------|------|
| `cmd` / `run` | `cmd` | `cmd` 为列表,`run` 为 shell 字符串 |
| `needs` | `depends_on` | 矩阵 job 依赖自动展开为所有变体 |
| `if` | `conditions` | `success()`/`always()`/`env.VAR`/`env.VAR == 'x'`/`env.VAR != 'x'` |
| `strategy.matrix` | 矩阵扇出 | 笛卡尔积,命名 `{job}_{key}-{value}` |
| `strategy.matrix.include` | 矩阵追加 | 额外组合(可引入新键) |
| `strategy.matrix.exclude` | 矩阵剔除 | 匹配所有 key-value 的组合被移除 |
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env`/`outputs` 中替换 |
| `outputs` | `outputs` | 任务声明的输出键值映射(元数据,不参与执行) |
| `needs: [{job: ..., if: ...}]` | 条件依赖 | `if` 不满足时跳过该依赖 |
| `continue-on-error` | `continue_on_error` | 字段名支持 hyphen/underscore |
| `runs-on` | `tags` | 追加到 tags 元组 |
| `defaults` | `GraphDefaults` | 图级默认值(retry/timeout/env/cwd 等) |
### 任务模板 —— task_template
`task_template` 工厂批量生成相似 TaskSpec:
@@ -393,15 +300,31 @@ runner.run_cli() # 解析 sys.argv 并执行
命令行用法:
```bash
pf pymake clean # 执行 clean 图
pf pymake build --strategy thread # 覆盖执行策略
pf pymake test --dry-run # 仅打印执行计划
pf pymake --list # 列出所有命令
pf pymake --quiet # 静默模式
python build.py clean # 执行 clean 图
python build.py build --strategy thread # 覆盖执行策略
python build.py test --dry-run # 仅打印执行计划
python build.py --list # 列出所有命令
python build.py --quiet # 静默模式
```
`verbose=True`(默认)时打印任务生命周期(开始/成功/失败/跳过)与命令输出;`--quiet` 关闭。
## 示例
仓库 `examples/` 目录包含完整示例:
- [`etl_pipeline.py`](examples/etl_pipeline.py) —— ETL 流水线(sequential
- [`parallel_run.py`](examples/parallel_run.py) —— 并行执行对比(thread vs sequential
- [`async_aggregation.py`](examples/async_aggregation.py) —— 异步聚合 + Context 注入
运行:
```bash
python examples/etl_pipeline.py
python examples/parallel_run.py
python examples/async_aggregation.py
```
## 断点续跑
```python
@@ -429,22 +352,6 @@ px.TaskSpec(
)
```
**LRU 驱逐**`MemoryBackend(maxsize=N)` 限制最大条目数,超限时按 LRU(最近最少使用)
策略驱逐最旧条目。`get`/`has` 命中会将条目移到最近使用位置:
```python
from pyflowx import MemoryBackend
backend = MemoryBackend(maxsize=100) # 最多 100 条,超出驱逐最旧
```
**手动失效**`invalidate(key)` 删除单个键的缓存,`clear()` 清除全部:
```python
backend.invalidate("fetch_user") # 删除单个键,返回是否已删除
backend.clear() # 清除全部
```
## 错误处理
所有错误都是 `PyFlowXError` 的子类:
@@ -474,7 +381,7 @@ except px.PyFlowXError:
| 特性 | PyFlowX | Airflow | Prefect | Dask |
|------|---------|---------|---------|------|
| 零样板 | 参数名即依赖 | 装饰器 + XCom | 装饰器 | 装饰器 |
| 运行时依赖 | rich + typer | 重型 | 中型 | 中型 |
| 运行时依赖 | 仅标准库 | 重型 | 中型 | 中型 |
| 类型安全 | mypy strict | 弱 | 中 | 中 |
| 异步原生 | 是 | 否 | 部分 | 否 |
| 断点续跑 | 内置 | 需配置 | 需配置 | 需配置 |
@@ -521,253 +428,6 @@ px.TaskSpec("low", fn=work, priority=0)
px.TaskSpec("high", fn=work, priority=10) # 同层内先执行
```
### 任务取消
`CancelToken` 提供线程安全的取消信号,传入 `run(cancel_event=...)` 即可外部取消正在执行的图。
`KeyboardInterrupt`(Ctrl+C)也会触发相同的优雅取消流程:
```python
token = px.CancelToken()
# 在另一个线程中触发取消
import threading
def cancel_later():
import time
time.sleep(2)
token.cancel("超时取消")
threading.Thread(target=cancel_later, daemon=True).start()
# 取消后:待运行任务标记 SKIPPED,运行中任务等待完成,report.success = False
report = px.run(graph, strategy="dependency", cancel_event=token)
for name, result in report.results.items():
print(f"{name}: {result.status.value}")
```
也支持标准 `threading.Event` 作为取消信号:
```python
event = threading.Event()
px.run(graph, cancel_event=event)
```
### 子图执行
`run(only=...)` / `run(tags=...)` 只运行指定任务及其传递依赖,适用于调试单个任务或增量执行:
```python
# 只运行 "report" 任务及其所有上游依赖
report = px.run(graph, only=["report"])
# 按标签过滤:只运行带 "ingest" 标签的任务及其依赖
report = px.run(graph, tags=["ingest"])
# 两者可组合(取并集)
report = px.run(graph, only=["report"], tags=["ingest"])
```
CLI 也支持 `--only` / `--tags`
```bash
pf yamlrun pipeline.yaml --only report,deploy
pf yamlrun pipeline.yaml --tags ingest,report
```
### 结果序列化
`RunReport` 支持将运行结果导出为 JSON / dict / CSV,便于持久化、跨进程传输或对接外部工具:
```python
report = px.run(graph, strategy="sequential")
# JSON 字符串(含 success/summary/results
text = report.to_json(indent=2)
# JSON 兼容字典
data = report.to_dict()
# CSV 表格导出(不含 value 列以避免特殊字符干扰)
csv_text = report.to_csv(include_value=False)
```
任务返回值可能是任意 Python 对象(不一定 JSON 可序列化)。默认策略:
可序列化原样保留,不可序列化回退到 `repr(value)`;调用方可通过
`value_serializer` 自定义:
```python
# 自定义值序列化(如把 Path 转为字符串)
text = report.to_json(value_serializer=str)
```
`from_json()` 可从 JSON 字符串重建 `RunReport`**仅供查询/分析**,不能重新
执行(`TaskSpec``fn`/`cmd` 等不可序列化字段无法恢复):
```python
restored = px.RunReport.from_json(text)
print(restored["double"]) # 任务返回值
print(restored.result_of("a").status) # TaskStatus
```
### CLI 可视化
`pf graph <yaml>` 在终端可视化任务图 DAG,无需运行即可检查依赖关系:
```bash
# 默认 ASCII 渲染:分层树(按标签着色) + 依赖关系表
pf graph pipeline.yaml
# 输出 Mermaid 语法(可粘贴到 mermaid.live
pf graph pipeline.yaml --format mermaid
# 自定义方向
pf graph pipeline.yaml --format mermaid --orientation LR
# 仅列出任务名
pf graph pipeline.yaml --format list
# 纯依赖关系表
pf graph pipeline.yaml --format deps
# 关闭标签着色(统一单色)
pf graph pipeline.yaml --color-by none
```
ASCII 模式默认 `--color-by tag`,按任务首标签选择 rich 颜色并在图下方
显示标签颜色图例,便于视觉分组。
`pf yamlrun` 新增 `--progress` 选项,启用 rich 进度条显示运行中任务、完成
数与耗时:
```bash
pf yamlrun pipeline.yaml --progress
```
### 任务列表过滤
`pf yamlrun --list` 支持 `--list-tag` / `--list-name` 过滤,便于大型 YAML
中快速定位任务:
```bash
# 列出所有任务
pf yamlrun pipeline.yaml --list
# 只列出含 ingest 标签的任务
pf yamlrun pipeline.yaml --list --list-tag ingest
# 按任务名子串过滤(大小写不敏感)
pf yamlrun pipeline.yaml --list --list-name extract
# 组合过滤(交集)
pf yamlrun pipeline.yaml --list --list-tag ingest --list-name extract
```
### 工具详情与 Shell 补全
`pf info [tool]` 显示工具详情:
```bash
# 列出所有工具的简表(含子命令数、描述)
pf info
# 显示指定工具的子命令详情
pf info clr
pf info gittool
```
`pf completion <shell>` 生成 shell 补全脚本,提升日常使用体验:
```bash
# bash —— 添加到 ~/.bashrc
pf completion bash >> ~/.bashrc
# zsh —— 添加到 ~/.zshrc
pf completion zsh >> ~/.zshrc
# fish —— 保存到 ~/.config/fish/completions/pf.fish
pf completion fish > ~/.config/fish/completions/pf.fish
```
补全脚本覆盖所有内建命令(`yamlrun`/`graph`/`info`/`completion`)与
`@px.tool` 注册的工具名;二级补全交给具体子命令的 `--help` 自描述。
### 失败诊断
运行失败时,`RunReport.diagnose()` 自动生成结构化诊断报告,包含:
- **根因任务**FAILED 且所有硬依赖非 FAILED 的任务(依赖链中最先出问题的环节)
- **依赖链回溯**:从根因到每个失败任务的路径
- **相似失败聚类**:按 `(异常类型, 消息前缀)` 聚类,发现批量失败模式
- **可操作建议**:识别 FileNotFoundError/ImportError/TimeoutError 等常见异常并给出提示
```python
report = px.run(graph, strategy="sequential")
if not report.success:
diag = report.diagnose()
if diag is not None:
print(diag.describe())
# 根因任务
print(diag.root_causes)
# 依赖链
for chain in diag.dependency_chains:
print(f"{chain.failed_task}: {' -> '.join(chain.chain)}")
```
`pf yamlrun` 失败时自动打印诊断摘要到 stderr:
```bash
pf yamlrun pipeline.yaml # 失败时自动输出诊断报告
```
`report.describe()` 在失败时也会附诊断摘要。
### 可观测性
每次 `run()` 自动生成 8 字符十六进制 `run_id`,作为本次运行的唯一追踪 ID
贯穿日志、序列化结果与诊断报告:
```python
report = px.run(graph)
print(report.run_id) # 如 "a3f4b2c1"
print(report.summary()["run_id"]) # summary 也含 run_id
```
`run_id` 通过结构化日志的 `extra` 字段注入所有执行器日志,便于跨系统关联。
日志格式遵循 `logging` 标准延迟格式化,`extra` 字段包括:
- `run_id` —— 本次运行 ID
- `task_name` —— 任务名(任务级日志)
- `status` —— 任务状态(`running`/`success`/`failed`/`skipped`
- `attempts` —— 尝试次数(失败日志)
- `error_type` —— 异常类型名(失败日志)
```python
import logging
logging.basicConfig(level=logging.INFO)
# 日志输出示例:
# INFO 运行开始: run_id=a3f4b2c1 strategy=dependency tasks=3
# INFO task 'a' skipped (cached)
# WARNING task 'b' failed (attempt 1/3): RuntimeError('boom'); retrying
# INFO 运行结束: run_id=a3f4b2c1 success=False tasks=3
```
`to_json()` / `to_dict()` / `from_json()` 同步保留 `run_id`,反序列化时可还原:
旧版 JSON(无 `run_id` 字段)会自动生成新 ID,向前兼容。
`report.profile(graph)` 返回 `ProfileReport`,基于 `started_at`/`finished_at`
做离线性能分析(关键路径、并行度、Top 瓶颈),零运行时开销:
```python
profile = report.profile(graph)
print(profile.describe()) # 人类可读报告
print(profile.critical_path) # 关键路径任务序列
bottlenecks = profile.top_bottlenecks(5) # Top-5 瓶颈任务
```
## 开发
```bash
@@ -778,17 +438,15 @@ uv sync --extra dev
uv run pytest --cov=pyflowx --cov-fail-under=95
# 类型检查
uv run pyrefly check .
uv run mypy
# 代码风格
uv run ruff check .
uv run ruff format --check .
uv run ruff check src tests examples
uv run ruff format --check src tests examples
```
## 模块结构
### 核心
| 模块 | 职责 |
|------|------|
| `task.py` | 纯数据结构:`TaskSpec``RetryPolicy``TaskHooks``TaskStatus` |
@@ -796,25 +454,12 @@ uv run ruff format --check .
| `compose.py` | 多图组合:`GraphComposer` / `compose` |
| `context.py` | 上下文注入:参数名→依赖解析 |
| `command.py` | 命令执行:`run_command`list/shell/Callable |
| `cancellation.py` | 任务取消:`CancelToken` 线程安全取消令牌 |
| `conditions.py` | 条件执行:内置条件与组合器 |
| `executors.py` | 执行器与 `run` 入口:四种策略共享模块级辅助verbose 统一应用到 spec |
| `executors.py` | 执行器与 `run` 入口:四种策略共享模块级辅助 |
| `storage.py` | 状态后端:`MemoryBackend` / `JSONBackend`batch flush |
| `yaml_loader.py` | YAML 任务编排:`load_yaml` / `parse_yaml_string`GitHub Actions 风格) |
| `runner.py` | CLI 运行器:`CliRunner` |
| `report.py` | 运行结果:`RunReport` / `TaskResult` |
| `tools.py` | `@px.tool` 装饰器:`ToolSpec` / `run_tool` / `list_tools` / `list_subcommands` |
| `profiling.py` | 性能分析:`Profiler` 任务耗时统计 |
| `errors.py` | 错误家族:`PyFlowXError` 子类 |
| `ops/` | CLI 工具模块集合(每个子模块用 `@px.tool` 注册一个工具) |
### CLI 工具
| 模块 | 职责 |
|------|------|
| `cli/pf.py` | 统一入口:`pf <tool> [command]`,按需导入 `ops/<tool>.py` 触发 `@px.tool` 注册 |
| `cli/legacy/profiler.py` | 性能分析 CLIlegacy |
| `cli/legacy/emlmanager.py` | 邮件管理 CLIlegacy |
## 许可证
-51
View File
@@ -1,51 +0,0 @@
"""PyFlowX 性能基准套件.
用法::
python -m benchmarks # 运行全部基准
python -m benchmarks graph # 仅图构建基准
python -m benchmarks execution # 仅执行基准
python -m benchmarks context # 仅上下文注入基准
python -m benchmarks storage # 仅状态后端基准
python -m benchmarks advanced # 仅高级基准
"""
from __future__ import annotations
import time
from collections.abc import Callable
from typing import Any
from rich.console import Console
from rich.table import Table
__all__ = ["print_results", "time_it"]
def time_it(fn: Callable[[], Any], iterations: int = 100, warmup: int = 5) -> tuple[float, float]:
"""计时工具:返回 (平均耗时 ms, 吞吐 ops/sec)."""
for _ in range(warmup):
fn()
times: list[float] = []
for _ in range(iterations):
t0 = time.perf_counter()
fn()
times.append(time.perf_counter() - t0)
avg = sum(times) / len(times)
return avg * 1000, 1.0 / avg if avg > 0 else float("inf")
def print_results(title: str, results: list[tuple[str, int, float, float]]) -> None:
"""打印格式化基准结果表."""
console = Console()
table = Table(title=title, show_header=True, header_style="bold")
table.add_column("场景", style="cyan", no_wrap=True)
table.add_column("迭代", justify="right")
table.add_column("平均耗时", justify="right", style="yellow")
table.add_column("吞吐", justify="right", style="green")
for name, iters, ms, ops in results:
ms_str = f"{ms:.3f} ms" if ms < 1 else f"{ms:.2f} ms"
ops_str = f"{ops:.0f} ops/s" if ops > 1000 else f"{ops:.1f} ops/s"
table.add_row(name, str(iters), ms_str, ops_str)
console.print(table)
console.print()
-449
View File
@@ -1,449 +0,0 @@
"""PyFlowX 性能基准套件.
用法::
python -m benchmarks # 运行全部基准
python -m benchmarks graph # 仅图构建基准
python -m benchmarks execution # 仅执行基准
python -m benchmarks context # 仅上下文注入基准
python -m benchmarks storage # 仅状态后端基准
python -m benchmarks advanced # 仅高级基准
"""
from __future__ import annotations
import math
import sys
import tempfile
from collections.abc import Callable
from pathlib import Path
from typing import Any
from rich.console import Console
import pyflowx as px
from benchmarks import print_results, time_it
from benchmarks.bench_advanced import run_advanced
from pyflowx import Graph, GraphDefaults, RetryPolicy, TaskSpec
from pyflowx._json import _HAS_ORJSON
from pyflowx._json import dumps as _dumps
from pyflowx._json import loads as _loads
from pyflowx.context import build_call_args
from pyflowx.storage import JSONBackend, MemoryBackend, SQLiteBackend
# ============================================================================
# 图生成工具
# ============================================================================
def make_chain(n: int) -> list[TaskSpec]:
"""生成 n 个任务的链式 DAG。"""
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
for i in range(1, n):
specs[i] = TaskSpec(f"t{i}", cmd=["true"], depends_on=(f"t{i - 1}",))
return specs
def make_diamond(n: int) -> list[TaskSpec]:
"""生成 n 个任务的菱形 DAG(每层宽度约 sqrt(n))。"""
width = max(1, int(math.sqrt(n)))
specs: list[TaskSpec] = []
prev_layer: list[str] = []
layer = 0
count = 0
while count < n:
cur_layer: list[str] = []
for j in range(width):
if count >= n:
break
name = f"l{layer}_t{j}"
deps = tuple(prev_layer) if prev_layer else ()
specs.append(TaskSpec(name, cmd=["true"], depends_on=deps))
cur_layer.append(name)
count += 1
prev_layer = cur_layer
layer += 1
return specs
def make_wide(n: int) -> list[TaskSpec]:
"""生成 n 个独立任务(无依赖,最大并行度)。"""
return [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
# ============================================================================
# 基准:图构建
# ============================================================================
def bench_construction() -> None:
"""图构建(from_specs + validate)基准。"""
results = []
for n in (10, 100, 500, 1000):
specs = make_chain(n)
ms, _ = time_it(lambda s=specs: Graph.from_specs(s), iterations=20)
results.append((f"chain({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
for n in (10, 100, 500, 1000):
specs = make_diamond(n)
ms, _ = time_it(lambda s=specs: Graph.from_specs(s), iterations=20)
results.append((f"diamond({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
print_results("图构建 (from_specs + validate)", results)
def bench_layers() -> None:
"""拓扑分层基准(冷启动 vs 缓存命中)。"""
results = []
for n in (100, 500, 1000):
specs = make_diamond(n)
graph = Graph.from_specs(specs)
def _cold(g: Graph = graph) -> None:
g._layers_cache = None # type: ignore[attr-defined]
g.layers()
ms_cold, ops_cold = time_it(_cold, iterations=50, warmup=5)
results.append((f"layers(cold,{n})", 50, ms_cold, ops_cold))
ms_hot, ops_hot = time_it(lambda g=graph: g.layers(), iterations=200, warmup=10)
results.append((f"layers(cached,{n})", 200, ms_hot, ops_hot))
print_results("拓扑分层 (layers)", results)
def bench_resolved_spec() -> None:
"""resolved_spec 缓存命中基准。"""
results = []
for n in (100, 500, 1000):
specs = make_chain(n)
defaults = GraphDefaults(retry=RetryPolicy(max_attempts=2))
graph = Graph.from_specs(specs, defaults=defaults)
name = f"t{n // 2}"
ms, ops = time_it(lambda g=graph, nm=name: g.resolved_spec(nm), iterations=500, warmup=20)
results.append((f"resolved_spec(cached,{n})", 500, ms, ops))
print_results("resolved_spec (缓存命中)", results)
def run_graph() -> None:
"""运行全部图基准。"""
bench_construction()
bench_layers()
bench_resolved_spec()
# ============================================================================
# 基准:任务执行
# ============================================================================
def bench_sequential() -> None:
"""sequential 策略执行基准。"""
results = []
def noop() -> None:
pass
for n in (50, 200, 500):
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="sequential"), iterations=10, warmup=2)
results.append((f"sequential({n})", 10, ms, ops))
print_results("执行策略: sequential", results)
def bench_thread() -> None:
"""thread 策略执行基准。"""
results = []
def noop() -> None:
pass
for n in (50, 200, 500):
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="thread"), iterations=10, warmup=2)
results.append((f"thread({n})", 10, ms, ops))
print_results("执行策略: thread", results)
def bench_async() -> None:
"""async 策略执行基准。"""
results = []
def noop() -> None:
pass
for n in (50, 200, 500):
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="async"), iterations=10, warmup=2)
results.append((f"async({n})", 10, ms, ops))
print_results("执行策略: async", results)
def bench_dependency() -> None:
"""dependency 策略执行基准。"""
results = []
def noop() -> None:
pass
for n in (50, 200, 500):
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="dependency"), iterations=10, warmup=2)
results.append((f"dependency({n})", 10, ms, ops))
print_results("执行策略: dependency", results)
def bench_cmd_execution() -> None:
"""cmd 任务执行基准(真实子进程)。"""
results = []
for n in (10, 50, 100):
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="sequential"), iterations=5, warmup=1)
results.append((f"cmd-sequential({n})", 5, ms, ops))
for n in (10, 50, 100):
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
graph = Graph.from_specs(specs)
ms, ops = time_it(lambda g=graph: px.run(g, strategy="thread", max_workers=8), iterations=5, warmup=1)
results.append((f"cmd-thread({n})", 5, ms, ops))
print_results("cmd 任务执行 (['true'])", results)
def run_execution() -> None:
"""运行全部执行基准。"""
bench_sequential()
bench_thread()
bench_async()
bench_dependency()
bench_cmd_execution()
# ============================================================================
# 基准:上下文注入
# ============================================================================
def bench_context_no_deps() -> None:
"""无依赖 fn 任务的上下文注入基准。"""
results = []
def noop() -> None:
pass
spec = TaskSpec("noop", fn=noop)
context: dict[str, Any] = {}
ms, ops = time_it(lambda s=spec, c=context: build_call_args(s, c), iterations=2000, warmup=100)
results.append(("fn(no-deps)", 2000, ms, ops))
# cmd 任务快速路径
spec_cmd = TaskSpec("cmd", cmd=["true"])
ms, ops = time_it(lambda s=spec_cmd, c=context: build_call_args(s, c), iterations=2000, warmup=100)
results.append(("cmd(fast-path)", 2000, ms, ops))
print_results("上下文注入 (build_call_args)", results)
def bench_context_with_deps() -> None:
"""有依赖 fn 任务的上下文注入基准。"""
results = []
def consumer(a: int, b: int) -> int:
return a + b
spec = TaskSpec("consumer", fn=consumer, depends_on=("a", "b"))
context = {"a": 1, "b": 2, "c": 3}
ms, ops = time_it(lambda s=spec, c=context: build_call_args(s, c), iterations=2000, warmup=100)
results.append(("fn(2-deps)", 2000, ms, ops))
# Context 标注
from pyflowx.task import Context
def ctx_fn(ctx: Context) -> int:
return sum(ctx.values())
spec_ctx = TaskSpec("ctx", fn=ctx_fn, depends_on=("a", "b"))
ms, ops = time_it(lambda s=spec_ctx, c=context: build_call_args(s, c), iterations=2000, warmup=100)
results.append(("fn(Context-annotated)", 2000, ms, ops))
# **kwargs
def kwargs_fn(**kwargs: int) -> int:
return sum(kwargs.values())
spec_kw = TaskSpec("kw", fn=kwargs_fn, depends_on=("a", "b"))
ms, ops = time_it(lambda s=spec_kw, c=context: build_call_args(s, c), iterations=2000, warmup=100)
results.append(("fn(**kwargs)", 2000, ms, ops))
print_results("上下文注入 (有依赖)", results)
def run_context() -> None:
"""运行全部上下文注入基准。"""
bench_context_no_deps()
bench_context_with_deps()
# ============================================================================
# 基准:状态后端
# ============================================================================
def bench_storage() -> None:
"""状态后端 save/load 基准。"""
results = []
# MemoryBackend
mem = MemoryBackend()
ms, ops = time_it(lambda: mem.save("key", "value"), iterations=1000, warmup=50)
results.append(("MemoryBackend.save", 1000, ms, ops))
ms, ops = time_it(mem.load, iterations=1000, warmup=50)
results.append(("MemoryBackend.load", 1000, ms, ops))
# JSONBackendbatch 模式)
tmp_dir = tempfile.mkdtemp()
json_path = str(Path(tmp_dir) / "state.json")
json_backend = JSONBackend(json_path)
with json_backend.batch():
for i in range(100):
json_backend.save(f"task_{i}", f"result_{i}")
def _json_save() -> None:
jb = JSONBackend(json_path)
with jb.batch():
for i in range(10):
jb.save(f"bench_{i}", f"val_{i}")
ms, ops = time_it(_json_save, iterations=50, warmup=5)
results.append(("JSONBackend.save(batch=10)", 50, ms, ops))
# 复杂 value(嵌套 dict)—— 展示 batch 模式延迟验证的优化效果
complex_value = {
"output": {"path": "/data/result.json", "size": 1024, "checksum": "abc123"},
"metrics": {"accuracy": 0.95, "latency_ms": 120, "samples": 1000},
"artifacts": [f"artifact_{j}.bin" for j in range(10)],
"metadata": {"version": "1.0", "tags": ["trained", "validated"], "created_at": "2026-07-07"},
}
def _json_save_complex() -> None:
jb = JSONBackend(json_path)
with jb.batch():
for i in range(10):
jb.save(f"bench_complex_{i}", complex_value)
ms, ops = time_it(_json_save_complex, iterations=50, warmup=5)
results.append(("JSONBackend.save(batch=10,complex)", 50, ms, ops))
ms, ops = time_it(json_backend.load, iterations=200, warmup=10)
results.append(("JSONBackend.load", 200, ms, ops))
# SQLiteBackend
db_path = str(Path(tmp_dir) / "state.db")
sqlite_backend = SQLiteBackend(db_path)
with sqlite_backend.batch():
for i in range(100):
sqlite_backend.save(f"task_{i}", f"result_{i}")
def _sqlite_save() -> None:
sb = SQLiteBackend(db_path)
with sb.batch():
for i in range(10):
sb.save(f"bench_{i}", f"val_{i}")
ms, ops = time_it(_sqlite_save, iterations=50, warmup=5)
results.append(("SQLiteBackend.save(batch=10)", 50, ms, ops))
ms, ops = time_it(sqlite_backend.load, iterations=200, warmup=10)
results.append(("SQLiteBackend.load", 200, ms, ops))
print_results("状态后端 (save/load)", results)
# 序列化基准:_json 抽象层 dumps/loads
ser_results = []
report_like = {
"run_id": "abc12345",
"success": True,
"results": [
{
"name": f"task_{i}",
"status": "success",
"attempts": 1,
"duration_seconds": 0.123 + i * 0.001,
"value": {"output": [i, i + 1, i + 2], "meta": {"tag": "api"}},
}
for i in range(100)
],
}
serialized = _dumps(report_like)
ms, ops = time_it(lambda: _dumps(report_like), iterations=200, warmup=10)
ser_results.append(("dumps(report-100)", 200, ms, ops))
ms, ops = time_it(lambda: _loads(serialized), iterations=200, warmup=10)
ser_results.append(("loads(report-100)", 200, ms, ops))
backend_name = "orjson" if _HAS_ORJSON else "stdlib"
print_results(f"JSON 序列化 (后端={backend_name})", ser_results)
# 清理临时目录
import shutil
shutil.rmtree(tmp_dir, ignore_errors=True)
def run_storage() -> None:
"""运行全部状态后端基准。"""
bench_storage()
# ============================================================================
# 主入口
# ============================================================================
BENCH_MODULES: dict[str, Callable[[], None]] = {
"graph": run_graph,
"execution": run_execution,
"context": run_context,
"storage": run_storage,
"advanced": run_advanced,
}
def main(argv: list[str] | None = None) -> int:
"""CLI 入口。"""
args = argv if argv is not None else sys.argv[1:]
console = Console()
console.print("[bold cyan]PyFlowX 性能基准套件[/bold cyan]\n")
if not args or args[0] in ("--all", "-a"):
for name, fn in BENCH_MODULES.items():
console.print(f"[bold]运行: {name}[/bold]")
fn()
elif args[0] in BENCH_MODULES:
BENCH_MODULES[args[0]]()
elif args[0] in ("--help", "-h"):
console.print("用法: python -m benchmarks [graph|execution|context|storage]")
console.print(" 无参数 = 运行全部基准")
else:
console.print(f"[red]未知基准模块: {args[0]}[/red]")
console.print(f"可用: {', '.join(BENCH_MODULES)}")
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
-309
View File
@@ -1,309 +0,0 @@
"""高级基准:条件评估/YAML 加载/通知器/run_iter/子图/取消/CPU+I/O 密集型任务.
这些基准覆盖核心调度引擎的扩展路径,与 bench_graph/bench_execution/bench_context/
bench_storage 互补,构成完整的性能度量体系。
"""
from __future__ import annotations
import contextlib
import tempfile
import time
from pathlib import Path
from typing import Any
import pyflowx as px
from benchmarks import print_results, time_it
from pyflowx import Graph, TaskSpec
from pyflowx.cancellation import CancelToken
from pyflowx.conditions import BuiltinConditions
from pyflowx.notification import CallbackNotifier, NotificationLevel
# ============================================================================
# 基准:条件评估
# ============================================================================
def bench_conditions() -> None:
"""条件评估基准(静态/上下文/复合)."""
results: list[tuple[str, int, float, float]] = []
# 静态条件(IS_LINUX
cond_static = BuiltinConditions.IS_LINUX()
ctx: dict[str, Any] = {}
ms, ops = time_it(lambda c=cond_static, x=ctx: c(x), iterations=10000, warmup=500)
results.append(("static(IS_LINUX)", 10000, ms, ops))
# 上下文条件(DEP_EQUALS
cond_dep = BuiltinConditions.DEP_EQUALS("a", 1)
ctx_dep = {"a": 1, "b": 2}
ms, ops = time_it(lambda c=cond_dep, x=ctx_dep: c(x), iterations=10000, warmup=500)
results.append(("DEP_EQUALS", 10000, ms, ops))
# 复合条件(AND(OR(NOT, DEP_TRUTHY), DEP_PRESENT)
cond_complex = BuiltinConditions.AND(
BuiltinConditions.OR(
BuiltinConditions.NOT(BuiltinConditions.IS_WINDOWS()),
BuiltinConditions.DEP_TRUTHY("a"),
),
BuiltinConditions.DEP_PRESENT("b"),
)
ms, ops = time_it(lambda c=cond_complex, x=ctx_dep: c(x), iterations=10000, warmup=500)
results.append(("AND(OR(NOT,DEP_TRUTHY),DEP_PRESENT)", 10000, ms, ops))
print_results("条件评估", results)
# ============================================================================
# 基准:YAML 加载
# ============================================================================
_YAML_TEMPLATE = """\
jobs:
{jobs}
defaults:
retry:
max_attempts: 2
backoff: 0.0
"""
def _make_yaml(n: int) -> str:
"""生成 n 个任务的 YAML(链式依赖)."""
lines = []
for i in range(n):
deps = f"needs: [t{i - 1}]" if i > 0 else ""
lines.append(f" t{i}:\n cmd: ['true']\n {deps}".rstrip())
return _YAML_TEMPLATE.format(jobs="\n".join(lines))
def bench_yaml_load() -> None:
"""YAML 加载基准(解析 + Graph 构建)."""
results: list[tuple[str, int, float, float]] = []
tmp_dir = tempfile.mkdtemp()
for n in (10, 50, 100):
yaml_text = _make_yaml(n)
yaml_path = Path(tmp_dir) / f"bench_{n}.yaml"
yaml_path.write_text(yaml_text, encoding="utf-8")
ms, _ops = time_it(lambda p=yaml_path: Graph.from_yaml(p), iterations=20, warmup=2)
results.append((f"yaml_load({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
print_results("YAML 加载 (from_yaml)", results)
import shutil
shutil.rmtree(tmp_dir, ignore_errors=True)
# ============================================================================
# 基准:通知器
# ============================================================================
def bench_notifiers() -> None:
"""通知器 notify 调用开销基准."""
results: list[tuple[str, int, float, float]] = []
# 构造一个真实 TaskEvent
from pyflowx.task import TaskEvent, TaskStatus
event = TaskEvent(task="t0", status=TaskStatus.SUCCESS, attempts=1)
# CallbackNotifier(无级别过滤)
calls: list[int] = [0]
def _cb(_e: Any) -> None:
calls[0] += 1
notifier = CallbackNotifier(_cb)
ms, ops = time_it(lambda e=event, n=notifier: n.notify(e), iterations=50000, warmup=1000)
results.append(("CallbackNotifier.notify", 50000, ms, ops))
# 级别过滤(仅 SUCCESS
notifier_filtered = CallbackNotifier(_cb, levels={NotificationLevel.SUCCESS})
ms, ops = time_it(lambda e=event, n=notifier_filtered: n.notify(e), iterations=50000, warmup=1000)
results.append(("CallbackNotifier(filtered).notify", 50000, ms, ops))
print_results("通知器 notify", results)
# ============================================================================
# 基准:run_iter 流式 API
# ============================================================================
def bench_run_iter() -> None:
"""run_iter 流式执行基准."""
results: list[tuple[str, int, float, float]] = []
def noop() -> None:
pass
for n in (50, 200, 500):
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
graph = Graph.from_specs(specs)
def _run_iter(g: Graph = graph) -> None:
list(px.run_iter(g, strategy="sequential"))
ms, ops = time_it(_run_iter, iterations=10, warmup=2)
results.append((f"run_iter(sequential,{n})", 10, ms, ops))
print_results("run_iter 流式执行", results)
# ============================================================================
# 基准:子图过滤
# ============================================================================
def bench_subgraph() -> None:
"""子图过滤基准(subgraph_with_deps 传递闭包计算)."""
results: list[tuple[str, int, float, float]] = []
for n in (100, 500, 1000):
specs = []
for i in range(n):
deps = (f"t{i - 1}",) if i > 0 else ()
specs.append(TaskSpec(f"t{i}", cmd=["true"], depends_on=deps))
graph = Graph.from_specs(specs)
# 取中点任务,触发前半部分传递闭包
target = f"t{n // 2}"
ms, ops = time_it(
lambda g=graph, t=target: g.subgraph_with_deps([t]),
iterations=50,
warmup=5,
)
results.append((f"subgraph_with_deps({n})", 50, ms, ops))
print_results("子图过滤 (subgraph_with_deps)", results)
# ============================================================================
# 基准:取消机制
# ============================================================================
def bench_cancellation() -> None:
"""取消机制基准(cancel_event 触发到返回)."""
results: list[tuple[str, int, float, float]] = []
def slow() -> None:
time.sleep(0.5)
for n in (50, 200):
specs = [TaskSpec(f"t{i}", fn=slow) for i in range(n)]
graph = Graph.from_specs(specs)
def _cancel_after(g: Graph = graph) -> None:
token = CancelToken()
# 立即取消,让所有任务变 SKIPPED
token.cancel()
with contextlib.suppress(Exception):
px.run(g, strategy="sequential", cancel_event=token)
ms, ops = time_it(_cancel_after, iterations=5, warmup=1)
results.append((f"cancel(immediate,{n})", 5, ms, ops))
print_results("取消机制 (cancel_event)", results)
# ============================================================================
# 基准:CPU 密集型任务
# ============================================================================
def _fib(n: int) -> int:
"""递归斐波那契(CPU 密集型)."""
if n < 2:
return n
return _fib(n - 1) + _fib(n - 2)
def bench_cpu_intensive() -> None:
"""CPU 密集型任务基准(递归斐波那契)."""
results: list[tuple[str, int, float, float]] = []
for n_tasks, fib_n in ((10, 20), (20, 20), (10, 25)):
specs = [TaskSpec(f"t{i}", fn=_fib, args=(fib_n,)) for i in range(n_tasks)]
graph = Graph.from_specs(specs)
# sequential
ms, ops = time_it(
lambda g=graph: px.run(g, strategy="sequential"),
iterations=3,
warmup=1,
)
results.append((f"cpu-seq({n_tasks}x fib{fib_n})", 3, ms, ops))
# threadCPU 密集型受 GIL 限制,验证是否退化)
ms, ops = time_it(
lambda g=graph: px.run(g, strategy="thread", max_workers=4),
iterations=3,
warmup=1,
)
results.append((f"cpu-thread({n_tasks}x fib{fib_n})", 3, ms, ops))
print_results("CPU 密集型 (递归斐波那契)", results)
# ============================================================================
# 基准:I/O 密集型任务
# ============================================================================
def bench_io_intensive() -> None:
"""I/O 密集型任务基准(sleep 模拟)."""
results: list[tuple[str, int, float, float]] = []
def io_sleep() -> None:
time.sleep(0.01) # 10ms 模拟 I/O
for n in (20, 50):
specs = [TaskSpec(f"t{i}", fn=io_sleep) for i in range(n)]
graph = Graph.from_specs(specs)
# sequential(总耗时 ≈ n * 10ms
ms, ops = time_it(
lambda g=graph: px.run(g, strategy="sequential"),
iterations=3,
warmup=1,
)
results.append((f"io-seq({n})", 3, ms, ops))
# thread(I/O 密集型应能并行,总耗时 ≈ 10ms)
ms, ops = time_it(
lambda g=graph: px.run(g, strategy="thread", max_workers=8),
iterations=3,
warmup=1,
)
results.append((f"io-thread({n})", 3, ms, ops))
# async(I/O 密集型应能并行)
ms, ops = time_it(
lambda g=graph: px.run(g, strategy="async"),
iterations=3,
warmup=1,
)
results.append((f"io-async({n})", 3, ms, ops))
print_results("I/O 密集型 (sleep 10ms)", results)
# ============================================================================
# 主入口
# ============================================================================
def run_advanced() -> None:
"""运行全部高级基准."""
bench_conditions()
bench_yaml_load()
bench_notifiers()
bench_run_iter()
bench_subgraph()
bench_cancellation()
bench_cpu_intensive()
bench_io_intensive()
-102
View File
@@ -1,102 +0,0 @@
API 参考
========
任务描述
--------
.. autoclass:: pyflowx.TaskSpec
:members:
:undoc-members:
:show-inheritance:
:exclude-members: args, kwargs
.. autoclass:: pyflowx.RetryPolicy
:members:
:undoc-members:
.. autoclass:: pyflowx.TaskHooks
:members:
:undoc-members:
.. autoclass:: pyflowx.TaskStatus
:members:
:undoc-members:
图构建
------
.. autoclass:: pyflowx.Graph
:members:
:undoc-members:
:exclude-members: from_specs, from_yaml
.. autoclass:: pyflowx.GraphDefaults
:members:
:undoc-members:
.. autofunction:: pyflowx.compose
.. autofunction:: pyflowx.task_template
执行
----
.. autofunction:: pyflowx.run
.. autoclass:: pyflowx.RunReport
:members:
:undoc-members:
.. autoclass:: pyflowx.TaskResult
:members:
:undoc-members:
@px.tool 工具
-------------
.. autoclass:: pyflowx.ToolSpec
:members:
:undoc-members:
.. autofunction:: pyflowx.tool
.. autofunction:: pyflowx.run_tool
.. autofunction:: pyflowx.list_tools
.. autofunction:: pyflowx.list_subcommands
命令执行
--------
.. autofunction:: pyflowx.run_command
CLI 运行器
----------
.. autoclass:: pyflowx.CliRunner
:members:
:undoc-members:
状态后端
--------
.. autoclass:: pyflowx.StateBackend
:members:
:undoc-members:
.. autoclass:: pyflowx.MemoryBackend
:members:
:undoc-members:
.. autoclass:: pyflowx.JSONBackend
:members:
:undoc-members:
错误家族
--------
.. autoexception:: pyflowx.PyFlowXError
.. autoexception:: pyflowx.DuplicateTaskError
.. autoexception:: pyflowx.MissingDependencyError
.. autoexception:: pyflowx.CycleError
.. autoexception:: pyflowx.TaskFailedError
.. autoexception:: pyflowx.TaskTimeoutError
.. autoexception:: pyflowx.InjectionError
.. autoexception:: pyflowx.StorageError
-45
View File
@@ -1,45 +0,0 @@
变更日志
========
0.4.5
-----
CLI 重构
~~~~~~~~
- 新增 ``pf`` 统一入口:通过 ``pf <tool> [command] [options]`` 调用所有工具
- 13 个工具迁移到 YAML 配置(filedate/filelevel/folderback/folderzip/screenshot/sshcopyid/lscalc/bumpversion/autofmt/piptool/packtool/pdftool/gittool
- YAML 配置支持 ``cli:`` 段声明命令行参数 schema,由 ``build_cli_parser`` 自动生成 argparse
- 删除 13 个冗余 ``.py`` 入口脚本,统一通过 ``pf`` 调用
- ``run()````verbose=True`` 时自动把 verbose 标记应用到所有 spec
- 全局选项 ``--verbose`` 改为 ``--quiet``(默认显示执行过程)
- ``cmd`` 任务成功时打印 stdout(此前被静默丢弃)
- ``gittool````CLEAN_EXCLUDES`` 数组变量配置 ``git clean -e`` 参数
YAML 任务编排
~~~~~~~~~~~~~
- 支持 ``variables`` 变量定义,``${VAR}`` 在 cmd/env/cwd 中替换
- 列表变量展开为 cmd 数组多个元素
- ``cli:`` 段支持 subcommands/positional/options 三级 schema
- 支持 ``type: path`` 自动转为 ``pathlib.Path``
文档
~~~~
- 搭建 Sphinx 文档,发布到 ReadTheDocs
- 更新 READMECLI 示例改为 ``pf`` 统一入口,模块结构表补全
0.3.x
-----
- 新增 YAML 任务编排(GitHub Actions 风格 schema
- 新增 ``fn:`` 函数引用与 ``register_fn`` / ``get_fn`` 注册中心
- 新增 ``compose`` / ``GraphComposer`` 多图组合
- 新增 ``task_template`` 任务模板工厂
- 新增 ``concurrency_key`` + ``concurrency_limits`` 并发限制
- 新增 ``JSONBackend`` 断点续跑与 ``batch()`` 批量落盘
- 新增 ``cache_key`` 缓存键函数
- 新增条件执行(``IS_WINDOWS`` / ``HAS_INSTALLED`` / ``ENV_VAR_EQUALS`` 等)
- 四种执行策略:``sequential`` / ``thread`` / ``async`` / ``dependency``
- 参数名即依赖的上下文注入机制
-65
View File
@@ -1,65 +0,0 @@
"""Sphinx 配置.
ReadTheDocs 构建 PyFlowX 文档站。
"""
from __future__ import annotations
import sys
from pathlib import Path
# 确保 src/ 在 sys.path 中, autodoc 能导入 pyflowx
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
from pyflowx import __version__
# -- 项目信息 --------------------------------------------------------------
project = "PyFlowX"
author = "pyflowx"
copyright = "2024, pyflowx"
release = __version__
version = __version__
# -- Sphinx 配置 -----------------------------------------------------------
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.viewcode",
"sphinx.ext.intersphinx",
"myst_parser",
]
# -- 主题 ------------------------------------------------------------------
html_theme = "sphinx_rtd_theme"
html_static_path = ["_static"]
# -- autodoc 配置 ----------------------------------------------------------
autodoc_default_options = {
"members": True,
"undoc-members": True,
"show-inheritance": True,
"member-order": "bysource",
}
autodoc_type_hints = "description"
autodoc_typehints_format = "short"
# -- napoleon 配置 (Google/NumPy docstring 兼容) --------------------------
napoleon_google_docstring = True
napoleon_numpy_docstring = True
napoleon_include_init_with_doc = False
napoleon_include_private_with_doc = False
napoleon_include_special_with_doc = True
# -- intersphinx -----------------------------------------------------------
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
}
# -- 全局选项 ---------------------------------------------------------------
language = "zh_CN"
master_doc = "index"
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
source_suffix = {
".rst": "restructuredtext",
".md": "markdown",
}
-158
View File
@@ -1,158 +0,0 @@
pf 统一 CLI 入口
================
所有工具通过 ``pf <tool> [command] [options]`` 调用。工具定义在 ``cli/configs/`` 目录下的 YAML 文件中。
基本用法
--------
.. code-block:: bash
pf # 列出所有可用工具
pf filedate # 查看 filedate 工具帮助
pf filedate add a.txt # 调用 filedate 的 add 子命令
pf gitt c # 调用 gittool 的 c 子命令
pf pymake b # 调用 pymake 的 b 别名
全局选项
--------
所有 YAML 工具支持以下全局选项:
.. list-table::
:header-rows: 1
:widths: 25 75
* - 选项
- 说明
* - ``--dry-run``
- 仅打印执行计划,不执行
* - ``--quiet`` / ``-q``
- 减少输出,不显示执行过程
* - ``--strategy``
- 执行策略(``sequential`` / ``thread`` / ``async`` / ``dependency``
* - ``--list``
- 列出所有任务名后退出
默认 ``verbose`` 开启,显示执行过程(任务开始/命令/返回码/任务成功)。``--quiet`` 关闭。
YAML 配置工具
--------------
.. list-table::
:header-rows: 1
:widths: 20 15 65
* - 工具
- 别名
- 说明
* - ``filedate``
- ``fd``
- 文件日期处理
* - ``filelevel``
- ``fl``
- 文件等级重命名
* - ``folderback``
- ``fb``
- 文件夹备份
* - ``folderzip``
- ``fz``
- 文件夹压缩
* - ``gittool``
- ``gitt``
- Git 执行工具
* - ``lscalc``
- ``ls``
- LS-DYNA 计算工具
* - ``packtool``
- ``pack``
- Python 打包工具
* - ``pdftool``
- ``pdf``
- PDF 文件工具集
* - ``piptool``
- ``pip``
- pip 包管理工具
* - ``screenshot``
- ``ss``
- 截图工具
* - ``sshcopyid``
- ``ssh``
- SSH 密钥部署工具
* - ``autofmt``
- ``af``
- 自动格式化工具
* - ``bumpversion``
- ``bump``
- 版本号自动管理工具
传统工具
--------
.. list-table::
:header-rows: 1
:widths: 20 80
* - 工具
- 说明
* - ``pymake``
- 构建工具(替代 Makefile),如 ``pf pymake b`` 构建
* - ``yamlrun``
- YAML pipeline 执行器,``pf yamlrun pipeline.yaml``
* - ``profiler``
- 性能分析
* - ``emlman``
- 邮件管理
* - ``reseticon``
- 重置图标缓存
自定义工具
----------
``cli/configs/`` 目录新建 ``<tool>.yaml`` 即可被 ``pf`` 自动发现:
.. code-block:: yaml
# cli/configs/mytool.yaml
strategy: sequential
variables:
MSG: "hello"
cli:
description: "我的工具"
usage: "pf mytool [command]"
subcommands:
greet:
help: "打招呼"
jobs:
greet:
cmd: ["echo", "${MSG}"]
执行::
pf mytool greet
CliRunner(编程式)
-------------------
``CliRunner`` 把多个 Graph 映射为命令行子命令,适合构建项目专属构建工具:
.. code-block:: python
runner = px.CliRunner(
strategy="sequential",
description="My Build Tool",
graphs={
"clean": clean_graph,
"build": build_graph,
"test": test_graph,
},
)
runner.run_cli() # 解析 sys.argv 并执行
命令行::
pf pymake clean
pf pymake build --strategy thread
pf pymake test --dry-run
pf pymake --list
pf pymake --quiet
-93
View File
@@ -1,93 +0,0 @@
执行策略与 run()
=================
``run()`` 是执行入口,支持四种策略:
.. code-block:: python
report = px.run(
graph,
strategy="async", # sequential | thread | async | dependency
max_workers=8, # thread 策略的线程池大小
concurrency_limits={"db": 2}, # 按 concurrency_key 限流
dry_run=False, # True = 仅打印计划
verbose=True, # True = 打印执行过程
on_event=callback, # 状态转换回调
state=px.JSONBackend("state.json"), # 断点续跑后端
continue_on_error=False, # True = 单任务失败不中断整体
)
策略对比
--------
.. list-table::
:header-rows: 1
:widths: 18 18 30 16 18
* - 策略
- 并发模型
- 适用场景
- 同步任务
- 异步任务
* - ``sequential``
- 串行
- 调试、CPU 密集
- 直接调用
- 事件循环
* - ``thread``
- 线程池
- I/O 密集同步
- 线程池
- 不支持
* - ``async``
- 事件循环
- I/O 密集异步
- 卸载到线程池
- 事件循环
* - ``dependency``
- 依赖驱动
- 最大化并行度
- 卸载到线程池
- 事件循环
所有策略都遵循 ``RetryPolicy````timeout``、上下文注入、状态后端、``concurrency_limits``
并发出 ``TaskEvent``RUNNING/SUCCESS/FAILED/SKIPPED)。``dependency`` 策略无层屏障:
任务在其所有硬依赖完成后立即启动。
上下文注入规则
--------------
按顺序求值:
1. **标注为 ``Context``** 的参数 → 接收完整上游结果映射
2. **名称匹配依赖** 的参数 → 接收该依赖的结果(含软依赖,缺失时注入默认值)
3. **``**kwargs``** 参数 → 接收所有依赖结果(dict)
4. **``TaskSpec.args`` / ``kwargs``** → 为非依赖参数提供静态值
.. code-block:: python
from typing import Any, Dict
def aggregate(ctx: px.Context) -> Dict[str, Any]:
"""ctx 包含所有 depends_on 任务的返回值。"""
return dict(ctx)
def merge(fetch_a: str, fetch_b: str) -> str:
"""fetch_a / fetch_b 自动注入。"""
return fetch_a + fetch_b
断点续跑
--------
.. code-block:: python
from pyflowx import JSONBackend
backend = JSONBackend("state.json", ttl=3600)
report = px.run(graph, strategy="sequential", state=backend)
``run()`` 内部以 ``backend.batch()`` 包裹整个执行:所有 ``save`` 延迟到运行结束时统一落盘一次。
缓存键:默认存储键为任务名。配置 ``cache_key`` 函数后,键为 ``"name:cache_key_value"``
完整 API 说明详见 :doc:`/api`
-50
View File
@@ -1,50 +0,0 @@
Graph —— DAG 构建
=================
``Graph`` 管理任务集合,提供建构建、校验、分层、可视化能力。
构建方式
--------
.. code-block:: python
# 图级默认值:TaskSpec 字段为 None 时回退
defaults = px.GraphDefaults(retry=px.RetryPolicy(max_attempts=2), timeout=60.0)
graph = px.Graph.from_specs([...], defaults=defaults) # 整批校验(推荐)
# 或增量构建
graph = px.Graph(defaults=defaults)
graph.add(px.TaskSpec("a", fn_a))
graph.add(px.TaskSpec("b", fn_b, ("a",)))
常用方法
--------
.. code-block:: python
graph.validate() # 显式校验(环检测)
graph.layers() # 拓扑分层(Kahn 算法)
graph.to_mermaid() # Mermaid 可视化
graph.describe() # 人类可读摘要
graph.subgraph(("api",)) # 按标签切片
graph.subgraph_by_names(("a", "b")) # 按名称切片
graph.map("fetch", [1, 2, 3], lambda i: TaskSpec(f"fetch_{i}", ...)) # 批量 fan-out
图组合
------
``compose`` / ``GraphComposer`` 把带字符串引用的多个图展开为纯 ``Graph``
.. code-block:: python
graphs = {
"build": px.Graph.from_specs([px.TaskSpec("b", cmd=["echo", "b"])]),
"all": px.Graph.from_specs(["build", px.TaskSpec("t", cmd=["echo", "t"])]),
}
resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
引用格式:``"command_name"``(整个图)或 ``"command_name.task_name"``(特定任务)。
``CliRunner`` 内部自动调用 ``compose``
完整方法说明详见 :doc:`/api`
-89
View File
@@ -1,89 +0,0 @@
TaskSpec —— 任务描述
=====================
``TaskSpec`` 是不可变的任务描述符(``Generic[T]``,返回类型一路传到 ``RunReport``),是唯一需要配置的东西。
主要参数说明:
.. code-block:: python
px.TaskSpec(
name="fetch_user", # 唯一标识
fn=fetch_user, # 同步或异步函数
cmd=["curl", "..."], # 或: 执行命令(覆盖 fn)
depends_on=("auth",), # 硬依赖(参与拓扑分层)
soft_depends_on=("cache",), # 软依赖(仅注入,不参与分层)
args=(uid,), # 静态位置参数(追加在注入参数后)
kwargs={"timeout": 30}, # 静态关键字参数
retry=px.RetryPolicy(max_attempts=3, delay=1.0, backoff=2.0),
timeout=30.0, # 超时秒数(None = 不限制)
tags=("api", "user"), # 自由标签,用于子图过滤
conditions=(is_prod,), # 条件函数列表(全部为 True 才执行)
priority=10, # 同层内优先级(高优先执行,默认 0)
concurrency_key="db", # 并发分组键(配合 concurrency_limits 限流)
cache_key=lambda ctx: str(ctx.get("uid")), # 缓存键函数
hooks=px.TaskHooks(pre_run=..., post_run=..., on_failure=...),
cwd=Path("/tmp"), # 命令工作目录(仅 cmd 模式)
env={"DEBUG": "1"}, # 环境变量覆盖
verbose=True, # 打印命令输出(仅 cmd 模式)
skip_if_missing=True, # 命令不存在时自动跳过(仅 list[str] cmd
allow_upstream_skip=False, # 上游 SKIPPED/FAILED 时是否仍执行
continue_on_error=False, # 本任务失败是否不中断整体
)
两种任务形态
------------
- **函数任务**``fn``):普通 Python 函数,参数名驱动自动注入
- **命令任务**``cmd``):执行外部命令,支持 ``list[str]````str``shell)、``Callable`` 三种形态
``skip_if_missing=True`` 时,``list[str]`` 类型的 ``cmd`` 会通过 ``shutil.which`` 检查命令是否存在,不存在则跳过任务(标记为 ``SKIPPED``)而非失败。
重试策略
--------
``RetryPolicy`` 配置重试次数、延迟、退避:
.. code-block:: python
retry = px.RetryPolicy(
max_attempts=3, # 最大尝试次数
delay=1.0, # 初始延迟秒数
backoff=2.0, # 退避倍数
jitter=0.1, # 随机抖动(避免惊群)
retry_on=(ConnectionError,), # 仅对这些异常重试
)
任务钩子
--------
``TaskHooks`` 在任务生命周期触发(异常仅记录,不影响任务状态):
.. code-block:: python
hooks = px.TaskHooks(
pre_run=lambda spec: print(f"start {spec.name}"),
post_run=lambda spec, value: print(f"done {spec.name}"),
on_failure=lambda spec, exc: alert(spec.name, exc),
)
px.TaskSpec("task", fn=work, hooks=hooks)
任务模板
--------
``task_template`` 工厂批量生成相似 TaskSpec
.. code-block:: python
fetch = px.task_template(
fn=fetch_url,
retry=px.RetryPolicy(max_attempts=5),
timeout=30.0,
tags=("api",),
)
graph = px.Graph.from_specs([
fetch("users", url="https://api.example.com/users"),
fetch("posts", url="https://api.example.com/posts"),
])
完整字段说明详见 :doc:`/api`
-164
View File
@@ -1,164 +0,0 @@
YAML 任务编排
=============
PyFlowX 支持 GitHub Actions 风格的声明式 YAML 任务编排,从 YAML 文件直接加载任务图。
编程式 API
----------
.. code-block:: python
import pyflowx as px
# 从 YAML 文件加载任务图
graph = px.Graph.from_yaml("pipeline.yaml")
report = px.run(graph, strategy="thread")
# 或用函数式 API
graph = px.load_yaml("pipeline.yaml")
# 从字符串解析
graph = px.parse_yaml_string("""
jobs:
hello:
cmd: ["echo", "hello"]
""")
YAML Schema
-----------
.. code-block:: yaml
strategy: thread # 图级默认策略
defaults: # 图级默认值
retry: {max_attempts: 3}
verbose: true
env: {CI: "true"}
variables: # 变量定义 (可在 cmd/env 中 ${VAR} 引用)
OUTPUT: "dist"
jobs:
setup:
cmd: ["git", "clone", "..."]
runs-on: linux
build:
needs: [setup] # 依赖列表
cmd: ["python", "-m", "build"]
timeout: 300
retry: {max_attempts: 2, delay: 1.0}
test:
needs: [build]
cmd: ["python${{ matrix.version }}", "-m", "pytest"]
strategy:
matrix: # 笛卡尔积展开为 6 个任务
version: ["3.8", "3.9", "3.10"]
os: ["linux", "macos"]
if: "env.CI" # 条件: 环境变量存在
lint:
needs: [build]
cmd: ["ruff", "check"]
if: "env.CI == 'true'"
deploy:
needs: [test, lint] # 矩阵依赖自动展开
cmd: ["twine", "upload"]
if: "env.DEPLOY_TOKEN != ''"
allow-upstream-skip: true
concurrency-key: deploy_lock
字段映射
--------
.. list-table::
:header-rows: 1
:widths: 30 30 40
* - YAML 字段
- TaskSpec 字段
- 说明
* - ``jobs.<id>``
- ``name``
- job ID 作为任务名
* - ``cmd`` / ``run``
- ``cmd``
- ``cmd`` 为列表形式,``run`` 为 shell 字符串
* - ``needs``
- ``depends_on``
- 依赖列表(矩阵任务自动展开)
* - ``if``
- ``conditions``
- ``success()`` / ``always()`` / ``env.VAR`` / ``env.VAR == 'x'``
* - ``strategy.matrix``
- 矩阵扇出
- 笛卡尔积展开为多个任务
* - ``${{ matrix.key }}``
- 占位符
- 在 cmd/run/cwd/env 中替换
* - ``timeout``
- ``timeout``
- 超时秒数
* - ``retry``
- ``retry``
- ``{max_attempts, delay, backoff, jitter}``
* - ``cwd``
- ``cwd``
- 工作目录
* - ``env``
- ``env``
- 环境变量
* - ``verbose``
- ``verbose``
- 详细输出
* - ``continue-on-error``
- ``continue_on_error``
- 失败不中止整图
* - ``skip-if-missing``
- ``skip_if_missing``
- 命令不存在时跳过
* - ``allow-upstream-skip``
- ``allow_upstream_skip``
- 上游跳过时仍执行
* - ``priority``
- ``priority``
- 同层优先级
* - ``concurrency-key``
- ``concurrency_key``
- 并发限制键
* - ``tags``
- ``tags``
- 自由标签
* - ``runs-on``
- ``tags``(追加)
- 运行环境标签
CLI 配置段(``cli:``
----------------------
工具 YAML 还可定义 ``cli:`` 段,声明命令行参数 schema,由 ``pf`` 自动解析:
.. code-block:: yaml
cli:
description: "FileDate - 文件日期处理工具"
usage: "pf filedate <command> [files...]"
subcommands:
add:
help: "添加日期前缀"
positional:
- name: FILES
nargs: "+"
type: path
help: "文件路径"
options:
- name: CLEAR
flag: "--clear"
action: store_true
help: "清除已有日期前缀"
支持的 ``type````str`` / ``int`` / ``float`` / ``path``
完整 API 说明详见 :doc:`/api`
-54
View File
@@ -1,54 +0,0 @@
PyFlowX 文档
============
PyFlowX 是一个轻量、类型安全的 DAG 任务调度器:**参数名就是依赖声明**
无需装饰器、无需样板包装器,写一个普通函数,框架按参数名自动注入上游结果。
特性
----
- **零样板** —— 参数名即依赖,框架自动注入上游结果
- **四种执行策略** —— sequential(串行)、thread(线程池)、async(事件循环)、dependency(依赖驱动,最大化并行)
- **类型安全** —— ``TaskSpec[T]`` 把返回类型一路传到 ``RunReport``
- **DAG 校验** —— 构建时即时校验重名、缺失依赖、环
- **自动分层** —— Kahn 算法分组,同层任务可并行
- **重试与超时** —— 每个任务独立配置 ``RetryPolicy````timeout``
- **并发限制** —— ``concurrency_key`` + ``concurrency_limits`` 按组限流
- **断点续跑** —— ``MemoryBackend`` / ``JSONBackend``,成功结果可缓存复用
- **命令任务** —— ``cmd`` 参数直接执行外部命令
- **条件执行** —— ``conditions`` 按平台、环境变量等条件跳过任务
- **pf 统一 CLI** —— ``pf <tool> [command]`` 调用所有工具
- **最小依赖** —— ``rich`` + ``typer`` + ``typing-extensions``3.13 以下)
文档导航
--------
.. toctree::
:maxdepth: 2
:caption: 入门
installation
quickstart
.. toctree::
:maxdepth: 2
:caption: 用户指南
guide/task
guide/graph
guide/execution
guide/cli
.. toctree::
:maxdepth: 2
:caption: 参考
api
changelog
索引
----
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
-51
View File
@@ -1,51 +0,0 @@
安装
====
PyFlowX 支持 Python 3.10+,运行时依赖 ``rich````typer````pyyaml````typing-extensions``3.13 以下)。
pip 安装
--------
.. code-block:: bash
pip install pyflowx
uv 安装
-------
推荐使用 `uv <https://docs.astral.sh/uv/>`_
.. code-block:: bash
uv add pyflowx
可选依赖
--------
``office`` —— PDF/图片处理(pdftool、screenshot 等工具需要):
.. code-block:: bash
pip install pyflowx[office]
``dev`` —— 开发工具链(ruff、pyrefly、pytest、tox 等):
.. code-block:: bash
pip install pyflowx[dev]
验证安装
--------
.. code-block:: bash
pf --version
输出示例::
PyFlowX 0.4.5
下一步
------
前往 :doc:`quickstart` 开始使用。
-87
View File
@@ -1,87 +0,0 @@
快速上手
========
核心思想:**参数名即依赖**。写一个普通函数,参数名匹配上游任务名,框架自动注入结果。
最小示例
--------
.. code-block:: python
import pyflowx as px
def extract() -> list[int]:
return [1, 2, 3]
# 参数名 extract 自动匹配上游任务名 → 自动注入
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
graph = px.Graph.from_specs([
px.TaskSpec("extract", extract),
px.TaskSpec("double", double, ("extract",)),
])
report = px.run(graph, strategy="sequential")
print(report["double"]) # [2, 4, 6]
三种任务形态
------------
1. **函数任务**``fn``):普通 Python 函数,参数名驱动自动注入
2. **命令任务**``cmd``):执行外部命令,支持 ``list[str]`` / ``str``shell/ ``Callable``
3. **YAML 声明式**:从 YAML 文件加载任务图
.. code-block:: python
graph = px.Graph.from_specs([
px.TaskSpec("list", cmd=["ls", "-la"]),
px.TaskSpec("greet", fn=lambda: "hello"),
])
执行策略
--------
PyFlowX 提供四种执行策略:
.. list-table::
:header-rows: 1
:widths: 20 20 60
* - 策略
- 并发模型
- 适用场景
* - ``sequential``
- 串行
- 调试、CPU 密集
* - ``thread``
- 线程池
- I/O 密集同步
* - ``async``
- 事件循环
- I/O 密集异步
* - ``dependency``
- 依赖驱动
- 最大化并行度(默认推荐)
.. code-block:: python
report = px.run(graph, strategy="dependency")
结果访问
--------
.. code-block:: python
report["task_name"] # 任务返回值
report.result_of("task_name") # 完整 TaskResult
report.success # 整体是否成功
report.summary() # 统计字典
report.failed_tasks() # 失败任务名列表
下一步
------
- :doc:`guide/task` —— TaskSpec 详细配置
- :doc:`guide/yaml` —— YAML 声明式任务编排
- :doc:`guide/cli` —— ``pf`` 统一 CLI 入口
+37 -18
View File
@@ -7,26 +7,49 @@ classifiers = [
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Topic :: Software Development :: Libraries :: Application Frameworks",
]
dependencies = [
"pyyaml>=6.0.1",
"rich>=13.7.0",
"typer>=0.24.0",
"typing-extensions>=4.13.2; python_version < '3.13'",
"graphlib_backport >= 1.0.0; python_version < '3.9'",
"typing-extensions>=4.13.2; python_version < '3.10'",
]
description = "Lightweight, type-safe DAG task scheduler with multi-strategy execution."
keywords = ["async", "dag", "scheduler", "task", "workflow"]
license = { text = "MIT" }
name = "pyflowx"
readme = "README.md"
requires-python = ">=3.10"
version = "0.4.8"
requires-python = ">=3.8"
version = "0.3.2"
[project.scripts]
emlman = "pyflowx.cli.legacy.emlmanager:main"
pf = "pyflowx.cli.pf:main"
pxp = "pyflowx.cli.legacy.profiler:main"
autofmt = "pyflowx.cli.autofmt:main"
bumpversion = "pyflowx.cli.bumpversion:main"
dockercmd = "pyflowx.cli.dev.dockercmd:main"
emlman = "pyflowx.cli.emlmanager:main"
filedate = "pyflowx.cli.filedate:main"
filelvl = "pyflowx.cli.filelevel:main"
foldback = "pyflowx.cli.folderback:main"
foldzip = "pyflowx.cli.folderzip:main"
gitt = "pyflowx.cli.gittool:main"
lscalc = "pyflowx.cli.lscalc:main"
msdown = "pyflowx.cli.llm.msdownload:main"
packtool = "pyflowx.cli.packtool:main"
pdftool = "pyflowx.cli.pdftool:main"
piptool = "pyflowx.cli.piptool:main"
pxp = "pyflowx.cli.profiler:main"
pymake = "pyflowx.cli.pymake:main"
reseticon = "pyflowx.cli.reseticoncache:main"
scrcap = "pyflowx.cli.screenshot:main"
sglang = "pyflowx.cli.llm.sglang:main"
sshcopy = "pyflowx.cli.sshcopyid:main"
# dev
envdev = "pyflowx.cli.dev.envdev:main"
# system
clr = "pyflowx.cli.system.clearscreen:main"
taskk = "pyflowx.cli.system.taskkill:main"
wch = "pyflowx.cli.system.which:main"
[project.optional-dependencies]
dev = [
@@ -44,8 +67,6 @@ dev = [
"tox-uv>=1.13.1",
"tox>=4.25.0",
]
docs = ["myst-parser>=3.0", "sphinx-rtd-theme>=2.0", "sphinx>=7.0"]
fast = ["orjson>=3.10.0"]
office = [
"pillow>=10.4.0",
"pymupdf>=1.24.11",
@@ -74,12 +95,12 @@ packages = ["src/pyflowx"]
pyflowx = { workspace = true }
[dependency-groups]
dev = ["pyflowx[dev,docs,fast,office]"]
dev = ["pyflowx[dev,office]"]
[tool.coverage.run]
branch = true
concurrency = ["thread"]
omit = ["src/pyflowx/cli/*", "tests/*"]
omit = ["src/pyflowx/cli/*", "src/pyflowx/examples/*", "tests/*"]
source = ["pyflowx"]
[tool.coverage.report]
@@ -99,7 +120,7 @@ markers = ["slow: marks tests as slow (deselect with
# Ruff 配置 - 与 .pre-commit-config.yaml 保持一致
[tool.ruff]
line-length = 120
target-version = "py310"
target-version = "py38"
[tool.ruff.lint]
ignore = [
@@ -132,11 +153,9 @@ select = [
]
[tool.ruff.lint.per-file-ignores]
"**/tests/**" = ["ARG001", "ARG002"]
"src/pyflowx/tools.py" = ["PLR0911"]
"**/tests/**" = ["ARG001", "ARG002"]
[tool.pyrefly]
preset = "strict"
project-includes = ["**/*.ipynb", "**/*.py*"]
python-version = "3.10"
search-path = ["."]
python-version = "3.8"
+13 -58
View File
@@ -13,7 +13,7 @@
* :class:`GraphDefaults` —— 图级默认值。
* :func:`compose` —— 编程式组合多图。
* :func:`task_template` —— 批量生成相似 TaskSpec 的工厂。
* 状态后端::class:`StateBackend`、:class:`MemoryBackend`、:class:`JSONBackend`、:class:`SQLiteBackend`
* 状态后端::class:`StateBackend`、:class:`MemoryBackend`、:class:`JSONBackend`。
快速上手
--------
@@ -58,12 +58,18 @@
from __future__ import annotations
from .cancellation import CancelToken
from .command import run_command
from .compose import GraphComposer, compose
from .conditions import IS_LINUX, IS_MACOS, IS_POSIX, IS_WINDOWS, BuiltinConditions, Condition, Constants
from .conditions import (
IS_LINUX,
IS_MACOS,
IS_POSIX,
IS_WINDOWS,
BuiltinConditions,
Condition,
Constants,
)
from .context import Context, build_call_args, describe_injection
from .diagnostics import DependencyChain, DiagnosticReport, FailureCluster, diagnose
from .errors import (
CycleError,
DuplicateTaskError,
@@ -74,31 +80,14 @@ from .errors import (
TaskFailedError,
TaskTimeoutError,
)
from .executors import Strategy, run, run_iter
from .executors import Strategy, run
from .graph import Graph, GraphDefaults
from .history import RunHistory
from .monitoring import MetricsCollector, health_check, start_metrics_server
from .notification import (
ALL_LEVELS,
CallbackNotifier,
DingTalkNotifier,
DiscordNotifier,
FeishuNotifier,
NotificationLevel,
Notifier,
SlackNotifier,
TelegramNotifier,
WebhookNotifier,
WeChatNotifier,
)
from .profiling import ProfileReport, TaskProfile
from .progress import ProgressCallback, RichProgressMonitor
from .report import RunReport
from .runner import CliExitCode, CliRunner
from .storage import JSONBackend, MemoryBackend, SQLiteBackend, StateBackend
from .storage import JSONBackend, MemoryBackend, StateBackend
from .task import (
CacheKeyFn,
LoopSpec,
RetryPolicy,
TaskCmd,
TaskEvent,
@@ -110,54 +99,34 @@ from .task import (
task,
task_template,
)
from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool
from .yaml_loader import load_yaml, parse_yaml_string
__version__ = "0.4.8"
__version__ = "0.4.2"
__all__ = [
"ALL_LEVELS",
"IS_LINUX",
"IS_MACOS",
"IS_POSIX",
"IS_WINDOWS",
"BuiltinConditions",
"CacheKeyFn",
"CallbackNotifier",
"CancelToken",
"CliExitCode",
"CliRunner",
"Condition",
"Constants",
"Context",
"CycleError",
"DependencyChain",
"DiagnosticReport",
"DingTalkNotifier",
"DiscordNotifier",
"DuplicateTaskError",
"FailureCluster",
"FeishuNotifier",
"Graph",
"GraphComposer",
"GraphDefaults",
"InjectionError",
"JSONBackend",
"LoopSpec",
"MemoryBackend",
"MetricsCollector",
"MissingDependencyError",
"NotificationLevel",
"Notifier",
"ProfileReport",
"ProgressCallback",
"PyFlowXError",
"RetryPolicy",
"RichProgressMonitor",
"RunHistory",
"RunReport",
"SQLiteBackend",
"SlackNotifier",
"StateBackend",
"StorageError",
"Strategy",
@@ -170,26 +139,12 @@ __all__ = [
"TaskSpec",
"TaskStatus",
"TaskTimeoutError",
"TelegramNotifier",
"ToolSpec",
"WeChatNotifier",
"WebhookNotifier",
"build_call_args",
"cmd",
"compose",
"describe_injection",
"diagnose",
"health_check",
"list_subcommands",
"list_tools",
"load_yaml",
"parse_yaml_string",
"run",
"run_command",
"run_iter",
"run_tool",
"start_metrics_server",
"task",
"task_template",
"tool",
]
-100
View File
@@ -1,100 +0,0 @@
"""JSON 序列化抽象层。
优先使用 orjson(性能更高),回退到标准库 json。对外提供与标准库
json 兼容的 ``dumps`` / ``loads`` / ``dump`` / ``load`` / ``JSONDecodeError``
接口,使调用方无感知切换。
orjson 与标准库的差异由本模块吸收:
- ``dumps`` 返回 ``str``orjson 原生返回 ``bytes``,此处统一 decode
- ``ensure_ascii`` 参数对 orjson 无效(始终输出 UTF-8),此处忽略
- ``indent`` 映射为 ``orjson.OPT_INDENT_2``
"""
from __future__ import annotations
__all__ = ["JSONDecodeError", "dump", "dumps", "load", "loads"]
from typing import IO, Any
try:
import orjson # type: ignore[import-not-found]
_HAS_ORJSON = True
except ImportError: # pragma: no cover
_HAS_ORJSON = False
if _HAS_ORJSON:
JSONDecodeError = orjson.JSONDecodeError # type: ignore[possibly-unbound]
_OPT_NON_STR_KEYS = orjson.OPT_NON_STR_KEYS # type: ignore[possibly-unbound]
def dumps(
obj: Any,
*,
ensure_ascii: bool = False, # noqa: ARG001 — 兼容标准库签名
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> str:
"""序列化为 JSON 字符串(orjson 后端)。"""
opts = _OPT_NON_STR_KEYS
if indent is not None:
opts |= orjson.OPT_INDENT_2 # type: ignore[possibly-unbound]
return orjson.dumps(obj, default=default, option=opts).decode("utf-8") # type: ignore[possibly-unbound]
def loads(s: str | bytes) -> Any:
"""从 JSON 字符串/字节反序列化(orjson 后端)。"""
return orjson.loads(s) # type: ignore[possibly-unbound]
def dump(
obj: Any,
fh: IO[str],
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> None:
"""序列化并写入文件句柄(orjson 后端)。"""
fh.write(dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default))
def load(fh: IO[str]) -> Any:
"""从文件句柄反序列化(orjson 后端)。"""
return orjson.loads(fh.read()) # type: ignore[possibly-unbound]
else: # pragma: no cover
import json as _stdlib
JSONDecodeError = _stdlib.JSONDecodeError
def dumps(
obj: Any,
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> str:
"""序列化为 JSON 字符串(标准库后端)。"""
return _stdlib.dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default)
def loads(s: str | bytes) -> Any:
"""从 JSON 字符串/字节反序列化(标准库后端)。"""
return _stdlib.loads(s)
def dump(
obj: Any,
fh: IO[str],
*,
ensure_ascii: bool = False,
indent: int | None = None,
default: Any = None,
**_kwargs: Any,
) -> None:
"""序列化并写入文件句柄(标准库后端)。"""
_stdlib.dump(obj, fh, ensure_ascii=ensure_ascii, indent=indent, default=default)
def load(fh: IO[str]) -> Any:
"""从文件句柄反序列化(标准库后端)。"""
return _stdlib.load(fh)
-114
View File
@@ -1,114 +0,0 @@
"""任务取消与优雅停止。
提供 :class:`CancelToken` 作为线程安全的取消信号,可传入 :func:`pyflowx.run`
实现外部取消或响应 ``KeyboardInterrupt``。
设计要点
--------
* 基于 :class:`threading.Event`,跨线程/协程安全。
* 轻量:不引入额外线程或回调,仅维护一个事件标志。
* 可组合:多个 CancelToken 可通过 :meth:`link` 联动。
"""
from __future__ import annotations
import threading
from typing import Any
__all__ = ["CancelToken"]
class CancelToken:
"""线程安全的取消令牌。
用法::
token = CancelToken()
# 在另一个线程中:
token.cancel()
# 传入 run()
report = px.run(graph, cancel_event=token)
也可直接用 :class:`threading.Event` 代替——本类仅提供语义更清晰的 API。
"""
__slots__ = ("_event", "_reason")
def __init__(self) -> None:
self._event = threading.Event()
self._reason: str | None = None
def cancel(self, reason: str | None = None) -> None:
"""设置取消信号。
Parameters
----------
reason:
可选的取消原因描述,供日志或调试使用。
"""
if reason is not None:
self._reason = reason
self._event.set()
@property
def is_cancelled(self) -> bool:
"""是否已取消。"""
return self._event.is_set()
@property
def reason(self) -> str | None:
"""取消原因(未取消时为 None)。"""
return self._reason
def wait(self, timeout: float | None = None) -> bool:
"""等待取消信号。
Returns
-------
bool
``True`` 表示已取消,``False`` 表示超时。
"""
return self._event.wait(timeout)
def link(self, event: threading.Event) -> None:
"""联动一个 ``threading.Event``:当 ``event`` 被设置时,本 token 也被取消。
适用于将标准 ``threading.Event`` 桥接到 CancelToken。
"""
def _watcher() -> None:
event.wait()
self.cancel()
t = threading.Thread(target=_watcher, daemon=True)
t.start()
def __repr__(self) -> str:
status = "cancelled" if self._event.is_set() else "active"
reason = f", reason={self._reason!r}" if self._reason else ""
return f"CancelToken({status}{reason})"
# 兼容 threading.Event 接口:run() 内部用 .is_set() 检查
def is_set(self) -> bool:
"""兼容 :meth:`threading.Event.is_set`。"""
return self._event.is_set()
# 允许 ``CancelToken | threading.Event`` 统一类型检查
def __bool__(self) -> bool:
return self._event.is_set()
def _is_cancelled(token: Any) -> bool:
"""统一检查 CancelToken 或 threading.Event 是否已取消。
Parameters
----------
token:
``None``、``CancelToken`` 或 ``threading.Event``。
"""
if token is None:
return False
if isinstance(token, CancelToken):
return token.is_cancelled
return bool(token.is_set())
@@ -1,60 +1,52 @@
"""autofmt - 自动格式化工具.
"""自动格式化工具模块.
提供格式化代码/代码检查/自动添加文档/同步配置 子命令.
提供 Python 代码自动格式化的常用功能封装,
支持 docstring 自动生成pyproject.toml 配置同步等功能.
"""
from __future__ import annotations
import argparse
import ast
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx.ops._common import IGNORE_PATTERNS
__all__ = [
"add_docstring",
"auto_add_docstrings",
"fmt",
"format_all",
"format_with_ruff",
"generate_module_docstring",
"lint",
"lint_with_ruff",
"sync_pyproject_config",
try:
import tomllib # noqa: F401
HAS_TOMLLIB = True
except ImportError:
HAS_TOMLLIB = False
# ============================================================================
# 配置
# ============================================================================
IGNORE_PATTERNS = [
"__pycache__",
"*.pyc",
"*.pyo",
".git",
".venv",
".idea",
".vscode",
"*.egg-info",
"dist",
"build",
".pytest_cache",
".tox",
".mypy_cache",
]
# ============================================================================
# 辅助函数
# ============================================================================
@px.tool("autofmt", subcommand="fmt", help="格式化代码")
def fmt(target: str = ".") -> None:
"""格式化代码 (ruff format).
Parameters
----------
target : str
目标路径 (默认: .)
"""
format_with_ruff(Path(target), fix=False)
@px.tool("autofmt", subcommand="lint", help="代码检查")
def lint(target: str = ".", fix: bool = False) -> None:
"""代码检查 (ruff check).
Parameters
----------
target : str
目标路径 (默认: .)
fix : bool
自动修复问题
"""
lint_with_ruff(Path(target), fix=fix)
def format_with_ruff(target: Path, fix: bool = True) -> None:
"""使用 ruff 格式化代码.
@@ -110,10 +102,12 @@ def add_docstring(file_path: Path, docstring: str) -> bool:
content = file_path.read_text(encoding="utf-8")
tree = ast.parse(content)
# 检查是否已有 docstring
first_node = tree.body[0] if tree.body else None
if first_node and isinstance(first_node, ast.Expr) and isinstance(first_node.value, ast.Constant):
return False
# 添加 docstring
lines = content.splitlines()
doc_lines = docstring.splitlines()
doc_lines.append("")
@@ -144,6 +138,7 @@ def generate_module_docstring(file_path: Path) -> str:
stem = file_path.stem
parent = file_path.parent.name
# 关键词匹配
keywords = {
"cli": f"Command-line interface for {parent}",
"gui": f"Graphical user interface for {parent}",
@@ -160,14 +155,13 @@ def generate_module_docstring(file_path: Path) -> str:
return f'"""{stem.replace("_", " ").title()} module."""'
@px.tool("autofmt", subcommand="doc", help="自动添加文档字符串")
def auto_add_docstrings(root_dir: Path = Path()) -> int:
def auto_add_docstrings(root_dir: Path) -> int:
"""自动为所有 Python 文件添加 docstring.
Parameters
----------
root_dir : Path
根目录 (默认: 当前目录)
根目录
Returns
-------
@@ -176,6 +170,7 @@ def auto_add_docstrings(root_dir: Path = Path()) -> int:
"""
count = 0
for py_file in root_dir.rglob("*.py"):
# 跳过忽略的文件
if any(pattern in str(py_file) for pattern in IGNORE_PATTERNS):
continue
@@ -187,20 +182,20 @@ def auto_add_docstrings(root_dir: Path = Path()) -> int:
return count
@px.tool("autofmt", subcommand="sync", help="同步 pyproject 配置")
def sync_pyproject_config(root_dir: Path = Path()) -> None:
def sync_pyproject_config(root_dir: Path) -> None:
"""同步 pyproject.toml 配置到子项目.
Parameters
----------
root_dir : Path
根目录 (默认: 当前目录)
根目录
"""
main_toml = root_dir / "pyproject.toml"
if not main_toml.exists():
print(f"主项目配置文件不存在: {main_toml}")
return
# 查找所有子项目的 pyproject.toml
sub_tomls = [p for p in root_dir.rglob("pyproject.toml") if p != main_toml and ".venv" not in str(p)]
if not sub_tomls:
@@ -209,6 +204,7 @@ def sync_pyproject_config(root_dir: Path = Path()) -> None:
print(f"找到 {len(sub_tomls)} 个子项目配置文件")
# 对每个子项目调用 ruff format
for sub_toml in sub_tomls:
subprocess.run(["ruff", "format", str(sub_toml)], check=False)
@@ -223,6 +219,64 @@ def format_all(root_dir: Path) -> None:
root_dir : Path
根目录
"""
# 使用 ruff format
subprocess.run(["ruff", "format", str(root_dir)], check=True)
# 使用 ruff check
subprocess.run(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], check=True)
print(f"格式化完成: {root_dir}")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""自动格式化工具主函数."""
parser = argparse.ArgumentParser(
description="AutoFmt - 自动格式化工具",
usage="autofmt <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# ruff format 命令
format_parser = subparsers.add_parser("fmt", help="使用 ruff 格式化代码")
format_parser.add_argument("--target", type=str, default=".", help="目标路径")
# ruff check 命令
lint_parser = subparsers.add_parser("lint", help="使用 ruff 检查代码")
lint_parser.add_argument("--target", type=str, default=".", help="目标路径")
lint_parser.add_argument("--fix", action="store_true", help="自动修复")
# 自动添加 docstring 命令
doc_parser = subparsers.add_parser("doc", help="自动添加 docstring")
doc_parser.add_argument("--root-dir", type=str, default=".", help="根目录")
# 同步配置命令
sync_parser = subparsers.add_parser("sync", help="同步 pyproject.toml 配置")
sync_parser.add_argument("--root-dir", type=str, default=".", help="根目录")
args = parser.parse_args()
if args.command == "fmt":
graph = px.Graph.from_specs([px.TaskSpec("ruff_format", cmd=["ruff", "format", args.target], verbose=True)])
elif args.command == "lint":
cmd = ["ruff", "check", args.target]
if args.fix:
cmd.extend(["--fix", "--unsafe-fixes"])
graph = px.Graph.from_specs([px.TaskSpec("ruff_check", cmd=cmd, verbose=True)])
elif args.command == "doc":
graph = px.Graph.from_specs([
px.TaskSpec("auto_docstring", fn=auto_add_docstrings, args=(Path(args.root_dir),), verbose=True)
])
elif args.command == "sync":
graph = px.Graph.from_specs([
px.TaskSpec("sync_config", fn=sync_pyproject_config, args=(Path(args.root_dir),), verbose=True)
])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
+263
View File
@@ -0,0 +1,263 @@
"""版本号自动管理工具.
使用 TaskSpec 模式实现, 支持语义化版本管理和多文件格式的版本号更新.
"""
from __future__ import annotations
import argparse
import re
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
BumpVersionType = Literal["patch", "minor", "major"]
# 针对不同文件类型的版本号匹配模式
# pyproject.toml: version = "X.Y.Z" 或 version = 'X.Y.Z'
_PYPROJECT_VERSION_PATTERN = re.compile(
r'(?:^|\n)\s*version\s*=\s*["\']'
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
r'["\']',
re.MULTILINE,
)
# __init__.py: __version__ = "X.Y.Z" 或 __version__ = 'X.Y.Z'
_INIT_VERSION_PATTERN = re.compile(
r'(?:^|\n)\s*__version__\s*=\s*["\']'
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
r'["\']',
re.MULTILINE,
)
def _get_pattern_for_file(file_name: str) -> re.Pattern[str] | None:
"""根据文件类型获取对应的正则表达式.
Parameters
----------
file_name : str
文件名
Returns
-------
re.Pattern[str] | None
对应的正则表达式,如果无法确定则返回 None
"""
if file_name == "pyproject.toml":
return _PYPROJECT_VERSION_PATTERN
if file_name == "__init__.py":
return _INIT_VERSION_PATTERN
return None
def _calculate_new_version(major: int, minor: int, patch: int, part: BumpVersionType) -> str:
"""计算新版本号.
Parameters
----------
major : int
当前主版本号
minor : int
当前次版本号
patch : int
当前补丁版本号
part : BumpVersionType
要更新的部分
Returns
-------
str
新版本号
"""
if part == "major":
return f"{major + 1}.0.0"
if part == "minor":
return f"{major}.{minor + 1}.0"
return f"{major}.{minor}.{patch + 1}"
def _build_replacement_string(original_match: str, new_version: str, file_name: str) -> str:
"""构建替换字符串,保留原始格式.
Parameters
----------
original_match : str
原始匹配的字符串
new_version : str
新版本号
file_name : str
文件名
Returns
-------
str
替换字符串
"""
quote_char = '"' if '"' in original_match else "'"
if file_name == "pyproject.toml":
prefix_match = re.match(r'(\s*version\s*=\s*)["\']', original_match)
prefix = prefix_match.group(1) if prefix_match else "version = "
return f"{prefix}{quote_char}{new_version}{quote_char}"
if file_name == "__init__.py":
prefix_match = re.match(r'(\s*__version__\s*=\s*)["\']', original_match)
prefix = prefix_match.group(1) if prefix_match else "__version__ = "
return f"{prefix}{quote_char}{new_version}{quote_char}"
return new_version
def bump_file_version(file_path: Path, part: BumpVersionType = "patch") -> str | None:
"""更新文件中的版本号.
Parameters
----------
file_path : Path
要更新的文件路径
part : BumpVersionType
版本部分: patch, minor, major
Returns
-------
str | None
更新后的新版本号,如果文件中未找到版本号则返回 None
"""
try:
content = file_path.read_text(encoding="utf-8")
except Exception as e:
print(f"读取文件 {file_path} 时出错: {e}")
raise
# 获取文件对应的正则表达式
pattern = _get_pattern_for_file(file_path.name)
# 对于未知文件类型,尝试两种模式
if pattern:
match = pattern.search(content)
else:
match = _PYPROJECT_VERSION_PATTERN.search(content) or _INIT_VERSION_PATTERN.search(content)
if not match:
print(f"文件 {file_path} 中未找到版本号模式")
return None
# 提取当前版本号
major = int(match.group("major"))
minor = int(match.group("minor"))
patch = int(match.group("patch"))
# 计算新版本号
new_version = _calculate_new_version(major, minor, patch, part)
# 构建替换字符串
original_match = match.group(0)
replacement = _build_replacement_string(original_match, new_version, file_path.name)
# 更新文件内容
content = content.replace(original_match, replacement)
try:
file_path.write_text(content, encoding="utf-8")
except Exception as e:
print(f"更新文件 {file_path} 版本号时出错: {e}")
raise
return new_version
def main() -> None:
"""版本号管理工具主函数."""
parser = argparse.ArgumentParser(description="BumpVersion - 版本号自动管理工具")
parser.add_argument(
"part",
type=str,
nargs="?",
default="patch",
choices=get_args(BumpVersionType),
help=f"版本部分: {get_args(BumpVersionType)}",
)
parser.add_argument(
"--no-tag",
action="store_true",
help="提交后不创建 git tag",
)
args = parser.parse_args()
part = args.part
# 搜索文件,排除常见的虚拟环境和缓存目录
ignore_dirs = {".venv", "venv", ".git", "__pycache__", ".tox", "node_modules", "build", "dist", ".eggs"}
all_files = set()
for pattern in ["__init__.py", "pyproject.toml"]:
for file in Path.cwd().rglob(pattern):
# 检查路径中是否包含需要忽略的目录
if not any(ignore_dir in file.parts for ignore_dir in ignore_dirs):
all_files.add(file)
if not all_files:
print("未找到包含版本号的文件")
return
print(f"找到 {len(all_files)} 个文件需要更新版本号")
for file in sorted(all_files):
print(f" - {file.relative_to(Path.cwd())}")
# 更新所有文件的版本号(使用顺序执行避免竞争条件)
# 使用相对于 cwd 的路径作为任务名,确保唯一性
graph = px.Graph.from_specs([
px.TaskSpec(
f"bump_{file.relative_to(Path.cwd())}".replace("\\", "_").replace("/", "_").replace(".", "_"),
fn=bump_file_version,
args=(file, part),
)
for file in all_files
])
report = px.run(graph, strategy="sequential")
# 收集新版本号(取第一个成功的结果)
new_version = None
for task_name in report:
result = report[task_name]
if result is not None:
new_version = result
break
if not new_version:
print("未能获取新版本号")
return
print(f"版本号已更新为: {new_version}")
# 提交修改并创建标签
tasks = [
px.TaskSpec("git_add", cmd=["git", "add", "."]),
px.TaskSpec(
"git_commit",
cmd=["git", "commit", "-m", f"bump version to {new_version}"],
depends_on=("git_add",),
),
]
if not args.no_tag:
tag_name = f"v{new_version}"
tasks.append(
px.TaskSpec(
"git_tag",
cmd=["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"],
depends_on=("git_commit",),
)
)
graph = px.Graph.from_specs(tasks)
px.run(graph, strategy="sequential")
if not args.no_tag:
print(f"已创建标签: v{new_version}")
+26
View File
@@ -0,0 +1,26 @@
from __future__ import annotations
from typing import Literal
import pyflowx as px
DockerMirrorType = Literal["tencent"]
DOCKER_MIRROR_URLS: dict[DockerMirrorType, str] = {"tencent": "ccr.ccs.tencentyun.com"}
def main():
# parser = argparse.ArgumentParser(description="Docker 命令行工具")
# parser.add_argument("--username", nargs="?", default="", type=str, help="Docker 用户名")
# args = parser.parse_args()
tasks: list[px.TaskSpec] = [
px.cmd(["docker", "login", "--username", "xxx", DOCKER_MIRROR_URLS["tencent"]], name="docker_login_tencent"),
]
alias: dict[str, str | list[str | px.TaskSpec] | px.TaskSpec | px.Graph] = {
"login": "docker_login_tencent",
}
runner = px.CliRunner(strategy="sequential", tasks=tasks, aliases=alias)
runner.run_cli()
+357
View File
@@ -0,0 +1,357 @@
from __future__ import annotations
import argparse
import getpass
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
from pyflowx.conditions import BuiltinConditions
from pyflowx.tasks.system import setenv_group, write_file
# ============================================================================
# Mirror 配置
# ============================================================================
DOWNLOAD_MIRROR_SCRIPT: str = "curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh"
INSTALL_MIRROR_SCRIPT: str = "sudo bash /tmp/linuxmirrors.sh"
# ============================================================================
# Python 配置
# ============================================================================
PyMirrorType = Literal["tsinghua", "aliyun", "huaweicloud", "ustc", "zju"]
PIP_INDEX_URLS: dict[PyMirrorType, str] = {
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
"huaweicloud": "https://mirrors.huaweicloud.com/repository/pypi/simple/",
"ustc": "https://pypi.mirrors.ustc.edu.cn/simple/",
"zju": "https://mirrors.zju.edu.cn/pypi/simple/",
}
PIP_TRUSTED_HOSTS: dict[PyMirrorType, str] = {
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
"aliyun": "mirrors.aliyun.com",
"huaweicloud": "mirrors.huaweicloud.com",
"ustc": "pypi.mirrors.ustc.edu.cn",
"zju": "mirrors.zju.edu.cn",
}
PIP_CONFIG_PATH = Path.home() / ".pip" / "pip.conf" if BuiltinConditions.IS_LINUX() else Path.home() / "pip" / "pip.ini"
UV_INDEX_URLS = PIP_INDEX_URLS
UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
# ============================================================================
# Conda 配置
# ============================================================================
CondaMirrorType = Literal["tsinghua", "ustc", "bsfu", "aliyun"]
CONDA_MIRROR_URLS: dict[CondaMirrorType, list[str]] = {
"tsinghua": [
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
],
"ustc": [
"https://mirrors.ustc.edu.cn/anaconda/pkgs/main/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/free/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/r/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/pytorch/",
],
"bsfu": [
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/main/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/free/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/r/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/pytorch/",
],
"aliyun": [
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
"https://mirrors.aliyun.com/anaconda/pkgs/r/",
"https://mirrors.aliyun.com/anaconda/pkgs/msys2/",
"https://mirrors.aliyun.com/anaconda/pkgs/pro/",
"https://mirrors.aliyun.com/anaconda/pkgs/dev/",
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
"https://mirrors.aliyun.com/anaconda/cloud/bioconda/",
"https://mirrors.aliyun.com/anaconda/cloud/menpo/",
"https://mirrors.aliyun.com/anaconda/cloud/pytorch/",
],
}
CONDA_CONFIG_PATH = Path.home() / ".condarc"
# ============================================================================
# Qt 配置
# ============================================================================
QT_LIBS: list[str] = [
"build-essential",
"libgl1",
"libegl1",
"libglib2.0-0",
"libfontconfig1",
"libfreetype6",
"libxkbcommon0",
"libdbus-1-3",
"libxcb-xinerama0",
"libxcb-icccm4",
"libxcb-image0",
"libxcb-keysyms1",
"libxcb-randr0",
"libxcb-render-util0",
"libxcb-shape0",
"libxcb-xfixes0",
"libxcb-cursor0",
]
CHINESE_FONTS: list[str] = [
"fonts-noto-cjk",
"fonts-wqy-microhei",
"fonts-wqy-zenhei",
"fonts-noto-color-emoji",
]
# ============================================================================
# Rust 配置
# ============================================================================
RustMirrorType = Literal["tsinghua", "ustc", "aliyun"]
RustVersionType = Literal["stable", "nightly", "beta"]
DEFAULT_RUST_VERSION: RustVersionType = "stable"
DEFAULT_MIRROR: RustMirrorType = "tsinghua"
RUSTUP_MIRRORS: dict[RustMirrorType, dict[str, str]] = {
"tsinghua": {
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
},
"aliyun": {
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
},
"ustc": {
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
},
}
RUSTUP_DOWNLOAD_URL_LINUX = "https://mirrors.aliyun.com/repo/rust/rustup-init.sh"
RUSTUP_DOWNLOAD_URL_WINDOWS = "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"
RUST_CONFIG_PATH = Path.home() / ".cargo" / "config.toml"
RUST_SCCACHE_DIR: Path = Path.home() / ".cargo" / "sccache"
RUST_SCCACHE_CACHE_SIZE: str = "20G"
def main() -> None:
"""主函数."""
parser = argparse.ArgumentParser(description="环境开发工具")
parser.add_argument(
"--python-mirror",
nargs="?",
type=str,
default="tsinghua",
choices=get_args(PyMirrorType),
help="Python 镜像源",
)
parser.add_argument(
"--conda-mirror",
nargs="?",
type=str,
default="tsinghua",
choices=get_args(CondaMirrorType),
help="Conda 镜镜像源",
)
parser.add_argument(
"--rust-mirror",
nargs="?",
type=str,
default=DEFAULT_MIRROR,
choices=get_args(RustMirrorType),
help="Rust 镜像源",
)
parser.add_argument(
"--rust-version",
nargs="?",
type=str,
default=DEFAULT_RUST_VERSION,
choices=get_args(RustVersionType),
help=f"Rust 版本, 推荐: {get_args(RustVersionType)}",
)
args = parser.parse_args()
python_mirror = args.python_mirror
conda_mirror_urls = CONDA_MIRROR_URLS[args.conda_mirror]
rust_mirror = args.rust_mirror
rust_version = args.rust_version
# 确保配置文件目录存在
PIP_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
CONDA_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
RUST_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
RUST_SCCACHE_DIR.mkdir(parents=True, exist_ok=True)
# 使用 conditions 自动控制任务执行
graph = px.Graph.from_specs([
# 系统镜像配置(仅 Linux 且未配置国内镜像)
px.TaskSpec(
"download_mirror",
cmd=DOWNLOAD_MIRROR_SCRIPT,
conditions=(
BuiltinConditions.IS_LINUX(),
BuiltinConditions.NOT(
BuiltinConditions.OR(
*[
BuiltinConditions.FILE_CONTENT_EXISTS(f, m)
for f in [
"/etc/apt/sources.list",
"/etc/apt/sources.list.d/ubuntu.sources",
]
for m in get_args(PyMirrorType)
],
)
),
),
verbose=True,
),
px.TaskSpec(
"install_mirror",
cmd=INSTALL_MIRROR_SCRIPT,
depends_on=("download_mirror",),
verbose=True,
),
# 安装 Qt 依赖(仅 Linux
px.TaskSpec(
"install_qt_libs",
cmd=["sudo", "apt", "install", "-y", *QT_LIBS],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_mirror",),
allow_upstream_skip=True,
verbose=True,
),
# 安装中文字体(仅 Linux
px.TaskSpec(
"install_fonts",
cmd=["sudo", "apt", "install", "-y", *CHINESE_FONTS],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_mirror",),
allow_upstream_skip=True,
verbose=True,
),
# 安装 Docker
px.TaskSpec(
"install_docker",
cmd=["sudo", "apt", "install", "-y", "docker-compose-v2"],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_mirror",),
allow_upstream_skip=True,
verbose=True,
),
px.TaskSpec(
"add_docker_group",
cmd=["sudo", "usermod", "-aG", "docker", getpass.getuser()],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_docker",),
allow_upstream_skip=True,
verbose=True,
),
px.TaskSpec(
"refresh_docker_group",
cmd=["newgrp", "docker"],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("add_docker_group",),
allow_upstream_skip=True,
verbose=True,
),
# 设置 Python 环境变量
*setenv_group({
"PIP_INDEX_URL": PIP_INDEX_URLS[python_mirror],
"PIP_TRUSTED_HOSTS": PIP_TRUSTED_HOSTS[python_mirror],
"UV_INDEX_URL": UV_INDEX_URLS[python_mirror],
"UV_PYTHON_INSTALL_MIRROR": UV_PYTHON_INSTALL_MIRROR,
"UV_HTTP_TIMEOUT": "600",
"UV_LINK_MODE": "copy",
}),
# 写入 Python 配置(仅当未配置)
write_file(
str(PIP_CONFIG_PATH),
f"[global]\nindex-url = {PIP_INDEX_URLS[python_mirror]}\ntrusted-host = {PIP_TRUSTED_HOSTS[python_mirror]}",
),
# 写入 Conda 配置(仅当未配置)
write_file(
str(CONDA_CONFIG_PATH),
"show_channel_urls: true\nchannels:\n - " + "\n - ".join(conda_mirror_urls) + "\n - defaults",
),
# 设置 Rust 镜像源
*setenv_group({
"RUSTUP_DIST_SERVER": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_DIST_SERVER"],
"RUSTUP_UPDATE_ROOT": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_UPDATE_ROOT"],
"RUST_SCCACHE_DIR": str(RUST_SCCACHE_DIR),
"RUST_SCCACHE_CACHE_SIZE": RUST_SCCACHE_CACHE_SIZE,
}),
# 写入 Rust 配置(仅当未配置)
write_file(
str(RUST_CONFIG_PATH),
f"""
[source.crates-io]
replace-with = '{rust_mirror}'
[source.{rust_mirror}]
registry = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
[registries.{rust_mirror}]
index = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
""",
),
# 下载 Rustup 安装脚本
px.TaskSpec(
"download_rustup",
cmd=["curl", "-fsSL", RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
conditions=(BuiltinConditions.IS_LINUX(), BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup"))),
verbose=True,
),
px.TaskSpec(
"download_rustup_win",
cmd=[
"powershell",
"-Command",
"Invoke-WebRequest",
"-Uri",
RUSTUP_DOWNLOAD_URL_WINDOWS,
"-OutFile",
"rustup-init.exe",
],
conditions=(
BuiltinConditions.IS_WINDOWS(),
BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup")),
),
verbose=True,
),
# 安装 Rust 工具链
px.TaskSpec(
"install_rust",
cmd=["rustup", "toolchain", "install", rust_version],
conditions=(BuiltinConditions.HAS_INSTALLED("rustup"),),
depends_on=("setenv_rustup_dist_server",),
allow_upstream_skip=True,
verbose=True,
),
])
px.run(graph, strategy="thread", verbose=True)
@@ -154,7 +154,7 @@ class EmailDatabase:
cursor.execute(query, (f"%{keyword}%", f"%{keyword}%", f"%{keyword}%", f"%{keyword}%", limit, offset))
columns = [description[0] for description in cursor.description]
return [dict(zip(columns, row, strict=False)) for row in cursor.fetchall()]
return [dict(zip(columns, row)) for row in cursor.fetchall()]
def get_grouped_emails(self) -> dict[str, list[dict[str, Any]]]:
"""获取按主题分组的邮件."""
@@ -165,7 +165,7 @@ class EmailDatabase:
cursor.execute(f"SELECT * FROM {TABLE_NAME} ORDER BY subject, date_parsed DESC")
columns = [description[0] for description in cursor.description]
emails = [dict(zip(columns, row, strict=False)) for row in cursor.fetchall()]
emails = [dict(zip(columns, row)) for row in cursor.fetchall()]
# 按主题分组
grouped: dict[str, list[dict[str, Any]]] = {}
@@ -567,15 +567,13 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
emails = self.db.search_emails(keyword, field, limit, offset)
total_count = self.db.get_email_count()
self._send_json_response(
{
"emails": emails,
"count": len(emails),
"total": total_count,
"limit": limit,
"offset": offset,
}
)
self._send_json_response({
"emails": emails,
"count": len(emails),
"total": total_count,
"limit": limit,
"offset": offset,
})
def _api_get_email(self, query_params: dict[str, list[str]]) -> None:
"""API: 获取单个邮件详情."""
@@ -602,7 +600,7 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
self._send_json_response({"error": "邮件不存在"}, 404)
return
email_data = dict(zip(columns, row, strict=False))
email_data = dict(zip(columns, row))
self._send_json_response({"email": email_data})
def _api_get_grouped_emails(self) -> None:
@@ -1,26 +1,18 @@
"""filedate - 文件日期处理工具.
"""文件日期处理工具.
提供添加日期前缀/清除日期前缀 子命令.
自动检测文件名的日期前缀,
并根据文件的实际创建或修改时间重命名文件.
"""
from __future__ import annotations
import argparse
import re
import time
from pathlib import Path
import pyflowx as px
__all__ = [
"DATE_PATTERN",
"SEP",
"add_date_prefix",
"get_file_timestamp",
"process_file_date",
"process_files_date",
"remove_date_prefix",
]
# ============================================================================
# 配置
# ============================================================================
@@ -30,7 +22,7 @@ SEP = "_"
# ============================================================================
# 公共函数
# 辅助函数
# ============================================================================
@@ -77,34 +69,11 @@ def process_file_date(filepath: Path, clear: bool = False) -> None:
if clear:
remove_date_prefix(filepath)
else:
# 先移除旧日期前缀,再添加新日期前缀
new_path = remove_date_prefix(filepath)
add_date_prefix(new_path)
@px.tool("filedate", subcommand="add", help="添加日期前缀")
def process_files_date_add(files: list[Path]) -> None:
"""添加日期前缀.
Parameters
----------
files : list[Path]
文件路径列表
"""
process_files_date(files, clear=False)
@px.tool("filedate", subcommand="clear", help="清除日期前缀")
def process_files_date_clear(files: list[Path]) -> None:
"""清除日期前缀.
Parameters
----------
files : list[Path]
文件路径列表
"""
process_files_date(files, clear=True)
def process_files_date(targets: list[Path], clear: bool = False) -> None:
"""批量处理文件日期前缀.
@@ -118,3 +87,51 @@ def process_files_date(targets: list[Path], clear: bool = False) -> None:
for target in targets:
if target.exists() and not target.name.startswith("."):
process_file_date(target, clear)
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""文件日期处理工具主函数."""
parser = argparse.ArgumentParser(
description="FileDate - 文件日期处理工具",
usage="filedate <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 添加日期前缀命令
add_parser = subparsers.add_parser("add", help="添加日期前缀")
add_parser.add_argument("files", nargs="+", help="文件路径")
# 清除日期前缀命令
clear_parser = subparsers.add_parser("clear", help="清除日期前缀")
clear_parser.add_argument("files", nargs="+", help="文件路径")
args = parser.parse_args()
if args.command == "add":
graph = px.Graph.from_specs([
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": False},
)
])
elif args.command == "clear":
graph = px.Graph.from_specs([
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": True},
)
])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
@@ -1,22 +1,16 @@
"""filelevel - 文件等级重命名工具.
"""文件等级重命名工具.
提供设置文件等级 子命令.
根据文件等级配置自动重命名文件,
支持多种等级标识和括号格式.
"""
from __future__ import annotations
import argparse
from pathlib import Path
import pyflowx as px
__all__ = [
"BRACKETS",
"LEVELS",
"process_file_level",
"process_files_level",
"remove_marks",
]
# ============================================================================
# 配置
# ============================================================================
@@ -33,7 +27,7 @@ BRACKETS: tuple[str, str] = (" ([_(【-", " )]_)】")
# ============================================================================
# 公共函数
# 辅助函数
# ============================================================================
@@ -75,34 +69,72 @@ def process_file_level(filepath: Path, level: int = 0) -> None:
filestem = filepath.stem
original_stem = filestem
# 移除所有等级标记
for level_names in LEVELS.values():
if level_names:
filestem = remove_marks(filestem, level_names.split(","))
# 移除数字标记
for digit in map(str, range(1, 10)):
filestem = remove_marks(filestem, [digit])
# 添加等级标记
if level > 0:
levelstr = LEVELS.get(str(level), "").split(",")[0]
if levelstr:
filestem = f"{filestem}({levelstr})"
# 重命名文件
if filestem != original_stem:
new_path = filepath.with_name(filestem + filepath.suffix)
filepath.rename(new_path)
print(f"重命名: {filepath} -> {new_path}")
@px.tool("filelevel", subcommand="set", help="设置文件等级")
def process_files_level(files: list[Path], level: int = 0) -> None:
def process_files_level(targets: list[Path], level: int = 0) -> None:
"""批量处理文件等级标记.
Parameters
----------
files : list[Path]
targets : list[Path]
文件路径列表
level : int
文件等级 (0-4)
"""
for target in files:
for target in targets:
process_file_level(target, level)
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""文件等级重命名工具主函数."""
parser = argparse.ArgumentParser(
description="FileLevel - 文件等级重命名工具",
usage="filelevel <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 设置等级命令
level_parser = subparsers.add_parser("set", help="设置文件等级")
level_parser.add_argument("files", nargs="+", help="文件路径")
level_parser.add_argument("--level", type=int, choices=[0, 1, 2, 3, 4], required=True, help="文件等级 (0-4)")
args = parser.parse_args()
if args.command == "set":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"process_files_level", fn=process_files_level, args=([Path(f) for f in args.files], args.level)
)
]
)
else:
parser.print_help()
return
px.run(graph, strategy="thread")
@@ -1,6 +1,7 @@
"""folderback - 文件夹备份工具.
"""文件夹备份工具.
备份当前文件夹到指定目录, 自动清理旧备份.
备份文件和文件夹为 zip 文件,
自动删除超过最大数量的旧备份文件.
"""
from __future__ import annotations
@@ -11,15 +12,8 @@ from pathlib import Path
import pyflowx as px
__all__ = [
"backup_folder",
"remove_dump",
"zip_target",
]
# ============================================================================
# 公共函数
# 辅助函数
# ============================================================================
@@ -46,18 +40,17 @@ def zip_target(src: Path, dst: Path, max_zip: int) -> None:
print(f"备份完成: {target_path}")
@px.tool("folderback", help="备份文件夹")
def backup_folder(src: str = ".", dst: str = "./backup", max_zip: int = 5) -> None:
def backup_folder(src: str, dst: str, max_zip: int = 5) -> None:
"""备份文件夹.
Parameters
----------
src : str
源文件夹路径 (默认: 当前目录)
源文件夹路径
dst : str
目标文件夹路径 (默认: ./backup)
目标文件夹路径
max_zip : int
最大备份数量 (默认: 5)
最大备份数量
"""
src_path = Path(src)
dst_path = Path(dst)
@@ -71,3 +64,22 @@ def backup_folder(src: str = ".", dst: str = "./backup", max_zip: int = 5) -> No
print(f"创建目标文件夹: {dst_path}")
zip_target(src_path, dst_path, max_zip)
@px.task
def folderback_default() -> None:
"""备份当前目录到 ./backup."""
backup_folder(".", "./backup", 5)
def main() -> None:
"""文件夹备份工具主函数."""
runner = px.CliRunner(
strategy="thread",
description="FolderBack - 文件夹备份工具",
aliases={
# 备份当前目录到 ./backup
"b": folderback_default,
},
)
runner.run_cli()
@@ -1,6 +1,7 @@
"""folderzip - 文件夹压缩工具.
"""文件夹压缩工具.
压缩当前目录下的所有文件夹为 zip 文件.
压缩目录下的所有文件/文件夹为 zip 文件,
默认压缩当前目录下的所有子文件夹.
"""
from __future__ import annotations
@@ -10,25 +11,18 @@ from pathlib import Path
import pyflowx as px
__all__ = [
"IGNORE_DIRS",
"IGNORE_EXT",
"IGNORE_FILES",
"archive_folder",
"zip_folders",
]
# ============================================================================
# 配置
# ============================================================================
IGNORE_DIRS: list[str] = [".git", ".idea", ".vscode", "__pycache__"]
IGNORE_FILES: list[str] = [".gitignore"]
IGNORE: list[str] = [*IGNORE_DIRS, *IGNORE_FILES]
IGNORE_EXT: list[str] = [".zip", ".rar", ".7z", ".tar", ".gz"]
# ============================================================================
# 公共函数
# 辅助函数
# ============================================================================
@@ -42,14 +36,13 @@ def archive_folder(folder: Path) -> None:
print(f"压缩完成: {folder.name}.zip")
@px.tool("folderzip", help="压缩文件夹")
def zip_folders(cwd: str = ".") -> None:
"""压缩目录下的所有文件夹.
Parameters
----------
cwd : str
工作目录 (默认: 当前目录)
工作目录
"""
cwd_path = Path(cwd)
if not cwd_path.exists():
@@ -62,3 +55,22 @@ def zip_folders(cwd: str = ".") -> None:
for dir_path in dirs:
archive_folder(dir_path)
@px.task
def folderzip_default() -> None:
"""压缩当前目录下的所有文件夹."""
zip_folders(".")
def main() -> None:
"""文件夹压缩工具主函数."""
runner = px.CliRunner(
strategy="thread",
description="FolderZip - 文件夹压缩工具",
aliases={
# 压缩当前目录下的所有文件夹
"z": folderzip_default,
},
)
runner.run_cli()
+107
View File
@@ -0,0 +1,107 @@
"""Git 工具模块.
提供 Git 仓库管理的常用操作封装,
支持初始化、提交、清理、推送等功能.
"""
from __future__ import annotations
from pathlib import Path
import pyflowx as px
EXCLUDE_DIRS = [
# 编辑器相关目录
".vscode",
".idea",
".editorconfig",
".trae",
".qoder",
# 项目相关目录
".venv",
".git",
".tox",
".pytest_cache",
"node_modules",
".ruff_cache",
]
EXCLUDE_CMDS = [arg for d in EXCLUDE_DIRS for arg in ["-e", d]]
def init_sub_dirs() -> None:
"""初始化子目录的Git仓库."""
sub_dirs = [subdir for subdir in Path.cwd().iterdir() if subdir.is_dir()]
for subdir in sub_dirs:
px.run(
px.Graph.from_specs([
px.TaskSpec(
"init",
cmd=["git", "init"],
conditions=(lambda _: not_has_git_repo(),),
cwd=subdir,
),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",)),
px.TaskSpec("commit", cmd=["git", "commit", "-m", "init commit"], depends_on=("add",)),
]),
)
@px.task(name="isub")
def isub() -> None:
"""初始化子目录的Git仓库."""
init_sub_dirs()
push: px.TaskSpec = px.TaskSpec("push", cmd=["git", "push"])
pull: px.TaskSpec = px.TaskSpec("pull", cmd=["git", "pull"])
kill_tgit: px.TaskSpec = px.TaskSpec("task_kill", cmd=["taskkill", "/f", "/t", "/im", "tgitcache.exe"])
def not_has_git_repo() -> bool:
"""检查当前目录没有Git仓库."""
return not Path.cwd().exists() or not (Path.cwd() / ".git").is_dir()
def has_files() -> bool:
"""检查当前目录是否有文件."""
return bool(list(Path.cwd().glob("*")))
def main() -> None:
"""Git工具主函数."""
runner = px.CliRunner(
strategy="thread",
description="Gittool - Git 执行工具.",
aliases={
# 添加并提交
"a": px.Graph.from_specs([
px.TaskSpec("add", cmd=["git", "add", "."], conditions=(lambda _: has_files(),)),
px.TaskSpec("commit", cmd=["git", "commit", "-m", "chore: update"], depends_on=("add",)),
]),
# 清理(chain: clean → status
"c": px.Graph().chain(
px.TaskSpec("clean", cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS]),
px.TaskSpec("status", cmd=["git", "status", "--porcelain"]),
),
# 初始化、添加并提交
"i": px.Graph.from_specs([
px.TaskSpec("init", cmd=["git", "init"], conditions=(lambda _: not_has_git_repo(),)),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",), conditions=(lambda _: has_files(),)),
px.TaskSpec(
"commit",
cmd=["git", "commit", "-m", "init commit"],
depends_on=("add",),
conditions=(lambda _: has_files(),),
),
]),
# 初始化子目录
"isub": isub,
# 推送
"p": push,
# 拉取
"pl": pull,
# 重启TGit缓存
"r": kill_tgit,
},
)
runner.run_cli()
-15
View File
@@ -1,15 +0,0 @@
"""legacy 子包 — 未迁移到 @px.tool 的独立工具.
每个子模块有自己的 ``main()`` 函数, 由 ``pf`` 入口通过
``_LEGACY_TOOLS`` 路由调用. 这些工具因逻辑复杂 (web 应用、runpy 注入等)
暂未用 ``@px.tool`` 装饰器重写.
子模块
------
- :mod:`emlmanager` —— EML 邮件管理 Web 应用
- :mod:`profiler` —— pxp 性能分析器 (monkey-patch + runpy)
"""
from __future__ import annotations
__all__: list[str] = []
View File
+41
View File
@@ -0,0 +1,41 @@
"""Download from ModelScopeHub."""
import argparse
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
DownloadType = Literal["model", "dataset", "space"]
def main():
parser = argparse.ArgumentParser(description="Download a model from ModelScopeHub.")
parser.add_argument("name", help="Target name.")
parser.add_argument("--type", "-t", nargs="?", default="model", choices=get_args(DownloadType), help="Target type.")
parser.add_argument("--dir", default=None, help="Download directory.")
args = parser.parse_args()
if not args.name:
parser.error("name is required")
download_dir: Path = Path(args.dir) if args.dir else Path.home() / ".models" / args.name.split("/")[-1]
download_dir.mkdir(parents=True, exist_ok=True)
graph = px.Graph.from_specs([
px.TaskSpec(
name="download",
cmd=[
"uvx",
"modelscope",
"download",
f"--{args.type}",
args.name,
"--local_dir",
str(download_dir),
],
verbose=True,
),
])
px.run(graph, strategy="thread", verbose=True)
+63
View File
@@ -0,0 +1,63 @@
"""使用 SGLang 运行本地模型."""
import argparse
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import BuiltinConditions, Constants
def main():
parser = argparse.ArgumentParser(description="启动 SGLang 服务")
parser.add_argument("--model", default="~/.models/Qwen2.5-Coder-32B-Instruct-AWQ", help="模型路径")
parser.add_argument("--port", type=int, default=8000, help="服务端口")
parser.add_argument("--ctx-len", type=int, default=28672, help="最大上下文长度")
parser.add_argument("--mem", type=float, default=0.75, help="显存占比 (0-1)")
parser.add_argument("--host", default="0.0.0.0", help="主机地址")
parser.add_argument("--log-level", default="info", help="日志级别")
args = parser.parse_args()
if not args.model:
parser.error("model is required")
model_dir = Path(args.model).expanduser()
if not model_dir.exists():
parser.error(f"Model directory {model_dir} does not exist.")
graph = px.Graph.from_specs([
px.TaskSpec(
name="download",
cmd=[
"uv",
"install",
"sglang[all]",
],
conditions=(BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("sglang")),),
verbose=True,
),
px.TaskSpec(
name="run",
cmd=[
"python" if Constants.IS_WINDOWS else "python3",
"-m",
"sglang.launch_server",
"--model-path",
str(model_dir),
"--host",
str(args.host),
"--port",
"8000",
"--mem-fraction-static",
str(args.mem),
"--context-length",
"32768",
"--tool-call-parser",
"qwen",
"--log-level",
str(args.log_level),
],
verbose=True,
),
])
px.run(graph, strategy="sequential", verbose=True)
+174
View File
@@ -0,0 +1,174 @@
"""LS-DYNA 计算工具.
用于管理 LS-DYNA 仿真计算任务,
支持启动、监控和管理计算进程.
"""
from __future__ import annotations
import argparse
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
# ============================================================================
# 配置
# ============================================================================
LS_DYNA_COMMANDS: dict[str, list[str]] = {
"windows": ["ls-dyna_mpp", "i=input.k", "ncpu=4"],
"linux": ["ls-dyna_mpp", "i=input.k", "ncpu=8"],
"macos": ["ls-dyna_mpp", "i=input.k", "ncpu=4"],
}
DEFAULT_INPUT_FILE: str = "input.k"
DEFAULT_NCPU: int = 4
# ============================================================================
# 辅助函数
# ============================================================================
def get_ls_dyna_command(input_file: str, ncpu: int) -> list[str]:
"""获取 LS-DYNA 命令.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
Returns
-------
list[str]
LS-DYNA 命令列表
"""
if Constants.IS_WINDOWS or Constants.IS_MACOS:
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
else:
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
def run_ls_dyna(input_file: str, ncpu: int = DEFAULT_NCPU) -> None:
"""运行 LS-DYNA 计算.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
"""
input_path = Path(input_file)
if not input_path.exists():
print(f"输入文件不存在: {input_path}")
return
cmd = get_ls_dyna_command(input_file, ncpu)
try:
subprocess.run(cmd, check=True)
print(f"LS-DYNA 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA 计算失败: {e}")
def run_ls_dyna_mpi(input_file: str, ncpu: int = DEFAULT_NCPU) -> None:
"""运行 LS-DYNA MPI 计算.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
"""
input_path = Path(input_file)
if not input_path.exists():
print(f"输入文件不存在: {input_path}")
return
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
try:
subprocess.run(cmd, check=True)
print(f"LS-DYNA MPI 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 mpirun 或 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA MPI 计算失败: {e}")
def check_ls_dyna_status() -> None:
"""检查 LS-DYNA 进程状态."""
try:
if Constants.IS_WINDOWS:
result = subprocess.run(
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
capture_output=True,
text=True,
check=True,
)
print(result.stdout)
else:
result = subprocess.run(
["pgrep", "-f", "ls-dyna"],
capture_output=True,
text=True,
check=False,
)
if result.stdout.strip():
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
else:
print("没有运行中的 LS-DYNA 进程")
except subprocess.CalledProcessError as e:
print(f"检查进程状态失败: {e}")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""LS-DYNA 计算工具主函数."""
parser = argparse.ArgumentParser(
description="LSCalc - LS-DYNA 计算工具",
usage="lscalc <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 运行计算命令
run_parser = subparsers.add_parser("run", help="运行 LS-DYNA 计算")
run_parser.add_argument("input_file", help="输入文件路径")
run_parser.add_argument("--ncpu", type=int, default=DEFAULT_NCPU, help="CPU 核心数")
# 运行 MPI 计算命令
mpi_parser = subparsers.add_parser("mpi", help="运行 LS-DYNA MPI 计算")
mpi_parser.add_argument("input_file", help="输入文件路径")
mpi_parser.add_argument("--ncpu", type=int, default=DEFAULT_NCPU, help="CPU 核心数")
# 检查进程状态命令
subparsers.add_parser("status", help="检查 LS-DYNA 进程状态")
args = parser.parse_args()
if args.command == "run":
graph = px.Graph.from_specs(
[px.TaskSpec("run_ls_dyna", fn=run_ls_dyna, args=(args.input_file,), kwargs={"ncpu": args.ncpu})]
)
elif args.command == "mpi":
graph = px.Graph.from_specs(
[px.TaskSpec("run_ls_dyna_mpi", fn=run_ls_dyna_mpi, args=(args.input_file,), kwargs={"ncpu": args.ncpu})]
)
elif args.command == "status":
graph = px.Graph.from_specs([px.TaskSpec("check_ls_dyna_status", fn=check_ls_dyna_status)])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
+349
View File
@@ -0,0 +1,349 @@
"""Python 打包工具模块.
提供 Python 项目打包的常用功能封装,
支持源码打包、依赖打包、嵌入式 Python 安装等功能.
"""
from __future__ import annotations
import argparse
import shutil
import subprocess
import zipfile
from pathlib import Path
import pyflowx as px
# ============================================================================
# 配置
# ============================================================================
DEFAULT_BUILD_DIR = ".pypack"
DEFAULT_DIST_DIR = "dist"
DEFAULT_LIB_DIR = "libs"
DEFAULT_CACHE_DIR = ".cache/pypack"
IGNORE_PATTERNS = [
"__pycache__",
"*.pyc",
"*.pyo",
".git",
".venv",
".idea",
".vscode",
"*.egg-info",
"dist",
"build",
".pytest_cache",
".tox",
".mypy_cache",
]
# ============================================================================
# 辅助函数
# ============================================================================
def pack_source(project_dir: Path, output_dir: Path) -> None:
"""打包项目源码.
Parameters
----------
project_dir : Path
项目目录
output_dir : Path
输出目录
"""
output_dir.mkdir(parents=True, exist_ok=True)
# 检测项目名称
pyproject_file = project_dir / "pyproject.toml"
project_name = project_dir.name
if pyproject_file.exists():
try:
import tomllib
content = pyproject_file.read_text(encoding="utf-8")
data = tomllib.loads(content)
project_name = data.get("project", {}).get("name", project_name)
except ImportError:
pass
# 打包源码
source_dir = output_dir / "src" / project_name
source_dir.mkdir(parents=True, exist_ok=True)
# 复制文件
src_subdir = project_dir / "src"
if src_subdir.exists():
shutil.copytree(
src_subdir,
source_dir / "src",
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
dirs_exist_ok=True,
)
else:
for item in project_dir.iterdir():
if item.name in IGNORE_PATTERNS or item.name.startswith("."):
continue
dst_item = source_dir / item.name
if item.is_dir():
shutil.copytree(
item,
dst_item,
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
dirs_exist_ok=True,
)
else:
shutil.copy2(item, dst_item)
print(f"源码打包完成: {source_dir}")
def pack_dependencies(lib_dir: Path, dependencies: list[str]) -> None:
"""打包项目依赖.
Parameters
----------
lib_dir : Path
依赖库目录
dependencies : list[str]
依赖列表
"""
lib_dir.mkdir(parents=True, exist_ok=True)
if not dependencies:
print("没有依赖需要打包")
return
# 使用 pip 安装依赖到目标目录
cmd = [
"pip",
"install",
"--target",
str(lib_dir),
"--no-compile",
"--no-warn-script-location",
]
cmd.extend(dependencies)
subprocess.run(cmd, check=True)
print(f"依赖打包完成: {lib_dir}")
def pack_wheel(project_dir: Path, output_dir: Path) -> None:
"""打包项目为 wheel 文件.
Parameters
----------
project_dir : Path
项目目录
output_dir : Path
输出目录
"""
output_dir.mkdir(parents=True, exist_ok=True)
# 使用 pip wheel 打包
cmd = [
"pip",
"wheel",
"--no-deps",
"--wheel-dir",
str(output_dir),
str(project_dir),
]
subprocess.run(cmd, check=True)
print(f"Wheel 打包完成: {output_dir}")
def install_embed_python(version: str, output_dir: Path) -> None:
"""安装嵌入式 Python.
Parameters
----------
version : str
Python 版本 (如: 3.10, 3.11)
output_dir : Path
输出目录
"""
import platform
output_dir.mkdir(parents=True, exist_ok=True)
# 构建下载 URL
arch = platform.machine().lower()
if arch in ["x86_64", "amd64"]:
arch = "amd64"
elif arch in ["arm64", "aarch64"]:
arch = "arm64"
# 解析完整版本号
version_map = {
"3.8": "3.8.10",
"3.9": "3.9.13",
"3.10": "3.10.11",
"3.11": "3.11.9",
"3.12": "3.12.4",
}
full_version = version_map.get(version, f"{version}.0")
# Windows 嵌入式 Python 下载 URL
url = f"https://www.python.org/ftp/python/{full_version}/python-{full_version}-embed-{arch}.zip"
# 下载并解压
cache_file = Path(DEFAULT_CACHE_DIR) / f"python-{full_version}-embed-{arch}.zip"
cache_file.parent.mkdir(parents=True, exist_ok=True)
if not cache_file.exists():
print(f"正在下载嵌入式 Python {full_version}...")
import urllib.request
urllib.request.urlretrieve(url, cache_file)
print(f"下载完成: {cache_file}")
# 解压
with zipfile.ZipFile(cache_file, "r") as zf:
zf.extractall(output_dir)
print(f"嵌入式 Python 安装完成: {output_dir}")
def create_zip_package(source_dir: Path, output_file: Path) -> None:
"""创建 ZIP 打包文件.
Parameters
----------
source_dir : Path
源目录
output_file : Path
输出文件
"""
output_file.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(output_file, "w", zipfile.ZIP_DEFLATED) as zf:
for file in source_dir.rglob("*"):
if file.is_file():
arcname = file.relative_to(source_dir)
zf.write(file, arcname)
print(f"ZIP 打包完成: {output_file}")
def clean_build_dir(build_dir: Path) -> None:
"""清理构建目录.
Parameters
----------
build_dir : Path
构建目录
"""
if build_dir.exists():
shutil.rmtree(build_dir)
print(f"清理完成: {build_dir}")
else:
print(f"目录不存在: {build_dir}")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""Python 打包工具主函数."""
parser = argparse.ArgumentParser(
description="PackTool - Python 打包工具",
usage="packtool <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 源码打包命令
src_parser = subparsers.add_parser("src", help="打包项目源码")
src_parser.add_argument("--project-dir", type=str, default=".", help="项目目录")
src_parser.add_argument("--output-dir", type=str, default=DEFAULT_BUILD_DIR, help="输出目录")
# 依赖打包命令
deps_parser = subparsers.add_parser("deps", help="打包项目依赖")
deps_parser.add_argument("--lib-dir", type=str, default=DEFAULT_LIB_DIR, help="依赖库目录")
deps_parser.add_argument("dependencies", nargs="*", help="依赖列表")
# Wheel 打包命令
wheel_parser = subparsers.add_parser("wheel", help="打包项目为 wheel 文件")
wheel_parser.add_argument("--project-dir", type=str, default=".", help="项目目录")
wheel_parser.add_argument("--output-dir", type=str, default=DEFAULT_DIST_DIR, help="输出目录")
# 嵌入式 Python 安装命令
embed_parser = subparsers.add_parser("embed", help="安装嵌入式 Python")
embed_parser.add_argument("--version", type=str, default="3.10", help="Python 版本")
embed_parser.add_argument("--output-dir", type=str, default="python", help="输出目录")
# ZIP 打包命令
zip_parser = subparsers.add_parser("zip", help="创建 ZIP 打包文件")
zip_parser.add_argument("--source-dir", type=str, default=".", help="源目录")
zip_parser.add_argument("--output-file", type=str, default="package.zip", help="输出文件")
# 清理命令
subparsers.add_parser("clean", help="清理构建目录")
args = parser.parse_args()
if args.command == "src":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pack_source",
fn=pack_source,
args=(Path(args.project_dir), Path(args.output_dir)),
)
]
)
elif args.command == "deps":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pack_deps",
fn=pack_dependencies,
args=(Path(args.lib_dir), args.dependencies),
)
]
)
elif args.command == "wheel":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pack_wheel",
fn=pack_wheel,
args=(Path(args.project_dir), Path(args.output_dir)),
)
]
)
elif args.command == "embed":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"install_embed",
fn=install_embed_python,
args=(args.version, Path(args.output_dir)),
)
]
)
elif args.command == "zip":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"create_zip",
fn=create_zip_package,
args=(Path(args.source_dir), Path(args.output_file)),
)
]
)
elif args.command == "clean":
graph = px.Graph.from_specs([px.TaskSpec("clean_build", fn=clean_build_dir, args=(Path(DEFAULT_BUILD_DIR),))])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
+523
View File
@@ -0,0 +1,523 @@
"""PDF 工具模块.
提供 PDF 文件操作的常用功能封装,
支持合并、拆分、压缩、加密、水印、OCR等功能.
"""
from __future__ import annotations
import argparse
from pathlib import Path
import pyflowx as px
try:
import fitz # PyMuPDF
HAS_PYMUPDF = True
except ImportError:
HAS_PYMUPDF = False
try:
import pypdf
HAS_PYPDF = True
except ImportError:
HAS_PYPDF = False
# ============================================================================
# 配置
# ============================================================================
PDF_SUFFIX = ".pdf"
DEFAULT_QUALITY = 75
DEFAULT_PASSWORD = ""
# ============================================================================
# 辅助函数
# ============================================================================
def pdf_merge(input_paths: list[Path], output_path: Path) -> None:
"""合并多个 PDF 文件."""
if not HAS_PYPDF:
print("未安装 pypdf 库,请安装: pip install pypdf")
return
writer = pypdf.PdfWriter()
for input_path in input_paths:
if input_path.exists():
reader = pypdf.PdfReader(str(input_path))
for page in reader.pages:
writer.add_page(page)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"合并完成: {output_path}")
def pdf_split(input_path: Path, output_dir: Path) -> None:
"""拆分 PDF 文件为单页."""
if not HAS_PYPDF:
print("未安装 pypdf 库,请安装: pip install pypdf")
return
reader = pypdf.PdfReader(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
for i, page in enumerate(reader.pages):
writer = pypdf.PdfWriter()
writer.add_page(page)
output_file = output_dir / f"{input_path.stem}_page_{i + 1}.pdf"
with open(output_file, "wb") as f:
writer.write(f)
print(f"拆分完成: {output_dir}")
def pdf_compress(input_path: Path, output_path: Path) -> None:
"""压缩 PDF 文件."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
doc.close()
original_size = input_path.stat().st_size
new_size = output_path.stat().st_size
ratio = (1 - new_size / original_size) * 100
print(f"压缩完成: {output_path} (缩小 {ratio:.1f}%)")
def pdf_encrypt(input_path: Path, output_path: Path, password: str) -> None:
"""加密 PDF 文件."""
if not HAS_PYPDF:
print("未安装 pypdf 库,请安装: pip install pypdf")
return
reader = pypdf.PdfReader(str(input_path))
writer = pypdf.PdfWriter()
for page in reader.pages:
writer.add_page(page)
writer.encrypt(user_password=password, owner_password=password, use_128bit=True)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"加密完成: {output_path}")
def pdf_decrypt(input_path: Path, output_path: Path, password: str) -> None:
"""解密 PDF 文件."""
if not HAS_PYPDF:
print("未安装 pypdf 库,请安装: pip install pypdf")
return
reader = pypdf.PdfReader(str(input_path))
if reader.is_encrypted:
reader.decrypt(password)
writer = pypdf.PdfWriter()
for page in reader.pages:
writer.add_page(page)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"解密完成: {output_path}")
def pdf_extract_text(input_path: Path, output_path: Path) -> None:
"""提取 PDF 文本."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
text = ""
for page in doc:
text += str(page.get_text()) + "\n\n"
doc.close()
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(text, encoding="utf-8")
print(f"文本提取完成: {output_path}")
def pdf_extract_images(input_path: Path, output_dir: Path) -> None:
"""提取 PDF 图片."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
image_count = 0
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
images = page.get_images(full=True)
for img_idx, img in enumerate(images):
xref = img[0]
base_image = doc.extract_image(xref)
image_data = base_image["image"]
image_ext = base_image["ext"]
image_path = output_dir / f"page_{page_num + 1}_img_{img_idx + 1}.{image_ext}"
image_path.write_bytes(image_data)
image_count += 1
doc.close()
print(f"图片提取完成: {output_dir} (共 {image_count} 张)")
def pdf_add_watermark(input_path: Path, output_path: Path, text: str = "CONFIDENTIAL") -> None:
"""添加 PDF 水印."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
for page in doc:
rect = page.rect
text_width = fitz.get_text_length(text, fontsize=48)
x = (rect.width - text_width) / 2
y = rect.height / 2
page.insert_text((x, y), text, fontsize=48, rotate=45, color=(0, 0, 0))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"水印添加完成: {output_path}")
def pdf_rotate(input_path: Path, output_path: Path, rotation: int = 90) -> None:
"""旋转 PDF 页面."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
for page in doc:
page.set_rotation(rotation)
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"旋转完成: {output_path}")
def pdf_crop(input_path: Path, output_path: Path, margins: tuple[int, int, int, int]) -> None:
"""裁剪 PDF 页面."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
left, top, right, bottom = margins
for page in doc:
rect = page.rect
new_rect = fitz.Rect(
rect.x0 + left,
rect.y0 + top,
rect.x1 - right,
rect.y1 - bottom,
)
page.set_cropbox(new_rect)
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"裁剪完成: {output_path}")
def pdf_info(input_path: Path) -> None:
"""显示 PDF 信息."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
print(f"文件: {input_path}")
print(f"页数: {doc.page_count}")
# pyrefly: ignore [missing-attribute]
print(f"标题: {doc.metadata.get('title', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"作者: {doc.metadata.get('author', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"创建日期: {doc.metadata.get('creationDate', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"修改日期: {doc.metadata.get('modDate', 'N/A')}")
print(f"文件大小: {input_path.stat().st_size / 1024:.1f} KB")
doc.close()
def pdf_ocr(input_path: Path, output_path: Path, lang: str = "chi_sim+eng") -> None:
"""PDF OCR 识别."""
try:
import pytesseract
from PIL import Image
except ImportError:
print("未安装 OCR 相关库,请安装: pip install pytesseract pillow")
return
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
new_doc = fitz.open()
for page in doc:
pix = page.get_pixmap()
img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
ocr_text = pytesseract.image_to_string(img, lang=lang)
new_page = new_doc.new_page(width=page.rect.width, height=page.rect.height)
new_page.insert_image(new_page.rect, pixmap=pix)
text_rect = fitz.Rect(0, 0, page.rect.width, page.rect.height)
# pyrefly: ignore [bad-argument-type]
new_page.insert_textbox(text_rect, ocr_text, fontname="china-ss", fontsize=11)
output_path.parent.mkdir(parents=True, exist_ok=True)
new_doc.save(str(output_path))
new_doc.close()
doc.close()
print(f"OCR 识别完成: {output_path}")
def pdf_reorder(input_path: Path, output_path: Path, order: list[int]) -> None:
"""重排 PDF 页面顺序."""
if not HAS_PYPDF:
print("未安装 pypdf 库,请安装: pip install pypdf")
return
reader = pypdf.PdfReader(str(input_path))
writer = pypdf.PdfWriter()
for page_num in order:
if 0 <= page_num < len(reader.pages):
writer.add_page(reader.pages[page_num])
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"重排完成: {output_path}")
def pdf_to_images(input_path: Path, output_dir: Path, dpi: int = 300) -> None:
"""PDF 转图片."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
pix = page.get_pixmap(dpi=dpi)
image_path = output_dir / f"{input_path.stem}_page_{page_num + 1}.png"
pix.save(str(image_path))
doc.close()
print(f"转换完成: {output_dir}")
def pdf_repair(input_path: Path, output_path: Path) -> None:
"""修复 PDF 文件."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
return
doc = fitz.open(str(input_path))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
doc.close()
print(f"修复完成: {output_path}")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None: # noqa: PLR0912
"""PDF 工具主函数."""
parser = argparse.ArgumentParser(
description="PDFTool - PDF 文件工具集",
usage="pdftool <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 合并 PDF 命令
merge_parser = subparsers.add_parser("m", help="合并 PDF 文件")
merge_parser.add_argument("inputs", nargs="+", help="输入 PDF 文件路径")
merge_parser.add_argument("--output", type=str, default="merged.pdf", help="输出文件路径")
# 拆分 PDF 命令
split_parser = subparsers.add_parser("s", help="拆分 PDF 文件为单页")
split_parser.add_argument("input", help="输入 PDF 文件路径")
split_parser.add_argument("--output-dir", type=str, default="split", help="输出目录")
# 压缩 PDF 命令
compress_parser = subparsers.add_parser("c", help="压缩 PDF 文件")
compress_parser.add_argument("input", help="输入 PDF 文件路径")
compress_parser.add_argument("--output", type=str, default="compressed.pdf", help="输出文件路径")
# 加密 PDF 命令
encrypt_parser = subparsers.add_parser("e", help="加密 PDF 文件")
encrypt_parser.add_argument("input", help="输入 PDF 文件路径")
encrypt_parser.add_argument("--output", type=str, default="encrypted.pdf", help="输出文件路径")
encrypt_parser.add_argument("--password", type=str, required=True, help="密码")
# 解密 PDF 命令
decrypt_parser = subparsers.add_parser("d", help="解密 PDF 文件")
decrypt_parser.add_argument("input", help="输入 PDF 文件路径")
decrypt_parser.add_argument("--output", type=str, default="decrypted.pdf", help="输出文件路径")
decrypt_parser.add_argument("--password", type=str, required=True, help="密码")
# 提取文本命令
extract_text_parser = subparsers.add_parser("xt", help="提取 PDF 文本")
extract_text_parser.add_argument("input", help="输入 PDF 文件路径")
extract_text_parser.add_argument("--output", type=str, default="output.txt", help="输出文件路径")
# 提取图片命令
extract_images_parser = subparsers.add_parser("xi", help="提取 PDF 图片")
extract_images_parser.add_argument("input", help="输入 PDF 文件路径")
extract_images_parser.add_argument("--output-dir", type=str, default="images", help="输出目录")
# 添加水印命令
watermark_parser = subparsers.add_parser("w", help="添加 PDF 水印")
watermark_parser.add_argument("input", help="输入 PDF 文件路径")
watermark_parser.add_argument("--output", type=str, default="watermarked.pdf", help="输出文件路径")
watermark_parser.add_argument("--text", type=str, default="CONFIDENTIAL", help="水印文本")
# 旋转 PDF 命令
rotate_parser = subparsers.add_parser("r", help="旋转 PDF 页面")
rotate_parser.add_argument("input", help="输入 PDF 文件路径")
rotate_parser.add_argument("--output", type=str, default="rotated.pdf", help="输出文件路径")
rotate_parser.add_argument("--rotation", type=int, default=90, help="旋转角度 (90, 180, 270)")
# 裁剪 PDF 命令
crop_parser = subparsers.add_parser("crop", help="裁剪 PDF 页面")
crop_parser.add_argument("input", help="输入 PDF 文件路径")
crop_parser.add_argument("--output", type=str, default="cropped.pdf", help="输出文件路径")
crop_parser.add_argument("--left", type=int, default=10, help="左边裁剪")
crop_parser.add_argument("--top", type=int, default=10, help="顶部裁剪")
crop_parser.add_argument("--right", type=int, default=10, help="右边裁剪")
crop_parser.add_argument("--bottom", type=int, default=10, help="底部裁剪")
# 显示信息命令
info_parser = subparsers.add_parser("i", help="显示 PDF 信息")
info_parser.add_argument("input", help="输入 PDF 文件路径")
# OCR 识别命令
ocr_parser = subparsers.add_parser("ocr", help="PDF OCR 识别")
ocr_parser.add_argument("input", help="输入 PDF 文件路径")
ocr_parser.add_argument("--output", type=str, default="ocr.pdf", help="输出文件路径")
ocr_parser.add_argument("--lang", type=str, default="chi_sim+eng", help="OCR 语言")
# 转换图片命令
to_images_parser = subparsers.add_parser("img", help="PDF 转图片")
to_images_parser.add_argument("input", help="输入 PDF 文件路径")
to_images_parser.add_argument("--output-dir", type=str, default="images", help="输出目录")
to_images_parser.add_argument("--dpi", type=int, default=300, help="图片 DPI")
# 修复 PDF 命令
repair_parser = subparsers.add_parser("repair", help="修复 PDF 文件")
repair_parser.add_argument("input", help="输入 PDF 文件路径")
repair_parser.add_argument("--output", type=str, default="repaired.pdf", help="输出文件路径")
args = parser.parse_args()
if args.command == "m":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_merge", fn=pdf_merge, args=([Path(p) for p in args.inputs], Path(args.output)))
])
elif args.command == "s":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_split", fn=pdf_split, args=(Path(args.input), Path(args.output_dir)))
])
elif args.command == "c":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_compress", fn=pdf_compress, args=(Path(args.input), Path(args.output)))
])
elif args.command == "e":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_encrypt", fn=pdf_encrypt, args=(Path(args.input), Path(args.output), args.password))
])
elif args.command == "d":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_decrypt", fn=pdf_decrypt, args=(Path(args.input), Path(args.output), args.password))
])
elif args.command == "xt":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_extract_text", fn=pdf_extract_text, args=(Path(args.input), Path(args.output)))
])
elif args.command == "xi":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_extract_images", fn=pdf_extract_images, args=(Path(args.input), Path(args.output_dir)))
])
elif args.command == "w":
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_watermark",
fn=pdf_add_watermark,
args=(Path(args.input), Path(args.output)),
kwargs={"text": args.text},
)
])
elif args.command == "r":
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_rotate",
fn=pdf_rotate,
args=(Path(args.input), Path(args.output)),
kwargs={"rotation": args.rotation},
)
])
elif args.command == "crop":
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_crop",
fn=pdf_crop,
args=(Path(args.input), Path(args.output)),
kwargs={"margins": (args.left, args.top, args.right, args.bottom)},
)
])
elif args.command == "i":
graph = px.Graph.from_specs([px.TaskSpec("pdf_info", fn=pdf_info, args=(Path(args.input),))])
elif args.command == "ocr":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_ocr", fn=pdf_ocr, args=(Path(args.input), Path(args.output)), kwargs={"lang": args.lang})
])
elif args.command == "img":
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_to_images",
fn=pdf_to_images,
args=(Path(args.input), Path(args.output_dir)),
kwargs={"dpi": args.dpi},
)
])
elif args.command == "repair":
graph = px.Graph.from_specs([
px.TaskSpec("pdf_repair", fn=pdf_repair, args=(Path(args.input), Path(args.output)))
])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
-657
View File
@@ -1,657 +0,0 @@
"""PyFlowX 统一 CLI 入口.
通过 ``pf <tool> [command] [options]`` 调用所有工具,
工具定义在 ``pyflowx.ops`` 子包中, 每个模块用 ``@px.tool`` 装饰器注册.
用法
----
pf # 列出所有可用工具
pf filedate # 查看 filedate 工具帮助
pf filedate add a.txt # 调用 filedate 的 add 子命令
pf pymake b # 调用 pymake 的 b 别名
"""
from __future__ import annotations
import contextlib
import difflib
import importlib
import sys
from collections.abc import Sequence
from typing import Any
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from rich.text import Text
from rich.tree import Tree
from pyflowx import __version__
from pyflowx.tools import _TOOL_REGISTRY, run_tool
class PfApp:
"""pf 统一入口应用.
路由 ``pf <tool> [command]`` 到 ``@px.tool`` 注册的工具或传统 Python 工具.
"""
# 工具名到 ops 模块名的映射 (支持短别名)
_TOOL_ALIASES: dict[str, str] = {
"autofmt": "autofmt",
"af": "autofmt",
"bump": "bumpversion",
"bumpversion": "bumpversion",
"bv": "bumpversion",
"clr": "clr",
"clearscreen": "clr",
"dockercmd": "dockercmd",
"docker": "dockercmd",
"envdev": "envdev",
"env": "envdev",
"filedate": "filedate",
"fd": "filedate",
"filelevel": "filelevel",
"fl": "filelevel",
"folderback": "folderback",
"foldback": "folderback",
"fb": "folderback",
"folderzip": "folderzip",
"foldzip": "folderzip",
"fz": "folderzip",
"git": "gittool",
"gitt": "gittool",
"gittool": "gittool",
"gt": "gittool",
"ls": "lscalc",
"lscalc": "lscalc",
"msdown": "msdownload",
"msdownload": "msdownload",
"msd": "msdownload",
"pack": "packtool",
"packtool": "packtool",
"pk": "packtool",
"pdf": "pdftool",
"pdftool": "pdftool",
"pt": "pdftool",
"pip": "piptool",
"pymake": "pymake",
"piptool": "piptool",
"pp": "piptool",
"reseticon": "reseticoncache",
"reseticoncache": "reseticoncache",
"ric": "reseticoncache",
"screenshot": "screenshot",
"scrcap": "screenshot",
"ss": "screenshot",
"se": "setenv",
"setenv": "setenv",
"sglang": "sglang",
"sg": "sglang",
"ssh": "sshcopyid",
"sshcopy": "sshcopyid",
"sshcopyid": "sshcopyid",
"sc": "sshcopyid",
"taskk": "taskkill",
"taskkill": "taskkill",
"tk": "taskkill",
"wch": "which",
"which": "which",
"wf": "writefile",
"writefile": "writefile",
}
# 传统工具: 有自己的 main() 函数 (无法 @px.tool 化的复杂逻辑)
_LEGACY_TOOLS: dict[str, str] = {
"emlman": "pyflowx.cli.legacy.emlmanager:main",
"profiler": "pyflowx.cli.legacy.profiler:main",
"pxp": "pyflowx.cli.legacy.profiler:main",
}
# 规范工具名 → 完整模块路径 (ops/ 按功能分组后的动态导入映射)
_TOOL_MODULES: dict[str, str] = {
"autofmt": "pyflowx.ops.dev.autofmt",
"bumpversion": "pyflowx.ops.dev.bumpversion",
"clr": "pyflowx.ops.system.clr",
"dockercmd": "pyflowx.ops.infra.dockercmd",
"envdev": "pyflowx.ops.infra.envdev",
"filedate": "pyflowx.ops.files.filedate",
"filelevel": "pyflowx.ops.files.filelevel",
"folderback": "pyflowx.ops.files.folderback",
"folderzip": "pyflowx.ops.files.folderzip",
"gittool": "pyflowx.ops.dev.gittool",
"lscalc": "pyflowx.ops.dev.lscalc",
"msdownload": "pyflowx.ops.infra.msdownload",
"packtool": "pyflowx.ops.dev.packtool",
"pdftool": "pyflowx.ops.files.pdftool",
"piptool": "pyflowx.ops.dev.piptool",
"pymake": "pyflowx.ops.dev.pymake",
"reseticoncache": "pyflowx.ops.system.reseticoncache",
"screenshot": "pyflowx.ops.files.screenshot",
"setenv": "pyflowx.ops.system.setenv",
"sglang": "pyflowx.ops.infra.sglang",
"sshcopyid": "pyflowx.ops.infra.sshcopyid",
"taskkill": "pyflowx.ops.system.taskkill",
"which": "pyflowx.ops.system.which",
"writefile": "pyflowx.ops.system.writefile",
}
def __init__(self, argv: Sequence[str] | None = None) -> None:
self._argv = list(argv) if argv is not None else sys.argv[1:]
self._console = Console()
self._err = Console(stderr=True)
def run(self) -> int:
"""主入口, 返回退出码."""
if not self._argv or self._argv[0] in ("--help", "-h"):
self._list_tools()
return 0
first = self._argv[0]
if first in ("--version", "-V"):
self._console.print(f"PyFlowX [bold cyan]{__version__}[/bold cyan]")
return 0
# 内建子命令(与 @px.tool 注册的 ops 工具同级)
builtin = {
"yamlrun": self._run_yaml,
"graph": self._run_graph,
"info": self._run_info,
"completion": self._run_completion,
}
if first in builtin:
return builtin[first](self._argv[1:])
rest = self._argv[1:]
resolved = self._resolve_tool(first)
if resolved is None:
self._print_unknown_tool(first)
return 1
kind, target = resolved
if kind == "legacy":
return self._run_legacy(target, rest)
return self._run_tool(target, rest)
# ------------------------------------------------------------------ #
# 工具列表 (rich)
# ------------------------------------------------------------------ #
def _list_tools(self) -> None:
"""rich 表格列出所有可用工具."""
self._console.print(
Panel(
Text(f"PyFlowX v{__version__}", style="bold cyan", justify="center"),
subtitle="[dim]pf <tool> [command] [options][/dim]",
)
)
table = Table(title="@px.tool 工具", show_header=True, header_style="bold", show_lines=False)
table.add_column("命令", style="cyan", no_wrap=True)
table.add_column("别名", style="dim", no_wrap=True)
table.add_column("说明")
for tool in sorted(set(self._TOOL_ALIASES.values())):
aliases = self._aliases_for(tool)
table.add_row(f"pf {tool}", ", ".join(aliases), self._tool_description(tool))
self._console.print(table)
if self._LEGACY_TOOLS:
legacy = Table(title="传统工具", show_header=True, header_style="bold", show_lines=False)
legacy.add_column("命令", style="cyan", no_wrap=True)
for tool in sorted(self._LEGACY_TOOLS):
legacy.add_row(f"pf {tool}")
self._console.print(legacy)
self._console.print("\n[bold]示例:[/bold]")
self._console.print(" [cyan]pf filedate add a.txt[/cyan] # 给文件添加日期前缀")
self._console.print(" [cyan]pf pymake b[/cyan] # 构建 Python 包")
self._console.print(" [cyan]pf gitt c[/cyan] # 清理并查看 git 状态")
self._console.print(" [cyan]pf yamlrun pipeline.yaml[/cyan] # 执行 YAML 任务图")
def _aliases_for(self, canonical: str) -> list[str]:
"""获取工具的别名 (不含规范名本身)."""
return sorted(a for a, t in self._TOOL_ALIASES.items() if t == canonical and a != canonical)
def _tool_description(self, tool_name: str) -> str:
"""获取工具描述 (从 _TOOL_REGISTRY 中已注册 ToolSpec 的 description/help)."""
with contextlib.suppress(ImportError, KeyError):
importlib.import_module(self._TOOL_MODULES[tool_name])
if tool_name not in _TOOL_REGISTRY:
return ""
subs = _TOOL_REGISTRY[tool_name]
for spec in subs.values():
if spec.description:
return spec.description
for spec in subs.values():
if not spec.hidden and spec.help:
return spec.help
return ""
# ------------------------------------------------------------------ #
# 路由
# ------------------------------------------------------------------ #
def _resolve_tool(self, name: str) -> tuple[str, str] | None:
"""解析工具名, 返回 (类型, 目标)."""
if name in self._TOOL_ALIASES:
return ("tool", self._TOOL_ALIASES[name])
if name in self._LEGACY_TOOLS:
return ("legacy", self._LEGACY_TOOLS[name])
return None
def _print_unknown_tool(self, name: str) -> None:
"""打印未知工具错误 + 模糊匹配建议."""
self._err.print(f"[red]错误:[/red] 未知工具 [yellow]{name!r}[/yellow]")
suggestions = difflib.get_close_matches(name, list(self._TOOL_ALIASES), n=3, cutoff=0.5)
if suggestions:
self._err.print(f"[dim]是否想用: {', '.join(suggestions)}[/dim]")
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
def _run_legacy(self, module_path: str, argv: list[str]) -> int:
"""运行传统工具的 main() 函数."""
module_name, func_name = module_path.split(":", 1)
module = importlib.import_module(module_name)
func = getattr(module, func_name)
original_argv = sys.argv
sys.argv = [f"pf {module_name.split('.')[-1]}", *argv]
try:
func()
return 0
except SystemExit as e:
return int(e.code) if e.code is not None else 0
finally:
sys.argv = original_argv
def _run_yaml(self, argv: list[str]) -> int:
"""执行 yamlrun 命令: 从 YAML 文件加载任务图并执行."""
import argparse
from typing import get_args
from pyflowx import Graph, run
from pyflowx.errors import PyFlowXError, TaskFailedError
from pyflowx.executors import Strategy
from pyflowx.runner import CliExitCode
parser = argparse.ArgumentParser(prog="pf yamlrun", description="执行 YAML 任务图")
_ = parser.add_argument("file", help="YAML 文件路径")
_ = parser.add_argument(
"--strategy",
choices=list(get_args(Strategy)),
default="dependency",
help="执行策略 (默认: %(default)s)",
)
_ = parser.add_argument("--dry-run", action="store_true", help="仅打印执行计划")
_ = parser.add_argument("--list", action="store_true", help="列出所有任务名")
_ = parser.add_argument("--list-tag", help="--list 模式下只显示含此标签的任务")
_ = parser.add_argument(
"--list-name",
help="--list 模式下按子串过滤任务名(大小写不敏感)",
)
_ = parser.add_argument("--quiet", action="store_true", help="静默模式")
_ = parser.add_argument("--progress", action="store_true", help="显示 rich 进度条")
_ = parser.add_argument("--only", help="只运行指定任务(逗号分隔)及其依赖")
_ = parser.add_argument("--tags", help="只运行匹配标签的任务(逗号分隔)及其依赖")
parsed = parser.parse_args(argv)
try:
graph = Graph.from_yaml(parsed.file)
except (OSError, ValueError, PyFlowXError) as e:
self._err.print(f"[red]错误:[/red] 加载 YAML 失败: {e}")
return CliExitCode.FAILURE.value
if parsed.list:
self._print_task_list(
graph,
tag_filter=parsed.list_tag,
name_filter=parsed.list_name,
)
return CliExitCode.SUCCESS.value
if parsed.dry_run:
self._console.print(graph.describe())
return CliExitCode.SUCCESS.value
only = parsed.only.split(",") if parsed.only else None
tags = parsed.tags.split(",") if parsed.tags else None
progress = parsed.progress and not parsed.quiet
try:
report = run(
graph,
strategy=parsed.strategy,
verbose=not parsed.quiet,
only=only,
tags=tags,
progress=progress,
)
if not report.success and not parsed.quiet:
self._print_diagnostics(report)
return CliExitCode.SUCCESS.value if report.success else CliExitCode.FAILURE.value
except KeyboardInterrupt:
print("\n操作已取消", file=sys.stderr)
return CliExitCode.INTERRUPTED.value
except PyFlowXError as e:
self._err.print(f"[red]错误:[/red] {e}")
# TaskFailedError 携带 report 时打印诊断摘要
if isinstance(e, TaskFailedError) and e.report is not None and not parsed.quiet:
self._print_diagnostics(e.report)
return CliExitCode.FAILURE.value
def _print_diagnostics(self, report: Any) -> None:
"""打印失败诊断摘要到 stderr。"""
diag = report.diagnose()
if diag is not None:
self._err.print()
self._err.print(diag.describe(), style="red")
def _print_task_list(
self,
graph: Any,
tag_filter: str | None,
name_filter: str | None,
) -> None:
"""打印任务列表,可按标签/名称子串过滤。"""
name_lower = name_filter.lower() if name_filter else None
for name in graph.names:
if name_lower is not None and name_filter is not None and name_lower not in name.lower():
continue
if tag_filter is not None:
spec = graph.spec(name)
if tag_filter not in spec.tags:
continue
self._console.print(name)
def _run_graph(self, argv: list[str]) -> int:
"""执行 graph 命令: 终端可视化 YAML 任务图 DAG."""
import argparse
from pyflowx import Graph
from pyflowx.errors import PyFlowXError
from pyflowx.runner import CliExitCode
parser = argparse.ArgumentParser(prog="pf graph", description="可视化 YAML 任务图 DAG")
_ = parser.add_argument("file", help="YAML 文件路径")
_ = parser.add_argument(
"--format",
choices=["ascii", "mermaid", "list", "deps"],
default="ascii",
help="输出格式 (默认: %(default)s)",
)
_ = parser.add_argument(
"--orientation",
default="TD",
help="Mermaid 方向: TD/TB/BT/LR/RL (默认: %(default)s)",
)
_ = parser.add_argument(
"--color-by",
choices=["tag", "none"],
default="tag",
help="ASCII 格式节点着色策略: tag(按首标签着色) / none(单色) (默认: %(default)s)",
)
parsed = parser.parse_args(argv)
try:
graph = Graph.from_yaml(parsed.file)
except (OSError, ValueError, PyFlowXError) as e:
self._err.print(f"[red]错误:[/red] 加载 YAML 失败: {e}")
return CliExitCode.FAILURE.value
fmt = parsed.format
try:
if fmt == "mermaid":
# Mermaid 输出不加颜色,便于复制粘贴到 mermaid.live
self._console.print(graph.to_mermaid(parsed.orientation), markup=False)
elif fmt == "list":
for name in graph.names:
self._console.print(name)
elif fmt == "deps":
self._render_deps_table(graph)
else:
self._render_ascii(graph, color_by=parsed.color_by)
except ValueError as e:
self._err.print(f"[red]错误:[/red] {e}")
return CliExitCode.FAILURE.value
return CliExitCode.SUCCESS.value
def _render_ascii(self, graph: Any, color_by: str = "tag") -> None:
"""用 rich 渲染分层 DAG 视图与依赖关系表。
``color_by="tag"`` 时按任务首标签选择 rich 颜色,便于视觉分组;
``color_by="none"`` 时统一使用 cyan。
"""
layers = graph.layers()
if not layers:
self._console.print("[dim]空图(无任务)[/dim]")
return
palette = ("cyan", "magenta", "yellow", "green", "blue", "red", "bright_cyan")
tag_colors: dict[str, str] = {}
counter = [0]
def _color_for(name: str) -> str:
if color_by == "none":
return "cyan"
spec = graph.spec(name)
if not spec.tags:
return "cyan"
tag = spec.tags[0]
if tag not in tag_colors:
tag_colors[tag] = palette[counter[0] % len(palette)]
counter[0] += 1
return tag_colors[tag]
tree = Tree(f"[bold cyan]Graph[/bold cyan] ({len(graph)} 任务, {len(layers)} 层)")
for i, layer in enumerate(layers, 1):
names = ", ".join(f"[{_color_for(n)}]{n}[/{_color_for(n)}]" for n in layer)
tree.add(f"[bold]Layer {i}[/bold] ({len(layer)}): {names}")
self._console.print(tree)
if tag_colors and color_by == "tag":
legend = " ".join(f"[{c}]{t}[/{c}]" for t, c in tag_colors.items())
self._console.print(f"[dim]标签颜色:[/dim] {legend}")
self._console.print()
self._render_deps_table(graph)
def _render_deps_table(self, graph: Any) -> None:
"""渲染任务依赖关系表."""
table = Table(title="任务依赖关系", show_header=True, header_style="bold")
table.add_column("任务", style="cyan", no_wrap=True)
table.add_column("硬依赖", style="yellow")
table.add_column("软依赖", style="dim")
table.add_column("标签", style="green")
for name in graph.names:
spec = graph.spec(name)
hard = ", ".join(spec.depends_on) if spec.depends_on else "-"
soft = ", ".join(spec.soft_depends_on) if spec.soft_depends_on else "-"
tags = ", ".join(spec.tags) if spec.tags else "-"
table.add_row(name, hard, soft, tags)
self._console.print(table)
def _run_tool(self, target: str, argv: list[str]) -> int:
"""运行 @px.tool 工具.
导入 ``pyflowx.ops.<group>.<target>`` 触发 ``@px.tool`` 注册, 然后调用 ``run_tool``.
"""
with contextlib.suppress(ImportError, KeyError):
importlib.import_module(self._TOOL_MODULES[target])
if target not in _TOOL_REGISTRY:
self._err.print(f"[red]错误:[/red] 未找到工具 [yellow]{target!r}[/yellow]")
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
return 1
return run_tool(target, argv)
# ------------------------------------------------------------------ #
# info: 工具详情
# ------------------------------------------------------------------ #
def _run_info(self, argv: list[str]) -> int:
"""显示工具详情: ``pf info [tool]``。
无参数时列出所有工具的简表(含子命令数、描述);
指定工具时显示该工具的所有子命令详情。
"""
if not argv:
self._print_all_tools_detail()
return 0
name = argv[0]
resolved = self._resolve_tool(name)
if resolved is None or resolved[0] != "tool":
self._err.print(f"[red]错误:[/red] 未知工具 [yellow]{name!r}[/yellow]")
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
return 1
canonical = resolved[1]
self._print_tool_detail(canonical)
return 0
def _print_all_tools_detail(self) -> None:
"""打印所有工具的详情表。"""
# 触发工具模块导入以填充 _TOOL_REGISTRY
for mod_path in self._TOOL_MODULES.values():
with contextlib.suppress(ImportError):
importlib.import_module(mod_path)
table = Table(title="工具详情", show_header=True, header_style="bold")
table.add_column("工具", style="cyan", no_wrap=True)
table.add_column("别名", style="dim")
table.add_column("子命令数", justify="right")
table.add_column("描述")
for canonical in sorted(set(self._TOOL_ALIASES.values())):
aliases = ", ".join(self._aliases_for(canonical))
subs = _TOOL_REGISTRY.get(canonical, {})
desc = self._tool_description(canonical)
table.add_row(canonical, aliases or "-", str(len(subs)), desc or "-")
self._console.print(table)
def _print_tool_detail(self, canonical: str) -> None:
"""打印指定工具的子命令详情。"""
with contextlib.suppress(ImportError, KeyError):
importlib.import_module(self._TOOL_MODULES[canonical])
subs = _TOOL_REGISTRY.get(canonical, {})
aliases = self._aliases_for(canonical)
self._console.print(f"[bold cyan]{canonical}[/bold cyan]", end="")
if aliases:
self._console.print(f" [dim]别名: {', '.join(aliases)}[/dim]")
else:
self._console.print()
desc = self._tool_description(canonical)
if desc:
self._console.print(f"[dim]描述: {desc}[/dim]")
if not subs:
self._console.print("[dim]无子命令(直接调用)[/dim]")
return
table = Table(title=f"{canonical} 子命令", show_header=True, header_style="bold")
table.add_column("子命令", style="cyan", no_wrap=True)
table.add_column("描述")
table.add_column("隐藏", justify="center")
for name in sorted(n for n in subs if n is not None):
spec = subs[name]
table.add_row(
name,
spec.description or spec.help or "-",
"" if spec.hidden else "-",
)
self._console.print(table)
# ------------------------------------------------------------------ #
# completion: shell 补全脚本生成
# ------------------------------------------------------------------ #
def _run_completion(self, argv: list[str]) -> int:
"""生成 shell 补全脚本: ``pf completion bash|zsh|fish``。"""
import argparse
parser = argparse.ArgumentParser(prog="pf completion", description="生成 shell 补全脚本")
_ = parser.add_argument(
"shell",
choices=["bash", "zsh", "fish"],
help="目标 shell",
)
parsed = parser.parse_args(argv)
tools = sorted(set(self._TOOL_ALIASES.values()))
builtin_cmds = ["yamlrun", "graph", "info", "completion"]
all_cmds = builtin_cmds + tools
if parsed.shell == "bash":
self._console.print(self._bash_completion(all_cmds), markup=False)
elif parsed.shell == "zsh":
self._console.print(self._zsh_completion(all_cmds), markup=False)
else:
self._console.print(self._fish_completion(all_cmds), markup=False)
return 0
@staticmethod
def _bash_completion(cmds: list[str]) -> str:
"""生成 bash 补全脚本。"""
words = " ".join(cmds)
return f"""# pf bash 补全 —— 添加到 ~/.bashrc 或保存到 /etc/bash_completion.d/pf
_pf_complete() {{
local cur prev
cur="${{COMP_WORDS[COMP_CWORD]}}"
prev="${{COMP_WORDS[COMP_CWORD-1]}}"
if [ "$COMP_CWORD" -eq 1 ]; then
COMPREPLY=($(compgen -W "{words}" -- "$cur"))
return 0
fi
# 二级补全:交给具体子命令(pf <tool> --help 自描述)
return 0
}}
complete -F _pf_complete pf
"""
@staticmethod
def _zsh_completion(cmds: list[str]) -> str:
"""生成 zsh 补全脚本。"""
words = " ".join(cmds)
return f"""# pf zsh 补全 —— 添加到 ~/.zshrc
#compdef pf
_pf() {{
local -a cmds
cmds=({words})
if [ "$CURRENT" -eq 2 ]; then
_describe 'command' cmds
return 0
fi
return 0
}}
_pf "$@"
"""
@staticmethod
def _fish_completion(cmds: list[str]) -> str:
"""生成 fish 补全脚本。"""
lines = ["# pf fish 补全 —— 保存到 ~/.config/fish/completions/pf.fish"]
for c in cmds:
lines.append(f'complete -c pf -n "__fish_use_subcommand" -a "{c}"')
return "\n".join(lines) + "\n"
def main() -> None:
"""pf 统一入口主函数."""
sys.exit(PfApp().run())
if __name__ == "__main__":
main()
+195
View File
@@ -0,0 +1,195 @@
"""pip 包管理工具模块.
提供 pip 包管理操作的封装,
支持安装、卸载、下载等功能.
"""
from __future__ import annotations
import argparse
import fnmatch
import subprocess
from pathlib import Path
import pyflowx as px
# ============================================================================
# 配置
# ============================================================================
PACKAGE_DIR = "packages"
REQUIREMENTS_FILE = "requirements.txt"
# 受保护的包名集合
_PROTECTED_PACKAGES: frozenset[str] = frozenset({
"pyflowx",
"bitool",
})
# ============================================================================
# 辅助函数
# ============================================================================
def _get_installed_packages() -> list[str]:
"""获取当前环境中所有已安装的包名."""
try:
result = subprocess.run(
["pip", "list", "--format=freeze"],
capture_output=True,
text=True,
check=True,
)
packages: list[str] = []
for line in result.stdout.strip().split("\n"):
if line and "==" in line:
pkg_name = line.split("==")[0].strip()
packages.append(pkg_name)
except (subprocess.SubprocessError, OSError):
return []
return packages
def _expand_wildcard_packages(pattern: str) -> list[str]:
"""展开通配符模式为实际的包名列表."""
if not any(char in pattern for char in ["*", "?", "[", "]"]):
return [pattern]
installed_packages = _get_installed_packages()
matched = [pkg for pkg in installed_packages if fnmatch.fnmatchcase(pkg.lower(), pattern.lower())]
return matched
def _filter_protected_packages(packages: list[str]) -> list[str]:
"""过滤掉受保护的包名."""
safe = [p for p in packages if p.lower() not in {p.lower() for p in _PROTECTED_PACKAGES}]
filtered = [p for p in packages if p.lower() in {p.lower() for p in _PROTECTED_PACKAGES}]
if filtered:
print(f"跳过受保护的包: {', '.join(filtered)}")
return safe
def pip_uninstall(pkg_names: list[str]) -> None:
"""卸载包."""
packages_to_uninstall: list[str] = []
for pattern in pkg_names:
packages_to_uninstall.extend(_expand_wildcard_packages(pattern))
packages_to_uninstall = _filter_protected_packages(packages_to_uninstall)
if not packages_to_uninstall:
return
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True)
def pip_reinstall(pkg_names: list[str], offline: bool = False) -> None:
"""重新安装包."""
safe_pkgs = _filter_protected_packages(pkg_names)
if not safe_pkgs:
print("所有指定的包均为受保护包, 跳过重装")
return
subprocess.run(["pip", "uninstall", "-y", *safe_pkgs], check=True)
options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run(["pip", "install", *options, *safe_pkgs], check=True)
def pip_download(pkg_names: list[str], offline: bool = False) -> None:
"""下载包."""
options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run(
["pip", "download", *pkg_names, *options, "-d", PACKAGE_DIR],
check=True,
)
def pip_freeze() -> None:
"""冻结依赖."""
result = subprocess.run(
["pip", "freeze", "--exclude-editable"],
capture_output=True,
text=True,
check=True,
)
Path(REQUIREMENTS_FILE).write_text(result.stdout)
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""pip 工具主函数."""
parser = argparse.ArgumentParser(
description="PipTool - pip 包管理工具",
usage="piptool <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 安装命令
install_parser = subparsers.add_parser("i", help="安装包")
install_parser.add_argument("packages", nargs="+", help="要安装的包名")
# 卸载命令
uninstall_parser = subparsers.add_parser("u", help="卸载包")
uninstall_parser.add_argument("packages", nargs="+", help="要卸载的包名 (支持通配符)")
# 重装命令
reinstall_parser = subparsers.add_parser("r", help="重新安装包")
reinstall_parser.add_argument("packages", nargs="+", help="要重装的包名")
reinstall_parser.add_argument("--offline", action="store_true", help="使用离线模式")
# 下载命令
download_parser = subparsers.add_parser("d", help="下载包")
download_parser.add_argument("packages", nargs="+", help="要下载的包名")
download_parser.add_argument("--offline", action="store_true", help="使用离线模式")
# 升级 pip 命令
subparsers.add_parser("up", help="升级 pip")
# 冻结依赖命令
subparsers.add_parser("f", help="冻结依赖到 requirements.txt")
args = parser.parse_args()
if args.command == "i":
graph = px.Graph.from_specs([px.TaskSpec("pip_install", cmd=["pip", "install", *args.packages], verbose=True)])
elif args.command == "u":
graph = px.Graph.from_specs([
px.TaskSpec("pip_uninstall", fn=pip_uninstall, args=(args.packages,), verbose=True)
])
elif args.command == "r":
graph = px.Graph.from_specs([
px.TaskSpec(
"pip_reinstall",
fn=pip_reinstall,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
])
elif args.command == "d":
graph = px.Graph.from_specs([
px.TaskSpec(
"pip_download",
fn=pip_download,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
])
elif args.command == "up":
graph = px.Graph.from_specs([
px.TaskSpec("pip_upgrade", cmd=["python", "-m", "pip", "install", "--upgrade", "pip"], verbose=True)
])
elif args.command == "f":
graph = px.Graph.from_specs([px.TaskSpec("pip_freeze", fn=pip_freeze, verbose=True)])
else:
parser.print_help()
return
px.run(graph, strategy="thread")
@@ -40,10 +40,10 @@ import webbrowser
from pathlib import Path
from typing import Any
from ... import executors as _executors
from ... import runner as _runner
from ...profiling import ProfileReport
from ...report import RunReport
from .. import executors as _executors
from .. import runner as _runner
from ..profiling import ProfileReport
from ..report import RunReport
def _build_parser() -> argparse.ArgumentParser:
+125
View File
@@ -0,0 +1,125 @@
"""Python 构建工具模块.
完全替代传统的 Makefile,
提供更好的跨平台兼容性和 Python 生态集成.
"""
from __future__ import annotations
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
# 项目根目录(pymake.py 在 src/pyflowx/cli,向上四层到达根目录)
ROOT_DIR = Path(__file__).parent.parent.parent.parent
MATURIN_BUILD_COMMAND = ["maturin", "build", "-r"]
if Constants.IS_WINDOWS:
MATURIN_BUILD_COMMAND.extend(["--target", "x86_64-win7-windows-msvc", "-Zbuild-std", "-i", "python3.8"])
# 扁平注册所有任务(px.cmd 自动从命令前两段推导 name)
# 所有任务指定 cwd=ROOT_DIR,确保在项目根目录执行
tasks: list[px.TaskSpec] = [
px.cmd(["uv", "build"], cwd=ROOT_DIR),
px.cmd(MATURIN_BUILD_COMMAND, cwd=ROOT_DIR),
px.cmd(["uv", "sync"], cwd=ROOT_DIR),
px.cmd(["gitt", "c"], name="git_clean", cwd=ROOT_DIR),
px.cmd(
["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"],
name="test",
cwd=ROOT_DIR,
),
px.cmd(
["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"],
name="test_fast",
cwd=ROOT_DIR,
),
px.cmd(
["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
name="test_coverage",
cwd=ROOT_DIR,
),
px.cmd(["pyrefly", "check", "."], cwd=ROOT_DIR),
px.cmd(["git", "add", "-A"], name="git_add_all", cwd=ROOT_DIR),
px.cmd(["bumpversion"], cwd=ROOT_DIR),
px.cmd(["bumpversion", "minor"], cwd=ROOT_DIR),
px.cmd(["git", "push"], cwd=ROOT_DIR),
px.cmd(["git", "push", "--tags"], name="git_push_tags", cwd=ROOT_DIR),
px.cmd(["hatch", "publish"], name="publish_python", cwd=ROOT_DIR),
px.cmd(["twine", "upload", "--disable-progress-bar"], name="twine_publish", cwd=ROOT_DIR),
]
# 单任务别名(alias 名与任务名相同):直接内联 TaskSpec,避免 str 自引用
aliases: dict[str, str | list[str | px.TaskSpec] | px.TaskSpec | px.Graph] = {
# 构建命令
"b": "uv_build",
"bc": "maturin_build",
"ba": ["b", "bc"],
# 安装命令
"sync": "uv_sync",
# 清理命令
"c": "git_clean",
# 开发工具
"bump": ["c", "tc", "git_add_all", "bumpversion"],
"bumpmi": "bumpversion_minor",
"cov": ["git_clean", "test_coverage"],
"doc": px.cmd(["sphinx-build", "-b", "html", "docs", "docs/_build"], name="doc", cwd=ROOT_DIR),
"lint": px.cmd(["ruff", "check", "--fix", "--unsafe-fixes"], name="lint", cwd=ROOT_DIR),
"pb": ["twine_publish", "publish_python"],
"t": "test",
"tf": "test_fast",
"tc": ["pyrefly_check", "lint"],
"tox": px.cmd(["tox", "-p", "auto"], name="tox", cwd=ROOT_DIR),
# 发布命令
"p": ["git_clean", "git_push", "git_push_tags"],
}
def main() -> None:
"""pymake 构建工具.
🔨 构建命令:
pymake b - 构建 Python 主包 (uv build)
pymake bc - 构建 Rust 核心模块 (maturin build)
pymake ba - 构建所有包 (先 Python 后 Rust)
📦 安装命令 (开发模式):
pymake sync - 安装依赖包 (uv sync)
🧹 清理命令:
pymake c - 清理所有构建产物 (gitt c)
🛠️ 开发工具:
pymake t - 运行测试 (pytest)
pymake tc - 运行测试并生成覆盖率报告
pymake tf - 运行快速测试 (pytest -m not slow)
pymake lint - 代码格式化与检查 (ruff)
pymake type - 类型检查 (mypy, ty)
pymake doc - 构建文档 (sphinx)
🔬 多版本测试:
pymake tox - 多版本 Python 测试 (tox -p auto)
📦 发布命令:
pymake pb - 发布到 PyPI (twine + hatch)
🔖 版本管理:
pymake bump - 自动升级版本号并提交修改 (清理 + 检查 + 格式化 + git add + bumpversion)
💡 常用工作流:
1. 日常开发: pymake lint && pymake t
2. 构建发布包: pymake ba
3. 多版本兼容性测试: pymake tox
4. 发布到 PyPI: pymake pb
📝 示例:
pymake ba # 构建所有包
pymake sync # 安装依赖
pymake t # 运行测试
pymake tox # 多版本兼容性测试
pymake lint # 格式化代码
pymake type # 类型检查
"""
runner = px.CliRunner(strategy="sequential", description="PyMake - Python 构建工具", tasks=tasks, aliases=aliases)
runner.run_cli()
+10
View File
@@ -0,0 +1,10 @@
from __future__ import annotations
import pyflowx as px
from pyflowx.tasks.system import reset_icon_cache
def main() -> None:
"""重启图标缓存工具主函数."""
graph = px.Graph.from_specs(reset_icon_cache())
px.run(graph, strategy="thread")
@@ -1,10 +1,11 @@
"""screenshot - 截图工具.
"""截图工具.
跨平台截图: Windows PowerShell, macOS screencapture, Linux gnome-screenshot/scrot.
跨平台截图工具, 支持全屏截图和区域截图.
"""
from __future__ import annotations
import argparse
import subprocess
from datetime import datetime
from pathlib import Path
@@ -12,6 +13,10 @@ from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
# ============================================================================
# 辅助函数
# ============================================================================
def get_screenshot_path(filename: str | None = None) -> Path:
"""获取截图保存路径.
@@ -35,7 +40,6 @@ def get_screenshot_path(filename: str | None = None) -> Path:
return screenshots_dir / filename
@px.tool("screenshot", subcommand="full", help="全屏截图")
def take_screenshot_full(filename: str | None = None) -> None:
"""全屏截图.
@@ -47,6 +51,7 @@ def take_screenshot_full(filename: str | None = None) -> None:
output_path = get_screenshot_path(filename)
if Constants.IS_WINDOWS:
# Windows: 使用 PowerShell 截图
ps_script = f"""
Add-Type -AssemblyName System.Windows.Forms
Add-Type -AssemblyName System.Drawing
@@ -61,8 +66,10 @@ $bitmap.Dispose()
"""
subprocess.run(["powershell", "-Command", ps_script], check=True)
elif Constants.IS_MACOS:
# macOS: 使用 screencapture
subprocess.run(["screencapture", "-x", str(output_path)], check=True)
else:
# Linux: 使用 gnome-screenshot 或 scrot
try:
subprocess.run(["gnome-screenshot", "-f", str(output_path)], check=True)
except FileNotFoundError:
@@ -71,7 +78,6 @@ $bitmap.Dispose()
print(f"截图已保存: {output_path}")
@px.tool("screenshot", subcommand="area", help="区域截图")
def take_screenshot_area(filename: str | None = None) -> None:
"""区域截图.
@@ -83,6 +89,7 @@ def take_screenshot_area(filename: str | None = None) -> None:
output_path = get_screenshot_path(filename)
if Constants.IS_WINDOWS:
# Windows: 使用 PowerShell 截图 (需要用户选择区域)
ps_script = f"""
Add-Type -AssemblyName System.Windows.Forms
Add-Type -AssemblyName System.Drawing
@@ -106,11 +113,51 @@ $bitmap.Dispose()
"""
subprocess.run(["powershell", "-Command", ps_script], check=True)
elif Constants.IS_MACOS:
# macOS: 使用 screencapture 交互模式
subprocess.run(["screencapture", "-i", str(output_path)], check=True)
else:
# Linux: 使用 gnome-screenshot 交互模式
try:
subprocess.run(["gnome-screenshot", "-a", "-f", str(output_path)], check=True)
except FileNotFoundError:
subprocess.run(["scrot", "-s", str(output_path)], check=True)
print(f"截图已保存: {output_path}")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""截图工具主函数."""
parser = argparse.ArgumentParser(
description="Screenshot - 截图工具",
usage="screenshot <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 全屏截图命令
full_parser = subparsers.add_parser("full", help="全屏截图")
full_parser.add_argument("--filename", type=str, help="文件名")
# 区域截图命令
area_parser = subparsers.add_parser("area", help="区域截图")
area_parser.add_argument("--filename", type=str, help="文件名")
args = parser.parse_args()
if args.command == "full":
graph = px.Graph.from_specs(
[px.TaskSpec("screenshot_full", fn=take_screenshot_full, kwargs={"filename": args.filename})]
)
elif args.command == "area":
graph = px.Graph.from_specs(
[px.TaskSpec("screenshot_area", fn=take_screenshot_area, kwargs={"filename": args.filename})]
)
else:
parser.print_help()
return
px.run(graph, strategy="thread")
@@ -1,20 +1,23 @@
"""sshcopyid - SSH 密钥部署工具.
"""SSH 密钥部署工具.
本地 SSH 公钥部署到远程服务器.
类似 ssh-copy-id, 自动 SSH 公钥部署到远程服务器,
支持密码认证和密钥认证两种方式.
"""
from __future__ import annotations
import argparse
import subprocess
import sys
from pathlib import Path
import pyflowx as px
__all__ = ["ssh_copy_id"]
# ============================================================================
# 辅助函数
# ============================================================================
@px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器")
def ssh_copy_id(
hostname: str,
username: str,
@@ -40,6 +43,7 @@ def ssh_copy_id(
timeout : int
SSH 操作超时秒数, 默认 30
"""
# 读取公钥
pub_key_path = Path(keypath).expanduser()
if not pub_key_path.exists():
print(f"公钥文件不存在: {pub_key_path}")
@@ -47,10 +51,12 @@ def ssh_copy_id(
pub_key = pub_key_path.read_text().strip()
# 构建部署脚本
script = f"""mkdir -p ~/.ssh && chmod 700 ~/.ssh
cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys
grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys"""
# 使用 sshpass 执行
try:
subprocess.run(
[
@@ -74,7 +80,7 @@ grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}'
)
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}")
except FileNotFoundError:
print(f"未找到 sshpass 工具, 请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
print(f"未找到 sshpass 工具请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
sys.exit(1)
except subprocess.TimeoutExpired:
print("SSH 连接超时")
@@ -82,3 +88,35 @@ grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}'
except subprocess.CalledProcessError as e:
print(f"SSH 执行失败: {e}")
sys.exit(1)
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""SSH 密钥部署工具主函数."""
parser = argparse.ArgumentParser(
description="SSHCopyID - SSH 密钥部署工具",
usage="sshcopyid <hostname> <username> <password> [--port PORT] [--keypath KEYPATH]",
)
parser.add_argument("hostname", type=str, help="远程服务器主机名或 IP 地址")
parser.add_argument("username", type=str, help="远程服务器用户名")
parser.add_argument("password", type=str, help="远程服务器密码")
parser.add_argument("--port", type=int, default=22, help="SSH 端口 (默认: 22)")
parser.add_argument("--keypath", type=str, default="~/.ssh/id_rsa.pub", help="公钥文件路径")
parser.add_argument("--timeout", type=int, default=30, help="SSH 操作超时秒数 (默认: 30)")
args = parser.parse_args()
graph = px.Graph.from_specs(
[
px.TaskSpec(
"ssh_deploy",
fn=ssh_copy_id,
args=(args.hostname, args.username, args.password),
kwargs={"port": args.port, "keypath": args.keypath, "timeout": args.timeout},
)
]
)
px.run(graph, strategy="thread")
View File
+15
View File
@@ -0,0 +1,15 @@
"""清屏工具.
跨平台清屏工具, 支持终端和控制台清屏.
"""
from __future__ import annotations
import pyflowx as px
from pyflowx.tasks.system import clr
def main() -> None:
"""清屏工具主函数."""
graph = px.Graph.from_specs([clr()])
px.run(graph, strategy="thread")
+40
View File
@@ -0,0 +1,40 @@
"""进程终止工具.
跨平台进程终止工具, 支持按名称终止进程.
用法: taskkill proc_name [proc_name ...]
"""
from __future__ import annotations
import argparse
import pyflowx as px
from pyflowx.conditions import Constants
def main() -> None:
"""进程终止工具主函数."""
parser = argparse.ArgumentParser(
description="TaskKill - 进程终止工具",
usage="taskkill <process_name> [process_name ...]",
)
parser.add_argument(
"process_names",
type=str,
nargs="+",
help="进程名称 (如: chrome.exe python node)",
)
args = parser.parse_args()
if Constants.IS_WINDOWS:
cmd = ["taskkill", "/f", "/im"]
else:
cmd = ["pkill", "-f"]
graph = px.Graph.from_specs(
[
px.TaskSpec(f"kill_{proc_name}", cmd=[*cmd, f"{proc_name}*"], verbose=True)
for proc_name in args.process_names
],
)
px.run(graph, strategy="thread")
+21
View File
@@ -0,0 +1,21 @@
"""命令查找工具.
跨平台查找可执行命令路径, 类似 Unix 的 which 命令.
"""
from __future__ import annotations
import argparse
import pyflowx as px
from pyflowx.tasks.system import which
def main() -> None:
"""命令查找工具主函数."""
parser = argparse.ArgumentParser(description="Which - 命令查找工具")
parser.add_argument("commands", nargs="+", help="要查找的命令名称, 如: python ls ps gcc...")
args = parser.parse_args()
graph = px.Graph.from_specs([which(cmd) for cmd in args.commands])
px.run(graph, strategy="thread")
+7 -14
View File
@@ -10,16 +10,12 @@ from __future__ import annotations
import os
import subprocess
from typing import Any, cast
from rich.console import Console
from typing import Any, List, Union, cast
from .task import TaskSpec
__all__ = ["run_command"]
_cmd_console = Console()
def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
"""执行 ``spec.cmd`` 指定的命令(list / shell 字符串 / 可调用对象)。
@@ -43,9 +39,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
if callable(cmd) and not isinstance(cmd, (list, str)):
name = getattr(cmd, "__name__", "callable")
if verbose:
_cmd_console.print(f"[cyan]▸[/cyan] 执行可调用命令: [bold]{name}[/bold]")
print(f"[verbose] 执行可调用命令: {name}", flush=True)
if cwd is not None:
_cmd_console.print(f" [dim]工作目录: {cwd}[/dim]")
print(f"[verbose] 工作目录: {cwd}", flush=True)
try:
return cmd()
except Exception as e:
@@ -62,9 +58,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
label = "Shell 命令"
if verbose:
_cmd_console.print(f"[cyan]▸[/cyan] {verb}: [bold]{cmd_str}[/bold]")
print(f"[verbose] {verb}: {cmd_str}", flush=True)
if cwd is not None:
_cmd_console.print(f" [dim]工作目录: {cwd}[/dim]")
print(f"[verbose] 工作目录: {cwd}", flush=True)
# 合并环境变量
run_env: dict[str, str] | None = None
@@ -74,7 +70,7 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
try:
result = subprocess.run(
cast(str | list[str], cmd),
cast(Union[str, List[str]], cmd),
shell=not is_list,
cwd=cwd,
env=run_env,
@@ -91,12 +87,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
raise RuntimeError(f"{label}执行异常: {cmd_str}: {e}") from e
if verbose:
style = "green" if result.returncode == 0 else "red"
_cmd_console.print(f"[{style}]返回码: {result.returncode}[/{style}]")
print(f"[verbose] 返回码: {result.returncode}", flush=True)
if result.returncode == 0:
if not verbose and result.stdout:
print(result.stdout, end="", flush=True)
return None
err_msg = f"{label}执行失败: `{cmd_str}`, 返回码: {result.returncode}"
+1 -2
View File
@@ -16,9 +16,8 @@ import os
import shutil
import subprocess
import sys
from collections.abc import Callable
from pathlib import Path
from typing import Any
from typing import Any, Callable
from .task import Condition, Context
+1 -50
View File
@@ -16,9 +16,8 @@ DAG 库中泛滥的样板包装器。
from __future__ import annotations
import inspect
from collections.abc import Mapping
from functools import lru_cache
from typing import Any
from typing import Any, Mapping
from .errors import InjectionError
from .task import Context, TaskSpec
@@ -44,28 +43,6 @@ def _signature(fn: Any) -> inspect.Signature:
return inspect.signature(fn)
@lru_cache(maxsize=1024)
def _fn_no_dep_injection(fn: Any) -> tuple[tuple[str, ...] | None, str | None]:
"""预计算 fn 在无依赖/无静态参数时的注入计划。
返回 ``(context_params, error_param)``
- ``context_params``: Context 标注参数名元组;``None`` 表示存在必填参数需走慢路径。
- ``error_param``: 第一个无默认值且非 Context 标注的参数名(用于错误信息);``None`` 表示无此类参数。
"""
sig = inspect.signature(fn)
context_params: list[str] = []
for pname, param in sig.parameters.items():
if _is_context_annotation(param.annotation):
context_params.append(pname)
continue
if param.default is inspect.Parameter.empty and param.kind not in (
inspect.Parameter.VAR_POSITIONAL,
inspect.Parameter.VAR_KEYWORD,
):
return None, pname
return tuple(context_params), None
def _is_context_annotation(annotation: Any) -> bool:
"""判断参数标注是否为(或指向)``Context``。"""
if annotation is Context:
@@ -76,28 +53,6 @@ def _is_context_annotation(annotation: Any) -> bool:
return name in ("Context", "Mapping")
def _try_fast_path(spec: TaskSpec[Any]) -> tuple[tuple[Any, ...], dict[str, Any]] | None:
"""尝试快速路径:cmd 无参任务或 fn 无依赖任务。
返回 ``None`` 表示快速路径不适用,需走慢路径。
"""
# 快速路径 1cmd 任务(无 fn)的 effective_fn 是无参闭包。
if spec.fn is None and spec.cmd is not None and not spec.args and not spec.kwargs:
return (), {}
# 快速路径 2:fn 任务无依赖、无静态 args/kwargs 时,跳过 dep_context/collisions/leftover 构建。
if not spec.depends_on and not spec.soft_depends_on and not spec.args and not spec.kwargs:
fn = spec.effective_fn
context_params, error_param = _fn_no_dep_injection(fn)
if error_param is not None:
raise InjectionError(
spec.name,
f"parameter {error_param!r} has no dependency, static value, or default.",
)
assert context_params is not None # error_param is None ⟺ context_params 非 None
return (), {p: {} for p in context_params}
return None
def build_call_args(
spec: TaskSpec[Any],
context: Mapping[str, Any],
@@ -107,10 +62,6 @@ def build_call_args(
``context`` 必须已包含所有硬依赖与软依赖的结果(软依赖被跳过时由
执行器填入 :attr:`TaskSpec.defaults` 中的默认值)。
"""
fast = _try_fast_path(spec)
if fast is not None:
return fast
fn = spec.effective_fn
sig = _signature(fn)
params = sig.parameters
-324
View File
@@ -1,324 +0,0 @@
"""失败诊断:从 :class:`RunReport` 提取结构化的根因分析。
本模块不依赖 :class:`Graph`,仅通过 ``TaskResult.spec.depends_on`` 重建
依赖关系,因此可对反序列化后的报告进行诊断。
核心概念
--------
* **根因任务(root cause**FAILED 状态且其所有硬依赖都是 SUCCESS
(或无依赖)的任务。它是依赖链中"最先出问题"的环节。
* **依赖链(dependency chain)**:从根因到某个失败任务的路径。
* **相似失败聚类**:按 ``(异常类型, 消息前缀)`` 聚类,发现批量失败模式。
* **根因提示**:识别常见异常类型(FileNotFoundError/ImportError 等),
给出可操作建议。
"""
from __future__ import annotations
from collections import defaultdict
from dataclasses import dataclass, field
from .report import RunReport
from .task import TaskStatus
@dataclass(frozen=True)
class FailureCluster:
"""相似失败的聚类。
属性
----
pattern:
异常类型名 + 消息前缀(如 ``"ValueError: invalid input"``)。
tasks:
同类失败的任务名列表。
sample_error:
代表性错误消息(``repr(error)``)。
"""
pattern: str
tasks: list[str]
sample_error: str
@dataclass(frozen=True)
class DependencyChain:
"""失败任务的依赖链分析。
属性
----
failed_task:
失败的任务名。
root_cause:
依赖链中最先失败的任务(无失败依赖的失败任务)。
``failed_task`` 自身即为根因时与 ``failed_task`` 相同。
chain:
从 ``root_cause`` 到 ``failed_task`` 的任务名路径(含两端)。
broken_at:
链中第一个非 SUCCESS 的任务(通常等于 ``root_cause``)。
"""
failed_task: str
root_cause: str | None
chain: list[str]
broken_at: str | None
@dataclass(frozen=True)
class DiagnosticReport:
"""运行失败的诊断报告。
属性
----
failed_tasks:
所有 FAILED 状态的任务名。
skipped_due_to_failure:
因上游失败而 SKIPPED 的任务名(reason 含"失败""取消")。
root_causes:
根因任务列表(FAILED 且所有硬依赖 SUCCESS)。
dependency_chains:
每个失败任务的依赖链分析。
failure_clusters:
相似失败聚类。
hints:
针对根因异常的可操作建议。
"""
failed_tasks: list[str]
skipped_due_to_failure: list[str]
root_causes: list[str]
dependency_chains: list[DependencyChain]
failure_clusters: list[FailureCluster]
hints: list[str] = field(default_factory=list)
def describe(self) -> str:
"""人类可读的多行诊断报告。"""
lines: list[str] = ["=== 失败诊断报告 ==="]
# 根因
if self.root_causes:
lines.append(f"\n根因任务 ({len(self.root_causes)}):")
for rc in self.root_causes:
lines.append(f" - {rc}")
else:
lines.append("\n根因任务: 无法识别(可能所有失败任务的依赖均未成功)")
# 依赖链
if self.dependency_chains:
lines.append(f"\n依赖链 ({len(self.dependency_chains)}):")
for dc in self.dependency_chains:
chain_str = " -> ".join(dc.chain) if dc.chain else dc.failed_task
lines.append(f" {dc.failed_task}: {chain_str}")
# 相似失败聚类
if self.failure_clusters:
lines.append(f"\n相似失败聚类 ({len(self.failure_clusters)}):")
for cluster in self.failure_clusters:
tasks_str = ", ".join(cluster.tasks)
lines.append(f" [{cluster.pattern}] ({len(cluster.tasks)} 任务): {tasks_str}")
# 跳过的任务
if self.skipped_due_to_failure:
lines.append(f"\n因失败而跳过的任务 ({len(self.skipped_due_to_failure)}):")
for name in self.skipped_due_to_failure:
lines.append(f" - {name}")
# 提示
if self.hints:
lines.append("\n建议:")
for hint in self.hints:
lines.append(f" * {hint}")
return "\n".join(lines)
# ---------------------------------------------------------------------- #
# 内部辅助
# ---------------------------------------------------------------------- #
_HINT_MAP: dict[str, str] = {
"FileNotFoundError": "文件不存在,检查路径配置与工作目录",
"ImportError": "模块导入失败,检查依赖是否安装(pip install / uv add",
"ModuleNotFoundError": "模块不存在,检查依赖安装或 PYTHONPATH",
"TimeoutError": "任务超时,考虑增加 timeout 配置或优化任务性能",
"TaskTimeoutError": "任务超时,考虑增加 timeout 配置或优化任务性能",
"ConnectionError": "网络连接失败,检查目标地址可达性与防火墙设置",
"PermissionError": "权限不足,检查文件/目录权限或以管理员身份运行",
"KeyError": "键缺失,检查上下文注入参数名是否匹配任务签名",
"InjectionError": "上下文注入失败,检查任务签名参数与上游任务名",
"StorageError": "状态后端错误,检查存储路径权限与磁盘空间",
"CycleError": "依赖图存在环,检查 depends_on 是否形成循环",
"MemoryError": "内存不足,考虑分批处理或减少并发数",
"OSError": "系统错误,检查文件描述符/磁盘空间/权限",
}
def _classify_error(error: BaseException | None) -> tuple[str, str]:
"""返回 ``(异常类型名, 消息前缀)`` 用于聚类。"""
if error is None:
return ("Unknown", "")
type_name = type(error).__name__
msg = str(error)
prefix = msg[:50] if len(msg) > 50 else msg
return (type_name, prefix)
def _find_root_cause(
failed_task: str,
report: RunReport,
) -> tuple[str | None, list[str]]:
"""回溯 ``failed_task`` 的依赖链,返回 (根因任务, 路径)。
路径从根因到 ``failed_task``(含两端)。根因为 FAILED 且所有硬依赖
非 FAILED 的任务;若 ``failed_task`` 自身满足条件,则为其自身。
"""
# BFS 反向遍历,找到链中"最深处"的根因
visited: set[str] = set()
# 用 DFS 记录路径:path 始终以当前 task 开头,向根因方向延伸
def _trace(task: str, path: list[str]) -> tuple[str | None, list[str]]:
if task in visited: # 防止循环依赖(diagnose 不依赖 Graph 校验,results 可能有环)
return (None, path)
visited.add(task)
result = report.results[task] # _trace 只接收 failed_tasks 或 failed_deps,均保证在 results 中
# 检查此任务是否为根因:FAILED 且所有硬依赖非 FAILED
deps = result.spec.depends_on
failed_deps = [d for d in deps if d in report.results and report.results[d].status == TaskStatus.FAILED]
if result.status == TaskStatus.FAILED and not failed_deps:
return (task, path)
# 递归向上找
for dep in failed_deps:
root, new_path = _trace(dep, [dep, *path])
if root is not None:
return (root, new_path)
# 递归未找到明确根因时,当前任务作为断点返回
return (task, path)
return _trace(failed_task, [failed_task])
def _generate_hints(report: RunReport, root_causes: list[str]) -> list[str]:
"""针对根因异常生成可操作建议。"""
hints: list[str] = []
seen_types: set[str] = set()
for rc_name in root_causes:
result = report.results.get(rc_name)
if result is None or result.error is None:
continue
type_name = type(result.error).__name__
if type_name in seen_types:
continue
seen_types.add(type_name)
# 精确匹配
if type_name in _HINT_MAP:
hints.append(f"{rc_name}: {_HINT_MAP[type_name]}")
continue
# 父类匹配
for parent in type(result.error).__mro__:
parent_name = parent.__name__
if parent_name in _HINT_MAP:
hints.append(f"{rc_name}: {_HINT_MAP[parent_name]}")
break
return hints
# ---------------------------------------------------------------------- #
# 公共 API
# ---------------------------------------------------------------------- #
def diagnose(report: RunReport) -> DiagnosticReport | None:
"""生成失败诊断报告。
运行成功(``report.success`` 为 ``True``)时返回 ``None``
失败时返回 :class:`DiagnosticReport`,包含根因分析、依赖链回溯、
相似失败聚类与可操作建议。
本函数仅依赖 ``report.results`` 中的 ``spec.depends_on`` 重建依赖
关系,不依赖 :class:`Graph` 对象,因此可对反序列化后的报告进行诊断。
"""
if report.success:
return None
failed_tasks = report.failed_tasks()
if not failed_tasks:
# success=False 但无 FAILED 任务(可能是用户中断等情况)
return DiagnosticReport(
failed_tasks=[],
skipped_due_to_failure=report.skipped_tasks(),
root_causes=[],
dependency_chains=[],
failure_clusters=[],
hints=["运行未成功但无失败任务,可能是被取消或中断"],
)
# 识别根因:FAILED 且所有硬依赖非 FAILED
root_causes: list[str] = []
for name in failed_tasks:
result = report.results[name]
failed_deps = [
d for d in result.spec.depends_on if d in report.results and report.results[d].status == TaskStatus.FAILED
]
if not failed_deps:
root_causes.append(name)
# 依赖链回溯
chains: list[DependencyChain] = []
for failed in failed_tasks:
root, path = _find_root_cause(failed, report)
# broken_at 是路径中第一个非 SUCCESS 的任务
broken_at: str | None = None
for task in path:
r = report.results.get(task)
if r is not None and r.status != TaskStatus.SUCCESS:
broken_at = task
break
chains.append(
DependencyChain(
failed_task=failed,
root_cause=root,
chain=path,
broken_at=broken_at,
)
)
# 相似失败聚类
clusters_map: dict[tuple[str, str], list[str]] = defaultdict(list)
sample_errors: dict[tuple[str, str], str] = {}
for name in failed_tasks:
result = report.results[name]
key = _classify_error(result.error)
clusters_map[key].append(name)
if key not in sample_errors:
sample_errors[key] = repr(result.error) if result.error else ""
clusters: list[FailureCluster] = [
FailureCluster(
pattern=f"{type_name}: {prefix}" if prefix else type_name,
tasks=sorted(tasks),
sample_error=sample_errors.get((type_name, prefix), ""),
)
for (type_name, prefix), tasks in sorted(clusters_map.items(), key=lambda x: -len(x[1]))
]
# 因失败而跳过的任务(reason 包含"失败"或"取消"或"skip"
skipped_due_to_failure: list[str] = []
for name, result in report.results.items():
if result.status == TaskStatus.SKIPPED:
reason = (result.reason or "").lower()
if any(kw in reason for kw in ("失败", "取消", "fail", "cancel", "abort")):
skipped_due_to_failure.append(name)
# 根因提示
hints = _generate_hints(report, root_causes)
return DiagnosticReport(
failed_tasks=failed_tasks,
skipped_due_to_failure=skipped_due_to_failure,
root_causes=root_causes,
dependency_chains=chains,
failure_clusters=clusters,
hints=hints,
)
+2 -9
View File
@@ -6,11 +6,7 @@
from __future__ import annotations
from collections.abc import Iterable
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .report import RunReport
from typing import Any, Iterable
class PyFlowXError(Exception):
@@ -51,8 +47,7 @@ class TaskFailedError(PyFlowXError):
"""任务耗尽所有重试后仍失败时抛出。
原始异常保留在 :attr:`__cause__` 上,同时通过 :attr:`cause` 暴露,
便于用户代码访问。``report`` 字段(可选)携带失败时的 :class:`RunReport`
便于调用方生成诊断报告。
便于用户代码访问。
"""
def __init__(
@@ -61,7 +56,6 @@ class TaskFailedError(PyFlowXError):
cause: BaseException,
attempts: int,
layer: int | None = None,
report: RunReport | None = None,
) -> None:
location = f" (layer {layer})" if layer is not None else ""
super().__init__(f"Task '{task}' failed after {attempts} attempt(s){location}: {cause}")
@@ -69,7 +63,6 @@ class TaskFailedError(PyFlowXError):
self.cause = cause
self.attempts = attempts
self.layer = layer
self.report = report
class TaskTimeoutError(PyFlowXError):
View File
+56
View File
@@ -0,0 +1,56 @@
"""Example 3: async aggregation with static args and Context injection.
Shows:
* async task functions executed with strategy="async".
* static positional args (TaskSpec.args) for parameterised tasks.
* Context annotation to receive the full upstream result mapping.
* on_event callback for real-time progress.
"""
from __future__ import annotations
import asyncio
from typing import Any
import pyflowx as px
async def fetch_user(uid: int) -> dict[str, Any]:
await asyncio.sleep(0.2)
return {"id": uid, "name": f"User{uid}"}
async def fetch_posts(uid: int) -> list[int]:
await asyncio.sleep(0.2)
return [uid, uid + 1]
# Context annotation → receives the full mapping of upstream results.
def aggregate(ctx: px.Context) -> dict[str, Any]:
return dict(ctx)
def main() -> None:
graph = px.Graph.from_specs([
# Static positional args parameterise the same function twice.
px.TaskSpec("fetch_user", fetch_user, args=(1,)),
px.TaskSpec("fetch_posts", fetch_posts, args=(1,)),
px.TaskSpec("aggregate", aggregate, depends_on=("fetch_user", "fetch_posts")),
])
print("=== Dry run ===")
_ = px.run(graph, strategy="async", dry_run=True)
events: list[px.TaskEvent] = []
print("\n=== Async execution ===")
report = px.run(graph, strategy="async", on_event=events.append)
for ev in events:
print(f" event: {ev.task} -> {ev.status.value}")
print(f"\naggregate = {report['aggregate']}")
print(report.describe())
if __name__ == "__main__":
main()
+77
View File
@@ -0,0 +1,77 @@
"""Example 1: ETL pipeline (sequential strategy).
Demonstrates the core PyFlowX workflow:
* Define tasks as plain functions.
* Declare the DAG with a list of TaskSpec.
* Parameter names == dependency names → automatic context injection,
no wrappers needed (contrast with flowweaver's get_task_result boilerplate).
* dry_run to preview, then execute and read typed results from RunReport.
"""
from __future__ import annotations
from typing import Any
import pyflowx as px
# --- task functions: pure, testable, no framework coupling ------------- #
def extract_customers() -> list[dict[str, Any]]:
return [
{"id": "C001", "name": "Alice"},
{"id": "C002", "name": "Bob"},
]
def extract_orders() -> list[dict[str, Any]]:
return [
{"id": "O001", "customer_id": "C001", "amount": 150.0},
{"id": "O002", "customer_id": "C002", "amount": 200.5},
]
# Parameter names match dependency names → automatic injection.
def transform(
extract_customers: list[dict[str, Any]],
extract_orders: list[dict[str, Any]],
) -> list[dict[str, Any]]:
cmap = {c["id"]: c for c in extract_customers}
return [{**o, "customer_name": cmap[o["customer_id"]]["name"]} for o in extract_orders if o["customer_id"] in cmap]
def load(transform: list[dict[str, Any]]) -> int:
print(f" loaded {len(transform)} records")
return len(transform)
def main() -> None:
graph = px.Graph.from_specs([
px.TaskSpec("extract_customers", extract_customers, tags=("extract",)),
px.TaskSpec("extract_orders", extract_orders, tags=("extract",)),
px.TaskSpec(
"transform",
transform,
depends_on=("extract_customers", "extract_orders"),
tags=("transform",),
),
px.TaskSpec(
"load", load, depends_on=("transform",), retry=px.RetryPolicy(max_attempts=1, delay=1.0), tags=("load",)
),
])
print("=== Execution plan ===")
print(graph.describe())
print("\n=== Dry run (no execution) ===")
_ = px.run(graph, strategy="sequential", dry_run=True)
print("\n=== Sequential execution ===")
report = px.run(graph, strategy="sequential")
print(report.describe())
print(f"\nload result = {report['load']}")
print(f"summary = {report.summary()}")
if __name__ == "__main__":
main()
+57
View File
@@ -0,0 +1,57 @@
"""Example 2: parallel execution (thread strategy).
Same DAG run with sequential vs. thread strategy to show layer-internal
parallelism. Tasks within a layer run concurrently; layers are barriers.
Layer 1: [fetch_a, fetch_b] (parallel)
Layer 2: [merge] (waits for both)
"""
from __future__ import annotations
import time
import pyflowx as px
def fetch_a() -> str:
time.sleep(0.5)
return "a"
def fetch_b() -> str:
time.sleep(0.5)
return "b"
def merge(fetch_a: str, fetch_b: str) -> str:
return fetch_a + fetch_b
def main() -> None:
graph = px.Graph.from_specs([
px.TaskSpec("fetch_a", fetch_a),
px.TaskSpec("fetch_b", fetch_b),
px.TaskSpec("merge", merge, depends_on=("fetch_a", "fetch_b")),
])
print("=== Mermaid diagram ===")
print(graph.to_mermaid("LR"))
print("\n=== Sequential (expect ~1.0s) ===")
start = time.time()
report_seq = px.run(graph, strategy="sequential")
t_seq = time.time() - start
print(f" result={report_seq['merge']} time={t_seq:.2f}s")
print("\n=== Threaded (expect ~0.5s) ===")
start = time.time()
report_thr = px.run(graph, strategy="thread", max_workers=2)
t_thr = time.time() - start
print(f" result={report_thr['merge']} time={t_thr:.2f}s")
print(f"\nspeedup = {t_seq / t_thr:.2f}x")
if __name__ == "__main__":
main()
+65 -547
View File
@@ -46,32 +46,20 @@ import concurrent.futures
import contextlib
import inspect
import logging
import queue
import threading
import time
from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping
from dataclasses import replace as dc_replace
from datetime import datetime
from pathlib import Path
from typing import Any, Literal, cast
from typing import Any, Awaitable, Callable, Literal, Mapping, cast
from rich.console import Console
from .cancellation import CancelToken, _is_cancelled
from .context import build_call_args, describe_injection
from .errors import TaskFailedError, TaskTimeoutError
from .graph import Graph
from .notification import Notifier
from .progress import ProgressCallback, RichProgressMonitor
from .report import RunReport
from .storage import StateBackend, resolve_backend
from .task import TaskEvent, TaskHooks, TaskResult, TaskSpec, TaskStatus
logger = logging.getLogger(__name__)
# verbose 模式输出用的 Console (不抢占用户 stdout, 仅格式化)
_verbose_console = Console()
# 进程池复用:同一次 run() 内的 process 任务共享一个 ProcessPoolExecutor。
# 模块级缓存避免每次任务都创建/销毁进程池的开销。
# run() 结束后通过 _shutdown_process_pool() 关闭(shutdown(wait=False) +
@@ -227,11 +215,7 @@ def _apply_cached(
result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中")
report.results[name] = result
_emit(on_event, result)
logger.info(
"task %r skipped (cached)",
name,
extra={"run_id": report.run_id, "task_name": name, "status": TaskStatus.SKIPPED.value},
)
logger.info("task %r skipped (cached)", name)
return True
@@ -291,17 +275,7 @@ def _prepare_for_execution(
reason=skip_reason,
)
_emit(on_event, result)
logger.info(
"task %r skipped (%s)",
spec.name,
skip_reason,
extra={
"run_id": report.run_id if report is not None else "-",
"task_name": spec.name,
"status": TaskStatus.SKIPPED.value,
"reason": skip_reason,
},
)
logger.info("task %r skipped (%s)", spec.name, skip_reason)
return result
@@ -323,30 +297,15 @@ def _finalize_failure(
layer_idx: int | None,
on_event: EventCallback | None,
continue_on_error: bool,
name: str,
report: RunReport | None,
) -> None:
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。
失败结果在抛出前写入 ``report.results[name]``,使流式 API
(:func:`run_iter`) 能在 re-raise 前yield 该结果。
"""
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。"""
result.status = TaskStatus.FAILED
result.finished_at = datetime.now()
if report is not None:
report.results[name] = result
_emit(on_event, result)
if continue_on_error:
logger.warning(
"task %r failed but continue_on_error=True; continuing.",
result.spec.name,
extra={
"run_id": report.run_id if report is not None else "-",
"task_name": result.spec.name,
"status": TaskStatus.FAILED.value,
"attempts": result.attempts,
"error_type": type(result.error).__name__ if result.error else "Unknown",
},
)
return
raise TaskFailedError(
@@ -354,7 +313,6 @@ def _finalize_failure(
cause=result.error if result.error is not None else RuntimeError("unknown"),
attempts=result.attempts,
layer=layer_idx,
report=report,
)
@@ -364,8 +322,6 @@ def _handle_failure(
exc: BaseException,
layer_idx: int | None,
on_event: EventCallback | None,
name: str,
report: RunReport | None,
) -> bool:
"""统一处理失败:超时转换、重试决策、finalize。
@@ -375,7 +331,6 @@ def _handle_failure(
``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。
"""
# asyncio.TimeoutError → TaskTimeoutError(统一异常类型)
run_id = report.run_id if report is not None else "-"
if isinstance(exc, asyncio.TimeoutError):
exc = TaskTimeoutError(spec.name, spec.timeout or 0.0)
logger.warning(
@@ -383,13 +338,6 @@ def _handle_failure(
spec.name,
result.attempts,
spec.retry.max_attempts,
extra={
"run_id": run_id,
"task_name": spec.name,
"status": TaskStatus.FAILED.value,
"attempts": result.attempts,
"error_type": "TaskTimeoutError",
},
)
else:
logger.warning(
@@ -398,19 +346,12 @@ def _handle_failure(
result.attempts,
spec.retry.max_attempts,
exc,
extra={
"run_id": run_id,
"task_name": spec.name,
"status": TaskStatus.FAILED.value,
"attempts": result.attempts,
"error_type": type(exc).__name__,
},
)
result.error = exc
if _should_retry(spec, result.attempts, exc):
return False
_run_hooks(spec.hooks, "on_failure", spec, exc)
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error, name, report)
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error)
return True
@@ -447,7 +388,7 @@ class SyncTaskRunner:
_mark_success(spec, result, value)
return result
except Exception as exc:
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
if _handle_failure(spec, result, exc, layer_idx, on_event):
return result
wait = spec.retry.wait_seconds(result.attempts)
if wait > 0:
@@ -474,7 +415,7 @@ class AsyncTaskRunner:
result: TaskResult[Any] = TaskResult(spec=spec)
result.started_at = datetime.now()
args, kwargs = build_call_args(spec, context)
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
_run_hooks(spec.hooks, "pre_run", spec)
_emit_running(on_event, spec)
@@ -486,7 +427,7 @@ class AsyncTaskRunner:
_mark_success(spec, result, value)
return result
except Exception as exc:
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
if _handle_failure(spec, result, exc, layer_idx, on_event):
return result
wait = spec.retry.wait_seconds(result.attempts)
if wait > 0:
@@ -572,10 +513,6 @@ def _filter_and_sort(
for name in layer:
spec = graph.resolved_spec(name)
specs[name] = spec
# 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
context[name] = report.results[name].value
continue
if not _apply_cached(name, spec, context, report, backend, on_event):
to_run.append(name)
return _sort_by_priority(to_run, specs)
@@ -662,7 +599,7 @@ class ThreadedLayerRunner:
backend: StateBackend,
layer_idx: int,
on_event: EventCallback | None,
pool: concurrent.futures.ThreadPoolExecutor,
max_workers: int,
concurrency_limits: Mapping[str, int],
) -> None:
to_run = _filter_and_sort(layer, graph, context, report, backend, on_event)
@@ -684,18 +621,21 @@ class ThreadedLayerRunner:
if sem is not None:
sem.release()
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
pool.submit(_run_threaded_task, name): name for name in to_run
}
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
try:
for fut in concurrent.futures.as_completed(future_to_name):
name = future_to_name[fut]
completed[name] = fut.result()
finally:
with lock:
for name, (task_ctx, result) in completed.items():
_store_result(name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event)
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
pool.submit(_run_threaded_task, name): name for name in to_run
}
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
try:
for fut in concurrent.futures.as_completed(future_to_name):
name = future_to_name[fut]
completed[name] = fut.result()
finally:
with lock:
for name, (task_ctx, result) in completed.items():
_store_result(
name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event
)
class AsyncLayerRunner:
@@ -726,30 +666,10 @@ class AsyncLayerRunner:
return task_ctx, result
results = await asyncio.gather(*[_run_async_task(name) for name in to_run])
for name, (task_ctx, result) in zip(to_run, results, strict=True):
for name, (task_ctx, result) in zip(to_run, results):
_store_result(name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event)
def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
"""从 ``dynamic`` 任务的返回值中提取新 specs。
仅当 ``spec.dynamic`` 为 ``True`` 且任务成功时提取。返回值可以是单个
:class:`TaskSpec` 或其列表。提取后,原结果的 ``value`` 被替换为生成的
任务名列表(便于查询),specs 列表返回给调用方注册调度。
"""
if not spec.dynamic or result.status != TaskStatus.SUCCESS:
return []
value = result.value
spawned: list[TaskSpec[Any]] = []
if isinstance(value, TaskSpec):
spawned = [value]
elif isinstance(value, list) and value and all(isinstance(x, TaskSpec) for x in value):
spawned = value
if not spawned:
return []
return spawned
class DependencyRunner:
"""依赖驱动调度:任务在硬/软依赖完成后立即启动,无层屏障。
@@ -768,17 +688,13 @@ class DependencyRunner:
backend: StateBackend,
on_event: EventCallback | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None:
all_names = list(graph.all_specs().keys())
semaphores = _build_semaphores(all_names, graph, asyncio.Semaphore, concurrency_limits)
futures: dict[str, asyncio.Task[TaskResult[Any]]] = {}
futures: dict[str, asyncio.Future[TaskResult[Any]]] = {}
async def _run_task(name: str) -> TaskResult[Any]:
spec = graph.resolved_spec(name)
# 检查点恢复:已在 report 中的 SUCCESS 任务直接返回
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
return report.results[name]
# 等待所有硬依赖完成
for dep in spec.depends_on:
if dep in futures:
@@ -788,244 +704,48 @@ class DependencyRunner:
if dep in futures:
await futures[dep]
# 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED
if _is_cancelled(cancel_event) and name not in report.results:
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus.SKIPPED,
finished_at=datetime.now(),
reason="执行被取消",
)
report.results[name] = result
_emit(on_event, result)
return result
task_ctx = _build_context(spec, context, report)
if _apply_cached(name, spec, context, report, backend, on_event):
return report.results[name]
sem = _get_sem(semaphores, spec)
result = await AsyncTaskRunner.run(spec, task_ctx, None, on_event, report, sem)
# 动态任务生成:提取 specs,替换 result.value 为生成任务名列表
spawned = _extract_dynamic_specs(result, spec)
if spawned:
spawned_names = [s.name for s in spawned]
result = dc_replace(result, value=spawned_names)
# 重新存储已修改的 result
context[name] = result.value
report.results[name] = result
_emit(on_event, result)
else:
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
# 注册并调度动态生成的 specs
for raw_spec in spawned:
# 确保新 spec 依赖生成方(保证顺序与上下文可见性)
final_spec = raw_spec
if name not in raw_spec.depends_on and name not in raw_spec.soft_depends_on:
final_spec = dc_replace(raw_spec, depends_on=(*raw_spec.depends_on, name))
graph._register_single(final_spec)
# 为新 spec 的 concurrency_key 创建信号量
if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores:
limit = concurrency_limits.get(final_spec.concurrency_key, 1)
semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit)
futures[final_spec.name] = loop.create_task(_run_task(final_spec.name))
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
return result
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
for name in all_names:
futures[name] = loop.create_task(_run_task(name))
# 循环等待所有 futures(含动态生成的),直到全部完成。
# asyncio.wait 不像 gather 自动传播异常,需手动检查并 re-raise,
# 匹配 gather 的 fail-fast 行为(首个异常即取消剩余任务并抛出)。
while futures:
await asyncio.wait(futures.values(), return_when=asyncio.FIRST_COMPLETED)
completed_names = [n for n, f in futures.items() if f.done()]
for done_name in completed_names:
f = futures[done_name]
exc = f.exception()
if exc is not None:
for remaining in futures.values():
if not remaining.done():
remaining.cancel()
raise exc
del futures[done_name]
await asyncio.gather(*futures.values())
# ---------------------------------------------------------------------- #
# 公共 API
# ---------------------------------------------------------------------- #
def _compose_callbacks(*callbacks: EventCallback | None) -> EventCallback | None:
"""组合多个回调为一个,按顺序调用;全为 None 时返回 None。"""
fns = [c for c in callbacks if c is not None]
if not fns:
return None
def _composed(event: TaskEvent) -> None:
for fn in fns:
fn(event)
return _composed
def _finish_monitoring(monitor: ProgressCallback | None, notifier_list: list[Notifier]) -> None:
"""执行结束时清理监控器与通知器资源。"""
if monitor is not None:
monitor.finish()
for n in notifier_list:
n.close()
def _resolve_progress(
progress: bool | ProgressCallback,
verbose: bool,
on_event: EventCallback | None,
notifiers: Notifier | list[Notifier] | None,
) -> tuple[ProgressCallback | None, list[Notifier], EventCallback | None]:
"""解析 ``progress``/``notifiers`` 参数,返回 (monitor, notifier_list, effective_callback)。
将 progress/verbose/on_event/notifiers 的回调组合逻辑从 :func:`run` 中提取出来,
控制 ``run`` 的分支数在 ruff PLR0912 阈值内。
"""
monitor: ProgressCallback | None = None
if progress is True:
monitor = RichProgressMonitor()
elif progress is not False:
monitor = progress
notifier_list: list[Notifier] = []
if notifiers is not None:
notifier_list = notifiers if isinstance(notifiers, list) else [notifiers]
callbacks: list[EventCallback] = []
if verbose:
callbacks.append(_make_verbose_callback(None))
if monitor is not None:
callbacks.append(monitor.on_event)
if on_event is not None:
callbacks.append(on_event)
for n in notifier_list:
callbacks.append(n.notify)
effective_callback = _compose_callbacks(*callbacks) if callbacks else None
return monitor, notifier_list, effective_callback
def _apply_subgraph_filter(
graph: Graph,
only: Iterable[str] | None,
tags: Iterable[str] | None,
) -> Graph:
"""根据 ``only``/``tags`` 过滤图,返回包含传递依赖的子图。
``only`` 与 ``tags`` 取并集:匹配任一条件的任务及其所有传递依赖
(硬依赖 + 软依赖)都会被包含在子图中,使子图可独立执行。
"""
names: set[str] = set()
if only is not None:
names.update(only)
if tags is not None:
tag_set = set(tags)
for name, spec in graph.all_specs().items():
if tag_set & set(spec.tags):
names.add(name)
if not names:
return Graph(defaults=graph.defaults)
return graph.subgraph_with_deps(names)
def _dispatch_strategy(
strategy: Strategy,
graph: Graph,
context: dict[str, Any],
report: RunReport,
backend: StateBackend,
effective_callback: EventCallback | None,
max_workers: int | None,
limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None,
) -> None:
"""按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。
``dynamic=True`` 任务仅在 ``dependency`` 策略下支持(需运行时扩展图)。
"""
has_dynamic = any(s.dynamic for s in graph.all_specs().values())
if has_dynamic and strategy != "dependency":
raise ValueError(
f"dynamic=True 任务仅支持 strategy='dependency',当前策略: {strategy!r}"
"动态任务生成需要运行时扩展图,层屏障模型(sequential/thread/async)不支持。"
)
if strategy == "sequential":
layers = graph.layers()
_drive_sequential(graph, layers, context, report, backend, effective_callback, cancel_event)
elif strategy == "thread":
layers = graph.layers()
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits, cancel_event)
elif strategy == "async":
layers = graph.layers()
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits, cancel_event))
elif strategy == "dependency":
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits, cancel_event))
else: # pragma: no cover - Strategy Literal 已穷尽所有取值
raise ValueError(f"Unknown strategy: {strategy!r}")
def _make_verbose_callback(on_event: EventCallback | None) -> EventCallback:
"""包装 on_event 回调, 在 verbose 模式下用 rich 打印任务生命周期。"""
"""包装 on_event 回调, 在 verbose 模式下打印任务生命周期。"""
def _verbose_callback(event: TaskEvent) -> None:
dur = f" ({event.duration:.3f}s)" if event.duration is not None else ""
if event.status == TaskStatus.RUNNING:
_verbose_console.print(f"[cyan]▸[/cyan] [bold]{event.task!r}[/bold] 开始执行...")
print(f"[verbose] 任务 {event.task!r} 开始执行...", flush=True)
elif event.status == TaskStatus.SUCCESS:
_verbose_console.print(f"[green]✓[/green] [bold]{event.task!r}[/bold] 成功[dim]{dur}[/dim]")
print(f"[verbose] 任务 {event.task!r} 成功{dur}", flush=True)
elif event.status == TaskStatus.FAILED:
err = f": {event.error}" if event.error else ""
_verbose_console.print(
f"[red]✗[/red] [bold]{event.task!r}[/bold] 失败[dim]{dur} (尝试 {event.attempts} 次)[/dim][red]{err}[/red]"
print(
f"[verbose] 任务 {event.task!r} 失败{dur} (尝试 {event.attempts} 次){err}",
flush=True,
)
elif event.status == TaskStatus.SKIPPED:
reason = f" ({event.reason})" if event.reason else ""
_verbose_console.print(f"[yellow]○[/yellow] [bold]{event.task!r}[/bold] 跳过[dim]{reason}[/dim]")
print(f"[verbose] 任务 {event.task!r} 跳过{reason}", flush=True)
if on_event is not None:
on_event(event)
return _verbose_callback
def _load_resume_report(source: RunReport | str | Path) -> RunReport:
"""从 RunReport 或 JSON 文件路径加载前次运行报告。"""
if isinstance(source, RunReport):
return source
return RunReport.from_json(Path(source).read_text())
def _apply_resume(
resume_from: RunReport | str | Path | None,
graph: Graph,
report: RunReport,
context: dict[str, Any],
) -> None:
"""从前次报告中预填充 SUCCESS 任务的 result 和 context。
执行器在遇到已存在 result 时自动跳过(_filter_and_sort / _run_task)。
"""
if resume_from is None:
return
prev_report = _load_resume_report(resume_from)
all_specs = graph.all_specs()
restored = 0
for name, result in prev_report.results.items():
if result.status == TaskStatus.SUCCESS and name in all_specs:
report.results[name] = dc_replace(result, reason="从检查点恢复")
context[name] = result.value
restored += 1
if restored:
logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored})
def run(
graph: Graph,
strategy: Strategy = "dependency",
@@ -1036,13 +756,6 @@ def run(
on_event: EventCallback | None = None,
state: StateBackend | None = None,
concurrency_limits: Mapping[str, int] | None = None,
progress: bool | ProgressCallback = False,
notifiers: Notifier | list[Notifier] | None = None,
cancel_event: threading.Event | CancelToken | None = None,
only: Iterable[str] | None = None,
tags: Iterable[str] | None = None,
resume_from: RunReport | str | Path | None = None,
_report: RunReport | None = None,
) -> RunReport:
"""执行图并返回 :class:`RunReport`。
@@ -1066,28 +779,6 @@ def run(
concurrency_limits:
``{concurrency_key: max_concurrent}`` 映射。具有相同
``concurrency_key`` 的任务共享信号量,限制同时运行实例数。
progress:
进度监控:``False``(默认)无监控;``True`` 使用
:class:`~pyflowx.progress.RichProgressMonitor` 显示终端进度条;
传入 :class:`~pyflowx.progress.ProgressCallback` 实例使用自定义监控器。
与 ``on_event`` 独立且可共存。
notifiers:
任务通知器:``None``(默认)无通知;传入 :class:`~pyflowx.notification.Notifier`
实例或列表。通知器的 ``notify`` 方法在每次任务事件时调用,``close`` 在
执行结束时调用。支持飞书/钉钉/企业微信 webhook 与 Python 回调。
cancel_event:
取消信号:传入 :class:`~pyflowx.cancellation.CancelToken` 或
:class:`threading.Event`,设置后待运行任务被标记为 SKIPPED,
运行中任务等待完成后返回部分结果。``KeyboardInterrupt`` 也会
触发相同的优雅取消流程。
only:
只运行指定任务名及其传递依赖。与 ``tags`` 取并集。
tags:
只运行匹配任意标签的任务及其传递依赖。与 ``only`` 取并集。
resume_from:
从之前的运行报告恢复:传入 :class:`RunReport` 或 JSON 文件路径。
前次运行中 SUCCESS 的任务会被跳过(标记 reason="从检查点恢复"),
FAILED/SKIPPED 的任务及其下游会被重新执行。
抛出
----
@@ -1101,190 +792,52 @@ def run(
_print_dry_run(graph, layers)
return RunReport(success=True)
# 子图过滤:only/tags 选择任务子集及其传递依赖
if only is not None or tags is not None:
graph = _apply_subgraph_filter(graph, only, tags)
# verbose 模式下, 把所有 spec 的 verbose 标记设为 True,
# 使 execute_command 打印执行命令与返回码 (任务生命周期由 callback 打印)
if verbose:
from dataclasses import replace
graph = Graph.from_specs([replace(s, verbose=True) if not s.verbose else s for s in graph.all_specs().values()])
# 入口统一校验一次:所有策略共用,避免 layers() / dependency 路径
# 各自重复调用 validate()。
graph.validate()
# 组合回调链:verbose 打印 + 进度监控 + 用户回调 + 通知器
monitor, notifier_list, effective_callback = _resolve_progress(progress, verbose, on_event, notifiers)
effective_callback: EventCallback | None = _make_verbose_callback(on_event) if verbose else on_event
backend = resolve_backend(state)
report = _report if _report is not None else RunReport()
report = RunReport()
context: dict[str, Any] = {}
limits = concurrency_limits or {}
# 检查点恢复:预填充 SUCCESS 任务的 result 和 context,执行器自动跳过。
_apply_resume(resume_from, graph, report, context)
logger.info(
"运行开始: run_id=%s strategy=%s tasks=%d",
report.run_id,
strategy,
len(graph),
extra={"run_id": report.run_id, "strategy": strategy, "total_tasks": len(graph)},
)
if monitor is not None:
monitor.start(len(graph))
# 内部取消令牌:KeyboardInterrupt 时设置,驱动各策略停止启动新任务。
internal_cancel = CancelToken()
effective_cancel: CancelToken | threading.Event | None = cancel_event
# backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend);
# MemoryBackend 为 no-op。即使中途抛出 TaskFailedErrorbatch 退出时
# 仍会 flush 一次,保留已成功任务的结果以便断点续跑。
try:
with backend.batch():
try:
_dispatch_strategy(
strategy, graph, context, report, backend, effective_callback, max_workers, limits, effective_cancel
)
except TaskFailedError:
report.success = False
raise
except KeyboardInterrupt:
# Ctrl+C:设置内部取消信号,驱动各策略停止启动新任务。
internal_cancel.cancel("KeyboardInterrupt")
report.success = False
logger.warning(
"执行被用户中断(KeyboardInterrupt),已取消待运行任务",
extra={"run_id": report.run_id, "status": "cancelled"},
)
finally:
# 关闭进程池:通知管理线程退出 + kill 工作进程,避免
# threading._shutdown 等待管理线程 join 工作进程导致 ~7s 阻塞。
_shutdown_process_pool()
finally:
_finish_monitoring(monitor, notifier_list)
# 取消(外部或 KeyboardInterrupt)后标记未完成任务为 SKIPPED 并修正报告
if _is_cancelled(effective_cancel) or internal_cancel.is_cancelled:
_mark_remaining_skipped(graph, report, effective_callback)
report.success = False
logger.info(
"运行结束: run_id=%s success=%s tasks=%d",
report.run_id,
report.success,
len(report.results),
extra={
"run_id": report.run_id,
"success": report.success,
"total_tasks": len(report.results),
},
)
return report
def run_iter(
graph: Graph,
strategy: Strategy = "dependency",
*,
max_workers: int | None = None,
verbose: bool = False,
state: StateBackend | None = None,
concurrency_limits: Mapping[str, int] | None = None,
) -> Iterator[tuple[str, TaskResult[Any]]]:
"""流式执行图,逐个 yield 完成任务的 ``(name, TaskResult)``。
与 :func:`run` 相同的执行语义,但以生成器形式逐个返回结果而非等待
全部完成后返回 :class:`RunReport`。适用于:
* 大量任务需要逐个处理结果(如写入数据库、发送通知)
* 内存敏感场景(不需要同时持有所有结果)
* 需要在任务完成时即时响应
内部通过后台线程执行 :func:`run` + 队列传递结果实现,不改变 ``run``
的执行逻辑。生成器耗尽后若执行过程抛异常则 re-raise。
参数与 :func:`run` 一致(不含 ``dry_run``/``on_event``/``progress``/
``notifiers``/``_report``,这些由流式机制内部管理)。
Yields
------
tuple[str, TaskResult]
``(task_name, result)``,按任务完成顺序 yield。终态事件
SUCCESS/FAILED/SKIPPED)均会 yield。
Raises
------
ValueError
``strategy`` 不被识别时。
TaskFailedError
任何任务耗尽重试后仍失败时(在 yield 完已失败任务后 re-raise)。
Example
-------
for name, result in px.run_iter(graph):
print(f"{name}: {result.status.value}")
"""
report = RunReport()
ready: queue.Queue[str | None] = queue.Queue()
error_box: list[BaseException | None] = [None]
yielded: set[str] = set()
_terminal = {TaskStatus.SUCCESS, TaskStatus.FAILED, TaskStatus.SKIPPED}
def _on_event(event: TaskEvent) -> None:
if event.status in _terminal:
ready.put(event.task)
def _worker() -> None:
with backend.batch():
try:
run(
graph,
strategy=strategy,
max_workers=max_workers,
verbose=verbose,
on_event=_on_event,
state=state,
concurrency_limits=concurrency_limits,
_report=report,
)
except BaseException as exc:
error_box[0] = exc
if strategy == "sequential":
layers = graph.layers()
_drive_sequential(graph, layers, context, report, backend, effective_callback)
elif strategy == "thread":
layers = graph.layers()
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits)
elif strategy == "async":
layers = graph.layers()
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits))
elif strategy == "dependency":
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits))
else:
raise ValueError(f"Unknown strategy: {strategy!r}")
except TaskFailedError:
report.success = False
raise
finally:
ready.put(None)
# 关闭进程池:通知管理线程退出 + kill 工作进程,避免
# threading._shutdown 等待管理线程 join 工作进程导致 ~7s 阻塞。
_shutdown_process_pool()
thread = threading.Thread(target=_worker, daemon=True)
thread.start()
try:
while True:
name = ready.get()
if name is None:
break
if name in yielded:
continue
yielded.add(name)
if name in report.results:
yield name, report.results[name]
finally:
thread.join(timeout=0)
if error_box[0] is not None:
raise error_box[0]
return report
def _print_dry_run(graph: Graph, layers: list[list[str]]) -> None:
"""打印执行计划但不运行任何任务。"""
_verbose_console.print(f"[bold]Dry run:[/bold] [cyan]{len(graph)}[/cyan] tasks, [cyan]{len(layers)}[/cyan] layers")
print(f"Dry run: {len(graph)} tasks, {len(layers)} layers")
for idx, layer in enumerate(layers, 1):
_verbose_console.print(f" [dim]Layer {idx}:[/dim] {layer}")
print(f" Layer {idx}: {layer}")
for name in layer:
_verbose_console.print(f" [cyan]-[/cyan] {describe_injection(graph.resolved_spec(name))}")
print(f" - {describe_injection(graph.resolved_spec(name))}")
def _drive_sequential(
@@ -1294,11 +847,8 @@ def _drive_sequential(
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
cancel_event: threading.Event | CancelToken | None = None,
) -> None:
for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event):
return
SequentialLayerRunner.execute(layer, graph, context, report, backend, idx, on_event)
@@ -1311,16 +861,10 @@ def _drive_threaded(
on_event: EventCallback | None,
max_workers: int | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None:
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
max_layer_size = max((len(layer) for layer in layers), default=1)
pool_workers = max_workers or max(1, min(32, max_layer_size))
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event):
return
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, pool, concurrency_limits)
for idx, layer in enumerate(layers, 1):
workers = max_workers or max(1, min(32, len(layer)))
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, workers, concurrency_limits)
async def _async_drive(
@@ -1331,32 +875,6 @@ async def _async_drive(
backend: StateBackend,
on_event: EventCallback | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None:
for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event):
return
await AsyncLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, concurrency_limits)
def _mark_remaining_skipped(
graph: Graph,
report: RunReport,
on_event: EventCallback | None,
) -> None:
"""将未执行的任务标记为 SKIPPED(取消场景)。
遍历图中所有任务,将尚未出现在 ``report.results`` 中的任务标记为
SKIPPED,使最终报告包含完整的任务状态映射。
"""
for name, spec in graph.all_specs().items():
if name in report.results:
continue
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus.SKIPPED,
finished_at=datetime.now(),
reason="执行被取消",
)
report.results[name] = result
_emit(on_event, result)
+30 -261
View File
@@ -1,7 +1,7 @@
"""DAG 构建、校验、分层与可视化。
使用自实现的 Kahn 算法进行拓扑排序。图以增量方式构建并即时校验,
使配置错误在构建时(而非执行时)快速失败。
使用标准库的 :mod:`graphlib`3.9+)或 :mod:`graphlib_backport`3.8
进行拓扑排序。图以增量方式构建并即时校验,使配置错误在构建时(而非执行时)快速失败。
支持:
* 图级默认值 :class:`GraphDefaults`TaskSpec 字段为 ``None`` 时回退。
@@ -18,60 +18,24 @@ __all__ = [
]
import inspect
from collections.abc import Callable, Iterable, Mapping, Sequence
import sys
from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any
from typing import Any, Callable, Iterable, Mapping, Sequence
from .errors import CycleError, DuplicateTaskError, MissingDependencyError
from .task import Context, RetryPolicy, TaskSpec
if sys.version_info >= (3, 9): # pragma: no cover
import graphlib # pyright: ignore[reportUnreachable]
def _topological_layers(deps: Mapping[str, tuple[str, ...]]) -> tuple[list[list[str]], list[str] | None]:
"""Kahn 算法分层拓扑排序。
_TopologicalSorter = graphlib.TopologicalSorter
else: # pragma: no cover
import graphlib # type: ignore[import-untyped]
返回 ``(layers, cycle_nodes)``:无环时 ``cycle_nodes`` 为 ``None``
有环时为参与环的未处理节点列表(非精确环路径,仅指示存在环)。
直接用 dict + list 计算入度与反向邻接,在 diamond(1000) 图上约
0.45 万 ops/s(含 Graph 构造开销)。
"""
# 入度(依赖数)与反向邻接表
in_degree: dict[str, int] = {}
dependents: dict[str, list[str]] = {}
for name, d in deps.items():
in_degree[name] = len(d)
dependents.setdefault(name, [])
for name, d in deps.items():
for dep in d:
if dep in dependents:
dependents[dep].append(name)
# 初始层:入度为 0 的节点
current = sorted(name for name, deg in in_degree.items() if deg == 0)
layers: list[list[str]] = []
processed = 0
while current:
layers.append(current)
nxt: list[str] = []
for node in current:
for dependent in dependents[node]:
in_degree[dependent] -= 1
if in_degree[dependent] == 0:
nxt.append(dependent)
processed += 1
nxt.sort()
current = nxt
if processed < len(deps):
cycle_nodes = [name for name, deg in in_degree.items() if deg > 0]
return layers, cycle_nodes
return layers, None
_TopologicalSorter = graphlib.TopologicalSorter # pragma: no cover
@dataclass(slots=True)
@dataclass
class GraphDefaults:
"""图级默认值。TaskSpec 对应字段为 ``None`` 时回退到此处。
@@ -168,7 +132,7 @@ def _make_namespaced_fn(orig_fn: Any, ns: str, dep_names: set[str]) -> Any:
return wrapper
@dataclass(slots=True)
@dataclass
class Graph:
"""校验后的有向无环任务图。
@@ -192,18 +156,6 @@ class Graph:
# 在 specs / defaults 变更时失效。
_resolved_cache: dict[str, TaskSpec[Any]] = field(default_factory=dict)
# layers() 缓存:避免重复 run() 调用时重算拓扑排序。
# 在 specs 变更时失效。
_layers_cache: list[list[str]] | None = field(default=None)
# loop 组映射:原 spec 名 → 展开后的实例名列表。
# 用于依赖重写:其他任务引用 loop 原名时,自动展开为依赖所有实例。
_loop_groups: dict[str, list[str]] = field(default_factory=dict)
# 任务分组映射:组名 → 组内任务名元组。
# 用于依赖重写:引用组名等价于依赖组内所有任务。
_groups: dict[str, tuple[str, ...]] = field(default_factory=dict)
# ------------------------------------------------------------------ #
# 构建
# ------------------------------------------------------------------ #
@@ -235,123 +187,13 @@ class Graph:
prev_name = current.name
return self
def pipeline(self, *specs: TaskSpec[Any]) -> Graph:
"""创建数据流水线:每个任务的结果通过上下文注入传给下一个。
与 :meth:`chain` 行为一致(依赖关系相同),但语义上强调数据流:
前驱任务的返回值会自动注入后继任务的同名参数。
后继任务的 ``fn`` 参数名需匹配前驱任务名才能接收数据。
``outputs`` 字段声明的命名输出可通过
:meth:`RunReport.output_of` 查询。
Examples
--------
>>> graph = px.Graph().pipeline(extract_spec, transform_spec, load_spec)
>>> report = px.run(graph)
>>> report.output_of("extract", "items")
"""
return self.chain(*specs)
def group(self, name: str, tasks: Sequence[str]) -> Graph:
"""声明任务分组,引用组名等价于依赖组内所有任务。
组名必须不与任何已注册任务名冲突。组内任务必须已注册(或在本批
``from_specs`` 收集阶段先于引用方注册)。声明后,后续 ``add`` 或
``from_specs`` 中 ``depends_on``/``soft_depends_on`` 引用组名时,
会被重写为依赖组内全部任务。
参数
----
name:
组名。不能与 ``self.specs`` 中任务名重复,也不能与已有组名重复。
tasks:
组内任务名序列。所有任务必须已在图中注册。
返回 ``self`` 支持链式调用。
Raises
------
ValueError
组名与任务名冲突、组名已存在、或组内任务未注册时。
"""
if not name:
raise ValueError("group.name 必须为非空字符串。")
if name in self.specs:
raise ValueError(f"组名 {name!r} 与已注册任务名冲突。")
if name in self._groups:
raise ValueError(f"组名 {name!r} 已存在。")
missing = [t for t in tasks if t not in self.specs]
if missing:
raise ValueError(f"{name!r} 引用了未注册的任务: {missing}")
if not tasks:
raise ValueError(f"{name!r} 不能为空。")
self._groups[name] = tuple(tasks)
return self
def _register(self, spec: TaskSpec[Any]) -> None:
"""注册单个 spec。若 ``spec.loop`` 非 ``None``,展开为多实例。"""
if spec.loop is not None:
for expanded in self._expand_loop_to_specs(spec):
self._register_single(expanded)
return
self._register_single(self._rewrite_deps(spec))
def _register_single(self, spec: TaskSpec[Any]) -> None:
"""注册单个已展开的 spec(无 loop)。"""
if spec.name in self.specs:
raise DuplicateTaskError(spec.name)
self.specs[spec.name] = spec
# 拓扑依赖仅含硬依赖;软依赖仅用于注入,不影响分层。
self.deps[spec.name] = spec.depends_on
self._resolved_cache.clear()
self._layers_cache = None
def _expand_loop_to_specs(self, spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
"""展开 ``spec.loop`` 为多实例列表,并记录到 ``_loop_groups``。
返回展开后的 spec 列表(``loop`` 字段已清空,``args`` 已追加 item)。
不写入 ``specs``/``deps``,由调用方注册。
"""
loop = spec.loop
assert loop is not None # 由 _register 保证
expanded: list[TaskSpec[Any]] = []
expanded_names: list[str] = []
for i, item in enumerate(loop.items):
key = loop.key_fn(i, item) if loop.key_fn is not None else f"{spec.name}_{i}"
new_spec = replace(spec, name=key, args=(*spec.args, item), loop=None)
new_spec = self._rewrite_deps(new_spec)
expanded.append(new_spec)
expanded_names.append(key)
self._loop_groups[spec.name] = expanded_names
return expanded
def _rewrite_deps(self, spec: TaskSpec[Any]) -> TaskSpec[Any]:
"""重写 spec 的依赖:引用 loop 组名或 group 组名时展开为所有实例。"""
if not self._loop_groups and not self._groups:
return spec
# 合并 loop 组与普通组:组名 → 展开后的任务名列表
combined: dict[str, Sequence[str]] = {}
combined.update(self._loop_groups)
combined.update(self._groups)
new_hard: list[str] = []
new_soft: list[str] = []
changed = False
for dep in spec.depends_on:
if dep in combined:
new_hard.extend(combined[dep])
changed = True
else:
new_hard.append(dep)
for dep in spec.soft_depends_on:
if dep in combined:
new_soft.extend(combined[dep])
changed = True
else:
new_soft.append(dep)
if not changed:
return spec
return replace(spec, depends_on=tuple(new_hard), soft_depends_on=tuple(new_soft))
@classmethod
def from_specs(
@@ -377,31 +219,15 @@ class Graph:
"""
graph = cls(defaults=defaults or GraphDefaults(), namespace=namespace)
pending_refs: list[str] = []
collected: list[TaskSpec[Any]] = []
for spec in specs:
if isinstance(spec, str):
pending_refs.append(spec)
elif isinstance(spec, TaskSpec):
collected.append(spec)
graph._register(spec)
else:
raise TypeError(f"from_specs 只接受 TaskSpec 或 str,收到: {type(spec)}")
# 两阶段:先展开所有 loop(记录 _loop_groups),再统一重写依赖并批量注册。
# 批量注册跳过每次 _register 的 cache 清空(N 次清空降为 0 次);
# cache 失效由后续 validate()/layers() 首次调用自然触发。
expanded: list[TaskSpec[Any]] = []
for spec in collected:
if spec.loop is not None:
expanded.extend(graph._expand_loop_to_specs(spec))
else:
expanded.append(graph._rewrite_deps(spec))
for spec in expanded:
if spec.name in graph.specs:
raise DuplicateTaskError(spec.name)
graph.specs[spec.name] = spec
graph.deps[spec.name] = spec.depends_on
if pending_refs:
graph._pending_refs = pending_refs
@@ -409,30 +235,6 @@ class Graph:
graph.validate()
return graph
@classmethod
def from_yaml(cls, path: str | Path) -> Graph:
"""从 YAML 文件加载任务图(GitHub Actions 风格)。
Parameters
----------
path:
YAML 文件路径。
Returns
-------
Graph
解析后的任务图,支持 ``jobs``/``needs``/``strategy.matrix``/
``if``/``continue-on-error``/``env``/``defaults`` 等字段。
Raises
------
ValueError
YAML 结构不符合 schema 时。
"""
from .yaml_loader import load_yaml
return load_yaml(path)
def add_subgraph(self, sub: Graph, *, namespace: str | None = None) -> Graph:
"""将子图合并到当前图,任务名加命名空间前缀避免冲突。
@@ -487,16 +289,14 @@ class Graph:
raise MissingDependencyError(name, dep)
def validate(self) -> None:
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。
顺带填充 :attr:`_layers_cache`(无环时),使后续 :meth:`layers`
直接命中缓存,避免 :func:`_topological_layers` 二次计算。
"""
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。"""
self._validate_references()
layers, cycle_nodes = _topological_layers(self.deps)
if cycle_nodes is not None:
raise CycleError(cycle_nodes)
self._layers_cache = layers
sorter = _TopologicalSorter(self.deps)
try:
sorter.prepare()
except graphlib.CycleError as exc: # type: ignore[name-defined]
cycle: Sequence[str] = exc.args[1] if len(exc.args) > 1 else []
raise CycleError(list(cycle)) from exc
# ------------------------------------------------------------------ #
# 内省
@@ -568,18 +368,19 @@ class Graph:
同层任务无相互硬依赖,可并发执行。软依赖不参与分层。
层按执行顺序返回。图有环时抛出 :class:`CycleError`。
结果按实例缓存;specs 变更时失效(:meth:`add` / :meth:`_register`)。
.. note::
本方法假定图已通过 :meth:`validate` 校验(由 :func:`pyflowx.run`
在入口统一执行一次)。若直接调用本方法,需自行先校验。
"""
if self._layers_cache is not None:
return self._layers_cache
result, cycle_nodes = _topological_layers(self.deps)
if cycle_nodes is not None:
raise CycleError(cycle_nodes)
self._layers_cache = result
sorter = _TopologicalSorter(self.deps)
result: list[list[str]] = []
sorter.prepare()
while sorter.is_active():
ready = list(sorter.get_ready())
ready.sort()
result.append(ready)
for node in ready:
sorter.done(node)
return result
# ------------------------------------------------------------------ #
@@ -608,38 +409,6 @@ class Graph:
]
return Graph.from_specs(kept, defaults=self.defaults)
def subgraph_with_deps(self, names: Iterable[str]) -> Graph:
"""返回包含 ``names`` 及其所有传递依赖的新图。
与 :meth:`subgraph_by_names` 不同,本方法会沿 ``depends_on`` 和
``soft_depends_on`` 向上遍历,确保被选中的任务所需的上游全部包含在内,
使子图可独立执行。
Parameters
----------
names:
需要执行的任务名。每个名称必须存在于当前图中。
"""
seeds: set[str] = set(names)
for n in seeds:
if n not in self.specs:
raise KeyError(f"Unknown task name: {n!r}")
# BFS 收集传递依赖(硬依赖 + 软依赖)
closure: set[str] = set()
queue: list[str] = list(seeds)
while queue:
name = queue.pop()
if name in closure:
continue
closure.add(name)
for dep in self.all_deps(name):
if dep not in closure:
queue.append(dep)
kept: list[TaskSpec[Any]] = [
_prune_deps(spec, lambda d: d in closure) for spec in self.specs.values() if spec.name in closure
]
return Graph.from_specs(kept, defaults=self.defaults)
# ------------------------------------------------------------------ #
# Fan-out / map-reduce
# ------------------------------------------------------------------ #
-87
View File
@@ -1,87 +0,0 @@
"""运行历史管理:存储、查询与比较多次运行的 :class:`RunReport`。
适用于:
* 断点续跑:从最近一次失败运行恢复,跳过已成功任务。
* 趋势分析:对比多次运行的任务耗时、状态变化。
* 审计追溯:按 run_id 查询历史报告。
用法
----
history = RunHistory(".pyflowx/runs")
history.save(report)
latest = history.latest()
if latest is not None and not latest.success:
# 从最近失败处恢复
px.run(graph, resume_from=latest)
"""
from __future__ import annotations
__all__ = ["RunHistory"]
from pathlib import Path
from .report import RunReport
class RunHistory:
"""运行历史记录管理器。
将 :class:`RunReport` 以 JSON 文件存储到指定目录,文件名为
``{run_id}.json``。支持按 run_id 加载、列出全部、查询最近一次、删除。
Parameters
----------
dir:
存储目录。不存在时自动创建。
"""
def __init__(self, dir: str | Path) -> None:
self.dir = Path(dir)
self.dir.mkdir(parents=True, exist_ok=True)
def save(self, report: RunReport) -> Path:
"""保存运行报告,返回文件路径。"""
path = self.dir / f"{report.run_id}.json"
path.write_text(report.to_json(), encoding="utf-8")
return path
def load(self, run_id: str) -> RunReport:
"""按 run_id 加载报告。
Raises
------
FileNotFoundError
指定 run_id 的文件不存在。
"""
path = self.dir / f"{run_id}.json"
if not path.exists():
raise FileNotFoundError(f"运行报告 {run_id!r} 不存在: {path}")
return RunReport.from_json(path.read_text(encoding="utf-8"))
def list_runs(self) -> list[str]:
"""列出全部 run_id(按文件名排序)。"""
return sorted(p.stem for p in self.dir.glob("*.json"))
def latest(self) -> RunReport | None:
"""返回最近修改的报告;目录为空时返回 ``None``。"""
runs = sorted(self.dir.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True)
if not runs:
return None
return RunReport.from_json(runs[0].read_text(encoding="utf-8"))
def delete(self, run_id: str) -> bool:
"""删除指定 run_id 的报告。返回是否实际删除。"""
path = self.dir / f"{run_id}.json"
if path.exists():
path.unlink()
return True
return False
def __len__(self) -> int:
return sum(1 for _ in self.dir.glob("*.json"))
def __contains__(self, run_id: object) -> bool:
if not isinstance(run_id, str):
return False
return (self.dir / f"{run_id}.json").exists()
-217
View File
@@ -1,217 +0,0 @@
"""监控导出:Prometheus 指标收集与健康检查。
提供零依赖的轻量监控方案:
* :class:`MetricsCollector` —— 通过 ``on_event`` 回调收集任务指标,
导出 Prometheus 文本格式。可配合 ``http.server`` 或 ``prometheus_client``
暴露指标端点。
* :func:`health_check` —— 分析 :class:`RunReport` 返回结构化健康状态。
用法
----
collector = px.MetricsCollector()
report = px.run(graph, on_event=collector.on_event)
print(collector.metrics_text())
print(px.health_check(report))
"""
from __future__ import annotations
__all__ = ["MetricsCollector", "health_check", "start_metrics_server"]
from collections.abc import Callable
from typing import Any
from .report import RunReport
from .task import TaskEvent, TaskStatus
class MetricsCollector:
"""收集任务执行指标,导出 Prometheus 文本格式。
作为 :func:`run` 的 ``on_event`` 回调使用,收集每个任务的生命周期
事件并聚合为 Prometheus 指标。线程安全(回调可能从多个线程调用)。
指标列表
--------
- ``pyflowx_task_total{task, status}`` — 任务计数器
- ``pyflowx_task_duration_seconds{task}`` — 任务耗时(gauge,最近值)
- ``pyflowx_task_duration_seconds_sum{task}`` — 任务累计耗时
- ``pyflowx_task_retries_total{task}`` — 重试次数
- ``pyflowx_run_total{status}`` — 运行计数器
"""
def __init__(self) -> None:
self._task_count: dict[str, dict[str, int]] = {}
self._task_duration_last: dict[str, float] = {}
self._task_duration_sum: dict[str, float] = {}
self._task_retries: dict[str, int] = {}
self._lock_counts: dict[str, int] = {}
def on_event(self, event: TaskEvent) -> None:
"""``on_event`` 回调:记录任务生命周期事件。"""
task = event.task
status = event.status.value
# 任务计数
self._task_count.setdefault(task, {})
self._task_count[task][status] = self._task_count[task].get(status, 0) + 1
# 耗时(仅在终态时记录)
if event.duration is not None and event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED):
self._task_duration_last[task] = event.duration
self._task_duration_sum[task] = self._task_duration_sum.get(task, 0.0) + event.duration
# 重试次数
if event.attempts > 1 and event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED):
self._task_retries[task] = self._task_retries.get(task, 0) + event.attempts - 1
def record_run(self, report: RunReport) -> None:
"""记录一次完整运行的结果。"""
status = "success" if report.success else "failed"
self._lock_counts[status] = self._lock_counts.get(status, 0) + 1
def metrics_text(self) -> str:
"""导出 Prometheus 文本格式指标字符串。"""
lines: list[str] = []
# pyflowx_task_total
if self._task_count:
lines.append("# HELP pyflowx_task_total Total tasks by status.")
lines.append("# TYPE pyflowx_task_total counter")
for task in sorted(self._task_count):
for status in sorted(self._task_count[task]):
count = self._task_count[task][status]
lines.append(f'pyflowx_task_total{{task="{task}",status="{status}"}} {count}')
# pyflowx_task_duration_seconds (last)
if self._task_duration_last:
lines.append("# HELP pyflowx_task_duration_seconds Last task duration in seconds.")
lines.append("# TYPE pyflowx_task_duration_seconds gauge")
for task in sorted(self._task_duration_last):
lines.append(f'pyflowx_task_duration_seconds{{task="{task}"}} {self._task_duration_last[task]}')
# pyflowx_task_duration_seconds_sum
if self._task_duration_sum:
lines.append("# HELP pyflowx_task_duration_seconds_sum Total task duration in seconds.")
lines.append("# TYPE pyflowx_task_duration_seconds_sum counter")
for task in sorted(self._task_duration_sum):
lines.append(f'pyflowx_task_duration_seconds_sum{{task="{task}"}} {self._task_duration_sum[task]}')
# pyflowx_task_retries_total
if self._task_retries:
lines.append("# HELP pyflowx_task_retries_total Total task retries.")
lines.append("# TYPE pyflowx_task_retries_total counter")
for task in sorted(self._task_retries):
lines.append(f'pyflowx_task_retries_total{{task="{task}"}} {self._task_retries[task]}')
# pyflowx_run_total
if self._lock_counts:
lines.append("# HELP pyflowx_run_total Total runs by status.")
lines.append("# TYPE pyflowx_run_total counter")
for status in sorted(self._lock_counts):
lines.append(f'pyflowx_run_total{{status="{status}"}} {self._lock_counts[status]}')
return "\n".join(lines) + "\n"
def reset(self) -> None:
"""清空所有收集的指标。"""
self._task_count.clear()
self._task_duration_last.clear()
self._task_duration_sum.clear()
self._task_retries.clear()
self._lock_counts.clear()
def health_check(report: RunReport) -> dict[str, Any]:
"""分析运行报告,返回结构化健康状态。
状态判定:
- ``healthy`` — 全部任务 SUCCESS
- ``degraded`` — 部分(非全部)任务 FAILED
- ``unhealthy`` — 全部任务 FAILED
- ``unknown`` — 无任务
Returns
-------
dict
包含 ``status``/``total``/``success``/``failed``/``skipped``
/``duration`` 字段。
"""
total = len(report.results)
if total == 0:
return {"status": "unknown", "total": 0, "message": "无任务结果"}
counts: dict[str, int] = {}
total_duration = 0.0
for r in report.results.values():
status = r.status.value
counts[status] = counts.get(status, 0) + 1
if r.duration is not None:
total_duration += r.duration
success = counts.get("success", 0)
failed = counts.get("failed", 0)
skipped = counts.get("skipped", 0)
if failed == 0:
status = "healthy"
elif failed < total:
status = "degraded"
else:
status = "unhealthy"
return {
"status": status,
"total": total,
"success": success,
"failed": failed,
"skipped": skipped,
"duration": round(total_duration, 3),
}
def start_metrics_server(collector: MetricsCollector, port: int = 9100, host: str = "0.0.0.0") -> Callable[[], None]:
"""启动简单 HTTP 服务器暴露 Prometheus 指标。
返回停止函数,调用后关闭服务器。阻塞调用方线程,建议在后台线程运行。
.. note::
使用标准库 ``http.server``,零依赖。生产环境建议配合
``prometheus_client`` 或反向代理使用。
"""
import sys
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
if sys.version_info >= (3, 12):
from typing import override
else:
from typing_extensions import override # pragma: no cover
class _Handler(BaseHTTPRequestHandler):
def do_GET(self) -> None:
if self.path == "/metrics":
body = collector.metrics_text().encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
else:
self.send_response(404)
self.end_headers()
@override
def log_message(self, format: str, *args: object) -> None:
pass # 静默访问日志
server = HTTPServer((host, port), _Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
def stop() -> None:
server.shutdown()
server.server_close()
return stop
-320
View File
@@ -1,320 +0,0 @@
"""任务通知系统。
在任务生命周期事件(RUNNING/SUCCESS/FAILED/SKIPPED)发生时发送通知到
外部系统。支持两类 sink
* :class:`CallbackNotifier` —— 包装 Python 回调,零外部依赖
* :class:`WebhookNotifier` 及其子类 —— 通过 HTTP POST 发送到飞书/钉钉/企业微信
通知系统与 :class:`pyflowx.progress.ProgressCallback` 独立且可共存。
``notifiers`` 参数在 :func:`pyflowx.run` 中传入,通知器的 ``notify`` 方法
被加入回调链,``close`` 在执行结束时调用。
设计要点
--------
* 事件过滤在 Notifier 内部,通过 ``levels`` 参数控制哪些事件触发通知。
* webhook 使用 stdlib :mod:`urllib.request`,零新依赖。
* 通知发送失败仅记录日志,不影响任务执行。
快速上手
--------
import pyflowx as px
# 1. 回调通知
notifier = px.CallbackNotifier(
lambda e: print(f"[通知] {e.task}: {e.status.value}")
)
px.run(graph, notifiers=notifier)
# 2. 飞书 webhook(仅失败时通知)
feishu = px.FeishuNotifier(
url="https://open.feishu.cn/open-apis/bot/v2/hook/xxx",
levels=(px.NotificationLevel.FAILED,),
)
px.run(graph, notifiers=[feishu])
"""
from __future__ import annotations
__all__ = [
"ALL_LEVELS",
"CallbackNotifier",
"DingTalkNotifier",
"DiscordNotifier",
"FeishuNotifier",
"NotificationLevel",
"Notifier",
"SlackNotifier",
"TelegramNotifier",
"WeChatNotifier",
"WebhookNotifier",
]
import logging
import sys
import urllib.error
import urllib.request
from collections.abc import Callable, Iterable
from enum import Enum
from typing import Any, Protocol, runtime_checkable
if sys.version_info >= (3, 12):
from typing import override
else:
from typing_extensions import override # pragma: no cover
from ._json import dumps
from .task import TaskEvent, TaskStatus
logger = logging.getLogger(__name__)
class NotificationLevel(Enum):
"""通知触发的事件级别。"""
RUNNING = "running"
SUCCESS = "success"
FAILED = "failed"
SKIPPED = "skipped"
_LEVEL_STATUS_MAP: dict[NotificationLevel, TaskStatus] = {
NotificationLevel.RUNNING: TaskStatus.RUNNING,
NotificationLevel.SUCCESS: TaskStatus.SUCCESS,
NotificationLevel.FAILED: TaskStatus.FAILED,
NotificationLevel.SKIPPED: TaskStatus.SKIPPED,
}
#: 所有事件级别的冻结集合。
ALL_LEVELS: frozenset[NotificationLevel] = frozenset(_LEVEL_STATUS_MAP.keys())
#: 各平台消息前缀图标。
_STATUS_ICONS: dict[TaskStatus, str] = {
TaskStatus.RUNNING: "",
TaskStatus.SUCCESS: "",
TaskStatus.FAILED: "",
TaskStatus.SKIPPED: "",
}
def _status_to_level(status: TaskStatus) -> NotificationLevel | None:
"""将 TaskStatus 映射为 NotificationLevel,无映射时返回 None。"""
for level, st in _LEVEL_STATUS_MAP.items():
if st == status:
return level
return None
def _format_event_message(event: TaskEvent) -> str:
"""格式化事件为人类可读消息文本(各平台共享)。"""
icon = _STATUS_ICONS.get(event.status, "")
dur = f" ({event.duration:.3f}s)" if event.duration is not None else ""
parts = [f"{icon} 任务 {event.task} {event.status.value}{dur}"]
if event.error:
parts.append(f" 错误: {event.error}")
if event.reason:
parts.append(f" 原因: {event.reason}")
if event.attempts > 1:
parts.append(f" 尝试次数: {event.attempts}")
return "\n".join(parts)
@runtime_checkable
class Notifier(Protocol):
"""通知器协议。
生命周期:执行期间多次 ``notify`` → 结束时 ``close``。
实现者应确保 ``notify`` 不抛异常(或仅在不可恢复时抛),
通知失败不应影响任务执行。
"""
def notify(self, event: TaskEvent) -> None:
"""处理单个任务事件。"""
...
def close(self) -> None:
"""释放资源(如关闭连接)。"""
...
class CallbackNotifier:
"""包装 Python 回调为 :class:`Notifier`。
Parameters
----------
callback:
接收 :class:`TaskEvent` 的回调函数。
levels:
触发通知的事件级别集合。默认 :data:`ALL_LEVELS`(全生命周期)。
"""
def __init__(
self,
callback: Callable[[TaskEvent], None],
levels: Iterable[NotificationLevel] = ALL_LEVELS,
) -> None:
self._callback = callback
self._levels = frozenset(levels)
def notify(self, event: TaskEvent) -> None:
level = _status_to_level(event.status)
if level is not None and level in self._levels:
self._callback(event)
def close(self) -> None:
"""无资源需释放。"""
class WebhookNotifier:
"""通过 HTTP POST 发送通知的基类。
使用 stdlib :mod:`urllib.request`,零外部依赖。子类覆盖
:meth:`_build_payload` 以适配不同平台的消息格式。
Parameters
----------
url:
webhook 接收地址。
levels:
触发通知的事件级别集合。默认 :data:`ALL_LEVELS`。
timeout:
HTTP 请求超时秒数。默认 ``10``。
"""
def __init__(
self,
url: str,
levels: Iterable[NotificationLevel] = ALL_LEVELS,
timeout: float = 10.0,
) -> None:
self._url = url
self._levels = frozenset(levels)
self._timeout = timeout
def notify(self, event: TaskEvent) -> None:
level = _status_to_level(event.status)
if level is None or level not in self._levels:
return
payload = self._build_payload(event)
self._send(payload)
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
"""构建请求体。子类覆盖以适配平台格式。"""
return {"text": _format_event_message(event)}
def _send(self, payload: dict[str, Any]) -> None:
"""发送 JSON POST 请求,失败仅记录日志。"""
data = dumps(payload, ensure_ascii=False).encode("utf-8")
req = urllib.request.Request(
self._url,
data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
_ = resp.read()
except (urllib.error.URLError, OSError) as exc:
logger.warning("webhook 通知发送失败 (%s): %s", self._url, exc)
def close(self) -> None:
"""无持久连接需关闭。"""
class FeishuNotifier(WebhookNotifier):
"""飞书(Lark)自定义机器人通知器。
消息格式:``{"msg_type": "text", "content": {"text": ...}}``。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {
"msg_type": "text",
"content": {"text": _format_event_message(event)},
}
class DingTalkNotifier(WebhookNotifier):
"""钉钉自定义机器人通知器。
消息格式:``{"msgtype": "text", "text": {"content": ...}}``。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {
"msgtype": "text",
"text": {"content": _format_event_message(event)},
}
class WeChatNotifier(WebhookNotifier):
"""企业微信群机器人通知器。
消息格式:``{"msgtype": "text", "text": {"content": ...}}``。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {
"msgtype": "text",
"text": {"content": _format_event_message(event)},
}
class SlackNotifier(WebhookNotifier):
"""Slack Incoming Webhook 通知器。
消息格式:``{"text": ...}``。Slack webhook 仅需 ``text`` 字段,
与基类格式一致,独立成类便于类型区分与后续扩展(如 channel/username)。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {"text": _format_event_message(event)}
class DiscordNotifier(WebhookNotifier):
"""Discord Webhook 通知器。
消息格式:``{"content": ...}``。Discord webhook 使用 ``content`` 字段
承载消息文本。
"""
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {"content": _format_event_message(event)}
class TelegramNotifier(WebhookNotifier):
"""Telegram Bot 通知器。
通过 ``sendMessage`` API 发送消息。需要在 ``__init__`` 中传入
``chat_id``(聊天/群组 ID),并使用 bot token 拼接 webhook URL
``https://api.telegram.org/bot<TOKEN>/sendMessage``。
消息格式:``{"chat_id": ..., "text": ...}``。
Parameters
----------
chat_id:
接收消息的聊天或群组 ID(数字字符串或 @channelusername)。
"""
def __init__(
self,
url: str,
chat_id: str,
levels: Iterable[NotificationLevel] = ALL_LEVELS,
timeout: float = 10.0,
) -> None:
super().__init__(url=url, levels=levels, timeout=timeout)
self._chat_id = chat_id
@override
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
return {
"chat_id": self._chat_id,
"text": _format_event_message(event),
}
-18
View File
@@ -1,18 +0,0 @@
"""ops 子包 — CLI 工具模块集合 (按功能分组).
每个子模块使用 ``@px.tool`` 装饰器注册 CLI 工具, 由 ``pf`` 入口按需
``importlib.import_module`` 触发注册. 子模块按功能分组到子目录.
子目录
------
- :mod:`files` —— 文件与文档操作 (filedate/filelevel/folderback/folderzip/pdftool/screenshot)
- :mod:`dev` —— 开发与构建工具 (autofmt/bumpversion/gittool/packtool/piptool/pymake/lscalc)
- :mod:`system` —— 系统工具 (clr/reseticoncache/taskkill/which)
- :mod:`infra` —— 基础设施与服务部署 (dockercmd/envdev/sshcopyid/msdownload/sglang)
:mod:`_common` 保留在根目录, 提供跨平台命令选择、平台守卫等共享辅助.
"""
from __future__ import annotations
__all__: list[str] = []
-66
View File
@@ -1,66 +0,0 @@
"""ops 共享辅助: 跨平台命令选择、平台守卫、共享忽略模式."""
from __future__ import annotations
from collections.abc import Sequence
from pyflowx.conditions import Constants
__all__ = ["IGNORE_PATTERNS", "ensure_platform", "platform_command"]
IGNORE_PATTERNS: list[str] = [
"__pycache__",
"*.pyc",
"*.pyo",
".git",
".venv",
".idea",
".vscode",
"*.egg-info",
"dist",
"build",
".pytest_cache",
".tox",
".mypy_cache",
]
def platform_command(windows: Sequence[str], posix: Sequence[str]) -> list[str]:
"""按当前平台返回对应命令.
Parameters
----------
windows : Sequence[str]
Windows 平台命令 (含参数).
posix : Sequence[str]
Linux/macOS 平台命令 (含参数).
Returns
-------
list[str]
当前平台对应的命令列表 (副本, 可安全修改).
"""
return list(windows) if Constants.IS_WINDOWS else list(posix)
def ensure_platform(predicate: bool, tool_name: str, platform_name: str) -> bool:
"""平台不满足时打印提示, 返回是否满足.
Parameters
----------
predicate : bool
平台匹配谓词 (如 ``Constants.IS_WINDOWS``).
tool_name : str
工具名称, 用于提示信息前缀.
platform_name : str
目标平台名称, 用于提示信息.
Returns
-------
bool
平台满足时返回 True, 否则打印 ``{tool_name}: 仅在 {platform_name} 上支持`` 并返回 False.
"""
if not predicate:
print(f"{tool_name}: 仅在 {platform_name} 上支持")
return False
return True
-5
View File
@@ -1,5 +0,0 @@
"""dev 子包 — 开发与构建工具."""
from __future__ import annotations
__all__: list[str] = []
-232
View File
@@ -1,232 +0,0 @@
"""版本号管理模块.
提供单文件版本号更新 (``bump_file_version``) 与项目级批量版本号同步
(``bump_project_version``) 能力. ``bump_project_version`` 通过 ``@px.tool``
注册为 CLI 工具, 可通过 ``pf bumpversion`` 调用.
设计要点
--------
``bump_project_version`` 采用 "先读取基准、再统一写入" 的两阶段策略:
先扫描所有 ``__init__.py`` / ``pyproject.toml`` 文件, 读取各自的版本号,
取最大值作为基准版本计算新版本号, 然后把新版本号统一写入所有文件,
避免文件间版本号不同步导致的跳号问题.
"""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
from typing import Literal
import pyflowx as px
__all__ = [
"BumpVersionType",
"bump_file_version",
"bump_project_version",
]
# ============================================================================
# 配置
# ============================================================================
BumpVersionType = Literal["patch", "minor", "major"]
_PYPROJECT_VERSION_PATTERN = re.compile(
r'(?:^|\n)\s*version\s*=\s*["\']'
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
r'["\']',
re.MULTILINE,
)
_INIT_VERSION_PATTERN = re.compile(
r'(?:^|\n)\s*__version__\s*=\s*["\']'
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
r'["\']',
re.MULTILINE,
)
_IGNORE_DIRS = frozenset({".venv", "venv", ".git", "__pycache__", ".tox", "node_modules", "build", "dist", ".eggs"})
# ============================================================================
# 私有辅助函数
# ============================================================================
def _get_pattern_for_file(file_name: str) -> re.Pattern[str] | None:
"""根据文件类型获取对应的正则表达式."""
if file_name == "pyproject.toml":
return _PYPROJECT_VERSION_PATTERN
if file_name == "__init__.py":
return _INIT_VERSION_PATTERN
return None
def _calculate_new_version(major: int, minor: int, patch: int, part: BumpVersionType) -> str:
"""计算新版本号."""
if part == "major":
return f"{major + 1}.0.0"
if part == "minor":
return f"{major}.{minor + 1}.0"
return f"{major}.{minor}.{patch + 1}"
def _build_replacement_string(original_match: str, new_version: str, file_name: str) -> str:
"""构建替换字符串, 保留原始格式."""
quote_char = '"' if '"' in original_match else "'"
key = "__version__" if file_name == "__init__.py" else "version"
prefix_match = re.match(rf"(\s*{key}\s*=\s*)[\"']", original_match)
prefix = prefix_match.group(1) if prefix_match else f"{key} = "
return f"{prefix}{quote_char}{new_version}{quote_char}"
def _read_version_tuple(file_path: Path) -> tuple[int, int, int] | None:
"""从文件中读取版本号, 返回 (major, minor, patch) 元组; 未找到返回 None.
读取失败时抛出 ``OSError`` / ``UnicodeDecodeError`` 由调用方处理.
"""
pattern = _get_pattern_for_file(file_path.name)
if pattern is None:
return None
content = file_path.read_text(encoding="utf-8")
match = pattern.search(content)
if not match:
return None
return int(match.group("major")), int(match.group("minor")), int(match.group("patch"))
def _write_version_to_file(file_path: Path, new_version: str) -> bool:
"""把新版本号写入指定文件; 成功返回 True, 未匹配到版本号返回 False."""
pattern = _get_pattern_for_file(file_path.name)
if pattern is None: # pragma: no cover - 调用方已保证 pattern 不为 None
return False
content = file_path.read_text(encoding="utf-8")
match = pattern.search(content)
if not match: # pragma: no cover - 调用方已通过 _read_version_tuple 验证
return False
replacement = _build_replacement_string(match.group(0), new_version, file_path.name)
content = content.replace(match.group(0), replacement)
try:
file_path.write_text(content, encoding="utf-8")
except OSError as e:
print(f"更新文件 {file_path} 版本号时出错: {e}")
raise
return True
# ============================================================================
# 公共函数
# ============================================================================
def bump_file_version(file_path: Path, part: BumpVersionType = "patch") -> str | None:
"""更新单个文件中的版本号.
读取文件当前版本号, 按 ``part`` 指定的部分递增, 写回文件.
Parameters
----------
file_path : Path
要更新的文件路径 (``pyproject.toml`` 或 ``__init__.py``)
part : BumpVersionType
版本部分: patch, minor, major
Returns
-------
str | None
更新后的新版本号; 文件中未找到版本号或读取失败时返回 None
"""
version_tuple = _read_version_tuple(file_path)
if version_tuple is None:
print(f"文件 {file_path} 中未找到版本号模式")
return None
major, minor, patch = version_tuple
new_version = _calculate_new_version(major, minor, patch, part)
if not _write_version_to_file(file_path, new_version): # pragma: no cover - _read_version_tuple 已验证
return None
return new_version
@px.tool("bumpversion", help="版本号自动管理")
def bump_project_version(part: BumpVersionType = "patch", no_tag: bool = False) -> str | None:
"""批量同步项目所有版本号文件并提交.
扫描当前目录下所有 ``__init__.py`` 和 ``pyproject.toml`` 文件
(排除虚拟环境和缓存目录), 先读取每个文件的当前版本号取最大值作为基准,
计算新版本号后统一写入所有文件, 最后执行 git add (按文件名) + commit + tag.
采用 "先读取基准、再统一写入" 的两阶段策略, 即使某些文件版本号不同步,
也能在一次 bump 后重新对齐, 避免跳号.
Parameters
----------
part : BumpVersionType
版本部分: patch, minor, major
no_tag : bool
提交后不创建 git tag
Returns
-------
str | None
更新后的新版本号; 未找到版本号文件时返回 None
"""
all_files: set[Path] = set()
for pattern in ("__init__.py", "pyproject.toml"):
for file in Path.cwd().rglob(pattern):
if not any(ignore_dir in file.parts for ignore_dir in _IGNORE_DIRS):
all_files.add(file)
if not all_files:
print("未找到包含版本号的文件")
return None
print(f"找到 {len(all_files)} 个文件需要更新版本号")
cwd = Path.cwd()
for file in sorted(all_files):
print(f" - {file.relative_to(cwd)}")
# 阶段 1: 读取所有文件版本号, 取最大值作为基准
versions: list[tuple[int, int, int]] = []
for file in sorted(all_files):
v = _read_version_tuple(file)
if v is not None:
versions.append(v)
if not versions:
print("未能从任何文件读取版本号")
return None
major, minor, patch = max(versions)
new_version = _calculate_new_version(major, minor, patch, part)
print(f"基准版本: {major}.{minor}.{patch} -> 新版本: {new_version}")
# 阶段 2: 统一写入新版本号到所有文件
for file in sorted(all_files):
_write_version_to_file(file, new_version)
# 阶段 3: git add (按文件名) + commit + tag
relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)]
subprocess.run(["git", "add", *relative_files], check=True)
subprocess.run(["git", "commit", "-m", f"bump version to {new_version}"], check=True)
if not no_tag:
tag_name = f"v{new_version}"
subprocess.run(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], check=True)
print(f"已创建标签: {tag_name}")
return new_version
-142
View File
@@ -1,142 +0,0 @@
"""gittool - Git 执行工具.
提供添加提交/清理/初始化/推送/拉取 子命令.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import pyflowx as px
EXCLUDE_DIRS = [
# 编辑器相关目录
".vscode",
".idea",
".editorconfig",
".trae",
".qoder",
# 项目相关目录
".venv",
".git",
".ruff_cache",
".tox",
"node_modules",
]
EXCLUDE_CMDS = [arg for d in EXCLUDE_DIRS for arg in ["-e", d]]
# ============================================================================
# 私有辅助函数
# ============================================================================
def not_has_git_repo() -> bool:
"""检查当前目录没有 Git 仓库."""
return not Path.cwd().exists() or not (Path.cwd() / ".git").is_dir()
def has_files() -> bool:
"""检查当前 Git 仓库是否有未提交的更改."""
try:
result = subprocess.run(
["git", "status", "--porcelain"],
capture_output=True,
text=True,
check=False,
)
return bool(result.stdout.strip())
except (subprocess.SubprocessError, OSError):
return False
# ============================================================================
# fn 子命令
# ============================================================================
@px.tool("gittool", subcommand="a", help="添加并提交")
def git_add_commit(message: str = "chore: update") -> None:
"""执行 git add + git commit (仅当有未提交更改时).
Parameters
----------
message : str
提交信息 (默认: ``chore: update``)
"""
if not has_files():
print("没有文件需要提交")
return
subprocess.run(["git", "add", "."], check=True)
subprocess.run(["git", "commit", "-m", message], check=True)
@px.tool("gittool", subcommand="i", help="初始化并提交")
def git_init_add_commit(message: str = "init commit") -> None:
"""执行 git init (若需) + git add + git commit (若有更改).
Parameters
----------
message : str
提交信息 (默认: ``init commit``)
"""
if not_has_git_repo():
subprocess.run(["git", "init"], check=True)
if has_files():
subprocess.run(["git", "add", "."], check=True)
subprocess.run(["git", "commit", "-m", message], check=True)
else:
print("没有文件需要提交")
@px.tool("gittool", subcommand="isub", help="初始化子目录")
def init_sub_dirs() -> None:
"""初始化子目录的 Git 仓库."""
sub_dirs = [subdir for subdir in Path.cwd().iterdir() if subdir.is_dir()]
for subdir in sub_dirs:
px.run(
px.Graph().chain(
px.cmd(["git", "init"], conditions=(lambda _: not_has_git_repo(),), cwd=subdir),
px.cmd(["git", "add", "."]),
px.cmd(["git", "commit", "-m", "init commit"]),
),
)
# ============================================================================
# cmd 子命令
# ============================================================================
@px.tool(
"gittool",
subcommand="clean",
help="清理未跟踪文件",
cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS],
hidden=True,
)
def clean() -> None:
"""清理 Git 未跟踪文件 (隐藏命令, 被 c 依赖)."""
@px.tool(
"gittool",
subcommand="c",
help="清理并查看状态",
cmd=["git", "status", "--porcelain"],
needs=["clean"],
)
def c() -> None:
"""清理未跟踪文件并查看 Git 状态."""
@px.tool("gittool", subcommand="p", help="推送", cmd=["git", "push"])
def p() -> None:
"""推送代码到远程仓库."""
@px.tool("gittool", subcommand="pl", help="拉取", cmd=["git", "pull"])
def pl() -> None:
"""从远程仓库拉取代码."""
-111
View File
@@ -1,111 +0,0 @@
"""lscalc - LS-DYNA 计算工具.
运行 LS-DYNA 计算 (单机/MPI) 与进程状态检查.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
_DEFAULT_NCPU: int = 4
def get_ls_dyna_command(input_file: str, ncpu: int) -> list[str]:
"""获取 LS-DYNA 命令.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
Returns
-------
list[str]
LS-DYNA 命令列表
"""
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
@px.tool("lscalc", subcommand="run", help="运行 LS-DYNA 计算")
def run_ls_dyna(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
"""运行 LS-DYNA 计算.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
"""
input_path = Path(input_file)
if not input_path.exists():
print(f"输入文件不存在: {input_path}")
return
cmd = get_ls_dyna_command(input_file, ncpu)
try:
subprocess.run(cmd, check=True)
print(f"LS-DYNA 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA 计算失败: {e}")
@px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算")
def run_ls_dyna_mpi(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
"""运行 LS-DYNA MPI 计算.
Parameters
----------
input_file : str
输入文件路径
ncpu : int
CPU 核心数
"""
input_path = Path(input_file)
if not input_path.exists():
print(f"输入文件不存在: {input_path}")
return
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
try:
subprocess.run(cmd, check=True)
print(f"LS-DYNA MPI 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 mpirun 或 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA MPI 计算失败: {e}")
@px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态")
def check_ls_dyna_status() -> None:
"""检查 LS-DYNA 进程状态."""
try:
if Constants.IS_WINDOWS:
result = subprocess.run(
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
capture_output=True,
text=True,
check=True,
)
print(result.stdout)
else:
result = subprocess.run(
["pgrep", "-f", "ls-dyna"],
capture_output=True,
text=True,
check=False,
)
if result.stdout.strip():
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
else:
print("没有运行中的 LS-DYNA 进程")
except subprocess.CalledProcessError as e:
print(f"检查进程状态失败: {e}")
-236
View File
@@ -1,236 +0,0 @@
"""packtool - Python 打包工具.
提供源码打包/依赖打包/wheel 构建/嵌入式 Python 安装/zip 包创建/清理 子命令.
"""
from __future__ import annotations
import platform
import shutil
import subprocess
import urllib.request
import zipfile
from pathlib import Path
import pyflowx as px
from pyflowx.ops._common import IGNORE_PATTERNS
__all__ = [
"clean_build_dir",
"create_zip_package",
"install_embed_python",
"pack_dependencies",
"pack_source",
"pack_wheel",
]
# ============================================================================
# 配置
# ============================================================================
DEFAULT_BUILD_DIR = ".pypack"
DEFAULT_DIST_DIR = "dist"
DEFAULT_LIB_DIR = "libs"
DEFAULT_CACHE_DIR = ".cache/pypack"
# ============================================================================
# 公共函数
# ============================================================================
@px.tool("packtool", subcommand="src", help="打包源码")
def pack_source(project_dir: Path = Path(), output_dir: Path = Path(".pypack")) -> None:
"""打包项目源码.
Parameters
----------
project_dir : Path
项目目录 (默认: 当前目录)
output_dir : Path
输出目录 (默认: .pypack)
"""
output_dir.mkdir(parents=True, exist_ok=True)
pyproject_file = project_dir / "pyproject.toml"
project_name = project_dir.name
if pyproject_file.exists():
try:
import tomllib
content = pyproject_file.read_text(encoding="utf-8")
data = tomllib.loads(content)
project_name = data.get("project", {}).get("name", project_name)
except ImportError:
pass
source_dir = output_dir / "src" / project_name
source_dir.mkdir(parents=True, exist_ok=True)
src_subdir = project_dir / "src"
if src_subdir.exists():
shutil.copytree(
src_subdir,
source_dir / "src",
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
dirs_exist_ok=True,
)
else:
for item in project_dir.iterdir():
if item.name in IGNORE_PATTERNS or item.name.startswith("."):
continue
dst_item = source_dir / item.name
if item.is_dir():
shutil.copytree(
item,
dst_item,
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
dirs_exist_ok=True,
)
else:
shutil.copy2(item, dst_item)
print(f"源码打包完成: {source_dir}")
@px.tool("packtool", subcommand="deps", help="打包依赖")
def pack_dependencies(lib_dir: Path = Path("libs"), dependencies: list[str] | None = None) -> None:
"""打包项目依赖.
Parameters
----------
lib_dir : Path
依赖库目录 (默认: libs)
dependencies : list[str] | None
依赖列表
"""
if dependencies is None:
dependencies = []
lib_dir.mkdir(parents=True, exist_ok=True)
if not dependencies:
print("没有依赖需要打包")
return
cmd = [
"pip",
"install",
"--target",
str(lib_dir),
"--no-compile",
"--no-warn-script-location",
]
cmd.extend(dependencies)
subprocess.run(cmd, check=True)
print(f"依赖打包完成: {lib_dir}")
@px.tool("packtool", subcommand="wheel", help="构建 wheel")
def pack_wheel(project_dir: Path = Path(), output_dir: Path = Path("dist")) -> None:
"""打包项目为 wheel 文件.
Parameters
----------
project_dir : Path
项目目录 (默认: 当前目录)
output_dir : Path
输出目录 (默认: dist)
"""
output_dir.mkdir(parents=True, exist_ok=True)
cmd = [
"pip",
"wheel",
"--no-deps",
"--wheel-dir",
str(output_dir),
str(project_dir),
]
subprocess.run(cmd, check=True)
print(f"Wheel 打包完成: {output_dir}")
@px.tool("packtool", subcommand="embed", help="安装嵌入式 Python")
def install_embed_python(version: str = "3.10", output_dir: Path = Path("python")) -> None:
"""安装嵌入式 Python.
Parameters
----------
version : str
Python 版本 (如: 3.10, 3.11), 默认 3.10
output_dir : Path
输出目录 (默认: python)
"""
output_dir.mkdir(parents=True, exist_ok=True)
arch = platform.machine().lower()
if arch in ["x86_64", "amd64"]:
arch = "amd64"
elif arch in ["arm64", "aarch64"]:
arch = "arm64"
version_map = {
"3.8": "3.8.10",
"3.9": "3.9.13",
"3.10": "3.10.11",
"3.11": "3.11.9",
"3.12": "3.12.4",
}
full_version = version_map.get(version, f"{version}.0")
url = f"https://www.python.org/ftp/python/{full_version}/python-{full_version}-embed-{arch}.zip"
cache_file = Path(DEFAULT_CACHE_DIR) / f"python-{full_version}-embed-{arch}.zip"
cache_file.parent.mkdir(parents=True, exist_ok=True)
if not cache_file.exists():
print(f"正在下载嵌入式 Python {full_version}...")
urllib.request.urlretrieve(url, cache_file)
print(f"下载完成: {cache_file}")
with zipfile.ZipFile(cache_file, "r") as zf:
zf.extractall(output_dir)
print(f"嵌入式 Python 安装完成: {output_dir}")
@px.tool("packtool", subcommand="zip", help="创建 zip 包")
def create_zip_package(source_dir: Path = Path(), output_file: Path = Path("package.zip")) -> None:
"""创建 ZIP 打包文件.
Parameters
----------
source_dir : Path
源目录 (默认: 当前目录)
output_file : Path
输出文件 (默认: package.zip)
"""
output_file.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(output_file, "w", zipfile.ZIP_DEFLATED) as zf:
for file in source_dir.rglob("*"):
if file.is_file():
arcname = file.relative_to(source_dir)
zf.write(file, arcname)
print(f"ZIP 打包完成: {output_file}")
@px.tool("packtool", subcommand="clean", help="清理构建目录")
def clean_build_dir(build_dir: Path = Path(".pypack")) -> None:
"""清理构建目录.
Parameters
----------
build_dir : Path
构建目录 (默认: .pypack)
"""
if build_dir.exists():
shutil.rmtree(build_dir)
print(f"清理完成: {build_dir}")
else:
print(f"目录不存在: {build_dir}")
-177
View File
@@ -1,177 +0,0 @@
"""piptool - pip 包管理工具.
提供安装/卸载/重装/下载/升级 pip/冻结依赖 子命令.
"""
from __future__ import annotations
import fnmatch
import subprocess
from pathlib import Path
import pyflowx as px
__all__ = [
"pip_download",
"pip_freeze",
"pip_install",
"pip_reinstall",
"pip_uninstall",
"pip_upgrade",
]
# ============================================================================
# 配置
# ============================================================================
PACKAGE_DIR = "packages"
REQUIREMENTS_FILE = "requirements.txt"
_PROTECTED_PACKAGES: frozenset[str] = frozenset(
{
"pyflowx",
"bitool",
}
)
# ============================================================================
# 私有辅助函数
# ============================================================================
def _get_installed_packages() -> list[str]:
"""获取当前环境中所有已安装的包名."""
try:
result = subprocess.run(
["pip", "list", "--format=freeze"],
capture_output=True,
text=True,
check=True,
)
packages: list[str] = []
for line in result.stdout.strip().split("\n"):
if line and "==" in line:
pkg_name = line.split("==")[0].strip()
packages.append(pkg_name)
except (subprocess.SubprocessError, OSError):
return []
return packages
def _expand_wildcard_packages(pattern: str) -> list[str]:
"""展开通配符模式为实际的包名列表."""
if not any(char in pattern for char in ["*", "?", "[", "]"]):
return [pattern]
installed_packages = _get_installed_packages()
matched = [pkg for pkg in installed_packages if fnmatch.fnmatchcase(pkg.lower(), pattern.lower())]
return matched
def _filter_protected_packages(packages: list[str]) -> list[str]:
"""过滤掉受保护的包名."""
safe = [p for p in packages if p.lower() not in {p.lower() for p in _PROTECTED_PACKAGES}]
filtered = [p for p in packages if p.lower() in {p.lower() for p in _PROTECTED_PACKAGES}]
if filtered:
print(f"跳过受保护的包: {', '.join(filtered)}")
return safe
# ============================================================================
# 公共函数
# ============================================================================
@px.tool("piptool", subcommand="i", help="安装包")
def pip_install(packages: list[str]) -> None:
"""安装包.
Parameters
----------
packages : list[str]
包名列表
"""
subprocess.run(["pip", "install", *packages], check=True)
print(f"安装完成: {', '.join(packages)}")
@px.tool("piptool", subcommand="u", help="卸载包")
def pip_uninstall(packages: list[str]) -> None:
"""卸载包.
Parameters
----------
packages : list[str]
包名列表
"""
packages_to_uninstall: list[str] = []
for pattern in packages:
packages_to_uninstall.extend(_expand_wildcard_packages(pattern))
packages_to_uninstall = _filter_protected_packages(packages_to_uninstall)
if not packages_to_uninstall:
return
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True)
@px.tool("piptool", subcommand="r", help="重装包")
def pip_reinstall(packages: list[str], offline: bool = False) -> None:
"""重新安装包.
Parameters
----------
packages : list[str]
包名列表
offline : bool
离线模式
"""
safe_ps = _filter_protected_packages(packages)
if not safe_ps:
print("所有指定的包均为受保护包, 跳过重装")
return
subprocess.run(["pip", "uninstall", "-y", *safe_ps], check=True)
options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run(["pip", "install", *options, *safe_ps], check=True)
@px.tool("piptool", subcommand="d", help="下载包")
def pip_download(packages: list[str], offline: bool = False) -> None:
"""下载包.
Parameters
----------
packages : list[str]
包名列表
offline : bool
离线模式
"""
options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run(
["pip", "download", *packages, *options, "-d", PACKAGE_DIR],
check=True,
)
@px.tool("piptool", subcommand="up", help="升级 pip")
def pip_upgrade() -> None:
"""升级 pip 到最新版本."""
subprocess.run(["python", "-m", "pip", "install", "--upgrade", "pip"], check=True)
print("pip 升级完成")
@px.tool("piptool", subcommand="f", help="导出依赖")
def pip_freeze() -> None:
"""冻结依赖到 requirements.txt."""
result = subprocess.run(
["pip", "freeze", "--exclude-editable"],
capture_output=True,
text=True,
check=True,
)
Path(REQUIREMENTS_FILE).write_text(result.stdout)
print(f"依赖已导出到 {REQUIREMENTS_FILE}")
-243
View File
@@ -1,243 +0,0 @@
"""pymake - 项目构建工具.
提供构建/测试/清理/文档/格式化/发布等子命令.
"""
from __future__ import annotations
__all__ = [
"b",
"ba",
"bc",
"bump",
"bumpmi",
"bumpversion",
"c",
"cov",
"doc",
"git_add_all",
"git_push",
"git_push_tags",
"lint",
"p",
"pb",
"publish_python",
"pyrefly_check",
"sync",
"t",
"tc",
"test_coverage",
"tf",
"tox",
"twine_publish",
]
from pathlib import Path
import pyflowx as px
# ============================================================================
# 单任务别名 (cmd 任务)
# ============================================================================
@px.tool("pymake", subcommand="b", help="构建 Python 主包 (uv build)", cmd=["uv", "build"])
def b(cwd: Path = Path()) -> None:
"""构建 Python 主包."""
@px.tool("pymake", subcommand="bc", help="构建 Rust 核心模块 (maturin build)", cmd=["maturin", "build", "-r"])
def bc(cwd: Path = Path()) -> None:
"""构建 Rust 核心模块."""
@px.tool("pymake", subcommand="sync", help="同步依赖 (uv sync)", cmd=["uv", "sync"])
def sync(cwd: Path = Path()) -> None:
"""同步依赖."""
@px.tool("pymake", subcommand="c", help="清理构建产物 (调用 gitt c)", cmd=["pf", "gitt", "c"])
def c(cwd: Path = Path()) -> None:
"""清理构建产物."""
@px.tool(
"pymake",
subcommand="t",
help="运行测试",
cmd=["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"],
)
def t(cwd: Path = Path()) -> None:
"""运行测试 (含并行)."""
@px.tool(
"pymake",
subcommand="tf",
help="快速测试 (无 slow)",
cmd=["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"],
)
def tf(cwd: Path = Path()) -> None:
"""快速测试 (无 slow, 无并行)."""
@px.tool("pymake", subcommand="bumpmi", help="升级次版本号 (bumpversion minor)", cmd=["pf", "bumpversion", "minor"])
def bumpmi(cwd: Path = Path()) -> None:
"""升级次版本号."""
@px.tool(
"pymake",
subcommand="doc",
help="构建 Sphinx 文档",
cmd=["sphinx-build", "-b", "html", "docs", "docs/_build"],
)
def doc(cwd: Path = Path()) -> None:
"""构建 Sphinx 文档."""
@px.tool("pymake", subcommand="lint", help="代码格式化与检查 (ruff)", cmd=["ruff", "check", "--fix", "--unsafe-fixes"])
def lint(cwd: Path = Path()) -> None:
"""代码格式化与检查."""
@px.tool("pymake", subcommand="tox", help="多版本测试 (tox)", cmd=["tox", "-p", "auto"])
def tox(cwd: Path = Path()) -> None:
"""多版本测试."""
# ============================================================================
# 内部 job (hidden, 不暴露为 subcommand)
# ============================================================================
@px.tool(
"pymake",
subcommand="bumpversion",
help="升级版本号 (patch)",
cmd=["pf", "bumpversion", "patch"],
needs=["git_add_all"],
hidden=True,
)
def bumpversion(cwd: Path = Path()) -> None:
"""升级版本号 (patch, 内部 job)."""
@px.tool(
"pymake",
subcommand="test_coverage",
help="测试并生成覆盖率",
cmd=["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
needs=["c"],
hidden=True,
)
def test_coverage(cwd: Path = Path()) -> None:
"""测试并生成覆盖率 (内部 job)."""
@px.tool(
"pymake",
subcommand="pyrefly_check",
help="pyrefly 类型检查",
cmd=["pyrefly", "check", "."],
hidden=True,
)
def pyrefly_check(cwd: Path = Path()) -> None:
"""pyrefly 类型检查 (内部 job)."""
@px.tool(
"pymake",
subcommand="git_add_all",
help="git add -A",
cmd=["git", "add", "-A"],
needs=["tc"],
hidden=True,
)
def git_add_all(cwd: Path = Path()) -> None:
"""git add -A (内部 job)."""
@px.tool("pymake", subcommand="git_push", help="git push", cmd=["git", "push"], hidden=True)
def git_push(cwd: Path = Path()) -> None:
"""git push (内部 job)."""
@px.tool(
"pymake",
subcommand="git_push_tags",
help="git push --tags",
cmd=["git", "push", "--tags"],
hidden=True,
)
def git_push_tags(cwd: Path = Path()) -> None:
"""git push --tags (内部 job)."""
@px.tool(
"pymake",
subcommand="twine_publish",
help="twine upload",
cmd=["twine", "upload", "--disable-progress-bar"],
hidden=True,
)
def twine_publish(cwd: Path = Path()) -> None:
"""twine upload (内部 job)."""
@px.tool("pymake", subcommand="publish_python", help="hatch publish", cmd=["hatch", "publish"], hidden=True)
def publish_python(cwd: Path = Path()) -> None:
"""hatch publish (内部 job)."""
# ============================================================================
# 聚合 job (有 needs 无 cmd)
# ============================================================================
@px.tool("pymake", subcommand="ba", help="构建所有包 (Python + Rust)", needs=["b", "bc"], strategy="thread")
def ba(cwd: Path = Path()) -> None:
"""构建所有包 (Python + Rust)."""
@px.tool("pymake", subcommand="bump", help="升级版本号 (清理 + 检查 + add + bumpversion)", needs=["bumpversion"])
def bump(cwd: Path = Path()) -> None:
"""升级版本号 (聚合)."""
@px.tool("pymake", subcommand="cov", help="测试并生成覆盖率", needs=["test_coverage"])
def cov(cwd: Path = Path()) -> None:
"""测试并生成覆盖率 (聚合)."""
@px.tool(
"pymake",
subcommand="tc",
help="类型检查 (pyrefly + ruff)",
needs=["c", "pyrefly_check", "lint"],
strategy="thread",
)
def tc(cwd: Path = Path()) -> None:
"""类型检查 (聚合)."""
@px.tool(
"pymake",
subcommand="p",
help="推送代码 (清理 + push + push tags)",
needs=["c", "git_push", "git_push_tags"],
strategy="thread",
)
def p(cwd: Path = Path()) -> None:
"""推送代码 (聚合)."""
@px.tool(
"pymake",
subcommand="pb",
help="发布到 PyPI (twine + hatch)",
needs=["twine_publish", "publish_python"],
strategy="thread",
)
def pb(cwd: Path = Path()) -> None:
"""发布到 PyPI (聚合)."""
-5
View File
@@ -1,5 +0,0 @@
"""files 子包 — 文件与文档操作工具."""
from __future__ import annotations
__all__: list[str] = []
-512
View File
@@ -1,512 +0,0 @@
"""pdftool - PDF 文件工具集.
提供 PDF 合并/拆分/压缩/加密/解密/提取文本/提取图片/水印/旋转/裁剪/
信息/OCR/转图片/修复 等子命令.
"""
from __future__ import annotations
from pathlib import Path
import pyflowx as px
__all__ = [
"pdf_add_watermark",
"pdf_compress",
"pdf_crop",
"pdf_decrypt",
"pdf_encrypt",
"pdf_extract_images",
"pdf_extract_text",
"pdf_info",
"pdf_merge",
"pdf_ocr",
"pdf_reorder",
"pdf_repair",
"pdf_rotate",
"pdf_split",
"pdf_to_images",
]
try:
import fitz # PyMuPDF
HAS_PYMUPDF = True
except ImportError:
HAS_PYMUPDF = False
try:
import pypdf
HAS_PYPDF = True
except ImportError:
HAS_PYPDF = False
def _require_pymupdf() -> bool:
"""PyMuPDF 未安装时打印提示, 返回是否可用."""
if not HAS_PYMUPDF:
print("未安装 PyMuPDF 库, 请安装: pip install PyMuPDF")
return False
return True
def _require_pypdf() -> bool:
"""pypdf 未安装时打印提示, 返回是否可用."""
if not HAS_PYPDF:
print("未安装 pypdf 库, 请安装: pip install pypdf")
return False
return True
@px.tool("pdftool", subcommand="m", help="合并 PDF")
def pdf_merge(input_paths: list[Path], output_path: Path = Path("merged.pdf")) -> None:
"""合并多个 PDF 文件.
Parameters
----------
input_paths : list[Path]
输入 PDF 文件列表
output_path : Path
输出文件 (默认: merged.pdf)
"""
if not _require_pypdf():
return
writer = pypdf.PdfWriter()
for input_path in input_paths:
if input_path.exists():
reader = pypdf.PdfReader(str(input_path))
for page in reader.pages:
writer.add_page(page)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"合并完成: {output_path}")
@px.tool("pdftool", subcommand="s", help="拆分 PDF")
def pdf_split(input_path: Path, output_dir: Path = Path("split")) -> None:
"""拆分 PDF 文件为单页.
Parameters
----------
input_path : Path
输入 PDF 文件
output_dir : Path
输出目录 (默认: split)
"""
if not _require_pypdf():
return
reader = pypdf.PdfReader(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
for i, page in enumerate(reader.pages):
writer = pypdf.PdfWriter()
writer.add_page(page)
output_file = output_dir / f"{input_path.stem}_page_{i + 1}.pdf"
with open(output_file, "wb") as f:
writer.write(f)
print(f"拆分完成: {output_dir}")
@px.tool("pdftool", subcommand="c", help="压缩 PDF")
def pdf_compress(input_path: Path, output_path: Path = Path("compressed.pdf")) -> None:
"""压缩 PDF 文件.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: compressed.pdf)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
doc.close()
original_size = input_path.stat().st_size
new_size = output_path.stat().st_size
ratio = (1 - new_size / original_size) * 100
print(f"压缩完成: {output_path} (缩小 {ratio:.1f}%)")
@px.tool("pdftool", subcommand="e", help="加密 PDF")
def pdf_encrypt(input_path: Path, output_path: Path = Path("encrypted.pdf"), password: str = "") -> None:
"""加密 PDF 文件.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: encrypted.pdf)
password : str
密码 (必填)
"""
if not password:
print("错误: --password 为必填参数")
return
if not _require_pypdf():
return
reader = pypdf.PdfReader(str(input_path))
writer = pypdf.PdfWriter()
for page in reader.pages:
writer.add_page(page)
writer.encrypt(user_password=password, owner_password=password, use_128bit=True)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"加密完成: {output_path}")
@px.tool("pdftool", subcommand="d", help="解密 PDF")
def pdf_decrypt(input_path: Path, output_path: Path = Path("decrypted.pdf"), password: str = "") -> None:
"""解密 PDF 文件.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: decrypted.pdf)
password : str
密码 (必填)
"""
if not password:
print("错误: --password 为必填参数")
return
if not _require_pypdf():
return
reader = pypdf.PdfReader(str(input_path))
if reader.is_encrypted:
reader.decrypt(password)
writer = pypdf.PdfWriter()
for page in reader.pages:
writer.add_page(page)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"解密完成: {output_path}")
@px.tool("pdftool", subcommand="xt", help="提取文本")
def pdf_extract_text(input_path: Path, output_path: Path = Path("output.txt")) -> None:
"""提取 PDF 文本.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: output.txt)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
text = ""
for page in doc:
text += str(page.get_text()) + "\n\n"
doc.close()
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(text, encoding="utf-8")
print(f"文本提取完成: {output_path}")
@px.tool("pdftool", subcommand="xi", help="提取图片")
def pdf_extract_images(input_path: Path, output_dir: Path = Path("images")) -> None:
"""提取 PDF 图片.
Parameters
----------
input_path : Path
输入 PDF 文件
output_dir : Path
输出目录 (默认: images)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
image_count = 0
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
images = page.get_images(full=True)
for img_idx, img in enumerate(images):
xref = img[0]
base_image = doc.extract_image(xref)
image_data = base_image["image"]
image_ext = base_image["ext"]
image_path = output_dir / f"page_{page_num + 1}_img_{img_idx + 1}.{image_ext}"
image_path.write_bytes(image_data)
image_count += 1
doc.close()
print(f"图片提取完成: {output_dir} (共 {image_count} 张)")
@px.tool("pdftool", subcommand="w", help="添加水印")
def pdf_add_watermark(
input_path: Path, output_path: Path = Path("watermarked.pdf"), text: str = "CONFIDENTIAL"
) -> None:
"""添加 PDF 水印.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: watermarked.pdf)
text : str
水印文字 (默认: CONFIDENTIAL)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
for page in doc:
rect = page.rect
text_width = fitz.get_text_length(text, fontsize=48)
x = (rect.width - text_width) / 2
y = rect.height / 2
page.insert_text((x, y), text, fontsize=48, rotate=45, color=(0, 0, 0))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"水印添加完成: {output_path}")
@px.tool("pdftool", subcommand="r", help="旋转 PDF")
def pdf_rotate(input_path: Path, output_path: Path = Path("rotated.pdf"), rotation: int = 90) -> None:
"""旋转 PDF 页面.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: rotated.pdf)
rotation : int
旋转角度 (默认: 90)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
for page in doc:
page.set_rotation(rotation)
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"旋转完成: {output_path}")
@px.tool("pdftool", subcommand="crop", help="裁剪 PDF")
def pdf_crop(
input_path: Path,
output_path: Path = Path("cropped.pdf"),
margins: tuple[int, int, int, int] = (10, 10, 10, 10),
) -> None:
"""裁剪 PDF 页面.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: cropped.pdf)
margins : tuple[int, int, int, int]
边距 (左, 上, 右, 下), 默认 (10, 10, 10, 10)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
left, top, right, bottom = margins
for page in doc:
rect = page.rect
new_rect = fitz.Rect(
rect.x0 + left,
rect.y0 + top,
rect.x1 - right,
rect.y1 - bottom,
)
page.set_cropbox(new_rect)
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path))
doc.close()
print(f"裁剪完成: {output_path}")
@px.tool("pdftool", subcommand="i", help="查看 PDF 信息")
def pdf_info(input_path: Path) -> None:
"""显示 PDF 信息.
Parameters
----------
input_path : Path
输入 PDF 文件
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
print(f"文件: {input_path}")
print(f"页数: {doc.page_count}")
# pyrefly: ignore [missing-attribute]
print(f"标题: {doc.metadata.get('title', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"作者: {doc.metadata.get('author', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"创建日期: {doc.metadata.get('creationDate', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"修改日期: {doc.metadata.get('modDate', 'N/A')}")
print(f"文件大小: {input_path.stat().st_size / 1024:.1f} KB")
doc.close()
@px.tool("pdftool", subcommand="ocr", help="PDF OCR 识别")
def pdf_ocr(input_path: Path, output_path: Path = Path("ocr.pdf"), lang: str = "chi_sim+eng") -> None:
"""PDF OCR 识别.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: ocr.pdf)
lang : str
识别语言 (默认: chi_sim+eng)
"""
try:
import pytesseract
from PIL import Image
except ImportError:
print("未安装 OCR 相关库, 请安装: pip install pytesseract pillow")
return
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
new_doc = fitz.open()
for page in doc:
pix = page.get_pixmap()
img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
ocr_text = pytesseract.image_to_string(img, lang=lang)
new_page = new_doc.new_page(width=page.rect.width, height=page.rect.height)
new_page.insert_image(new_page.rect, pixmap=pix)
text_rect = fitz.Rect(0, 0, page.rect.width, page.rect.height)
# pyrefly: ignore [bad-argument-type]
new_page.insert_textbox(text_rect, ocr_text, fontname="china-ss", fontsize=11)
output_path.parent.mkdir(parents=True, exist_ok=True)
new_doc.save(str(output_path))
new_doc.close()
doc.close()
print(f"OCR 识别完成: {output_path}")
def pdf_reorder(input_path: Path, output_path: Path, order: list[int]) -> None:
"""重排 PDF 页面顺序.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件
order : list[int]
页面顺序列表 (0-based)
"""
if not _require_pypdf():
return
reader = pypdf.PdfReader(str(input_path))
writer = pypdf.PdfWriter()
for page_num in order:
if 0 <= page_num < len(reader.pages):
writer.add_page(reader.pages[page_num])
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "wb") as f:
writer.write(f)
print(f"重排完成: {output_path}")
@px.tool("pdftool", subcommand="img", help="PDF 转图片")
def pdf_to_images(input_path: Path, output_dir: Path = Path("images"), dpi: int = 300) -> None:
"""PDF 转图片.
Parameters
----------
input_path : Path
输入 PDF 文件
output_dir : Path
输出目录 (默认: images)
dpi : int
DPI (默认: 300)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
pix = page.get_pixmap(dpi=dpi)
image_path = output_dir / f"{input_path.stem}_page_{page_num + 1}.png"
pix.save(str(image_path))
doc.close()
print(f"转换完成: {output_dir}")
@px.tool("pdftool", subcommand="repair", help="修复 PDF")
def pdf_repair(input_path: Path, output_path: Path = Path("repaired.pdf")) -> None:
"""修复 PDF 文件.
Parameters
----------
input_path : Path
输入 PDF 文件
output_path : Path
输出文件 (默认: repaired.pdf)
"""
if not _require_pymupdf():
return
doc = fitz.open(str(input_path))
output_path.parent.mkdir(parents=True, exist_ok=True)
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
doc.close()
print(f"修复完成: {output_path}")
-5
View File
@@ -1,5 +0,0 @@
"""infra 子包 — 基础设施与服务部署工具."""
from __future__ import annotations
__all__: list[str] = []
-37
View File
@@ -1,37 +0,0 @@
"""dockercmd - Docker 镜像登录工具.
登录腾讯云 Docker 镜像仓库.
"""
from __future__ import annotations
import getpass
import subprocess
import pyflowx as px
_DOCKER_MIRROR_TENCENT: str = "ccr.ccs.tencentyun.com"
@px.tool("dockercmd", subcommand="login", help="登录腾讯云 Docker 镜像仓库")
def docker_login_tencent(username: str = "") -> None:
"""登录腾讯云 Docker 镜像仓库.
Parameters
----------
username : str
Docker 用户名 (为空时由 docker 交互式提示输入)
"""
user = username or getpass.getuser()
try:
subprocess.run(
["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT],
check=True,
)
except subprocess.CalledProcessError as e:
print(f"docker login 失败 (returncode={e.returncode}): {e}")
return
except FileNotFoundError:
print("docker 命令未找到,请确认 Docker 已安装并在 PATH 中")
return
print(f"已登录腾讯云镜像仓库 (用户: {user})")
-418
View File
@@ -1,418 +0,0 @@
"""envdev - 开发环境镜像源配置工具.
配置 Python / Conda / Rust 镜像源 (Linux 还会安装 Qt 库、中文字体、Docker).
所有镜像源参数互不影响, 可单独使用.
Linux 专用操作 (系统镜像/Qt/字体/Docker) 在非 Linux 平台上由函数内部跳过.
"""
from __future__ import annotations
import getpass
import os
import shutil
import subprocess
from pathlib import Path
from typing import Literal
import pyflowx as px
from pyflowx.conditions import Constants
from pyflowx.ops._common import ensure_platform
__all__ = [
"download_rustup_script",
"install_linux_docker",
"install_linux_fonts",
"install_linux_qt_libs",
"install_rust_toolchain",
"setup_conda_mirror",
"setup_linux_system_mirror",
"setup_python_mirror",
"setup_rust_mirror",
]
# ============================================================================
# 配置常量
# ============================================================================
PyMirrorType = Literal["tsinghua", "aliyun", "huaweicloud", "ustc", "zju"]
CondaMirrorType = Literal["tsinghua", "ustc", "bsfu", "aliyun"]
RustMirrorType = Literal["tsinghua", "ustc", "aliyun"]
_PIP_INDEX_URLS: dict[str, str] = {
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
"huaweicloud": "https://mirrors.huaweicloud.com/repository/pypi/simple/",
"ustc": "https://pypi.mirrors.ustc.edu.cn/simple/",
"zju": "https://mirrors.zju.edu.cn/pypi/simple/",
}
_PIP_TRUSTED_HOSTS: dict[str, str] = {
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
"aliyun": "mirrors.aliyun.com",
"huaweicloud": "mirrors.huaweicloud.com",
"ustc": "pypi.mirrors.ustc.edu.cn",
"zju": "mirrors.zju.edu.cn",
}
_UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
_CONDA_MIRROR_URLS: dict[str, list[str]] = {
"tsinghua": [
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
],
"ustc": [
"https://mirrors.ustc.edu.cn/anaconda/pkgs/main/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/free/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/r/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/pytorch/",
],
"bsfu": [
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/main/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/free/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/r/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/pytorch/",
],
"aliyun": [
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
"https://mirrors.aliyun.com/anaconda/pkgs/r/",
"https://mirrors.aliyun.com/anaconda/pkgs/msys2/",
"https://mirrors.aliyun.com/anaconda/pkgs/pro/",
"https://mirrors.aliyun.com/anaconda/pkgs/dev/",
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
"https://mirrors.aliyun.com/anaconda/cloud/bioconda/",
"https://mirrors.aliyun.com/anaconda/cloud/menpo/",
"https://mirrors.aliyun.com/anaconda/cloud/pytorch/",
],
}
_RUSTUP_MIRRORS: dict[str, dict[str, str]] = {
"tsinghua": {
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
},
"aliyun": {
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
},
"ustc": {
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
},
}
_RUST_SCCACHE_DIR: Path = Path.home() / ".cargo" / "sccache"
_RUST_SCCACHE_CACHE_SIZE: str = "20G"
_QT_LIBS: list[str] = [
"build-essential",
"libgl1",
"libegl1",
"libglib2.0-0",
"libfontconfig1",
"libfreetype6",
"libxkbcommon0",
"libdbus-1-3",
"libxcb-xinerama0",
"libxcb-icccm4",
"libxcb-image0",
"libxcb-keysyms1",
"libxcb-randr0",
"libxcb-render-util0",
"libxcb-shape0",
"libxcb-xfixes0",
"libxcb-cursor0",
]
_CHINESE_FONTS: list[str] = [
"fonts-noto-cjk",
"fonts-wqy-microhei",
"fonts-wqy-zenhei",
"fonts-noto-color-emoji",
]
_DOWNLOAD_MIRROR_SCRIPT: str = "curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh"
_INSTALL_MIRROR_SCRIPT: str = "sudo bash /tmp/linuxmirrors.sh"
_RUSTUP_DOWNLOAD_URL_LINUX: str = "https://mirrors.aliyun.com/repo/rust/rustup-init.sh"
_RUSTUP_DOWNLOAD_URL_WINDOWS: str = "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"
# ============================================================================
# 私有辅助
# ============================================================================
def _pip_config_path() -> Path:
"""返回当前平台的 pip 配置文件路径."""
if Constants.IS_LINUX:
return Path.home() / ".pip" / "pip.conf"
return Path.home() / "pip" / "pip.ini"
# ============================================================================
# 镜像源配置函数
# ============================================================================
@px.tool("envdev", subcommand="setup-python", help="配置 Python 镜像源")
def setup_python_mirror(mirror: str = "tsinghua") -> None:
"""配置 Python 镜像源 (设置环境变量 + 写入 pip 配置文件).
设置 ``PIP_INDEX_URL`` / ``PIP_TRUSTED_HOSTS`` / ``UV_INDEX_URL`` /
``UV_PYTHON_INSTALL_MIRROR`` 等环境变量, 并写入 pip 配置文件.
Parameters
----------
mirror : str
镜像源名称: tsinghua/aliyun/huaweicloud/ustc/zju (默认: tsinghua)
"""
if mirror not in _PIP_INDEX_URLS:
print(f"未知 Python 镜像源: {mirror}")
return
index_url = _PIP_INDEX_URLS[mirror]
trusted_host = _PIP_TRUSTED_HOSTS[mirror]
os.environ["PIP_INDEX_URL"] = index_url
os.environ["PIP_TRUSTED_HOSTS"] = trusted_host
os.environ["UV_INDEX_URL"] = index_url
os.environ["UV_PYTHON_INSTALL_MIRROR"] = _UV_PYTHON_INSTALL_MIRROR
os.environ["UV_HTTP_TIMEOUT"] = "600"
os.environ["UV_LINK_MODE"] = "copy"
config_path = _pip_config_path()
config_path.parent.mkdir(parents=True, exist_ok=True)
content = f"[global]\nindex-url = {index_url}\ntrusted-host = {trusted_host}\n"
config_path.write_text(content, encoding="utf-8")
print(f"Python 镜像源已配置: {mirror} -> {config_path}")
@px.tool("envdev", subcommand="setup-conda", help="配置 Conda 镜像源")
def setup_conda_mirror(mirror: str = "tsinghua") -> None:
"""配置 Conda 镜像源 (写入 ~/.condarc).
Parameters
----------
mirror : str
镜像源名称: tsinghua/ustc/bsfu/aliyun (默认: tsinghua)
"""
if mirror not in _CONDA_MIRROR_URLS:
print(f"未知 Conda 镜像源: {mirror}")
return
urls = _CONDA_MIRROR_URLS[mirror]
config_path = Path.home() / ".condarc"
config_path.parent.mkdir(parents=True, exist_ok=True)
content = "show_channel_urls: true\nchannels:\n - " + "\n - ".join(urls) + "\n - defaults\n"
config_path.write_text(content, encoding="utf-8")
print(f"Conda 镜像源已配置: {mirror} -> {config_path}")
@px.tool("envdev", subcommand="setup-rust", help="配置 Rust 镜像源")
def setup_rust_mirror(mirror: str = "tsinghua", version: str = "stable") -> None:
"""配置 Rust 镜像源 (设置环境变量 + 写入 cargo config + 创建 sccache 目录).
设置 ``RUSTUP_DIST_SERVER`` / ``RUSTUP_UPDATE_ROOT`` / ``RUST_SCCACHE_DIR``
等环境变量, 写入 ``~/.cargo/config.toml``, 并创建 sccache 缓存目录.
Parameters
----------
mirror : str
镜像源名称: tsinghua/ustc/aliyun (默认: tsinghua)
version : str
Rust 版本 (未使用, 保留以与原 envdev 参数对齐)
"""
del version # 兼容旧参数, 实际安装由独立 job 处理
if mirror not in _RUSTUP_MIRRORS:
print(f"未知 Rust 镜像源: {mirror}")
return
mirrors = _RUSTUP_MIRRORS[mirror]
os.environ["RUSTUP_DIST_SERVER"] = mirrors["RUSTUP_DIST_SERVER"]
os.environ["RUSTUP_UPDATE_ROOT"] = mirrors["RUSTUP_UPDATE_ROOT"]
os.environ["RUST_SCCACHE_DIR"] = str(_RUST_SCCACHE_DIR)
os.environ["RUST_SCCACHE_CACHE_SIZE"] = _RUST_SCCACHE_CACHE_SIZE
_RUST_SCCACHE_DIR.mkdir(parents=True, exist_ok=True)
config_path = Path.home() / ".cargo" / "config.toml"
config_path.parent.mkdir(parents=True, exist_ok=True)
registry = mirrors["TOML_REGISTRY"]
content = (
f"\n[source.crates-io]\nreplace-with = '{mirror}'\n\n"
f'[source.{mirror}]\nregistry = "sparse+{registry}"\n\n'
f'[registries.{mirror}]\nindex = "sparse+{registry}"\n'
)
config_path.write_text(content, encoding="utf-8")
print(f"Rust 镜像源已配置: {mirror} -> {config_path}")
# ============================================================================
# Rust 工具链安装
# ============================================================================
@px.tool("envdev", subcommand="download-rustup", help="下载 Rustup 安装脚本")
def download_rustup_script() -> None:
"""下载 Rustup 安装脚本 (跨平台, 已安装 rustup 时跳过).
Linux 下载 ``rustup-init.sh``, Windows 下载 ``rustup-init.exe``.
"""
if shutil.which("rustup") is not None:
print("rustup 已安装, 跳过下载")
return
if Constants.IS_WINDOWS:
print("下载 rustup-init.exe...")
subprocess.run(
[
"powershell",
"-Command",
"Invoke-WebRequest",
"-Uri",
_RUSTUP_DOWNLOAD_URL_WINDOWS,
"-OutFile",
"rustup-init.exe",
],
check=False,
)
else:
print("下载 rustup-init.sh...")
subprocess.run(
["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
check=False,
)
@px.tool(
"envdev",
subcommand="install-rust",
help="安装 Rust 工具链",
needs=["setup-rust", "download-rustup"],
allow_upstream_skip=True,
)
def install_rust_toolchain(version: str = "stable") -> None:
"""安装 Rust 工具链 (rustup 未安装时跳过).
Parameters
----------
version : str
Rust 版本: ``stable`` / ``nightly`` / ``beta`` (默认: ``stable``)
"""
if shutil.which("rustup") is None:
print("rustup 未安装, 跳过工具链安装")
return
subprocess.run(["rustup", "toolchain", "install", version], check=False)
print(f"Rust 工具链 {version} 安装完成")
# ============================================================================
# Linux 专用函数
# ============================================================================
@px.tool("envdev", subcommand="setup-linux-mirror", help="配置 Linux 系统镜像源")
def setup_linux_system_mirror() -> None:
"""下载并安装 Linux 系统镜像源 (仅 Linux, 已配置国内镜像时跳过).
检查 ``/etc/apt/sources.list`` 与 ``/etc/apt/sources.list.d/ubuntu.sources``
是否已配置国内镜像, 已配置则跳过; 未配置则下载并执行 linuxmirrors 脚本.
"""
if not ensure_platform(Constants.IS_LINUX, "setup_linux_system_mirror", "Linux"):
return
apt_files = ["/etc/apt/sources.list", "/etc/apt/sources.list.d/ubuntu.sources"]
mirror_keys = list(_PIP_INDEX_URLS.keys())
already_configured = False
for apt_file in apt_files:
try:
content = Path(apt_file).read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
if any(mirror in content for mirror in mirror_keys):
already_configured = True
break
if already_configured:
print("已配置国内镜像源, 跳过系统镜像配置")
return
print("下载 linuxmirrors 脚本...")
subprocess.run(_DOWNLOAD_MIRROR_SCRIPT, shell=True, check=False)
print("安装 linuxmirrors...")
subprocess.run(_INSTALL_MIRROR_SCRIPT, shell=True, check=False)
@px.tool(
"envdev",
subcommand="install-qt-libs",
help="安装 Qt 依赖库",
needs=["setup-linux-mirror"],
allow_upstream_skip=True,
)
def install_linux_qt_libs() -> None:
"""安装 Qt 依赖库 (仅 Linux)."""
if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"):
return
subprocess.run(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False)
print("Qt 依赖库安装完成")
@px.tool(
"envdev",
subcommand="install-fonts",
help="安装中文字体",
needs=["setup-linux-mirror"],
allow_upstream_skip=True,
)
def install_linux_fonts() -> None:
"""安装中文字体 (仅 Linux)."""
if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"):
return
subprocess.run(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False)
print("中文字体安装完成")
@px.tool(
"envdev",
subcommand="install-docker",
help="安装 Docker",
needs=["setup-linux-mirror"],
allow_upstream_skip=True,
)
def install_linux_docker() -> None:
"""安装 Docker (仅 Linux)."""
if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"):
return
subprocess.run(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False)
subprocess.run(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False)
print("Docker 安装完成 (需重新登录以生效 docker 用户组)")
-46
View File
@@ -1,46 +0,0 @@
"""msdownload - ModelScope 模型/数据集下载工具.
ModelScope 下载模型数据集或空间.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import pyflowx as px
__all__ = ["msdownload_run"]
@px.tool("msdownload", help="ModelScope 模型/数据集下载工具")
def msdownload_run(name: str, target_type: str = "model", download_dir: str | None = None) -> None:
"""从 ModelScope 下载模型/数据集/空间.
Parameters
----------
name : str
目标名称 (: ``Qwen/Qwen2.5-Coder-32B-Instruct``)
target_type : str
目标类型: ``model`` / ``dataset`` / ``space`` (默认: ``model``)
download_dir : str | None
下载目录; None 时默认 ``~/.models/<name 最后一段>``
"""
if not name:
print("msdownload: name 不能为空")
return
if download_dir:
out_dir = Path(download_dir)
else:
out_dir = Path.home() / ".models" / name.rsplit("/", 1)[-1]
out_dir.mkdir(parents=True, exist_ok=True)
cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)]
print(f"下载 {target_type}: {name} -> {out_dir}")
try:
subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
print(f"下载失败 (returncode={e.returncode}): {e}")
except FileNotFoundError:
print("uvx 命令未找到,请确认 uv 已安装并在 PATH 中")
-97
View File
@@ -1,97 +0,0 @@
"""sglang - SGLang 本地模型服务工具.
提供安装与启动 SGLang 服务的子命令.
"""
from __future__ import annotations
import shutil
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx.ops._common import platform_command
__all__ = [
"install_sglang",
"run_sglang",
]
@px.tool("sglang", subcommand="install", help="安装 sglang")
def install_sglang() -> None:
"""安装 sglang (若未安装).
通过 ``shutil.which`` 检测 sglang 是否已安装, 未安装时执行 ``uv install sglang[all]``.
"""
if shutil.which("sglang") is not None:
print("sglang 已安装, 跳过安装步骤")
return
print("正在安装 sglang[all]...")
try:
subprocess.run(["uv", "install", "sglang[all]"], check=True)
except subprocess.CalledProcessError as e:
print(f"安装失败 (returncode={e.returncode}): {e}")
except FileNotFoundError:
print("uv 命令未找到,请确认 uv 已安装并在 PATH 中")
@px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"])
def run_sglang(
model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ",
port: int = 8000,
ctx_len: int = 32768,
mem_fraction: float = 0.75,
host: str = "0.0.0.0",
log_level: str = "info",
) -> None:
"""启动 SGLang 本地模型服务.
Parameters
----------
model : str
模型路径 (默认: ``~/.models/Qwen2.5-Coder-32B-Instruct-AWQ``)
port : int
服务端口 (默认: 8000)
ctx_len : int
最大上下文长度 (默认: 32768)
mem_fraction : float
显存占比 0-1 (默认: 0.75)
host : str
主机地址 (默认: 0.0.0.0)
log_level : str
日志级别 (默认: info)
"""
model_dir = Path(model).expanduser()
if not model_dir.exists():
print(f"模型目录不存在: {model_dir}")
return
python_bin = platform_command(["python"], ["python3"])[0]
cmd = [
python_bin,
"-m",
"sglang.launch_server",
"--model-path",
str(model_dir),
"--host",
host,
"--port",
str(port),
"--mem-fraction-static",
str(mem_fraction),
"--context-length",
str(ctx_len),
"--tool-call-parser",
"qwen",
"--log-level",
log_level,
]
print(f"启动 SGLang: {model_dir} (port={port}, ctx={ctx_len}, mem={mem_fraction})")
try:
subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
print(f"SGLang 启动失败 (returncode={e.returncode}): {e}")
except FileNotFoundError:
print(f"{python_bin} 命令未找到,请确认 Python 已安装并在 PATH 中")
-5
View File
@@ -1,5 +0,0 @@
"""system 子包 — 系统工具."""
from __future__ import annotations
__all__: list[str] = []
-21
View File
@@ -1,21 +0,0 @@
"""clr - 清屏工具.
跨平台清屏: Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
"""
from __future__ import annotations
import subprocess
import pyflowx as px
from pyflowx.ops._common import platform_command
@px.tool("clr", help="清屏 (跨平台)")
def clear_screen_run() -> None:
"""清屏 (跨平台).
Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
"""
# 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍
subprocess.run(platform_command(["cls"], ["clear"]), check=False)
-52
View File
@@ -1,52 +0,0 @@
"""reseticoncache - 重置 Windows 图标缓存工具.
执行流程: 杀掉 explorer 删除 IconCache.db 删除 iconcache* 重启 explorer.
仅在 Windows 上执行, Windows 平台打印提示并跳过.
"""
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
from pyflowx.ops._common import ensure_platform
@px.tool("reseticoncache", help="重置 Windows 图标缓存")
def reset_icon_cache_run() -> None:
"""重置 Windows 图标缓存.
执行流程: 杀掉 explorer 删除 IconCache.db 删除 iconcache* 重启 explorer.
仅在 Windows 上执行, Windows 平台打印提示并跳过.
"""
if not ensure_platform(Constants.IS_WINDOWS, "reset_icon_cache", "Windows"):
return
local_app_data = os.environ.get("LOCALAPPDATA", "")
if not local_app_data:
print("reset_icon_cache: LOCALAPPDATA 环境变量未设置")
return
icon_cache_db = Path(local_app_data) / "IconCache.db"
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
print("正在终止 explorer 进程...")
subprocess.run(["taskkill", "/f", "/im", "explorer.exe"], check=False)
if icon_cache_db.exists():
print(f"删除图标缓存: {icon_cache_db}")
subprocess.run(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False)
if explorer_cache_dir.exists():
print(f"清理 Explorer 缓存: {explorer_cache_dir}")
subprocess.run(
["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
check=False,
)
print("重启 explorer...")
subprocess.run(["cmd", "/c", "start", "explorer.exe"], check=False)
print("图标缓存已重置")
-29
View File
@@ -1,29 +0,0 @@
"""setenv - 设置环境变量工具.
设置当前进程的环境变量, 支持不覆盖已有值的 ``default`` 模式.
"""
from __future__ import annotations
import os
import pyflowx as px
@px.tool("setenv", help="设置环境变量")
def setenv_run(name: str, value: str, default: bool = False) -> None:
"""设置环境变量.
Parameters
----------
name : str
环境变量名
value : str
环境变量值
default : bool
True 时使用 ``setdefault`` 不覆盖已有值 (默认 False)
"""
if default:
os.environ.setdefault(name, value)
else:
os.environ[name] = value
-35
View File
@@ -1,35 +0,0 @@
"""taskkill - 进程终止工具.
跨平台按名称终止进程: Windows ``taskkill``, Linux/macOS ``pkill``.
"""
from __future__ import annotations
import subprocess
import pyflowx as px
from pyflowx.ops._common import platform_command
@px.tool("taskkill", help="按名称终止进程 (跨平台)")
def taskkill_run(process_names: list[str]) -> None:
"""按名称终止进程 (跨平台).
Windows 使用 ``taskkill /f /im <name>*``,
Linux/macOS 使用 ``pkill -f <name>*``.
Parameters
----------
process_names : list[str]
进程名称列表 (: ``["chrome.exe", "python"]``)
"""
cmd_prefix = platform_command(["taskkill", "/f", "/im"], ["pkill", "-f"])
for name in process_names:
print(f"终止进程: {name}")
# pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode
result = subprocess.run([*cmd_prefix, f"{name}*"], check=False)
if result.returncode == 0:
print(f" 已发送终止信号: {name}")
else:
print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}")
-36
View File
@@ -1,36 +0,0 @@
"""which - 命令查找工具.
跨平台查找可执行命令路径: Windows ``where``, Linux/macOS ``which``.
"""
from __future__ import annotations
import subprocess
import pyflowx as px
from pyflowx.ops._common import platform_command
@px.tool("which", help="查找可执行命令路径 (跨平台)")
def which_run(commands: list[str]) -> None:
"""查找可执行命令路径 (跨平台).
Windows 使用 ``where``, Linux/macOS 使用 ``which``.
对每个命令打印 ``<cmd> -> <path>`` ``<cmd> -> 未找到``.
Parameters
----------
commands : list[str]
要查找的命令名称列表
"""
which_cmd = platform_command(["where"], ["which"])[0]
for cmd in commands:
# where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode
result = subprocess.run([which_cmd, cmd], capture_output=True, text=True, check=False)
if result.returncode == 0:
# Windows 的 where 可能返回多行, 取第一个
path = result.stdout.strip().split("\n")[0].strip()
print(f"{cmd} -> {path}")
else:
print(f"{cmd} -> 未找到")
-26
View File
@@ -1,26 +0,0 @@
"""writefile - 写入文件工具.
将文本内容写入指定路径的文件, 支持指定编码.
"""
from __future__ import annotations
from pathlib import Path
import pyflowx as px
@px.tool("writefile", help="写入文件")
def write_file_run(path: str, content: str, encoding: str = "utf-8") -> None:
"""写入文件.
Parameters
----------
path : str
目标文件路径
content : str
写入内容
encoding : str
文件编码 (默认 ``utf-8``)
"""
Path(path).write_text(content, encoding=encoding)
-144
View File
@@ -1,144 +0,0 @@
"""实时进度监控。
提供两种监控形式
* :class:`RichProgressMonitor` 基于 ``rich.Progress`` 的终端进度条
通过 ``run(graph, progress=True)`` 开箱即用
* :class:`ProgressCallback` 协议定义 ``start`` / ``on_event`` / ``finish``
三个生命周期方法供程序化集成如接入 Web 仪表盘日志聚合系统
复用 :class:`pyflowx.task.TaskEvent` 事件机制与现有 ``on_event`` 回调独立
且可共存``on_event`` 是简单逐事件函数``ProgressCallback`` 是带生命周期的
监控器需要知道总任务数以渲染进度条
快速上手
--------
import pyflowx as px
# 1. 开箱即用
report = px.run(graph, progress=True)
# 2. 自定义监控器
class MyMonitor(px.ProgressCallback):
def start(self, total: int) -> None:
print(f"开始执行 {total} 个任务")
def on_event(self, event: px.TaskEvent) -> None:
print(f"{event.task}: {event.status.value}")
def finish(self) -> None:
print("执行完成")
report = px.run(graph, progress=MyMonitor())
"""
from __future__ import annotations
__all__ = [
"ProgressCallback",
"RichProgressMonitor",
]
from typing import Any, Protocol, runtime_checkable
from rich.console import Console
from rich.progress import (
BarColumn,
MofNCompleteColumn,
Progress,
SpinnerColumn,
TaskID,
TaskProgressColumn,
TextColumn,
TimeElapsedColumn,
)
from .task import TaskEvent, TaskStatus
@runtime_checkable
class ProgressCallback(Protocol):
"""进度监控回调协议。
生命周期``start`` 多次 ``on_event`` ``finish``
实现者可在 ``start`` 中初始化资源如打开进度条 ``finish`` 中清理
"""
def start(self, total: int) -> None:
"""执行开始时调用,``total`` 为图中的任务总数。"""
... # pragma: no cover
def on_event(self, event: TaskEvent) -> None:
"""每次任务状态转换时调用。"""
... # pragma: no cover
def finish(self) -> None:
"""执行结束(无论成功/失败)时调用。"""
... # pragma: no cover
class RichProgressMonitor:
"""基于 :class:`rich.progress.Progress` 的默认终端进度监控器。
显示运行中任务名完成数/总数进度条百分比与耗时
终态事件SUCCESS/FAILED/SKIPPED均推进进度
Parameters
----------
console:
可选的 :class:`rich.console.Console`用于测试注入
默认创建新 Console
"""
def __init__(self, console: Console | None = None) -> None:
self._progress = Progress(
SpinnerColumn(),
TextColumn("[bold blue]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
TaskProgressColumn(),
TimeElapsedColumn(),
console=console,
)
self._task_id: TaskID | None = None
self._completed: int = 0
self._total: int = 0
self._running: set[str] = set()
def start(self, total: int) -> None:
"""启动进度条。"""
self._total = total
self._progress.start()
self._task_id = self._progress.add_task("工作流执行中", total=total)
def on_event(self, event: TaskEvent) -> None:
"""处理任务事件,更新进度条。"""
if event.status == TaskStatus.RUNNING:
self._running.add(event.task)
self._update_description()
elif event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED, TaskStatus.SKIPPED):
self._running.discard(event.task)
self._completed += 1
if self._task_id is not None:
self._progress.advance(self._task_id)
self._update_description()
def finish(self) -> None:
"""停止进度条。"""
self._progress.stop()
def _update_description(self) -> None:
"""更新进度条描述:显示运行中任务名或完成统计。"""
if self._task_id is None:
return
if self._running:
shown = sorted(self._running)[:3]
suffix = f" (+{len(self._running) - 3})" if len(self._running) > 3 else ""
desc = f"运行中: {', '.join(shown)}{suffix}"
else:
desc = f"已完成 {self._completed}/{self._total}"
self._progress.update(self._task_id, description=desc)
def __enter__(self) -> RichProgressMonitor:
return self
def __exit__(self, *exc: Any) -> None:
self.finish()
+4 -408
View File
@@ -6,64 +6,10 @@
from __future__ import annotations
import csv
import html
import io
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any
from uuid import uuid4
from typing import Any, Iterator
from ._json import dumps, loads
from .task import TaskResult, TaskSpec, TaskStatus
if TYPE_CHECKING:
from .graph import Graph
from .profiling import ProfileReport
def _noop_fn() -> None:
"""反序列化重建报告时的占位 ``fn``。"""
return None
def _resolve_output_path(result: Any, path: str) -> Any:
"""从任务结果中按点分路径提取命名输出值。
路径语法
- ``"$"`` 整个结果
- ``"key"`` ``result["key"]``
- ``"a.b.c"`` ``result["a"]["b"]["c"]``
适用于 dict 结果 dict 结果仅支持 ``"$"``
"""
if path == "$":
return result
value = result
for key in path.split("."):
value = value[key]
return value
def _serialize_value(
value: Any,
value_serializer: Callable[[Any], Any] | None = None,
) -> Any:
"""序列化任务返回值为 JSON 兼容形式。
``None`` 直接返回提供 ``value_serializer`` 时由其转换否则尝试
JSON 序列化失败则回退到 ``repr(value)``保证任意值都能落盘
"""
if value is None:
return None
if value_serializer is not None:
return value_serializer(value)
try:
_ = dumps(value)
except (TypeError, ValueError):
return repr(value)
return value
from .task import TaskResult, TaskStatus
@dataclass
@@ -76,13 +22,10 @@ class RunReport:
任务名 -> :class:`TaskResult` 的映射插入顺序与任务完成顺序一致
success:
当且仅当所有非跳过任务都以 ``SUCCESS`` 结束时为 ``True``
run_id:
8 字符十六进制运行 ID自动生成用于日志追踪与跨系统关联
"""
results: dict[str, TaskResult[Any]] = field(default_factory=dict)
success: bool = True
run_id: str = field(default_factory=lambda: uuid4().hex[:8])
# ---- 类型化访问 --------------------------------------------------- #
def __getitem__(self, name: str) -> Any:
@@ -97,34 +40,6 @@ class RunReport:
"""返回 ``name`` 的完整 :class:`TaskResult`。"""
return self.results[name]
def output_of(self, task_name: str, output_name: str) -> Any:
"""获取任务的命名输出。
根据 :attr:`TaskSpec.outputs` 中声明的路径从任务结果中提取
例如 ``outputs={"artifact": "path"}`` 声明后
``report.output_of("build", "artifact")`` 返回 ``result["path"]``
Parameters
----------
task_name:
任务名
output_name:
``outputs`` 映射中的键名
Raises
------
KeyError
任务不存在任务未声明 outputs output_name 未声明时
"""
result = self.results.get(task_name)
if result is None:
raise KeyError(f"任务 {task_name!r} 不在报告中。")
spec = result.spec
if spec.outputs is None or output_name not in spec.outputs:
raise KeyError(f"任务 {task_name!r} 未声明输出 {output_name!r}")
path = spec.outputs[output_name]
return _resolve_output_path(result.value, path)
def __contains__(self, name: Any) -> bool:
return name in self.results
@@ -144,7 +59,6 @@ class RunReport:
if r.duration is not None:
total_duration += r.duration
return {
"run_id": self.run_id,
"success": self.success,
"total_tasks": len(self.results),
"by_status": counts,
@@ -172,328 +86,10 @@ class RunReport:
return {name: (r.duration or 0.0) for name, r in self.results.items()}
def describe(self) -> str:
"""用于调试的人类可读多行报告。
失败时``success=False``在末尾附诊断摘要便于快速定位根因
"""
lines: list[str] = [f"RunReport(run_id={self.run_id}, success={self.success})"]
"""用于调试的人类可读多行报告。"""
lines: list[str] = [f"RunReport(success={self.success})"]
for name, r in self.results.items():
dur = f"{r.duration:.3f}s" if r.duration is not None else "-"
err = f" error={r.error!r}" if r.error else ""
lines.append(f" {name}: {r.status.value} ({dur} attempts={r.attempts}){err}")
if not self.success:
diag = self.diagnose()
if diag is not None:
lines.append("")
lines.append(diag.describe())
return "\n".join(lines)
# ---- 诊断 --------------------------------------------------------- #
def diagnose(self) -> Any:
"""生成失败诊断报告。
运行成功``success`` ``True``时返回 ``None``失败时返回
:class:`pyflowx.diagnostics.DiagnosticReport`包含根因分析
依赖链回溯相似失败聚类与可操作建议
返回类型标注为 ``Any`` 以避免 ``report`` ``diagnostics`` 模块
循环导入运行时返回 ``DiagnosticReport | None``
"""
from .diagnostics import diagnose as _diagnose
return _diagnose(self)
# ---- 性能剖面 ----------------------------------------------------- #
def profile(self, graph: Graph) -> ProfileReport:
"""生成性能剖面报告。
基于 ``started_at``/``finished_at`` 时间戳进行离线分析零运行时
开销需要传入执行时的 :class:`Graph` 对象以计算依赖关系与关键路径
参数
----
graph:
执行 ``self`` 时使用的图
返回
----
:class:`pyflowx.profiling.ProfileReport`
示例
----
report = px.run(graph)
profile = report.profile(graph)
print(profile.describe())
"""
from .profiling import ProfileReport
return ProfileReport.from_report(self, graph)
# ---- 序列化 ------------------------------------------------------- #
def to_dict(
self,
value_serializer: Callable[[Any], Any] | None = None,
) -> dict[str, Any]:
"""转换为 JSON 兼容的字典。
参数
----
value_serializer:
自定义任务值序列化函数``None`` 时尝试 JSON 兼容转换
失败回退到 ``repr()``
返回结构包含顶层 ``run_id````success````summary`` ``results``
列表按完成顺序保序每个结果项含 name/status/value/error/
attempts/started_at/finished_at/duration_seconds/reason/tags/
depends_on 字段
"""
results_list: list[dict[str, Any]] = []
for name, r in self.results.items():
results_list.append(
{
"name": name,
"status": r.status.value,
"value": _serialize_value(r.value, value_serializer),
"error": repr(r.error) if r.error else None,
"attempts": r.attempts,
"started_at": r.started_at.isoformat() if r.started_at else None,
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
"duration_seconds": r.duration,
"reason": r.reason,
"tags": list(r.spec.tags),
"depends_on": list(r.spec.depends_on),
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
}
)
return {
"run_id": self.run_id,
"success": self.success,
"summary": self.summary(),
"results": results_list,
}
def to_json(
self,
indent: int = 2,
value_serializer: Callable[[Any], Any] | None = None,
) -> str:
"""序列化为 JSON 字符串。
参数
----
indent:
JSON 缩进空格数``0`` 表示紧凑单行
value_serializer:
自定义任务值序列化函数透传给 :meth:`to_dict`
"""
return dumps(
self.to_dict(value_serializer),
ensure_ascii=False,
indent=indent if indent > 0 else None,
)
def to_csv(self, include_value: bool = True) -> str:
"""导出为 CSV 字符串。
列顺序``name, status, attempts, duration_seconds, started_at,
finished_at, error, reason`` + 可选 ``value``
参数
----
include_value:
是否包含 value 任务返回值可能含逗号/换行符干扰解析
不需要值时设为 ``False`` 得到更干净输出
"""
buf = io.StringIO()
writer = csv.writer(buf)
header = [
"name",
"status",
"attempts",
"duration_seconds",
"started_at",
"finished_at",
"error",
"reason",
]
if include_value:
header.append("value")
writer.writerow(header)
for name, r in self.results.items():
row = [
name,
r.status.value,
r.attempts,
f"{r.duration:.6f}" if r.duration is not None else "",
r.started_at.isoformat() if r.started_at else "",
r.finished_at.isoformat() if r.finished_at else "",
repr(r.error) if r.error else "",
r.reason or "",
]
if include_value:
row.append(_serialize_value(r.value))
writer.writerow(row)
return buf.getvalue()
def to_html(
self,
include_value: bool = True,
value_serializer: Callable[[Any], Any] | None = None,
) -> str:
"""导出为自包含的 HTML 报告字符串。
生成完整的 HTML5 文档内联 CSS零外部依赖适合直接在浏览器
打开或邮件附件分享所有用户内容任务名错误原因均经
:func:`html.escape` 转义避免 XSS
参数
----
include_value:
是否在表格中展示任务返回值值可能较大或含特殊字符
不需要时设为 ``False`` 得到更紧凑报告
value_serializer:
自定义任务值序列化函数透传给 :func:`_serialize_value`
``None`` 时尝试 JSON 兼容转换失败回退到 ``repr()``
"""
summary = self.summary()
status_badge_class = {
TaskStatus.SUCCESS: "status-success",
TaskStatus.FAILED: "status-failed",
TaskStatus.SKIPPED: "status-skipped",
TaskStatus.RUNNING: "status-running",
TaskStatus.PENDING: "status-pending",
}
parts: list[str] = [
"<!DOCTYPE html>",
'<html lang="zh-CN">',
"<head>",
'<meta charset="utf-8">',
'<meta name="viewport" content="width=device-width, initial-scale=1">',
f"<title>PyFlowX 运行报告 {html.escape(self.run_id)}</title>",
"<style>",
"body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;"
"margin:0;padding:20px;background:#f5f5f5;color:#222;}",
".container{max-width:1100px;margin:0 auto;}",
".summary{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));"
"gap:12px;margin-bottom:24px;}",
".card{background:#fff;border-radius:8px;padding:16px;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
".card h3{margin:0 0 8px;font-size:13px;color:#666;font-weight:600;text-transform:uppercase;}",
".card .value{font-size:22px;font-weight:700;color:#222;}",
".card.success .value{color:#16a34a;}",
".card.failure .value{color:#dc2626;}",
"table{width:100%;border-collapse:collapse;background:#fff;border-radius:8px;"
"overflow:hidden;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
"th,td{padding:10px 12px;text-align:left;border-bottom:1px solid #eee;font-size:13px;}",
"th{background:#fafafa;font-weight:600;color:#555;}",
"tr:last-child td{border-bottom:none;}",
"td.name{font-family:monospace;font-weight:600;}",
".badge{display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;}",
".status-success{background:#dcfce7;color:#16a34a;}",
".status-failed{background:#fee2e2;color:#dc2626;}",
".status-skipped{background:#fef9c3;color:#ca8a04;}",
".status-running{background:#dbeafe;color:#2563eb;}",
".status-pending{background:#f3f4f6;color:#6b7280;}",
".error{color:#dc2626;font-family:monospace;font-size:12px;}",
".reason{color:#ca8a04;font-size:12px;}",
".value-cell{font-family:monospace;font-size:12px;max-width:300px;"
"overflow:hidden;text-overflow:ellipsis;white-space:nowrap;}",
".empty{padding:32px;text-align:center;color:#999;}",
"</style>",
"</head>",
"<body>",
'<div class="container">',
f"<h1>PyFlowX 运行报告 <code>{html.escape(self.run_id)}</code></h1>",
'<div class="summary">',
self._render_summary_card(
"运行状态", "成功" if self.success else "失败", "success" if self.success else "failure"
),
self._render_summary_card("任务总数", str(summary["total_tasks"])),
self._render_summary_card("总耗时", f"{summary['total_duration_seconds']:.3f}s"),
self._render_summary_card("成功", str(summary["by_status"].get("success", 0))),
self._render_summary_card("失败", str(summary["by_status"].get("failed", 0))),
self._render_summary_card("跳过", str(summary["by_status"].get("skipped", 0))),
"</div>", # .summary
]
if not self.results:
parts.append('<div class="empty">本次运行无任务结果。</div>')
else:
header_cells = [
"<th>任务名</th>",
"<th>状态</th>",
"<th>尝试次数</th>",
"<th>耗时(s)</th>",
"<th>开始时间</th>",
"<th>结束时间</th>",
"<th>错误</th>",
"<th>原因</th>",
]
if include_value:
header_cells.append("<th>返回值</th>")
parts.append("<table>")
parts.append("<thead><tr>" + "".join(header_cells) + "</tr></thead>")
parts.append("<tbody>")
for name, r in self.results.items():
badge_cls = status_badge_class.get(r.status, "status-pending")
cells = [
f'<td class="name">{html.escape(name)}</td>',
f'<td><span class="badge {badge_cls}">{html.escape(r.status.value)}</span></td>',
f"<td>{r.attempts}</td>",
f"<td>{r.duration:.6f}</td>" if r.duration is not None else "<td>-</td>",
f"<td>{html.escape(r.started_at.isoformat())}</td>" if r.started_at else "<td>-</td>",
f"<td>{html.escape(r.finished_at.isoformat())}</td>" if r.finished_at else "<td>-</td>",
f'<td class="error">{html.escape(repr(r.error))}</td>' if r.error else "<td>-</td>",
f'<td class="reason">{html.escape(r.reason)}</td>' if r.reason else "<td>-</td>",
]
if include_value:
val = _serialize_value(r.value, value_serializer)
val_str = dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
cells.append(f'<td class="value-cell">{html.escape(val_str)}</td>')
parts.append("<tr>" + "".join(cells) + "</tr>")
parts.append("</tbody></table>")
parts.extend(["</div>", "</body>", "</html>"])
return "\n".join(parts)
@staticmethod
def _render_summary_card(label: str, value: str, modifier: str = "") -> str:
"""渲染单个汇总卡片 HTML 片段。"""
cls = f"card {modifier}" if modifier else "card"
return f'<div class="{cls}"><h3>{html.escape(label)}</h3><div class="value">{html.escape(value)}</div></div>'
@classmethod
def from_json(cls, text: str) -> RunReport:
"""从 JSON 字符串重建 :class:`RunReport`。
注意
----
只能重建可序列化字段:class:`TaskSpec` ``fn``/``cmd``
不可序列化字段无法恢复重建的 spec 仅含 ``name``/``tags``/
``depends_on``重建后的 report 仅供查询/分析**不能用于重新执行**
"""
data = loads(text)
report = cls(
success=data.get("success", True),
run_id=data.get("run_id", uuid4().hex[:8]),
)
for entry in data.get("results", []):
spec: TaskSpec[Any] = TaskSpec(
name=entry["name"],
fn=_noop_fn,
tags=tuple(entry.get("tags", ())),
depends_on=tuple(entry.get("depends_on", ())),
outputs=dict(entry["outputs"]) if entry.get("outputs") else None,
)
started = entry.get("started_at")
finished = entry.get("finished_at")
result: TaskResult[Any] = TaskResult(
spec=spec,
status=TaskStatus(entry["status"]),
value=entry.get("value"),
attempts=entry.get("attempts", 0),
started_at=datetime.fromisoformat(started) if started else None,
finished_at=datetime.fromisoformat(finished) if finished else None,
reason=entry.get("reason"),
)
report.results[spec.name] = result
return report
+40 -4
View File
@@ -14,10 +14,9 @@ from __future__ import annotations
import argparse
import enum
import sys
from collections.abc import Sequence
from dataclasses import dataclass, field
from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any, get_args
from typing import Any, Sequence, get_args
from .compose import GraphComposer
from .errors import PyFlowXError
@@ -36,6 +35,39 @@ class CliExitCode(enum.IntEnum):
INTERRUPTED = 130 # 与 POSIX 信号中断一致
def _apply_verbose_to_graph(graph: Graph, verbose: bool) -> Graph:
"""创建新图, 其中所有 TaskSpec 的 verbose 字段被设置为指定值.
使用 ``dataclasses.replace`` 在不可变的 TaskSpec 上创建带 verbose 标记的副本.
依赖关系标签等元数据全部保留.
Note
-----
``_wrap_cmd`` 不再闭包捕获 ``verbose`` 此函数不再是必需的
直接翻转 ``spec.verbose`` 即可生效保留是为了向后兼容现有调用与测试
TaskSpec 仍是 frozen dataclass故仍用 ``replace`` 创建副本
Parameters
----------
graph : Graph
原始图.
verbose : bool
要设置的 verbose .
Returns
-------
Graph
所有 spec verbose 字段已更新的新图.
"""
new_specs: list[TaskSpec[Any]] = []
for spec in graph.all_specs().values():
if spec.verbose == verbose:
new_specs.append(spec)
else:
new_specs.append(replace(spec, verbose=verbose))
return Graph.from_specs(new_specs)
@dataclass
class CliRunner:
"""命令行运行器: 根据用户输入执行对应的任务流图.
@@ -264,8 +296,12 @@ class CliRunner:
# 确定是否 verbose: --quiet 覆盖默认值
verbose = self.verbose and not parsed.quiet
# 执行对应的图 (verbose 标记由 run() 统一应用到各 spec)
# 对图应用 verbose 设置 (重建带 verbose 标记的 spec)
graph = self.graphs[parsed.command]
if verbose:
graph = _apply_verbose_to_graph(graph, verbose=True)
# 执行对应的图
try:
report = run(
graph,
+15 -218
View File
@@ -13,23 +13,20 @@
from __future__ import annotations
import sqlite3
import json
import sys
import threading
import time
from abc import ABC, abstractmethod
from collections import OrderedDict
from collections.abc import Iterator, Mapping
from contextlib import AbstractContextManager, contextmanager, nullcontext
from collections.abc import Iterator
from contextlib import contextmanager, nullcontext
from pathlib import Path
from typing import Any
from typing import Any, ContextManager, Mapping
if sys.version_info >= (3, 12):
from typing import override
else:
from typing_extensions import override # pragma: no cover
from ._json import JSONDecodeError, dump, dumps, load, loads
from .errors import StorageError
@@ -59,14 +56,6 @@ class StateBackend(ABC):
def clear(self) -> None:
"""清除所有存储状态。"""
@abstractmethod
def invalidate(self, key: str) -> bool:
"""删除单个键的存储结果。
返回 ``True`` 表示键存在且已删除``False`` 表示键不存在
:meth:`clear` 不同仅删除指定键而非全部
"""
def flush(self) -> None: # noqa: B027
"""将内存中暂存的状态持久化到外部介质。
@@ -74,7 +63,7 @@ class StateBackend(ABC):
:class:`JSONBackend` :meth:`batch` 期间会延迟落盘需在退出时调用
"""
def batch(self) -> AbstractContextManager[None]:
def batch(self) -> ContextManager[None]:
"""返回一个上下文管理器,期间 :meth:`save` 可延迟 :meth:`flush`。
默认实现为 no-op :class:`MemoryBackend`:class:`JSONBackend`
@@ -115,10 +104,6 @@ class _TTLStateBackendMixin(StateBackend):
def _clear_raw(self) -> None:
"""清空所有记录。"""
@abstractmethod
def _delete_raw(self, key: str) -> bool:
"""删除单条记录。返回 ``True`` 表示存在且已删除,``False`` 表示不存在。"""
# ---- 共享实现 ---- #
def _now(self) -> float:
"""当前时间戳,默认为 wall-clock 秒。"""
@@ -154,10 +139,6 @@ class _TTLStateBackendMixin(StateBackend):
def clear(self) -> None:
self._clear_raw()
@override
def invalidate(self, key: str) -> bool:
return self._delete_raw(key)
class MemoryBackend(_TTLStateBackendMixin):
"""进程内 dict 后端。进程退出即丢失。
@@ -167,20 +148,11 @@ class MemoryBackend(_TTLStateBackendMixin):
ttl:
条目存活秒数``None`` 表示永不过期``has`` 在条目超过 ttl
返回 ``False``但不主动删除下次 ``save`` 覆盖
maxsize:
最大条目数``None`` 表示不限超出时按 LRU最近最少使用
策略驱逐最旧条目每次 ``get``/``has`` 命中会将条目移到末尾
最近使用``save`` 新增条目时若超限则驱逐头部条目
"""
def __init__(
self,
ttl: float | None = None,
maxsize: int | None = None,
) -> None:
self._store: OrderedDict[str, tuple[Any, float]] = OrderedDict()
def __init__(self, ttl: float | None = None) -> None:
self._store: dict[str, tuple[Any, float]] = {}
self._ttl = ttl
self._maxsize = maxsize
@override
def _now(self) -> float:
@@ -188,23 +160,11 @@ class MemoryBackend(_TTLStateBackendMixin):
@override
def _get_raw(self, key: str) -> tuple[Any, float] | None:
entry = self._store.get(key)
if entry is None:
return None
# LRU:命中时移到末尾(最近使用)
self._store.move_to_end(key)
return entry
return self._store.get(key)
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
if key in self._store:
# 已存在:更新并移到末尾
self._store.move_to_end(key)
self._store[key] = (value, ts)
# LRU 驱逐:超限时移除头部(最旧)
if self._maxsize is not None:
while len(self._store) > self._maxsize:
self._store.popitem(last=False)
@override
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
@@ -215,13 +175,6 @@ class MemoryBackend(_TTLStateBackendMixin):
def _clear_raw(self) -> None:
self._store.clear()
@override
def _delete_raw(self, key: str) -> bool:
if key in self._store:
del self._store[key]
return True
return False
class JSONBackend(_TTLStateBackendMixin):
"""基于文件的 JSON 存储,用于跨进程续跑。
@@ -249,7 +202,7 @@ class JSONBackend(_TTLStateBackendMixin):
return
try:
with open(self._path, encoding="utf-8") as fh:
data: Any = load(fh)
data: Any = json.load(fh)
if isinstance(data, dict):
# 兼容纯值格式与带元数据格式
self._store = {}
@@ -258,14 +211,14 @@ class JSONBackend(_TTLStateBackendMixin):
self._store[k] = v
else:
self._store[k] = {"value": v, "ts": time.time()}
except (OSError, JSONDecodeError) as exc:
except (OSError, json.JSONDecodeError) as exc:
raise StorageError(f"cannot read state file {self._path!r}", exc) from exc
def _flush(self) -> None:
tmp = self._path + ".tmp"
try:
with open(tmp, "w", encoding="utf-8") as fh:
dump(self._store, fh, ensure_ascii=False, indent=2)
json.dump(self._store, fh, ensure_ascii=False, indent=2)
_ = Path(tmp).replace(Path(self._path))
except (OSError, TypeError) as exc:
raise StorageError(f"cannot write state file {self._path!r}", exc) from exc
@@ -290,34 +243,17 @@ class JSONBackend(_TTLStateBackendMixin):
def _clear_raw(self) -> None:
self._store.clear()
@override
def _delete_raw(self, key: str) -> bool:
if key in self._store:
del self._store[key]
return True
return False
@override
def clear(self) -> None:
super().clear()
self._flush()
@override
def invalidate(self, key: str) -> bool:
removed = super().invalidate(key)
if removed and not self._defer_flush:
self._flush()
return removed
@override
def save(self, key: str, value: Any) -> None:
# batch 模式下延迟序列化验证到 flush 时(一次性 json.dump 整个 _store),
# 避免 N 次 save 的 N 次 json.dumps 开销;非 batch 模式仍即时验证以提供精确错误。
if not self._defer_flush:
try:
_ = dumps(value)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
try:
_ = json.dumps(value)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
super().save(key, value)
if not self._defer_flush:
self._flush()
@@ -332,8 +268,6 @@ class JSONBackend(_TTLStateBackendMixin):
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
将整次运行 N 个任务的 N 次全量落盘降为 1
batch 期间 ``save`` 跳过逐次 JSON 序列化验证 flush 时的
``json.dump`` 统一捕获不可序列化的值
"""
self._defer_flush = True
try:
@@ -343,143 +277,6 @@ class JSONBackend(_TTLStateBackendMixin):
self._flush()
class SQLiteBackend(_TTLStateBackendMixin):
"""基于 SQLite 文件的状态后端。
相比 :class:`JSONBackend` 的全量加载/落盘:class:`SQLiteBackend` 按键
查询按行写入适合大规模任务状态10k+ 条目使用标准库
:mod:`sqlite3`无新依赖
值以 JSON 文本存入 ``value`` 因此与 :class:`JSONBackend` 一样要求
结果可 JSON 序列化``ts`` 列存储写入时间戳用于 TTL 判断
线程安全内部用 :class:`threading.RLock` 序列化所有 sqlite3 调用
sqlite3 连接对象本身非线程安全支持 ``thread``/``async``/``dependency``
策略下的并发存取
Parameters
----------
path:
SQLite 文件路径``":memory:"`` 创建内存数据库进程退出丢失
适合测试
ttl:
条目存活秒数``None`` 表示永不过期``has`` 在条目超过 ttl
返回 ``False``但不主动删除下次 ``save`` 覆盖
"""
def __init__(self, path: str, ttl: float | None = None) -> None:
self._path = path
self._ttl = ttl
self._lock = threading.RLock()
self._in_batch = False
try:
self._conn = sqlite3.connect(path, check_same_thread=False)
with self._lock:
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.execute("PRAGMA synchronous=NORMAL")
self._conn.execute(
"CREATE TABLE IF NOT EXISTS state (key TEXT PRIMARY KEY, value TEXT NOT NULL, ts REAL NOT NULL)"
)
self._conn.execute("CREATE INDEX IF NOT EXISTS idx_state_ts ON state(ts)")
self._conn.commit()
except sqlite3.DatabaseError as exc:
raise StorageError(f"cannot open sqlite state file {path!r}", exc) from exc
@override
def _get_raw(self, key: str) -> tuple[Any, float] | None:
with self._lock:
try:
row = self._conn.execute("SELECT value, ts FROM state WHERE key = ?", (key,)).fetchone()
except sqlite3.DatabaseError as exc:
raise StorageError(f"cannot read key {key!r} from sqlite state", exc) from exc
if row is None:
return None
value_text, ts = row
try:
value = loads(value_text)
except JSONDecodeError as exc:
raise StorageError(f"cannot decode value for key {key!r}", exc) from exc
return value, float(ts)
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
try:
value_text = dumps(value, ensure_ascii=False)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
with self._lock:
try:
self._conn.execute(
"INSERT OR REPLACE INTO state (key, value, ts) VALUES (?, ?, ?)",
(key, value_text, ts),
)
if not self._in_batch:
self._conn.commit()
except sqlite3.DatabaseError as exc:
raise StorageError(f"cannot write key {key!r} to sqlite state", exc) from exc
@override
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
with self._lock:
try:
rows = self._conn.execute("SELECT key, value, ts FROM state").fetchall()
except sqlite3.DatabaseError as exc:
raise StorageError("cannot iterate sqlite state", exc) from exc
for k, value_text, ts in rows:
try:
value = loads(value_text)
except JSONDecodeError as exc:
raise StorageError(f"cannot decode value for key {k!r}", exc) from exc
yield k, value, float(ts)
@override
def _clear_raw(self) -> None:
with self._lock:
try:
self._conn.execute("DELETE FROM state")
self._conn.commit()
except sqlite3.DatabaseError as exc:
raise StorageError("cannot clear sqlite state", exc) from exc
@override
def _delete_raw(self, key: str) -> bool:
with self._lock:
try:
cursor = self._conn.execute("DELETE FROM state WHERE key = ?", (key,))
if not self._in_batch:
self._conn.commit()
return cursor.rowcount > 0
except sqlite3.DatabaseError as exc:
raise StorageError(f"cannot delete key {key!r} from sqlite state", exc) from exc
@override
def flush(self) -> None:
with self._lock:
try:
self._conn.commit()
except sqlite3.DatabaseError as exc:
raise StorageError("cannot flush sqlite state", exc) from exc
@override
@contextmanager
def batch(self) -> Iterator[None]:
"""进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。
将整次运行 N 个任务的 N commit 降为 1
"""
self._in_batch = True
try:
yield
finally:
self._in_batch = False
self.flush()
def close(self) -> None:
"""关闭数据库连接,释放文件句柄。"""
with self._lock:
self._conn.close()
def resolve_backend(backend: StateBackend | None) -> StateBackend:
"""返回 ``backend``;为 ``None`` 时返回新的 :class:`MemoryBackend`。"""
return backend if backend is not None else MemoryBackend()
+28 -87
View File
@@ -22,15 +22,22 @@ import os
import shutil
import sys
import threading
from collections.abc import Callable, Coroutine, Generator, Iterable, Mapping
from contextlib import AbstractContextManager, contextmanager
from contextlib import contextmanager
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from functools import cached_property
from pathlib import Path
from typing import (
Any,
Callable,
ContextManager,
Coroutine,
Generator,
Generic,
List,
Mapping,
Union,
cast,
)
@@ -42,18 +49,25 @@ else:
T = TypeVar("T", default=Any)
# 任务可调用对象可以是同步或异步的。显式保留联合类型,让 mypy 理解两种形态。
TaskFn = Callable[..., T] | Callable[..., Coroutine[Any, Any, T]]
TaskFn = Union[
Callable[..., T],
Callable[..., Coroutine[Any, Any, T]],
]
# 跨任务结果映射。值刻意使用 ``Any``,因为不同任务返回不同类型;
# 单任务类型由函数签名本身保留。
Context = Mapping[str, Any]
# 命令类型支持
TaskCmd = list[str] | str | Callable[..., Any]
TaskCmd = Union[
List[str], # 命令列表, 如 ["ls", "-la"]
str, # shell 命令字符串
Callable[..., Any], # Python 函数
]
# 执行策略:sequential/thread/async 为层屏障模型,dependency 为依赖驱动模型。
Strategy = Union[str, "StrategyKind"]
StrategyKind = Any # 占位,避免循环;executors 模块用 Literal 约束
Strategy = str | StrategyKind
logger = logging.getLogger(__name__)
@@ -64,65 +78,6 @@ Condition = Callable[[Context], bool]
CacheKeyFn = Callable[[Context], str]
# ---------------------------------------------------------------------- #
# 任务循环
# ---------------------------------------------------------------------- #
@dataclass(frozen=True, slots=True)
class LoopSpec:
"""任务循环展开规格。
添加到 :class:`Graph` ``loop`` ``None`` :class:`TaskSpec`
会被展开为多实例每个 ``item`` 生成一个新 spec``item`` 追加到
``args`` 末尾``loop`` 字段清空
展开后的实例名默认为 ``f"{name}_{i}"````i`` 0 起的索引
也可通过 ``key_fn`` 自定义其他任务引用 loop 原名时``Graph``
会自动重写为依赖所有展开实例
参数
----
items:
待迭代的项元组
key_fn:
接受 ``(index, item)``返回实例名``None`` 时用
``f"{name}_{i}"``
Examples
--------
>>> from pyflowx import LoopSpec
>>> spec = px.task(process, loop=LoopSpec.range(3))
>>> # 等价于手动创建 process_0, process_1, process_2 三个任务
"""
items: tuple[Any, ...]
key_fn: Callable[[int, Any], str] | None = None
def __post_init__(self) -> None:
if not self.items:
raise ValueError("LoopSpec.items 不能为空。")
@classmethod
def range(cls, *args: int) -> LoopSpec:
"""构造整数序列循环(签名与 :func:`range` 一致)。
支持三种形式
- ``range(stop)`` ``[0, 1, ..., stop-1]``
- ``range(start, stop)`` ``[start, ..., stop-1]``
- ``range(start, stop, step)`` 按步长递增/递减
"""
# classmethod 内的 ``range`` 查找内置函数,不引用本类 LoopSpec.range
return cls(items=tuple(range(*args)))
@classmethod
def from_iterable(
cls,
items: Iterable[Any],
key_fn: Callable[[int, Any], str] | None = None,
) -> LoopSpec:
"""从可迭代对象构造循环。"""
return cls(items=tuple(items), key_fn=key_fn)
def _format_skip_reason(failed_conditions: list[str]) -> str:
"""格式化跳过原因:≤2 个全展示,>2 个仅展示前 2 个并附总数。"""
if len(failed_conditions) <= 2:
@@ -133,7 +88,7 @@ def _format_skip_reason(failed_conditions: list[str]) -> str:
# ---------------------------------------------------------------------- #
# 重试策略
# ---------------------------------------------------------------------- #
@dataclass(frozen=True, slots=True)
@dataclass(frozen=True)
class RetryPolicy:
"""任务失败重试策略。
@@ -196,7 +151,7 @@ class RetryPolicy:
# ---------------------------------------------------------------------- #
# 任务钩子
# ---------------------------------------------------------------------- #
@dataclass(frozen=True, slots=True)
@dataclass(frozen=True)
class TaskHooks:
"""任务生命周期钩子。
@@ -220,7 +175,7 @@ class TaskStatus(Enum):
SKIPPED = "skipped" # 用于断点续跑与子图过滤
@dataclass(frozen=True, slots=True)
@dataclass(frozen=True)
class TaskSpec(Generic[T]):
"""单个 DAG 节点的不可变描述。
@@ -303,9 +258,6 @@ class TaskSpec(Generic[T]):
同步任务的执行器``"thread"``默认线程池/ ``"process"``
进程池绕过 GIL适合 CPU 密集型``fn`` 须可 pickle/
``"inline"``直接在事件循环线程调用最快但会阻塞循环
outputs:
任务声明的输出键值映射YAML 编排场景使用仅作为元数据携带
不参与执行可通过 ``report.result_of(name).spec.outputs`` 查询
"""
name: str
@@ -332,9 +284,6 @@ class TaskSpec(Generic[T]):
cache_key: CacheKeyFn | None = None
hooks: TaskHooks = field(default_factory=TaskHooks)
executor: str = "thread" # "thread" | "process" | "inline"
outputs: Mapping[str, str] | None = None
loop: LoopSpec | None = None
dynamic: bool = False
def __post_init__(self) -> None:
if not self.name:
@@ -351,7 +300,7 @@ class TaskSpec(Generic[T]):
if self.fn is None and self.cmd is None:
raise ValueError(f"TaskSpec '{self.name}': 必须提供 fn 或 cmd 参数。")
@property
@cached_property
def effective_fn(self) -> TaskFn[T]:
"""获取有效的执行函数。
@@ -359,10 +308,8 @@ class TaskSpec(Generic[T]):
包装函数在每次调用时从 ``self`` 读取 ``verbose``/``cwd``/``env``/
``timeout``避免闭包捕获运行期参数使翻转字段无需重建 spec
.. note::
使用 ``@property`` 而非 ``cached_property`` 以兼容 ``slots=True``
内存优化10k+ 任务场景闭包创建极轻量仅捕获 ``self``
重复访问的开销可忽略
结果按实例缓存:func:`functools.cached_property`frozen dataclass
字段不可变``_wrap_cmd`` 生成的闭包稳定无需每次访问重建
"""
if self.cmd is not None:
return self._wrap_cmd()
@@ -429,7 +376,7 @@ class TaskSpec(Generic[T]):
return shutil.which(cmd[0]) is not None
return True
def env_context(self) -> AbstractContextManager[None]:
def env_context(self) -> ContextManager[None]:
"""返回临时应用 ``env`` 与 ``cwd`` 的上下文管理器。
``fn`` 任务生效``cmd`` 任务在 :func:`_run_command` 中直接
@@ -534,9 +481,6 @@ def task(
continue_on_error: bool = False,
cache_key: CacheKeyFn | None = None,
hooks: TaskHooks | None = None,
outputs: Mapping[str, str] | None = None,
loop: LoopSpec | None = None,
dynamic: bool = False,
name: str | None = None,
) -> Any:
"""装饰器:将函数转为 :class:`TaskSpec`。
@@ -578,9 +522,6 @@ def task(
continue_on_error=continue_on_error,
cache_key=cache_key,
hooks=hooks if hooks is not None else TaskHooks(),
outputs=dict(outputs) if outputs else None,
loop=loop,
dynamic=dynamic,
)
if fn is None and cmd is None:
@@ -656,7 +597,7 @@ def task_template(
return _factory
@dataclass(slots=True)
@dataclass
class TaskResult(Generic[T]):
"""运行期间产生的可变单任务记录。"""
@@ -677,7 +618,7 @@ class TaskResult(Generic[T]):
return (self.finished_at - self.started_at).total_seconds()
@dataclass(frozen=True, slots=True)
@dataclass(frozen=True)
class TaskEvent:
"""执行期间向观察者发出的不可变事件。"""
+1
View File
@@ -0,0 +1 @@

Some files were not shown because too many files have changed in this diff Show More