Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 02539d60fd |
+2
-1
@@ -38,8 +38,9 @@ env
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# 文档(按需保留)
|
||||
# 文档与示例(按需保留)
|
||||
docs
|
||||
examples
|
||||
|
||||
# 系统文件
|
||||
.DS_Store
|
||||
|
||||
@@ -27,14 +27,14 @@ jobs:
|
||||
run: uv build
|
||||
|
||||
- name: Publish to pypi
|
||||
run: uv publish --token '${{ secrets.PYPI_TOKEN }}'
|
||||
run: uv publish --token '${{ secrets.PYPI_API_TOKEN }}'
|
||||
|
||||
- name: Create Gitea Release & Upload Assets
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG_NAME: ${{ github.ref_name }}
|
||||
REPO: ${{ github.repository }}
|
||||
GITEA_URL: http://172.17.0.1:3000
|
||||
GITEA_URL: http://gitea:3000
|
||||
run: |
|
||||
set -e
|
||||
# 1. 创建 Release
|
||||
|
||||
@@ -11,7 +11,3 @@ wheels/
|
||||
.coverage
|
||||
.idea
|
||||
*_profile.html
|
||||
|
||||
# Sphinx 文档构建输出
|
||||
docs/_build/
|
||||
.trae/refs
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
# ReadTheDocs 配置
|
||||
# https://docs.readthedocs.io/en/stable/config-file/v2.html
|
||||
version: 2
|
||||
|
||||
# 构建配置
|
||||
build:
|
||||
os: ubuntu-24.04
|
||||
tools:
|
||||
python: "3.11"
|
||||
|
||||
# Python 依赖与构建命令
|
||||
python:
|
||||
install:
|
||||
- method: pip
|
||||
path: .
|
||||
extra_requirements:
|
||||
- docs
|
||||
|
||||
# Sphinx 构建
|
||||
sphinx:
|
||||
configuration: docs/conf.py
|
||||
builder: html
|
||||
fail_on_warning: false
|
||||
@@ -1,93 +0,0 @@
|
||||
# 迭代 21 — P9 功能扩展(任务编排/数据流/持久化/监控)
|
||||
|
||||
## 本轮目标
|
||||
|
||||
延续 P8 性能优化后,扩展 PyFlowX 在四个方向的核心能力:
|
||||
|
||||
1. **P9.1 任务编排增强**:任务分组(`Graph.group`)+ 动态任务生成(`TaskSpec(dynamic=True)`)。
|
||||
2. **P9.2 数据流增强**:`RunReport.output_of` 命名输出提取 + `Graph.pipeline` 语义别名。
|
||||
3. **P9.3 持久化与恢复**:`run(resume_from=...)` 检查点恢复 + `RunHistory` 历史管理。
|
||||
4. **P9.4 监控导出**:`MetricsCollector`(Prometheus 文本)+ `health_check` + `start_metrics_server`(零依赖 HTTP)。
|
||||
|
||||
## 改动文件清单
|
||||
|
||||
### 新增
|
||||
|
||||
- `src/pyflowx/monitoring.py` — `MetricsCollector` / `health_check` / `start_metrics_server`。
|
||||
- `src/pyflowx/history.py` — `RunHistory` 文件系统历史管理。
|
||||
- `tests/test_group.py` — 13 用例(P9.1.2 任务分组)。
|
||||
- `tests/test_dynamic.py` — 11 用例(P9.1.3 动态任务)。
|
||||
- `tests/test_dataflow.py` — 12 用例(P9.2 数据流)。
|
||||
- `tests/test_persistence.py` — 15 用例(P9.3 检查点/历史)。
|
||||
- `tests/test_monitoring.py` — 16 用例(P9.4 监控导出)。
|
||||
|
||||
### 修改
|
||||
|
||||
- `src/pyflowx/task.py` — `TaskSpec` 新增 `dynamic: bool = False` 字段;`task()` 工厂新增 `dynamic` 参数。
|
||||
- `src/pyflowx/graph.py` — 新增 `group()` 方法;`_rewrite_deps_for_loops` 改名 `_rewrite_deps`,统一处理 `_loop_groups` + `_groups`;新增 `pipeline()` 语义别名。
|
||||
- `src/pyflowx/report.py` — 新增 `_resolve_output_path` 辅助 + `RunReport.output_of(name, output)` 方法。
|
||||
- `src/pyflowx/executors.py` —
|
||||
- `DependencyRunner.execute` 增加 while 循环 + `f.exception()` 手动传播(替代 `asyncio.gather`)。
|
||||
- 新增 `_extract_dynamic_specs` 辅助;`_run_task` 检测动态生成、注册派生 spec、启动派生 task。
|
||||
- `_dispatch_strategy` 增加 `has_dynamic` 校验(仅 `dependency` 策略支持)。
|
||||
- 新增 `_load_resume_report` + `_apply_resume` 辅助;`run()` 接受 `resume_from` 参数。
|
||||
- `_filter_and_sort` + `_run_task` 检查点跳过逻辑。
|
||||
- `src/pyflowx/__init__.py` — 导出 `RunHistory` / `MetricsCollector` / `health_check` / `start_metrics_server`。
|
||||
|
||||
## 关键决策与依据
|
||||
|
||||
### 1. 动态任务仅支持 `dependency` 策略
|
||||
|
||||
`dynamic=True` 允许任务 `fn` 运行时返回 `TaskSpec | list[TaskSpec]`,DependencyRunner 中途注册并调度派生任务。层模型(sequential/thread/async)的层屏障与动态扩展语义冲突,因此 `_dispatch_strategy` 显式拒绝其他策略。
|
||||
|
||||
依据:层模型按拓扑分层执行,动态任务可能跨层;依赖驱动模型无层屏障,可自然容纳运行时新增节点。
|
||||
|
||||
### 2. `asyncio.wait` 替代 `asyncio.gather` + 手动异常传播
|
||||
|
||||
`asyncio.gather` 在 first-failure 时取消其他任务并 re-raise,但所有 task 必须在调用前确定。动态任务场景下 `futures` 字典会随执行增长,必须用 while 循环 + `asyncio.wait(return_when=FIRST_COMPLETED)` 增量调度。
|
||||
|
||||
陷阱:`asyncio.wait` **不会**自动 re-raise task 异常。需手动 `f.exception()` 检查并显式 `raise`,同时取消未完成任务,才能与 `gather` 的 fail-fast 行为一致。该陷阱导致 7 个测试失败后定位修复。
|
||||
|
||||
### 3. 检查点恢复:仅恢复 SUCCESS 任务
|
||||
|
||||
`resume_from` 接受 `RunReport | str | Path`,仅恢复 `status == SUCCESS` 且名称在当前图中的任务。FAILED/SKIPPED 任务会重新执行。`_apply_resume` 抽取为独立函数以控制 `run()` 的 PLR0912 分支数(≤12)。
|
||||
|
||||
### 4. `metrics_text` 按 section 条件化输出
|
||||
|
||||
最初实现总是输出 `# HELP` / `# TYPE` 行(即便无数据)。这导致 `reset()` 后文本仍包含 `pyflowx_task_total` 字样,违反测试预期。改为所有 section(含 task_total/duration/duration_sum)都按数据存在性条件化,与 retries/run_total 一致。
|
||||
|
||||
### 5. `start_metrics_server` 用 `http.server` 零依赖
|
||||
|
||||
未引入 `prometheus_client` 等三方库,仅用标准库 `http.server.HTTPServer` + `BaseHTTPRequestHandler`。生产环境可叠加反向代理或替换为 `prometheus_client`。`@override` 装饰器按 Python 版本条件导入(`typing` 3.12+ / `typing_extensions` <3.12)。
|
||||
|
||||
### 6. `RunHistory` 文件命名与 `__contains__`/`__len__`
|
||||
|
||||
按 `{run_id}.json` 命名存储;`__contains__` / `__len__` 委托文件存在性检查,避免维护内存索引。`latest()` 按 mtime 排序取最新。
|
||||
|
||||
## 验证结果
|
||||
|
||||
```
|
||||
ruff check . → All checks passed!
|
||||
ruff format --check → 126 files already formatted
|
||||
pyrefly check . → 0 errors (68 suppressed)
|
||||
pytest --cov=pyflowx → 1525 passed in 11.56s
|
||||
coverage → 97.58%(branch;门槛 95%)
|
||||
```
|
||||
|
||||
各模块覆盖率(P9 新增部分):
|
||||
- `monitoring.py` 99%(仅 `typing_extensions` 回退分支未覆盖)
|
||||
- `history.py` 95%
|
||||
- `executors.py` 97%(dynamic/resume 分支已覆盖)
|
||||
- `report.py` 99%(`output_of` 路径解析已覆盖)
|
||||
- `graph.py` 98%(`group`/`pipeline`/`_rewrite_deps` 已覆盖)
|
||||
|
||||
## 遗留事项
|
||||
|
||||
- 监控导出未实现 OpenTelemetry 协议(当前仅 Prometheus 文本格式)。如需 OTLP,可在后续迭代引入 `opentelemetry-sdk`(违反零依赖原则,需用户确认)。
|
||||
- `RunHistory` 未实现压缩/归档;大量历史运行可能占用较多磁盘。后续可加 `max_entries` 或滚动归档。
|
||||
- 动态任务当前仅在 DependencyRunner 中实现;如层模型用户也需要,需额外设计跨层调度策略。
|
||||
- `_apply_resume` 仅恢复 `report.results`,未恢复 `backend` 缓存;如需检查点+缓存联合恢复,需额外设计。
|
||||
|
||||
## 归档说明
|
||||
|
||||
iter-21 暂不归档(前 5 次 iter-16~20 已归档至 `.trae/skills/pyflowx-development/SKILL.md`)。下次清理窗口为 iter-26 前夕。
|
||||
@@ -1,23 +0,0 @@
|
||||
# 开发流程约束
|
||||
|
||||
本规则补充 `self-driven.md`,定义执行过程记录与规则变更的两条硬约束。
|
||||
|
||||
## 执行过程记录
|
||||
|
||||
- **每次迭代**(一个完整的"计划 → 实现 → 测试 → 文档 → 验证"闭环)必须记录到
|
||||
`.trae/docs/` 目录,文件名形如 `iter-NN-<主题>.md`,`NN` 为零填充序号。
|
||||
- 记录内容至少包括:本轮目标、改动文件清单、关键决策与依据、验证结果、遗留事项。
|
||||
- **每 5 次迭代后**清理 `.trae/docs/`:
|
||||
- 把需要长期保留的整合性内容(可复用模式、踩坑总结、设计决策)
|
||||
归档到 `.trae/skills/` 对应技能文档中;
|
||||
- 其余过程性记录删除;
|
||||
- docs 目录只保留最近未归档的迭代记录(即第 6 次迭代开始前,清理 1–5 号记录)。
|
||||
- 归档动作需在迭代记录中标注"已归档至 skills/<文件>"。
|
||||
|
||||
## 规则变更约束
|
||||
|
||||
- 任何对 `.trae/rules/` 下文件的修改(新增、编辑、删除)**必须先询问用户**,
|
||||
获得明确授权后方可变动;**未授权前不得擅自修改 rules**。
|
||||
- 用户在对话中显式要求写入的规则内容(如"把 X 写入 rules")视为已授权,
|
||||
可直接写入,写入后在回复中说明改了哪个文件、加了什么。
|
||||
- 规则变更后需同步更新 `project_memory.md` 中的"开发约定"小节。
|
||||
@@ -25,7 +25,7 @@ uvx --from pyflowx pymake cov
|
||||
- **最低 Python 3.8**:用 `from __future__ import annotations` 延迟注解求值;
|
||||
按版本用 `typing.List`(3.8) → 内置泛型(3.9) → `X | Y`(3.10) → `typing.override`(3.12)。
|
||||
- **版本守卫**:`if sys.version_info >= (3, X):` 引入高版本 API;低版本回退分支加 `# pragma: no cover`。
|
||||
- **零运行时依赖**:仅依赖标准库(需 `typing-extensions` 用于 `override`/`TypeVar` 前向兼容)。
|
||||
- **零运行时依赖**:仅依赖标准库(3.8 需 `graphlib_backport`、`typing-extensions`)。
|
||||
新增依赖须审慎,优先用标准库。
|
||||
|
||||
## 类型注解
|
||||
@@ -150,7 +150,7 @@ uvx --from pyflowx pymake cov
|
||||
|
||||
## Git 与提交
|
||||
|
||||
- **自动提交**:任务完成后自动 `git add`(按文件名)+ `git commit` + `git push`(仅当分支已跟踪远程时执行 push;新分支跳过 push 并在总结中说明)。
|
||||
- **不自动提交/push**:除非用户明确要求。
|
||||
- **不修改 git config**。
|
||||
- **不运行破坏性命令**(`push --force`/`reset --hard`/`clean -f`)除非用户明确要求。
|
||||
- **staging**:按文件名添加,不用 `git add -A`/`git add .`,避免误加敏感文件。
|
||||
|
||||
@@ -1,151 +0,0 @@
|
||||
---
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# 自驱动开发规则
|
||||
|
||||
本规则定义一种"目标驱动、闭环执行"的工作模式:仅在任务开始时与用户确认一次目标与边界,后续由 Agent 自主完成"计划 → 编码 → 测试 → 文档 → 验证"的迭代循环,直到用户目标达成。
|
||||
|
||||
## 核心原则
|
||||
|
||||
- **目标导向**:始终以用户最终目标为准绳,所有阶段产出都应服务于该目标。
|
||||
- **闭环执行**:每个子任务必须走完"计划 → 实现 → 测试 → 文档 → 验证"五步;禁止跳步留半成品。
|
||||
- **自主决策**:初始确认之后,实现路径、API 形态、重构范围、文件命名、测试组织、错误修复策略等由 Agent 自行决断,不再逐项请示。**可逆操作(编辑文件、运行测试、修复 lint、调整实现)直接执行,不询问**;只有不可逆/高风险操作才暂停。
|
||||
- **透明沟通**:每个阶段开始前用一句话说明意图;关键节点(完成、阻塞、转向)给简短更新;不复述内部思考,**不在收尾时停下询问"是否继续"或"是否提交"**——直接输出总结并结束。
|
||||
- **安全边界**:仅在高风险、不可逆操作或真正阻塞时才暂停找用户。
|
||||
|
||||
## 初始确认(一次性,仅在最开始)
|
||||
|
||||
任务启动时,用 `AskUserQuestion` 一次性确认以下信息(已由项目规范覆盖的不必重复确认):
|
||||
|
||||
1. **目标与范围**:要解决什么问题?交付物是什么?显式列出不在范围内的内容。
|
||||
2. **验收标准**:怎样算"完成"?可观测的判定条件(功能、性能、覆盖率阈值)。
|
||||
3. **特殊约束**:除 `python-standards.md` 之外的约束(兼容性、依赖限制、API 兼容策略等)。
|
||||
4. **测试要求**:覆盖率门槛(项目默认 ≥95%,branch);是否需要新增 `slow` 标记。
|
||||
|
||||
**git commit/push 不在确认范围内**:任务完成后自动 commit + push(仅当分支已跟踪远程时执行 push;新分支跳过 push 并在总结中说明),遵循 `.trae/rules/git-commit-message.md` 风格。仅 force-push、reset --hard、clean -f、修改 git config 等真正破坏性操作才需暂停确认。
|
||||
|
||||
确认后,将目标与验收标准固化进 `TaskCreate` 任务列表,后续不再就同一信息反复询问。
|
||||
|
||||
## 迭代循环
|
||||
|
||||
下列五个阶段构成一个完整闭环。未达验收标准时,回到「计划」开启下一轮;达标准时,进入「收尾」。
|
||||
|
||||
### 1. 计划(Plan)
|
||||
|
||||
- 用 Explore/Glob/Grep 研究相关代码与既有模式,避免凭空设计。
|
||||
- 用 `TaskCreate` 把目标拆为可独立验证的子任务;每完成一项立即 `TaskUpdate` 为 completed。
|
||||
- 优先复用现有抽象;不为本轮假想需求设计接口。
|
||||
- 不过早抽象:三处相似才考虑提取,否则就地写。
|
||||
|
||||
### 2. 实现(Code)
|
||||
|
||||
- 严格遵守 `.trae/rules/python-standards.md` 与既有代码风格。
|
||||
- 优先 Edit 现有文件;新增文件需有明确职责边界。
|
||||
- 不引入运行时依赖(项目零依赖原则);确需引入须在计划阶段说明。
|
||||
- 公共 API 必须有完整类型注解与中文 docstring。
|
||||
- 不写未被要求的功能、不为未来场景预留扩展点。
|
||||
|
||||
### 3. 测试(Test)
|
||||
|
||||
- 新增/修改的公共 API 必须配套测试;优先通过公共接口测试,故障注入可访问私有属性并在 docstring 注明。
|
||||
- Mock 优先级:`monkeypatch` > 内联 stub > `unittest.mock` > `pytest-mock`;禁用 `@patch` 装饰器。
|
||||
- 必跑校验(每次修改后):
|
||||
|
||||
```bash
|
||||
uvx --from pyflowx pymake tc
|
||||
uvx --from pyflowx pymake cov
|
||||
```
|
||||
|
||||
- 测试失败时定位根因再修复,不通过放宽断言或 `# pragma: no cover` 绕过。
|
||||
- 覆盖率不得低于上一次的值(项目门槛 95%,branch)。
|
||||
|
||||
### 4. 文档(Docs)
|
||||
|
||||
- 同步更新 docstring、README、模块结构说明。
|
||||
- 行为变更须同步更新 `.agents/skills/pyflowx-development/SKILL.md` 中的对应章节。
|
||||
- 跨会话有价值的设计决策、约束、陷阱,追加到 memory(`project_memory.md` 或对应 `topics.md`)。
|
||||
- 不主动新建 `*.md` 文档;除非用户明确要求。
|
||||
|
||||
### 5. 验证(Verify)
|
||||
|
||||
- 逐条对照初始确认的「验收标准」核验;未满足则回到「计划」继续下一轮。
|
||||
- 全套门禁通过:ruff、pyrefly、pytest、coverage。
|
||||
- 给出本轮变更清单(改了哪些文件、为什么)。
|
||||
|
||||
## 多阶段项目
|
||||
|
||||
当项目按阶段(P0/P1/P2/...)划分时,初始确认的目标是**整个项目**,而非单个阶段。
|
||||
|
||||
- **阶段是里程碑,不是边界**:确认"推进 P1"只是指定下一步重点,不缩小整体目标范围;
|
||||
也不意味着 P1 完成后就可以停下。整体目标 = 所有阶段交付完毕。
|
||||
- **阶段完成不是停止条件**:某阶段验收标准全部满足后,**自动进入下一阶段的「计划」步骤**,
|
||||
不停下来输出"是否继续"或"是否扩展范围"的询问。仅在阶段切换时用一句话说明
|
||||
"进入 Pn,重点:…",然后继续自驱动迭代。
|
||||
- **「收尾」仅适用于整个项目完成**:只有所有阶段(含最后一个)都交付后,才执行
|
||||
「收尾」输出最终总结。单阶段完成 → 进入下一阶段的「计划」,不触发收尾。
|
||||
- **跨阶段依赖**:若下一阶段依赖外部资源(如新依赖审批、环境配置),属"不可恢复的失败"
|
||||
暂停条件;否则一律连续推进。
|
||||
- **阶段范围超出预期**:执行中发现某阶段需要显著扩大范围或改变方向(如 P2 本来只做
|
||||
表格抽取,发现必须先重写 IR),属"超出初始确认范围"暂停条件,需找用户确认。
|
||||
|
||||
## 暂停条件(仅在以下情况中断自驱动找用户)
|
||||
|
||||
1. **歧义无法自决**:需求存在多种合理解读且无既有约定可循。
|
||||
2. **高风险/不可逆操作**:删除非临时文件、`git push --force`、`reset --hard`、删表、修改 CI 配置、修改 git config、卸载依赖等。**普通 `git commit`/`push` 不属于此类**(任务完成后自动执行)。
|
||||
3. **不可恢复的失败**:根因不在本仓库、需外部环境/权限配合、或经两轮尝试仍无法定位。
|
||||
4. **超出初始确认范围**:用户目标在执行中发现需要显著扩大范围或改变方向。
|
||||
5. **用户主动询问**:用户在对话中提出新问题或要求澄清。
|
||||
|
||||
**注意**:"目标已达成"**不是**暂停条件——但"目标已达成"指**整个项目目标**(所有阶段交付完毕),而非单个阶段验收满足。单阶段验收满足后应自动进入下一阶段(见「多阶段项目」),不触发收尾。
|
||||
|
||||
非以上情况,一律继续自驱动,不要为"求确认"而暂停。
|
||||
|
||||
## 决策判据:该问还是自决
|
||||
|
||||
遇到不确定时,按以下顺序判断:
|
||||
|
||||
1. **是否不可逆/高风险?** 是 → 暂停确认(如删除文件、`push --force`、修改 CI 配置、卸载依赖)。否 → 继续。
|
||||
2. **是否在初始确认范围内?** 是 → 按确认执行,不询问。否 → 视为"超出初始确认范围",暂停。
|
||||
3. **是否有既有约定可循?** 是 → 按约定执行(参考 `python-standards.md`、`project_memory.md`)。否 → 视为"歧义无法自决",暂停。
|
||||
4. **是否可逆?** 是 → 直接执行,即使结果可能不完美(可在后续迭代修正)。否 → 暂停。
|
||||
|
||||
**可直接自决(不询问)的典型情况**:
|
||||
|
||||
- 测试失败、覆盖率不达标、lint/类型检查报错 → 定位根因并修复。
|
||||
- 代码风格选择(命名、模块划分、参数顺序)→ 自决。
|
||||
- 文件编辑、运行测试、运行校验命令 → 直接执行。
|
||||
- 任务完成后输出收尾总结 → 直接输出,不询问下一步。
|
||||
- 显式指定 `name` 参数以保持测试兼容性 → 自决。
|
||||
- 重命名局部变量以避免遮蔽 → 自决。
|
||||
|
||||
**必须暂停询问的典型情况**:
|
||||
|
||||
- 删除非临时文件、重命名公共模块/包。
|
||||
- `git push --force`、`reset --hard`、`clean -f`、修改 git config(普通 commit/push 自动执行,无需询问)。
|
||||
- 引入新的运行时依赖(违反项目零依赖原则)。
|
||||
- 修改 CI 配置、pre-commit 钩子、pyproject.toml 的工具链配置。
|
||||
- 卸载或降级既有依赖。
|
||||
|
||||
## 沟通风格
|
||||
|
||||
- 阶段切换时一句话说明即可;不要把内部推理写给用户看。
|
||||
- 完成子任务后用一两句总结改了什么、下一步做什么。
|
||||
- 遇到阻塞时直接说明:卡在哪、试了什么、需要用户做什么。
|
||||
- **不在收尾时询问"是否需要提交"或"是否扩展范围"**——直接输出总结并结束。用户后续若有新需求,由用户主动提出。
|
||||
- 不使用 emoji,除非用户明确要求。
|
||||
|
||||
## 工具使用
|
||||
|
||||
- 独立操作尽量并行调用(多个 Read/Grep/Glob 一批发出)。
|
||||
- 用 `TaskCreate`/`TaskUpdate` 维护进度,不批量推迟标记。
|
||||
- 长命令用后台运行(`run_in_background`),完成会自动通知。
|
||||
- 文件操作一律用专用工具:Read/Edit/Write/Glob/Grep,不用 `cat`/`sed`/`grep`/`find`。
|
||||
|
||||
## 收尾
|
||||
|
||||
- **仅当整个项目所有阶段都交付后**,才执行收尾:直接输出最终总结并结束任务(交付物、关键决策、遗留事项)。
|
||||
- 单阶段完成**不属于收尾**——见「多阶段项目」,应自动进入下一阶段的「计划」。
|
||||
- **自动提交**:收尾时自动 `git add`(按文件名)+ `git commit`(遵循 `.trae/rules/git-commit-message.md` 风格)+ `git push`(仅当分支已跟踪远程时执行;新分支跳过 push 并在总结中说明);**不询问**"是否需要提交"或"是否扩展范围"。
|
||||
- 若验收标准未全部满足,回到「计划」继续下一轮,不停下询问。
|
||||
- 将本次会话的关键产出与决策更新到 memory,便于后续会话续接。
|
||||
@@ -1,603 +0,0 @@
|
||||
---
|
||||
name: "pyflowx-development"
|
||||
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化相关的功能时参考。"
|
||||
---
|
||||
|
||||
# PyFlowX 开发知识库
|
||||
|
||||
本技能归档自迭代 06-20 的过程记录,按主题分类整理可复用知识。
|
||||
过程性细节(覆盖率数字、命令输出)已剔除,仅保留架构模式、设计依据与陷阱总结。
|
||||
|
||||
## 一、兼容性与代码清理
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **`typing_extensions` 版本守卫不可移除**:`override`(3.12+)与
|
||||
`TypeVar(default=)`(3.13+)在 3.10/3.11/3.12 上仍需 `typing_extensions`
|
||||
回退。守卫是前向兼容代码,不是 3.8 死代码。
|
||||
`pyproject.toml` 声明 `typing-extensions>=4.13.2; python_version < '3.13'`。
|
||||
- **`from __future__ import annotations` 保留**:PEP 563 在 3.10+ 仍提供
|
||||
前向引用与延迟求值价值。统一移除需逐文件审计前向引用、自引用类型,
|
||||
风险高收益低。
|
||||
- **`rules/` 修改必须先获用户授权**:`.trae/rules/` 下文件修改受
|
||||
`dev-workflow.md` 约束,未授权前不得擅自变动,即便描述已过时
|
||||
(如 "最低 Python 3.8" 实际已升至 >=3.10)。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **不要凭"3.8 兼容代码"假设做清理**:调研后才发现 `typing_extensions`
|
||||
守卫实际是 3.12/3.13 前向兼容,删除会破坏 3.10-3.12 上的 `override`
|
||||
与 `TypeVar(default=)`。清理前必须验证"过期代码"的真实用途。
|
||||
- **README/文档与代码状态必须同步**:未实现的特性声明(如未落地的
|
||||
YAML 编排章节)、过期的覆盖率徽章、错误的依赖描述都会误导用户。
|
||||
清理时优先对齐 README 与 `pyproject.toml`。
|
||||
|
||||
## 二、YAML 任务编排(GitHub Actions 风格)
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **Matrix 笛卡尔积展开**:多键矩阵用 `itertools.product` 展开为笛卡尔积,
|
||||
命名 `{job_id}_{key}-{value}`(多键用 `_` 连接)。
|
||||
如 `version: ["3.10","3.11"]` × `os: ["linux","macos"]` 展开 4 个任务
|
||||
`test_version-3.10_os-linux` 等。
|
||||
- **矩阵依赖自动展开**:若 job A `needs: [B]`,B 是矩阵 job,A 依赖 B 的
|
||||
所有变体(无需手动列出每个变体)。
|
||||
- **`${{ matrix.key }}` 占位符替换**:在 cmd/run/cwd/env 的字符串值中
|
||||
统一替换矩阵占位符。
|
||||
- **字段名 hyphen/underscore 兼容**:YAML 中 `continue-on-error` 与
|
||||
`continue_on_error` 等价,读取时统一规范化。
|
||||
- **cmd vs run 区分**:
|
||||
- `cmd: [...]` → list 形式(无 shell,直接 exec)
|
||||
- `run: "..."` → shell 字符串形式
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **条件解析映射到内置 Conditions**:
|
||||
- `success()` / `always()` → 无条件(空 conditions)
|
||||
- `env.VAR` → `ENV_VAR_EXISTS("VAR")`
|
||||
- `env.VAR == 'x'` → `ENV_VAR_EQUALS("VAR", "x")`
|
||||
- `env.VAR != 'x'` → `NOT(ENV_VAR_EQUALS("VAR", "x"))`
|
||||
- `failure()` → 不支持,抛 `ValueError`(PyFlowX 不支持 failure 触发)
|
||||
- **yamlrun 作为特殊命令而非 `@px.tool`**:CLI 路由
|
||||
`pf yamlrun <file> [--strategy S] [--dry-run] [--list] [--quiet] [--progress]`
|
||||
作为 `PfApp` 的 builtin 特殊命令注册,与 `pf graph` 同级,不通过 `@px.tool` 装饰器。
|
||||
- **Graph.from_yaml 委托 yaml_loader**:`Graph.from_yaml(path)` classmethod
|
||||
委托给 `yaml_loader.load_yaml(path)` 函数式 API,保持 Graph 类职责单一。
|
||||
|
||||
### Matrix include/exclude 语义(iter-19)
|
||||
|
||||
- **exclude**: 组合的所有 key-value 对都匹配则剔除(`combo.items() >= exclude.items()`)。
|
||||
如 `exclude: [{os: macos, version: "3.8"}]` 剔除 macos+3.8 组合。
|
||||
- **include**: 追加额外组合,可引入基础矩阵之外的键。
|
||||
如基础矩阵 `version: ["3.10"]`,`include: [{version: "3.11", experimental: true}]`
|
||||
追加 3.11 实验组合。
|
||||
- **无基础矩阵键时返回空列表而非 `[{}]`**:避免产生空名任务(`job_`)。
|
||||
|
||||
### 条件依赖(iter-19)
|
||||
|
||||
- **`needs` 项支持 `{job: name, if: expr}` 格式**:条件不满足时跳过该依赖
|
||||
(不加入 `depends_on`)。
|
||||
- 复用 `_parse_condition` 解析 `if` 表达式;条件求值时传空上下文 `{}`
|
||||
(`env` 条件直接读 `os.environ`)。
|
||||
- **典型场景**:仅在特定矩阵变体下才需要的依赖,如
|
||||
`needs: [{job: setup, if: env.DEPLOY == "true"}]`。
|
||||
|
||||
### outputs 字段(iter-19)
|
||||
|
||||
- `TaskSpec` 新增 `outputs: Mapping[str, str] | None` 作为元数据,
|
||||
仅携带不参与执行,可通过 `spec.outputs` 查询。
|
||||
- 支持 `${{ matrix.* }}` 占位符替换。
|
||||
- `to_dict` / `from_json` 往返保留;序列化向前兼容(旧 JSON 无此字段时回退 None)。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **YAML 加载失败统一包装**:`yaml_loader._safe_load` 应将 `yaml.YAMLError`
|
||||
包装为 `ValueError` 抛出,CLI 层统一捕获处理。否则 YAML 语法错误会
|
||||
以底层异常形式泄漏到用户面前。
|
||||
- **不可达防御代码应清理**:`_cartesian_product` 与 `_parse_condition` 中
|
||||
的防御性分支若逻辑上不可达,应删除而非保留 `# pragma: no cover`。
|
||||
|
||||
## 三、性能优化与基准套件
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **独立 `benchmarks/` 目录**(非 pytest 测试),用 `time.perf_counter` 计时,
|
||||
通过 `python -m benchmarks` CLI 入口运行。覆盖:
|
||||
- 图构建:10/100/500/1000 节点 DAG 的构建 + validate + layers
|
||||
- 任务执行:空 fn 任务 × 100/500,四种策略对比
|
||||
- 上下文注入:有/无依赖、有/无 Context 标注
|
||||
- 状态后端:MemoryBackend vs JSONBackend vs SQLiteBackend
|
||||
- **`layers()` 缓存模式**:Graph.layers() 每次 run() 都重算拓扑排序,
|
||||
应缓存。模式:
|
||||
- 添加 `_layers_cache` 私有字段
|
||||
- 首次调用计算并缓存
|
||||
- `add()` / `clear()` 等变更方法中失效缓存
|
||||
- 参照 `resolved_spec` 已有缓存模式
|
||||
- **cmd 任务快速路径**:cmd 任务的 `effective_fn` 是无参闭包 `_run()`,
|
||||
无需上下文注入。检测 `spec.fn is None and spec.cmd is not None` 时
|
||||
直接返回 `((), {})`,跳过 `build_call_args` 的签名内省与 dep_context 构建。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **性能特征(500 空任务)**:
|
||||
- `sequential`:514 ops/s(最快,无并发开销)—— 小任务/无 I/O 场景首选
|
||||
- `thread`:93 ops/s(线程池开销)—— I/O 密集型场景
|
||||
- `async`:44 ops/s(事件循环开销)—— 高并发 I/O 场景
|
||||
- `dependency`:42 ops/s(最大并行度但调度开销高)—— 复杂依赖图
|
||||
- **状态后端性能特征**:
|
||||
- `MemoryBackend`:save/load 600万/447万 ops/s(首选,无持久化需求时)
|
||||
- `JSONBackend`:save(batch=10)/load 3913/11.9万 ops/s
|
||||
- `SQLiteBackend`:save(batch=10)/load 9657/1.5万 ops/s(save 更快,load 较慢)
|
||||
- **缓存优化收益**:`layers()` 缓存命中 ~1500万 ops/s(~50000x 加速);
|
||||
cmd 快速路径 1130万 ops/s vs fn 无依赖 144万 ops/s(~8x 加速)。
|
||||
|
||||
### 自实现 Kahn 算法拓扑排序(iter-20)
|
||||
|
||||
- **方案**:模块级 `_topological_layers(deps)` 函数用 dict + list 直接计算
|
||||
入度与反向邻接,按层 BFS 处理。返回 `(layers, cycle_nodes)`:
|
||||
- 无环时 `cycle_nodes = None`
|
||||
- 有环时 `cycle_nodes` 为入度仍 >0 的节点列表(非精确环路径,仅指示存在环)
|
||||
- **额外优化**:`validate()` 顺带填充 `_layers_cache`,使后续 `layers()`
|
||||
直接命中缓存,避免二次计算 `_topological_layers`。
|
||||
- **结果**:diamond(1000) 图构建约 2.21ms(~452 ops/s)。
|
||||
|
||||
### build_call_args 双快速路径(iter-17)
|
||||
|
||||
- **快速路径 1(cmd 任务)**:`spec.fn is None and spec.cmd is not None and
|
||||
not spec.args and not spec.kwargs` → 直接返回 `((), {})`。
|
||||
- **快速路径 2(fn 无依赖)**:`not spec.depends_on and not spec.soft_depends_on
|
||||
and not spec.args and not spec.kwargs` → 调用 `_fn_no_dep_injection(fn)`
|
||||
获取 `@lru_cache(maxsize=1024)` 缓存的注入计划 `(context_params, error_param)`。
|
||||
- `context_params`:Context 标注参数名元组;存在必填无默认值参数时返回 `None`
|
||||
(需走慢路径)。
|
||||
- `error_param`:第一个无默认值且非 Context 标注的参数名(用于错误信息)。
|
||||
- **不变量**:`error_param is None ⟺ context_params 非 None`,用 `assert`
|
||||
辅助 pyrefly 类型收窄。
|
||||
- **提取到 `_try_fast_path` 函数**:避免 `build_call_args` 分支数超 PLR0912
|
||||
阈值 12。
|
||||
- **不可哈希 fn 回退**:`_cached_signature` 抛 TypeError 时回退到直接内省。
|
||||
|
||||
### JSONBackend batch 延迟验证(iter-17)
|
||||
|
||||
- **问题**:`save` 原本每次都 `json.dumps(value)` 验证可序列化性,batch
|
||||
模式下 N 次 save 产生 N 次验证开销,但 `flush` 只做一次 `json.dump`。
|
||||
- **方案**:`_defer_flush=True` 时跳过 `json.dumps` 验证,让 `flush` 的
|
||||
`json.dump` 统一捕获 `TypeError`/`ValueError`。非 batch 模式仍即时验证
|
||||
以提供精确错误位置。
|
||||
- **依据**:非 batch 调用方期望即时错误反馈;batch 调用方接受延迟错误以换吞吐。
|
||||
|
||||
### from_specs 批量注册(iter-17)
|
||||
|
||||
- **问题**:`from_specs` 原本对每个 spec 调 `_register`,而 `_register`
|
||||
每次都 `_resolved_cache.clear()` + `_layers_cache = None`,N 个 spec
|
||||
产生 N 次缓存清空。
|
||||
- **方案**:直接写入 `graph.specs` / `graph.deps` 字典,重复名检测在循环内做。
|
||||
缓存失效由后续 `validate()` / `layers()` 首次调用自然触发(cache 为空时
|
||||
自动重算)。
|
||||
- **不扩展到 `add`/`chain`**:增量 API 需即时校验以快速失败;批量构造可
|
||||
接受延迟校验。
|
||||
|
||||
### orjson 可选依赖抽象层(iter-20)
|
||||
|
||||
- **方案**:`_json.py` 模块 `try: import orjson except ImportError: import json`
|
||||
回退。导出 `dumps`/`loads`/`dump`/`load`/`JSONDecodeError`,签名与 stdlib
|
||||
兼容。
|
||||
- **安装**:`pip install pyflowx[fast]` 启用 orjson(`[project.optional-dependencies]
|
||||
fast = ["orjson>=3.10.0"]`)。
|
||||
- **orjson 兼容性差异**:
|
||||
- `dumps` 返回 `bytes` → wrapper 中 `.decode("utf-8")` 统一为 `str`
|
||||
- 无 `ensure_ascii` 参数 → wrapper 接受并忽略(`# noqa: ARG001`),orjson 始终 UTF-8
|
||||
- 无 `indent` 参数 → 映射为 `OPT_INDENT_2`
|
||||
- 紧凑模式无空格(`[1,2,3]` vs stdlib `[1, 2, 3]`)→ 测试断言改为格式无关
|
||||
- `JSONDecodeError` 直接复用 orjson 的(是 stdlib `json.JSONDecodeError` 子类)
|
||||
- **pyrefly 兼容**:`import orjson` 在 try 块内,所有 `orjson.xxx` 引用加
|
||||
`# type: ignore[possibly-unbound]`;orjson 未安装时 `# type: ignore[import-not-found]`。
|
||||
|
||||
### slots=True 内存优化(iter-20)
|
||||
|
||||
- **方案**:所有 dataclass 添加 `slots=True`,消除 per-instance `__dict__`
|
||||
开销(约 100-150 bytes/instance)。
|
||||
- **应用范围**:`Graph`/`GraphDefaults`/`TaskSpec`/`TaskResult`/`RetryPolicy`/
|
||||
`TaskHooks`/`TaskEvent`。
|
||||
- **冲突处理**:`frozen=True` + `slots=True` + `cached_property` 不兼容
|
||||
(无 `__dict__` 存缓存,frozen 禁止设 slot)→ 将 `TaskSpec.effective_fn`
|
||||
从 `@cached_property` 改为 `@property`。闭包创建极轻量(仅捕获 `self`),
|
||||
重复访问开销可忽略。
|
||||
- **结果**:10k TaskSpec 929 → 785 bytes/instance(15.5% 减少,1.38 MB 节省)。
|
||||
|
||||
### 线程池跨层复用(iter-20)
|
||||
|
||||
- **问题**:`thread` 策略每层创建新 `ThreadPoolExecutor`,N 层图创建/销毁
|
||||
N 次线程池。
|
||||
- **方案**:`_drive_threaded` 创建一个池,跨所有层复用;`max_workers` 取
|
||||
最大层宽 `max((len(layer) for layer in layers), default=1)`,上限 32。
|
||||
- **API 变更**:`ThreadedLayerRunner.execute` 签名 `max_workers: int` →
|
||||
`pool: concurrent.futures.ThreadPoolExecutor`,调用方负责池生命周期。
|
||||
- **额外**:`asyncio.get_event_loop()` → `asyncio.get_running_loop()`,
|
||||
兼容 Python 3.12+ 弃用警告(调用点均在 `asyncio.run()` 内的协程中)。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **`lru_cache` 对签名内省有 dict lookup 开销**:即便 `functools.lru_cache`
|
||||
缓存了 `_signature(fn)`,每次仍有 dict lookup。对热路径(如 cmd 任务)
|
||||
应在更外层短路,避免进入 `build_call_args`。
|
||||
- **`frozen=True` + `slots=True` + `cached_property` 三者不兼容**:选其中
|
||||
两个。若需 slots 内存优化,将 `cached_property` 改为 `@property`(轻量
|
||||
闭包场景)或外部 dict 缓存(重计算场景)。
|
||||
- **orjson 紧凑格式与 stdlib 不同**:跨后端测试需用格式无关断言
|
||||
(`assert "[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
|
||||
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
|
||||
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
|
||||
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
|
||||
|
||||
## 四、任务取消与优雅停止
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`CancelToken` 基于 `threading.Event`**:线程安全取消令牌,可跨线程/协程使用。
|
||||
接口:
|
||||
- `cancel()` 设置取消状态
|
||||
- `is_cancelled` 属性检查
|
||||
- `wait(timeout)` 等待取消信号
|
||||
- **统一取消检查**:`_is_cancelled(effective_cancel) or internal_cancel.is_cancelled`
|
||||
统一外部取消与 KeyboardInterrupt 的内部取消,避免各处写两段检查逻辑。
|
||||
- **各策略取消点设计**:
|
||||
- `sequential`:每个任务执行前检查
|
||||
- `thread`:每层提交前检查;已提交任务等待完成(不强制中断)
|
||||
- `async`:每个 task 创建前检查;用 `asyncio.CancelledError`
|
||||
- `dependency`:每个 `_run_task` 创建前检查
|
||||
- **KeyInterrupt 优雅处理模式**:
|
||||
```python
|
||||
try:
|
||||
execute(...)
|
||||
except KeyboardInterrupt:
|
||||
cancel_event.set()
|
||||
# 等待运行中任务完成(最多 5 秒)
|
||||
# 标记未完成任务为 SKIPPED
|
||||
report.success = False
|
||||
finally:
|
||||
cleanup(...) # 进程池、监控器、通知器
|
||||
```
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **取消后待运行任务标记 SKIPPED,运行中任务等待完成**:不强制中断运行中
|
||||
的任务,避免资源泄漏与状态不一致。这是"优雅停止"的核心定义。
|
||||
- **`run()` 接受可选 `cancel_event`**:调用方未提供时内部创建。这让
|
||||
外部代码可以预先持有 cancel_event 引用,在任意时刻触发取消。
|
||||
- **提取 `_dispatch_strategy` 辅助函数控制分支数**:`run()` 函数中
|
||||
四种策略的分发若用 if/elif 链会导致 PLR0912(分支过多)。提取辅助
|
||||
函数将 `run()` 的 return 数量控制在 12 以内。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **资源清理必须在 finally 块**:进程池、监控器、通知器若在 try 块内
|
||||
清理,KeyboardInterrupt 会跳过清理导致泄漏。
|
||||
- **`PLR0912`/`PLR0911` 分支/返回过多**:复杂调度逻辑容易触发。优先
|
||||
提取辅助函数(如 `_dispatch_strategy`、`_print_diagnostics`),
|
||||
而非用 `# noqa` 抑制。合并语义相同的分支(如 no-args 与 --help
|
||||
合并)也是有效手段。
|
||||
|
||||
## 五、子图执行
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`Graph.subgraph_with_deps(names)` 传递闭包**:从 `names` 出发,
|
||||
沿 `depends_on` + `soft_depends_on` 向上遍历,收集所有必须先完成的任务。
|
||||
返回新的 Graph 或任务名集合。
|
||||
- **`run(only, tags)` 参数组合**:
|
||||
- `only`:任务名列表,运行这些任务及其传递依赖
|
||||
- `tags`:标签列表,运行匹配任意标签的任务及其传递依赖
|
||||
- 两者可组合:并集后计算传递闭包
|
||||
- 都为 None 时运行全图(保持现有行为不变)
|
||||
- **CLI 选项设计**:`pf yamlrun pipeline.yaml --only task1,task2 --tags ingest,report`
|
||||
逗号分隔,与执行过滤解耦。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **传递闭包同时包含软依赖**:`subgraph_with_deps` 沿 `depends_on` 与
|
||||
`soft_depends_on` 双向遍历。软依赖也是必须先完成的任务,不可遗漏。
|
||||
|
||||
## 六、结果序列化
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **值序列化默认策略**(处理任意 Python 对象):
|
||||
1. `None` 直接返回 `None`
|
||||
2. 可 JSON 序列化的原样返回
|
||||
3. 不可序列化的回退到 `repr(value)`
|
||||
4. 调用方可通过 `value_serializer` 自定义
|
||||
- **`to_dict` 结构用列表保序**:`results` 为列表(非 dict),便于
|
||||
CSV/JSON 数组导出,保留执行顺序。
|
||||
字段:`name/status/value/error/attempts/started_at/finished_at/duration_seconds/reason/tags/depends_on`。
|
||||
- **CSV 导出省略 value 列**:`include_value=False` 时省略 value 列,
|
||||
因为值可能含逗号/换行符干扰 CSV 解析。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **`from_json` 仅供查询,不能重新执行**:反序列化只重建可序列化字段,
|
||||
`TaskSpec` 的 `fn`/`cmd` 等不可恢复,使用 noop 占位。重建后的 report
|
||||
仅供查询/分析,不能用于重新执行。这是明确的契约边界。
|
||||
- **`datetime` 通过 `fromisoformat` 还原**:ISO 8601 字符串往返转换,
|
||||
避免自定义时间格式。
|
||||
|
||||
### HTML 报告导出设计(iter-19)
|
||||
|
||||
- **自包含 HTML5 文档**:内联 CSS,无外部依赖,便于邮件附件/嵌入式展示。
|
||||
- **XSS 防护**:所有用户内容(任务名/错误/原因/值)经 `html.escape` 转义。
|
||||
- **状态徽章着色**:success 绿 / failed 红 / skipped 黄 / running 蓝,
|
||||
通过 CSS class 区分。
|
||||
- **`_render_summary_card` 静态方法**:渲染汇总卡片(总数/成功/失败/跳过/
|
||||
耗时),避免在 `to_html` 主流程中重复 HTML 拼接。
|
||||
- **value 列渲染**:尝试 `dumps(value)` 序列化后转义;失败回退 `repr(value)`。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **`_noop_fn` 占位函数体无法覆盖**:与 `task.py:_task_noop` 同模式,
|
||||
占位函数的函数体在测试中无法触发,属于可接受的覆盖率缺口。
|
||||
- **orjson 紧凑格式与 stdlib 不同**:HTML 报告中 value 列断言需格式无关
|
||||
(`"[1,2,3]" in s or "[1, 2, 3]" in s`),不能硬编码空格。
|
||||
|
||||
## 七、CLI 可视化与体验
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`pf graph` 多格式输出**:
|
||||
- `--format ascii`(默认):`rich.tree.Tree` 按层分组 + `rich.table.Table` 显示依赖/标签
|
||||
- `--format mermaid`:复用 `Graph.to_mermaid()` 直接打印(不加颜色,便于复制到 mermaid.live)
|
||||
- `--format list`:仅列任务名(与 `yamlrun --list` 一致)
|
||||
- `--format deps`:纯依赖关系表
|
||||
- `--orientation`:Mermaid 方向(TD/TB/BT/LR/RL)
|
||||
- **`--color-by tag` 着色模式**:维护 `tag_colors` 字典(标签 → rich 颜色名),
|
||||
首次见到标签时从调色板循环分配。无标签任务回退默认色。`color_by="none"`
|
||||
时统一颜色,不显示图例。
|
||||
- **`--list` 过滤与执行过滤解耦**:
|
||||
- `--only`/`--tags` 控制执行范围
|
||||
- `--list-tag`/`--list-name` 仅控制 `--list` 模式的显示
|
||||
- 名称匹配大小写不敏感(子串),标签匹配精确,两者可组合(交集)
|
||||
- **`pf info [tool]` 工具详情**:
|
||||
- 无参数 → 所有工具简表(工具/别名/子命令数/描述)
|
||||
- 带参数 → 工具详情(别名、描述、子命令表)
|
||||
- 支持别名解析(`pf info clearscreen` 等价于 `pf info clr`)
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **shell 补全不依赖第三方库**:直接生成静态脚本
|
||||
- bash: `complete -F _pf_complete pf` + `compgen -W "..."`
|
||||
- zsh: `#compdef pf` + `_describe 'command' cmds`
|
||||
- fish: 每个命令一行 `complete -c pf -n "__fish_use_subcommand" -a "..."`
|
||||
工具名来自 `_TOOL_ALIASES` 规范名集合,二级补全交给子命令的 `--help`。
|
||||
- **`--progress` 与 `--quiet` 互斥**:quiet 时无进度条,避免输出冲突。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **闭包内可变计数器用 `counter[0]` 单元素 list**:pyrefly 在闭包内对
|
||||
父作用域变量的读-写支持不佳,`nonlocal` 会触发类型检查告警。改用
|
||||
`counter = [0]` 单元素 list,闭包内 `counter[0] += 1` 修改的是 list
|
||||
内容(非绑定),pyrefly 接受。
|
||||
- **`_TOOL_REGISTRY` 的 key 类型为 `str | None`**:排序时必须过滤 `None`,
|
||||
否则 `sorted()` 会因类型比较失败抛错。
|
||||
|
||||
## 八、错误诊断
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **诊断数据结构(frozen dataclass)**:
|
||||
- `FailureCluster`:`pattern`(异常类型+消息前缀)/`tasks`/`sample_error`
|
||||
- `DependencyChain`:`failed_task`/`root_cause`/`chain`/`broken_at`
|
||||
- `DiagnosticReport`:`failed_tasks`/`skipped_due_to_failure`/`root_causes`/`dependency_chains`/`failure_clusters`/`hints`
|
||||
- **根因识别算法**:
|
||||
- **根因任务**:FAILED 状态且其所有硬依赖都是 SUCCESS(或无依赖)
|
||||
- **依赖链**:从根因到失败任务的路径,用 BFS 反向遍历依赖图
|
||||
- **broken_at**:链中第一个非 SUCCESS 的任务(通常是根因本身)
|
||||
- **相似失败聚类**:按 `(异常类型名, 消息前 50 字符)` 聚类,
|
||||
便于发现批量失败模式。
|
||||
- **根因提示表**:识别常见异常类型并给出建议
|
||||
- `FileNotFoundError` → 检查路径配置
|
||||
- `ImportError`/`ModuleNotFoundError` → 检查依赖安装
|
||||
- `TimeoutError`/`TaskTimeoutError` → 考虑增加 timeout 或优化性能
|
||||
- `ConnectionError` → 检查网络连通性
|
||||
- `PermissionError` → 检查权限
|
||||
- `KeyError` → 检查上下文注入参数
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **`TaskFailedError` 携带 `report` 字段**:失败时 `_finalize_failure`
|
||||
传入 report,CLI 层可从异常对象直接获取诊断信息,无需重复构造。
|
||||
`report` 字段可选(`Optional[RunReport]`),向后兼容。
|
||||
- **`report.diagnose()` 成功时返回 `None`**:明确区分"无诊断"与"空诊断",
|
||||
避免调用方误判。
|
||||
- **CLI 失败时自动打印诊断摘要到 stderr**:`pf yamlrun` 失败时
|
||||
(`report.success == False`)自动调用 `_print_diagnostics` 辅助方法,
|
||||
无需用户额外参数。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **CLI 异常处理合并同类**:`TaskFailedError` 与 `PyFlowXError` 的
|
||||
处理逻辑相似,应合并到同一 except 块,提取 `_print_diagnostics`
|
||||
辅助方法。重复的 except 分支会增加 PLR0911 风险。
|
||||
|
||||
## 九、观测性
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`run_id` 短码生成**:`uuid4().hex[:8]` 生成 8 字符十六进制短码,
|
||||
足以区分并发运行,便于人工阅读。`RunReport.run_id` 字段默认自动生成,
|
||||
显式指定时尊重调用方。
|
||||
- **结构化日志 extra 字段标准化**(不用 `LoggerAdapter`):
|
||||
- `run_id` — 运行 ID(所有日志)
|
||||
- `task_name` — 任务名(任务级日志)
|
||||
- `status` — 任务状态(success/failed/skipped/cancelled)
|
||||
- `attempts` — 尝试次数(失败日志)
|
||||
- `error_type` — 异常类型名(失败日志)
|
||||
- `strategy` / `total_tasks` — run() 入口/结束日志
|
||||
- `run_id` 从 `report` 参数读取,无 report 上下文时填 `"-"`
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **放弃 `LoggerAdapter` 方案**:会强制改写所有 `logger.xxx` 调用点,
|
||||
且对模块级函数(`_apply_cached`/`_handle_failure`)签名侵入大。
|
||||
改用直接在每条日志显式 `extra={...}`,散点 extra 已足够,无需过度抽象。
|
||||
- **`profile()` 便捷方法委托 ProfileReport**:
|
||||
```python
|
||||
def profile(self, graph: Graph) -> ProfileReport:
|
||||
from .profiling import ProfileReport
|
||||
|
||||
return ProfileReport.from_report(self, graph)
|
||||
```
|
||||
`Graph` / `ProfileReport` 仅在 `TYPE_CHECKING` 块导入,避免循环依赖;
|
||||
`from __future__ import annotations` 使注解延迟求值。
|
||||
- **序列化向前兼容**:`from_json()` 优先恢复 `run_id`,缺失时自动生成
|
||||
新 ID。旧版 JSON(无 `run_id`)可正常加载。
|
||||
|
||||
### 通知器平台扩展(iter-19)
|
||||
|
||||
- **复用 `WebhookNotifier` 基类**:Slack/Discord/Telegram/Feishu/DingTalk/
|
||||
WeChatNotifier 均继承 `WebhookNotifier`,仅覆盖 `_build_payload(payload: dict) -> dict`
|
||||
适配平台格式。零新依赖(stdlib `urllib.request`)。
|
||||
- **TelegramNotifier 覆盖 `__init__`**:额外需要 `chat_id` 参数,
|
||||
URL 中拼接 `bot_token`。
|
||||
- **`Notifier` Protocol 生命周期**:`notify(level, payload)` + `close()`。
|
||||
`NotificationLevel` 枚举(RUNNING/SUCCESS/FAILED/SKIPPED)+ `ALL_LEVELS`
|
||||
frozenset;`levels` 参数内部过滤。
|
||||
- **WebhookNotifier 通用字段**:`url`/`secret`/`levels`/`timeout`,
|
||||
`_send` 用 `urllib.request.urlopen` 发送 JSON POST,`_build_payload`
|
||||
默认透传(子类按平台格式重写)。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **循环依赖用 `TYPE_CHECKING` 守卫 + 延迟导入**:`report.py` 依赖
|
||||
`profiling.py` 的 `ProfileReport` 与 `graph.py` 的 `Graph`,但运行时
|
||||
不需要。`TYPE_CHECKING` 块导入 + `from __future__ import annotations`
|
||||
注解延迟求值,是打破循环依赖的标准模式。
|
||||
- **不要为未来场景预留扩展点**:`LoggerAdapter` + `extra` 的统一封装
|
||||
属于"未来可考虑"项,当前散点 extra 已足够,不应预先抽象。
|
||||
|
||||
## 十、API 设计通用模式
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **classmethod 委托函数式 API**:如 `Graph.from_yaml(path)` 委托
|
||||
`yaml_loader.load_yaml(path)`,保持类职责单一,函数式 API 可独立复用。
|
||||
- **frozen dataclass 表达不可变结果**:`FailureCluster`/
|
||||
`DependencyChain`/`DiagnosticReport` 等只读结果用 `@dataclass(frozen=True)`,
|
||||
自动获得 `__hash__` 与 `__eq__`,适合作为字典 key 或集合元素。
|
||||
- **列表保序而非 dict**:当顺序有意义(如执行顺序、依赖链路径)时,
|
||||
用 list 而非 dict 存储结果,便于序列化数组导出。
|
||||
- **可选参数 None 哨兵**:`run(only=None, tags=None)`,None 表示
|
||||
"不限制",与空列表 `[]` 语义区分(空列表通常表示"无匹配")。
|
||||
- **提取辅助函数控制分支数**:`run()`、CLI 命令分发等复杂函数容易
|
||||
触发 PLR0911/PLR0912。优先提取 `_dispatch_strategy`/
|
||||
`_print_diagnostics`/`_print_task_list` 等辅助函数,而非用 noqa 抑制
|
||||
或合并语义不同的分支。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **公共 API 必须有完整类型注解与中文 docstring**:项目既有风格,
|
||||
`python-standards.md` 硬约束。
|
||||
- **零运行时依赖原则(YAML 例外)**:除 PyYAML(YAML 编排必需)、
|
||||
rich + typer(CLI 必需)、typing-extensions(前向兼容)、orjson(可选
|
||||
加速,`pip install pyflowx[fast]`)外,不引入运行时依赖。新增依赖须
|
||||
审慎,优先用标准库。
|
||||
|
||||
## 十一、状态后端与缓存
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`StateBackend` 抽象基类**:`get`/`put`/`has`/`iter`/`clear` +
|
||||
`invalidate(key) -> bool`(iter-19 新增)。`invalidate` 返回是否实际
|
||||
删除,便于调用方判断缓存命中。
|
||||
- **`_TTLStateBackendMixin` 4 原语**:`_get_raw`/`_put_raw`/`_iter_raw`/
|
||||
`_clear_raw`/`_delete_raw`(iter-19 新增)。子类只需实现这 5 个原语,
|
||||
`get`/`put`/`has`/`iter`/`clear`/`invalidate` 由 mixin 统一组合 TTL 判断。
|
||||
- **`MemoryBackend` LRU 驱逐(iter-19)**:
|
||||
- 用 `OrderedDict` 替代普通 dict
|
||||
- `_get_raw` 命中时 `move_to_end(key)` 更新访问顺序
|
||||
- `_put_raw` 超限时 `popitem(last=False)` 驱逐头部(最旧)
|
||||
- `maxsize=None` 时不驱逐(默认行为,向后兼容)
|
||||
- **`SQLiteBackend` WAL 模式 + RLock**:线程安全;`batch()` 通过
|
||||
`_in_batch` 标志延迟 commit,不持锁。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **`JSONBackend.invalidate` 覆盖**:非 batch 模式下需 `flush()` 落盘,
|
||||
否则删除只在内存中生效,进程崩溃后状态丢失。batch 模式下标记
|
||||
`_dirty=True`,由 `flush()` 统一处理。
|
||||
- **`SQLiteBackend._delete_raw` 用 `cursor.rowcount > 0`**:判断是否
|
||||
实际删除,避免先 `has` 后 `delete` 的双次查询。
|
||||
- **`maxsize` 默认 None 而非 0**:0 会立即驱逐所有项,等价于禁用缓存;
|
||||
None 表示无限制,与历史行为兼容。
|
||||
|
||||
## 十二、CLI 工具错误处理
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`subprocess.run` 的 `check` 参数分级策略**(iter-16):
|
||||
- **失败需感知**(dockercmd/msdownload/sglang)→ `check=True` +
|
||||
`try/except subprocess.CalledProcessError` 打印 rich 错误
|
||||
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查
|
||||
`returncode` 打印结果(pkill 返回 1 表示无匹配进程,非错误)
|
||||
- **失败无关紧要**(clr/which)→ `check=False`,加注释说明原因
|
||||
- **错误路径统一处理模式**:
|
||||
```python
|
||||
try:
|
||||
subprocess.run(cmd, check=True, ...)
|
||||
except subprocess.CalledProcessError as e:
|
||||
_console.print(f"[red]命令失败 (returncode={e.returncode})[/]")
|
||||
except FileNotFoundError:
|
||||
_console.print(f"[red]命令未找到: {cmd[0]}[/]")
|
||||
```
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **rich Console 而非 print**:CLI 工具错误用 `rich.console.Console`
|
||||
打印带颜色输出,与项目其他 CLI 工具一致。
|
||||
- **`check=True` + try/except 优于 `check=False` + 手动检查**:除非
|
||||
命令返回非零是预期行为(如 taskkill pkill 无匹配、which 未找到命令),
|
||||
否则用 `check=True` 让 subprocess 抛异常,统一在 except 中处理。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
|
||||
装饰器与 `mock.patch.object` 上下文。mock subprocess 异常用
|
||||
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
|
||||
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
|
||||
- **mock 异常抛出技巧**:`lambda *_, **__: (_ for _ in ()).throw(err)`
|
||||
利用生成器仅在迭代时抛异常的特性,让无参 lambda 能抛出指定异常。
|
||||
|
||||
## 十三、测试与质量保障
|
||||
|
||||
### 可复用模式
|
||||
|
||||
- **`# pragma: no cover` 清理策略**(iter-18):按 `python-standards.md`
|
||||
"pragma: no cover 是清理信号,应激活或删除"原则,逐处分析:
|
||||
- **逻辑上不可达** → 删除(如 `if result is None` 在已过滤的列表后)
|
||||
- **防御性代码但实际可触发** → 激活:移除 pragma,写测试覆盖该分支
|
||||
- **调用方已保证的前置条件** → 保留 pragma,加注释说明依赖契约
|
||||
- **覆盖率缺口分析方法**:
|
||||
1. 跑 `pytest --cov=src/pyflowx --cov-report=term-missing`
|
||||
2. 找出 `Stmts`/`Miss` 列中 Miss > 0 的文件
|
||||
3. 看 `Missing` 列定位未覆盖行号
|
||||
4. 判断是"未触发分支"还是"不可达代码"
|
||||
5. 前者补测试,后者清 pragma 或删代码
|
||||
- **测试命名**:`test_<被测对象>_<场景>`,如 `test_json_backend_flush_type_error`。
|
||||
|
||||
### 设计决策
|
||||
|
||||
- **公共 API 优先测试**:用 `has`/`get` 等公共接口测试,不访问私有方法。
|
||||
故障注入等场景可临时访问私有属性,docstring 注明原因。
|
||||
- **Mock 优先级**:`monkeypatch` > 内联 stub > `unittest.mock` >
|
||||
`pytest-mock`。禁用 `@patch` 装饰器、`mock.patch.object` 上下文、
|
||||
`pytest-mock` 的 `mocker` fixture。
|
||||
- **`slow` 标记**:耗时测试加 `@pytest.mark.slow`,CI 可 `-m "not slow"` 跳过。
|
||||
|
||||
### 踩坑总结
|
||||
|
||||
- **monkeypatch 目标需跟随 import 路径**:当模块从 `import json` 改为
|
||||
`from ._json import dump`,测试中 `monkeypatch.setattr(json, "dump", ...)`
|
||||
失效,需改为 `monkeypatch.setattr(storage_mod, "dump", ...)`。
|
||||
- **占位函数体无法覆盖**:`_noop_fn` 等占位函数的函数体在测试中无法触发
|
||||
(参数已校验),属可接受的覆盖率缺口,加 `# pragma: no cover` 并注释原因。
|
||||
- **不可哈希参数回退分支**:`lru_cache` 缓存的函数遇到不可哈希参数会抛
|
||||
TypeError,需 try/except 回退到慢路径。该回退分支需单独写测试(构造
|
||||
不可哈希 fn 触发)。
|
||||
@@ -5,8 +5,7 @@
|
||||
[](https://github.com/gookeryoung/pyflowx/actions/workflows/ci.yml)
|
||||
[](https://pypi.org/project/pyflowx/)
|
||||
[](https://pypi.org/project/pyflowx/)
|
||||
[](https://pyflowx.readthedocs.io/zh/latest/)
|
||||
[](https://github.com/gookeryoung/pyflowx)
|
||||
[](https://github.com/gookeryoung/pyflowx)
|
||||
[](https://github.com/gookeryoung/pyflowx/blob/main/LICENSE)
|
||||
|
||||
PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖声明**。无需装饰器、
|
||||
@@ -23,9 +22,8 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
|
||||
- **软依赖** —— `soft_depends_on` 仅用于上下文注入,不参与拓扑分层
|
||||
- **并发限制** —— `concurrency_key` + `concurrency_limits` 按组限流
|
||||
- **任务钩子** —— `TaskHooks`(pre_run/post_run/on_failure)生命周期回调
|
||||
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘;`invalidate(key)` 单键失效
|
||||
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘
|
||||
- **缓存键** —— `cache_key` 函数基于输入计算稳定键,使不同输入产生独立缓存
|
||||
- **缓存驱逐** —— `MemoryBackend(maxsize=N)` LRU 驱逐最旧条目;TTL 过期自动忽略
|
||||
- **命令任务** —— `cmd` 参数直接执行外部命令,支持列表/shell/可调用对象
|
||||
- **条件执行** —— `conditions` 参数按平台、环境变量、应用安装等条件跳过任务
|
||||
- **图组合** —— `compose` / `GraphComposer` 编程式展开多图字符串引用
|
||||
@@ -33,18 +31,7 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
|
||||
- **图级默认值** —— `GraphDefaults` 统一配置 retry/timeout/concurrency 等
|
||||
- **CLI 运行器** —— `CliRunner` 把多个图映射为命令行子命令,替代 Makefile
|
||||
- **可观测** —— `on_event` 回调(RUNNING/SUCCESS/FAILED/SKIPPED)、`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
|
||||
- **进度监控** —— `run(progress=True)` 开箱即用的 rich 进度条;`ProgressCallback` 协议供程序化集成
|
||||
- **任务通知** —— `Notifier` 协议支持飞书/钉钉/企业微信/Slack/Discord/Telegram webhook 与 Python 回调,全生命周期事件可配置
|
||||
- **流式获取** —— `run_iter()` 生成器逐个 yield 任务结果,适用于大量任务逐个处理
|
||||
- **任务取消** —— `CancelToken` 线程安全取消令牌,支持外部取消与 `KeyboardInterrupt` 优雅停止
|
||||
- **子图执行** —— `run(only=[...], tags=[...])` 按名称或标签运行图的子集,自动包含传递依赖
|
||||
- **结果序列化** —— `RunReport.to_json()` / `to_dict()` / `to_csv()` / `to_html()` 多格式导出运行结果,`from_json()` 支持反序列化重建;HTML 报告自包含内联 CSS,适合浏览器直接打开或邮件分享
|
||||
- **CLI 可视化** —— `pf graph <yaml>` 终端 DAG 渲染(ASCII 分层树按标签着色 / Mermaid / 依赖表),`pf yamlrun --progress` rich 进度条,`pf info [tool]` 工具详情,`pf completion bash|zsh|fish` shell 补全
|
||||
- **失败诊断** —— `RunReport.diagnose()` 自动分析根因、依赖链回溯、相似失败聚类与可操作建议;CLI 失败时自动打印诊断摘要
|
||||
- **可观测性** —— `run_id` 贯穿日志/序列化/诊断;结构化日志 `extra` 含 `task_name`/`status`/`attempts`/`error_type`;`profile(graph)` 离线性能分析(关键路径/并行度/瓶颈)
|
||||
- **状态后端** —— `MemoryBackend` / `JSONBackend` / `SQLiteBackend`(WAL 模式,适合大规模任务)
|
||||
- **YAML 任务编排** —— GitHub Actions 风格 `jobs`/`needs`/`strategy.matrix`/`if` 条件,`pf yamlrun pipeline.yaml` 一键执行
|
||||
- **最小依赖** —— `rich` + `typer` + `typing-extensions`(3.13 以下)+ `pyyaml`;安装 `orjson` 可加速 JSON 序列化(`pip install pyflowx[fast]`)
|
||||
- **零运行时依赖** —— 仅依赖标准库(3.8 需 `graphlib_backport`)
|
||||
- **97% 测试覆盖** —— 分支覆盖率 >= 95%
|
||||
|
||||
## 安装
|
||||
@@ -158,86 +145,6 @@ resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
|
||||
引用格式:`"command_name"`(整个图)或 `"command_name.task_name"`(特定任务)。
|
||||
`CliRunner` 内部自动调用 `compose`。
|
||||
|
||||
### YAML 任务编排
|
||||
|
||||
GitHub Actions 风格的 YAML 文件直接加载为 `Graph`,支持 `jobs`/`needs`/`strategy.matrix`/`if`/`continue-on-error`/`env`/`defaults`,以及 `matrix.include`/`exclude` 过滤、条件依赖(`needs: [{job: ..., if: ...}]`)、`outputs` 声明:
|
||||
|
||||
```yaml
|
||||
# pipeline.yaml
|
||||
strategy: thread
|
||||
defaults:
|
||||
retry: {max_attempts: 3}
|
||||
env: {CI: "true"}
|
||||
|
||||
jobs:
|
||||
setup:
|
||||
cmd: ["echo", "setup"]
|
||||
runs-on: linux
|
||||
|
||||
build:
|
||||
needs: [setup]
|
||||
cmd: ["echo", "build-${{ matrix.version }}"]
|
||||
strategy:
|
||||
matrix:
|
||||
version: ["3.10", "3.11", "3.12"]
|
||||
os: ["linux", "macos"]
|
||||
exclude:
|
||||
- version: "3.10"
|
||||
os: "macos"
|
||||
include:
|
||||
- version: "3.13"
|
||||
os: "linux"
|
||||
if: "env.CI"
|
||||
outputs:
|
||||
version: "${{ matrix.version }}"
|
||||
|
||||
deploy:
|
||||
needs:
|
||||
- job: build
|
||||
if: "env.BRANCH == 'main'"
|
||||
cmd: ["echo", "deploy"]
|
||||
|
||||
test:
|
||||
needs: [build]
|
||||
cmd: ["echo", "test"]
|
||||
continue-on-error: true
|
||||
```
|
||||
|
||||
```python
|
||||
import pyflowx as px
|
||||
|
||||
graph = px.Graph.from_yaml("pipeline.yaml")
|
||||
# 或从字符串解析:graph = px.parse_yaml_string("...")
|
||||
report = px.run(graph, strategy="thread")
|
||||
```
|
||||
|
||||
CLI 一键执行:
|
||||
|
||||
```bash
|
||||
pf yamlrun pipeline.yaml # 执行
|
||||
pf yamlrun pipeline.yaml --dry-run # 仅预览,不执行
|
||||
pf yamlrun pipeline.yaml --list # 列出所有任务
|
||||
pf yamlrun pipeline.yaml --quiet # 静默模式
|
||||
pf yamlrun pipeline.yaml --strategy sequential # 覆盖策略
|
||||
```
|
||||
|
||||
**字段映射**:
|
||||
|
||||
| YAML 字段 | TaskSpec 字段 | 说明 |
|
||||
|-----------|---------------|------|
|
||||
| `cmd` / `run` | `cmd` | `cmd` 为列表,`run` 为 shell 字符串 |
|
||||
| `needs` | `depends_on` | 矩阵 job 依赖自动展开为所有变体 |
|
||||
| `if` | `conditions` | `success()`/`always()`/`env.VAR`/`env.VAR == 'x'`/`env.VAR != 'x'` |
|
||||
| `strategy.matrix` | 矩阵扇出 | 笛卡尔积,命名 `{job}_{key}-{value}` |
|
||||
| `strategy.matrix.include` | 矩阵追加 | 额外组合(可引入新键) |
|
||||
| `strategy.matrix.exclude` | 矩阵剔除 | 匹配所有 key-value 的组合被移除 |
|
||||
| `${{ matrix.key }}` | 占位符 | `cmd`/`run`/`cwd`/`env`/`outputs` 中替换 |
|
||||
| `outputs` | `outputs` | 任务声明的输出键值映射(元数据,不参与执行) |
|
||||
| `needs: [{job: ..., if: ...}]` | 条件依赖 | `if` 不满足时跳过该依赖 |
|
||||
| `continue-on-error` | `continue_on_error` | 字段名支持 hyphen/underscore |
|
||||
| `runs-on` | `tags` | 追加到 tags 元组 |
|
||||
| `defaults` | `GraphDefaults` | 图级默认值(retry/timeout/env/cwd 等) |
|
||||
|
||||
### 任务模板 —— task_template
|
||||
|
||||
`task_template` 工厂批量生成相似 TaskSpec:
|
||||
@@ -393,15 +300,31 @@ runner.run_cli() # 解析 sys.argv 并执行
|
||||
命令行用法:
|
||||
|
||||
```bash
|
||||
pf pymake clean # 执行 clean 图
|
||||
pf pymake build --strategy thread # 覆盖执行策略
|
||||
pf pymake test --dry-run # 仅打印执行计划
|
||||
pf pymake --list # 列出所有命令
|
||||
pf pymake --quiet # 静默模式
|
||||
python build.py clean # 执行 clean 图
|
||||
python build.py build --strategy thread # 覆盖执行策略
|
||||
python build.py test --dry-run # 仅打印执行计划
|
||||
python build.py --list # 列出所有命令
|
||||
python build.py --quiet # 静默模式
|
||||
```
|
||||
|
||||
`verbose=True`(默认)时打印任务生命周期(开始/成功/失败/跳过)与命令输出;`--quiet` 关闭。
|
||||
|
||||
## 示例
|
||||
|
||||
仓库 `examples/` 目录包含完整示例:
|
||||
|
||||
- [`etl_pipeline.py`](examples/etl_pipeline.py) —— ETL 流水线(sequential)
|
||||
- [`parallel_run.py`](examples/parallel_run.py) —— 并行执行对比(thread vs sequential)
|
||||
- [`async_aggregation.py`](examples/async_aggregation.py) —— 异步聚合 + Context 注入
|
||||
|
||||
运行:
|
||||
|
||||
```bash
|
||||
python examples/etl_pipeline.py
|
||||
python examples/parallel_run.py
|
||||
python examples/async_aggregation.py
|
||||
```
|
||||
|
||||
## 断点续跑
|
||||
|
||||
```python
|
||||
@@ -429,22 +352,6 @@ px.TaskSpec(
|
||||
)
|
||||
```
|
||||
|
||||
**LRU 驱逐**:`MemoryBackend(maxsize=N)` 限制最大条目数,超限时按 LRU(最近最少使用)
|
||||
策略驱逐最旧条目。`get`/`has` 命中会将条目移到最近使用位置:
|
||||
|
||||
```python
|
||||
from pyflowx import MemoryBackend
|
||||
|
||||
backend = MemoryBackend(maxsize=100) # 最多 100 条,超出驱逐最旧
|
||||
```
|
||||
|
||||
**手动失效**:`invalidate(key)` 删除单个键的缓存,`clear()` 清除全部:
|
||||
|
||||
```python
|
||||
backend.invalidate("fetch_user") # 删除单个键,返回是否已删除
|
||||
backend.clear() # 清除全部
|
||||
```
|
||||
|
||||
## 错误处理
|
||||
|
||||
所有错误都是 `PyFlowXError` 的子类:
|
||||
@@ -474,7 +381,7 @@ except px.PyFlowXError:
|
||||
| 特性 | PyFlowX | Airflow | Prefect | Dask |
|
||||
|------|---------|---------|---------|------|
|
||||
| 零样板 | 参数名即依赖 | 装饰器 + XCom | 装饰器 | 装饰器 |
|
||||
| 运行时依赖 | rich + typer | 重型 | 中型 | 中型 |
|
||||
| 运行时依赖 | 仅标准库 | 重型 | 中型 | 中型 |
|
||||
| 类型安全 | mypy strict | 弱 | 中 | 中 |
|
||||
| 异步原生 | 是 | 否 | 部分 | 否 |
|
||||
| 断点续跑 | 内置 | 需配置 | 需配置 | 需配置 |
|
||||
@@ -521,253 +428,6 @@ px.TaskSpec("low", fn=work, priority=0)
|
||||
px.TaskSpec("high", fn=work, priority=10) # 同层内先执行
|
||||
```
|
||||
|
||||
### 任务取消
|
||||
|
||||
`CancelToken` 提供线程安全的取消信号,传入 `run(cancel_event=...)` 即可外部取消正在执行的图。
|
||||
`KeyboardInterrupt`(Ctrl+C)也会触发相同的优雅取消流程:
|
||||
|
||||
```python
|
||||
token = px.CancelToken()
|
||||
|
||||
# 在另一个线程中触发取消
|
||||
import threading
|
||||
|
||||
|
||||
def cancel_later():
|
||||
import time
|
||||
|
||||
time.sleep(2)
|
||||
token.cancel("超时取消")
|
||||
|
||||
|
||||
threading.Thread(target=cancel_later, daemon=True).start()
|
||||
|
||||
# 取消后:待运行任务标记 SKIPPED,运行中任务等待完成,report.success = False
|
||||
report = px.run(graph, strategy="dependency", cancel_event=token)
|
||||
for name, result in report.results.items():
|
||||
print(f"{name}: {result.status.value}")
|
||||
```
|
||||
|
||||
也支持标准 `threading.Event` 作为取消信号:
|
||||
|
||||
```python
|
||||
event = threading.Event()
|
||||
px.run(graph, cancel_event=event)
|
||||
```
|
||||
|
||||
### 子图执行
|
||||
|
||||
`run(only=...)` / `run(tags=...)` 只运行指定任务及其传递依赖,适用于调试单个任务或增量执行:
|
||||
|
||||
```python
|
||||
# 只运行 "report" 任务及其所有上游依赖
|
||||
report = px.run(graph, only=["report"])
|
||||
|
||||
# 按标签过滤:只运行带 "ingest" 标签的任务及其依赖
|
||||
report = px.run(graph, tags=["ingest"])
|
||||
|
||||
# 两者可组合(取并集)
|
||||
report = px.run(graph, only=["report"], tags=["ingest"])
|
||||
```
|
||||
|
||||
CLI 也支持 `--only` / `--tags`:
|
||||
|
||||
```bash
|
||||
pf yamlrun pipeline.yaml --only report,deploy
|
||||
pf yamlrun pipeline.yaml --tags ingest,report
|
||||
```
|
||||
|
||||
### 结果序列化
|
||||
|
||||
`RunReport` 支持将运行结果导出为 JSON / dict / CSV,便于持久化、跨进程传输或对接外部工具:
|
||||
|
||||
```python
|
||||
report = px.run(graph, strategy="sequential")
|
||||
|
||||
# JSON 字符串(含 success/summary/results)
|
||||
text = report.to_json(indent=2)
|
||||
|
||||
# JSON 兼容字典
|
||||
data = report.to_dict()
|
||||
|
||||
# CSV 表格导出(不含 value 列以避免特殊字符干扰)
|
||||
csv_text = report.to_csv(include_value=False)
|
||||
```
|
||||
|
||||
任务返回值可能是任意 Python 对象(不一定 JSON 可序列化)。默认策略:
|
||||
可序列化原样保留,不可序列化回退到 `repr(value)`;调用方可通过
|
||||
`value_serializer` 自定义:
|
||||
|
||||
```python
|
||||
# 自定义值序列化(如把 Path 转为字符串)
|
||||
text = report.to_json(value_serializer=str)
|
||||
```
|
||||
|
||||
`from_json()` 可从 JSON 字符串重建 `RunReport`,**仅供查询/分析**,不能重新
|
||||
执行(`TaskSpec` 的 `fn`/`cmd` 等不可序列化字段无法恢复):
|
||||
|
||||
```python
|
||||
restored = px.RunReport.from_json(text)
|
||||
print(restored["double"]) # 任务返回值
|
||||
print(restored.result_of("a").status) # TaskStatus
|
||||
```
|
||||
|
||||
### CLI 可视化
|
||||
|
||||
`pf graph <yaml>` 在终端可视化任务图 DAG,无需运行即可检查依赖关系:
|
||||
|
||||
```bash
|
||||
# 默认 ASCII 渲染:分层树(按标签着色) + 依赖关系表
|
||||
pf graph pipeline.yaml
|
||||
|
||||
# 输出 Mermaid 语法(可粘贴到 mermaid.live)
|
||||
pf graph pipeline.yaml --format mermaid
|
||||
|
||||
# 自定义方向
|
||||
pf graph pipeline.yaml --format mermaid --orientation LR
|
||||
|
||||
# 仅列出任务名
|
||||
pf graph pipeline.yaml --format list
|
||||
|
||||
# 纯依赖关系表
|
||||
pf graph pipeline.yaml --format deps
|
||||
|
||||
# 关闭标签着色(统一单色)
|
||||
pf graph pipeline.yaml --color-by none
|
||||
```
|
||||
|
||||
ASCII 模式默认 `--color-by tag`,按任务首标签选择 rich 颜色并在图下方
|
||||
显示标签颜色图例,便于视觉分组。
|
||||
|
||||
`pf yamlrun` 新增 `--progress` 选项,启用 rich 进度条显示运行中任务、完成
|
||||
数与耗时:
|
||||
|
||||
```bash
|
||||
pf yamlrun pipeline.yaml --progress
|
||||
```
|
||||
|
||||
### 任务列表过滤
|
||||
|
||||
`pf yamlrun --list` 支持 `--list-tag` / `--list-name` 过滤,便于大型 YAML
|
||||
中快速定位任务:
|
||||
|
||||
```bash
|
||||
# 列出所有任务
|
||||
pf yamlrun pipeline.yaml --list
|
||||
|
||||
# 只列出含 ingest 标签的任务
|
||||
pf yamlrun pipeline.yaml --list --list-tag ingest
|
||||
|
||||
# 按任务名子串过滤(大小写不敏感)
|
||||
pf yamlrun pipeline.yaml --list --list-name extract
|
||||
|
||||
# 组合过滤(交集)
|
||||
pf yamlrun pipeline.yaml --list --list-tag ingest --list-name extract
|
||||
```
|
||||
|
||||
### 工具详情与 Shell 补全
|
||||
|
||||
`pf info [tool]` 显示工具详情:
|
||||
|
||||
```bash
|
||||
# 列出所有工具的简表(含子命令数、描述)
|
||||
pf info
|
||||
|
||||
# 显示指定工具的子命令详情
|
||||
pf info clr
|
||||
pf info gittool
|
||||
```
|
||||
|
||||
`pf completion <shell>` 生成 shell 补全脚本,提升日常使用体验:
|
||||
|
||||
```bash
|
||||
# bash —— 添加到 ~/.bashrc
|
||||
pf completion bash >> ~/.bashrc
|
||||
|
||||
# zsh —— 添加到 ~/.zshrc
|
||||
pf completion zsh >> ~/.zshrc
|
||||
|
||||
# fish —— 保存到 ~/.config/fish/completions/pf.fish
|
||||
pf completion fish > ~/.config/fish/completions/pf.fish
|
||||
```
|
||||
|
||||
补全脚本覆盖所有内建命令(`yamlrun`/`graph`/`info`/`completion`)与
|
||||
`@px.tool` 注册的工具名;二级补全交给具体子命令的 `--help` 自描述。
|
||||
|
||||
### 失败诊断
|
||||
|
||||
运行失败时,`RunReport.diagnose()` 自动生成结构化诊断报告,包含:
|
||||
|
||||
- **根因任务**:FAILED 且所有硬依赖非 FAILED 的任务(依赖链中最先出问题的环节)
|
||||
- **依赖链回溯**:从根因到每个失败任务的路径
|
||||
- **相似失败聚类**:按 `(异常类型, 消息前缀)` 聚类,发现批量失败模式
|
||||
- **可操作建议**:识别 FileNotFoundError/ImportError/TimeoutError 等常见异常并给出提示
|
||||
|
||||
```python
|
||||
report = px.run(graph, strategy="sequential")
|
||||
if not report.success:
|
||||
diag = report.diagnose()
|
||||
if diag is not None:
|
||||
print(diag.describe())
|
||||
# 根因任务
|
||||
print(diag.root_causes)
|
||||
# 依赖链
|
||||
for chain in diag.dependency_chains:
|
||||
print(f"{chain.failed_task}: {' -> '.join(chain.chain)}")
|
||||
```
|
||||
|
||||
`pf yamlrun` 失败时自动打印诊断摘要到 stderr:
|
||||
|
||||
```bash
|
||||
pf yamlrun pipeline.yaml # 失败时自动输出诊断报告
|
||||
```
|
||||
|
||||
`report.describe()` 在失败时也会附诊断摘要。
|
||||
|
||||
### 可观测性
|
||||
|
||||
每次 `run()` 自动生成 8 字符十六进制 `run_id`,作为本次运行的唯一追踪 ID,
|
||||
贯穿日志、序列化结果与诊断报告:
|
||||
|
||||
```python
|
||||
report = px.run(graph)
|
||||
print(report.run_id) # 如 "a3f4b2c1"
|
||||
print(report.summary()["run_id"]) # summary 也含 run_id
|
||||
```
|
||||
|
||||
`run_id` 通过结构化日志的 `extra` 字段注入所有执行器日志,便于跨系统关联。
|
||||
日志格式遵循 `logging` 标准延迟格式化,`extra` 字段包括:
|
||||
|
||||
- `run_id` —— 本次运行 ID
|
||||
- `task_name` —— 任务名(任务级日志)
|
||||
- `status` —— 任务状态(`running`/`success`/`failed`/`skipped`)
|
||||
- `attempts` —— 尝试次数(失败日志)
|
||||
- `error_type` —— 异常类型名(失败日志)
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
# 日志输出示例:
|
||||
# INFO 运行开始: run_id=a3f4b2c1 strategy=dependency tasks=3
|
||||
# INFO task 'a' skipped (cached)
|
||||
# WARNING task 'b' failed (attempt 1/3): RuntimeError('boom'); retrying
|
||||
# INFO 运行结束: run_id=a3f4b2c1 success=False tasks=3
|
||||
```
|
||||
|
||||
`to_json()` / `to_dict()` / `from_json()` 同步保留 `run_id`,反序列化时可还原:
|
||||
旧版 JSON(无 `run_id` 字段)会自动生成新 ID,向前兼容。
|
||||
|
||||
`report.profile(graph)` 返回 `ProfileReport`,基于 `started_at`/`finished_at`
|
||||
做离线性能分析(关键路径、并行度、Top 瓶颈),零运行时开销:
|
||||
|
||||
```python
|
||||
profile = report.profile(graph)
|
||||
print(profile.describe()) # 人类可读报告
|
||||
print(profile.critical_path) # 关键路径任务序列
|
||||
bottlenecks = profile.top_bottlenecks(5) # Top-5 瓶颈任务
|
||||
```
|
||||
|
||||
## 开发
|
||||
|
||||
```bash
|
||||
@@ -778,17 +438,15 @@ uv sync --extra dev
|
||||
uv run pytest --cov=pyflowx --cov-fail-under=95
|
||||
|
||||
# 类型检查
|
||||
uv run pyrefly check .
|
||||
uv run mypy
|
||||
|
||||
# 代码风格
|
||||
uv run ruff check .
|
||||
uv run ruff format --check .
|
||||
uv run ruff check src tests examples
|
||||
uv run ruff format --check src tests examples
|
||||
```
|
||||
|
||||
## 模块结构
|
||||
|
||||
### 核心
|
||||
|
||||
| 模块 | 职责 |
|
||||
|------|------|
|
||||
| `task.py` | 纯数据结构:`TaskSpec`、`RetryPolicy`、`TaskHooks`、`TaskStatus` |
|
||||
@@ -796,25 +454,12 @@ uv run ruff format --check .
|
||||
| `compose.py` | 多图组合:`GraphComposer` / `compose` |
|
||||
| `context.py` | 上下文注入:参数名→依赖解析 |
|
||||
| `command.py` | 命令执行:`run_command`(list/shell/Callable) |
|
||||
| `cancellation.py` | 任务取消:`CancelToken` 线程安全取消令牌 |
|
||||
| `conditions.py` | 条件执行:内置条件与组合器 |
|
||||
| `executors.py` | 执行器与 `run` 入口:四种策略共享模块级辅助;verbose 统一应用到 spec |
|
||||
| `executors.py` | 执行器与 `run` 入口:四种策略共享模块级辅助 |
|
||||
| `storage.py` | 状态后端:`MemoryBackend` / `JSONBackend`(batch flush) |
|
||||
| `yaml_loader.py` | YAML 任务编排:`load_yaml` / `parse_yaml_string`(GitHub Actions 风格) |
|
||||
| `runner.py` | CLI 运行器:`CliRunner` |
|
||||
| `report.py` | 运行结果:`RunReport` / `TaskResult` |
|
||||
| `tools.py` | `@px.tool` 装饰器:`ToolSpec` / `run_tool` / `list_tools` / `list_subcommands` |
|
||||
| `profiling.py` | 性能分析:`Profiler` 任务耗时统计 |
|
||||
| `errors.py` | 错误家族:`PyFlowXError` 子类 |
|
||||
| `ops/` | CLI 工具模块集合(每个子模块用 `@px.tool` 注册一个工具) |
|
||||
|
||||
### CLI 工具
|
||||
|
||||
| 模块 | 职责 |
|
||||
|------|------|
|
||||
| `cli/pf.py` | 统一入口:`pf <tool> [command]`,按需导入 `ops/<tool>.py` 触发 `@px.tool` 注册 |
|
||||
| `cli/legacy/profiler.py` | 性能分析 CLI(legacy) |
|
||||
| `cli/legacy/emlmanager.py` | 邮件管理 CLI(legacy) |
|
||||
|
||||
## 许可证
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
"""PyFlowX 性能基准套件.
|
||||
|
||||
用法::
|
||||
|
||||
python -m benchmarks # 运行全部基准
|
||||
python -m benchmarks graph # 仅图构建基准
|
||||
python -m benchmarks execution # 仅执行基准
|
||||
python -m benchmarks context # 仅上下文注入基准
|
||||
python -m benchmarks storage # 仅状态后端基准
|
||||
python -m benchmarks advanced # 仅高级基准
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
__all__ = ["print_results", "time_it"]
|
||||
|
||||
|
||||
def time_it(fn: Callable[[], Any], iterations: int = 100, warmup: int = 5) -> tuple[float, float]:
|
||||
"""计时工具:返回 (平均耗时 ms, 吞吐 ops/sec)."""
|
||||
for _ in range(warmup):
|
||||
fn()
|
||||
times: list[float] = []
|
||||
for _ in range(iterations):
|
||||
t0 = time.perf_counter()
|
||||
fn()
|
||||
times.append(time.perf_counter() - t0)
|
||||
avg = sum(times) / len(times)
|
||||
return avg * 1000, 1.0 / avg if avg > 0 else float("inf")
|
||||
|
||||
|
||||
def print_results(title: str, results: list[tuple[str, int, float, float]]) -> None:
|
||||
"""打印格式化基准结果表."""
|
||||
console = Console()
|
||||
table = Table(title=title, show_header=True, header_style="bold")
|
||||
table.add_column("场景", style="cyan", no_wrap=True)
|
||||
table.add_column("迭代", justify="right")
|
||||
table.add_column("平均耗时", justify="right", style="yellow")
|
||||
table.add_column("吞吐", justify="right", style="green")
|
||||
for name, iters, ms, ops in results:
|
||||
ms_str = f"{ms:.3f} ms" if ms < 1 else f"{ms:.2f} ms"
|
||||
ops_str = f"{ops:.0f} ops/s" if ops > 1000 else f"{ops:.1f} ops/s"
|
||||
table.add_row(name, str(iters), ms_str, ops_str)
|
||||
console.print(table)
|
||||
console.print()
|
||||
@@ -1,449 +0,0 @@
|
||||
"""PyFlowX 性能基准套件.
|
||||
|
||||
用法::
|
||||
|
||||
python -m benchmarks # 运行全部基准
|
||||
python -m benchmarks graph # 仅图构建基准
|
||||
python -m benchmarks execution # 仅执行基准
|
||||
python -m benchmarks context # 仅上下文注入基准
|
||||
python -m benchmarks storage # 仅状态后端基准
|
||||
python -m benchmarks advanced # 仅高级基准
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import sys
|
||||
import tempfile
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
import pyflowx as px
|
||||
from benchmarks import print_results, time_it
|
||||
from benchmarks.bench_advanced import run_advanced
|
||||
from pyflowx import Graph, GraphDefaults, RetryPolicy, TaskSpec
|
||||
from pyflowx._json import _HAS_ORJSON
|
||||
from pyflowx._json import dumps as _dumps
|
||||
from pyflowx._json import loads as _loads
|
||||
from pyflowx.context import build_call_args
|
||||
from pyflowx.storage import JSONBackend, MemoryBackend, SQLiteBackend
|
||||
|
||||
# ============================================================================
|
||||
# 图生成工具
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def make_chain(n: int) -> list[TaskSpec]:
|
||||
"""生成 n 个任务的链式 DAG。"""
|
||||
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
|
||||
for i in range(1, n):
|
||||
specs[i] = TaskSpec(f"t{i}", cmd=["true"], depends_on=(f"t{i - 1}",))
|
||||
return specs
|
||||
|
||||
|
||||
def make_diamond(n: int) -> list[TaskSpec]:
|
||||
"""生成 n 个任务的菱形 DAG(每层宽度约 sqrt(n))。"""
|
||||
width = max(1, int(math.sqrt(n)))
|
||||
specs: list[TaskSpec] = []
|
||||
prev_layer: list[str] = []
|
||||
layer = 0
|
||||
count = 0
|
||||
while count < n:
|
||||
cur_layer: list[str] = []
|
||||
for j in range(width):
|
||||
if count >= n:
|
||||
break
|
||||
name = f"l{layer}_t{j}"
|
||||
deps = tuple(prev_layer) if prev_layer else ()
|
||||
specs.append(TaskSpec(name, cmd=["true"], depends_on=deps))
|
||||
cur_layer.append(name)
|
||||
count += 1
|
||||
prev_layer = cur_layer
|
||||
layer += 1
|
||||
return specs
|
||||
|
||||
|
||||
def make_wide(n: int) -> list[TaskSpec]:
|
||||
"""生成 n 个独立任务(无依赖,最大并行度)。"""
|
||||
return [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:图构建
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_construction() -> None:
|
||||
"""图构建(from_specs + validate)基准。"""
|
||||
results = []
|
||||
for n in (10, 100, 500, 1000):
|
||||
specs = make_chain(n)
|
||||
ms, _ = time_it(lambda s=specs: Graph.from_specs(s), iterations=20)
|
||||
results.append((f"chain({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
|
||||
|
||||
for n in (10, 100, 500, 1000):
|
||||
specs = make_diamond(n)
|
||||
ms, _ = time_it(lambda s=specs: Graph.from_specs(s), iterations=20)
|
||||
results.append((f"diamond({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
|
||||
|
||||
print_results("图构建 (from_specs + validate)", results)
|
||||
|
||||
|
||||
def bench_layers() -> None:
|
||||
"""拓扑分层基准(冷启动 vs 缓存命中)。"""
|
||||
results = []
|
||||
for n in (100, 500, 1000):
|
||||
specs = make_diamond(n)
|
||||
graph = Graph.from_specs(specs)
|
||||
|
||||
def _cold(g: Graph = graph) -> None:
|
||||
g._layers_cache = None # type: ignore[attr-defined]
|
||||
g.layers()
|
||||
|
||||
ms_cold, ops_cold = time_it(_cold, iterations=50, warmup=5)
|
||||
results.append((f"layers(cold,{n})", 50, ms_cold, ops_cold))
|
||||
|
||||
ms_hot, ops_hot = time_it(lambda g=graph: g.layers(), iterations=200, warmup=10)
|
||||
results.append((f"layers(cached,{n})", 200, ms_hot, ops_hot))
|
||||
|
||||
print_results("拓扑分层 (layers)", results)
|
||||
|
||||
|
||||
def bench_resolved_spec() -> None:
|
||||
"""resolved_spec 缓存命中基准。"""
|
||||
results = []
|
||||
for n in (100, 500, 1000):
|
||||
specs = make_chain(n)
|
||||
defaults = GraphDefaults(retry=RetryPolicy(max_attempts=2))
|
||||
graph = Graph.from_specs(specs, defaults=defaults)
|
||||
name = f"t{n // 2}"
|
||||
ms, ops = time_it(lambda g=graph, nm=name: g.resolved_spec(nm), iterations=500, warmup=20)
|
||||
results.append((f"resolved_spec(cached,{n})", 500, ms, ops))
|
||||
|
||||
print_results("resolved_spec (缓存命中)", results)
|
||||
|
||||
|
||||
def run_graph() -> None:
|
||||
"""运行全部图基准。"""
|
||||
bench_construction()
|
||||
bench_layers()
|
||||
bench_resolved_spec()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:任务执行
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_sequential() -> None:
|
||||
"""sequential 策略执行基准。"""
|
||||
results = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
for n in (50, 200, 500):
|
||||
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="sequential"), iterations=10, warmup=2)
|
||||
results.append((f"sequential({n})", 10, ms, ops))
|
||||
|
||||
print_results("执行策略: sequential", results)
|
||||
|
||||
|
||||
def bench_thread() -> None:
|
||||
"""thread 策略执行基准。"""
|
||||
results = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
for n in (50, 200, 500):
|
||||
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="thread"), iterations=10, warmup=2)
|
||||
results.append((f"thread({n})", 10, ms, ops))
|
||||
|
||||
print_results("执行策略: thread", results)
|
||||
|
||||
|
||||
def bench_async() -> None:
|
||||
"""async 策略执行基准。"""
|
||||
results = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
for n in (50, 200, 500):
|
||||
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="async"), iterations=10, warmup=2)
|
||||
results.append((f"async({n})", 10, ms, ops))
|
||||
|
||||
print_results("执行策略: async", results)
|
||||
|
||||
|
||||
def bench_dependency() -> None:
|
||||
"""dependency 策略执行基准。"""
|
||||
results = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
for n in (50, 200, 500):
|
||||
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="dependency"), iterations=10, warmup=2)
|
||||
results.append((f"dependency({n})", 10, ms, ops))
|
||||
|
||||
print_results("执行策略: dependency", results)
|
||||
|
||||
|
||||
def bench_cmd_execution() -> None:
|
||||
"""cmd 任务执行基准(真实子进程)。"""
|
||||
results = []
|
||||
for n in (10, 50, 100):
|
||||
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="sequential"), iterations=5, warmup=1)
|
||||
results.append((f"cmd-sequential({n})", 5, ms, ops))
|
||||
|
||||
for n in (10, 50, 100):
|
||||
specs = [TaskSpec(f"t{i}", cmd=["true"]) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
ms, ops = time_it(lambda g=graph: px.run(g, strategy="thread", max_workers=8), iterations=5, warmup=1)
|
||||
results.append((f"cmd-thread({n})", 5, ms, ops))
|
||||
|
||||
print_results("cmd 任务执行 (['true'])", results)
|
||||
|
||||
|
||||
def run_execution() -> None:
|
||||
"""运行全部执行基准。"""
|
||||
bench_sequential()
|
||||
bench_thread()
|
||||
bench_async()
|
||||
bench_dependency()
|
||||
bench_cmd_execution()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:上下文注入
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_context_no_deps() -> None:
|
||||
"""无依赖 fn 任务的上下文注入基准。"""
|
||||
results = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
spec = TaskSpec("noop", fn=noop)
|
||||
context: dict[str, Any] = {}
|
||||
ms, ops = time_it(lambda s=spec, c=context: build_call_args(s, c), iterations=2000, warmup=100)
|
||||
results.append(("fn(no-deps)", 2000, ms, ops))
|
||||
|
||||
# cmd 任务快速路径
|
||||
spec_cmd = TaskSpec("cmd", cmd=["true"])
|
||||
ms, ops = time_it(lambda s=spec_cmd, c=context: build_call_args(s, c), iterations=2000, warmup=100)
|
||||
results.append(("cmd(fast-path)", 2000, ms, ops))
|
||||
|
||||
print_results("上下文注入 (build_call_args)", results)
|
||||
|
||||
|
||||
def bench_context_with_deps() -> None:
|
||||
"""有依赖 fn 任务的上下文注入基准。"""
|
||||
results = []
|
||||
|
||||
def consumer(a: int, b: int) -> int:
|
||||
return a + b
|
||||
|
||||
spec = TaskSpec("consumer", fn=consumer, depends_on=("a", "b"))
|
||||
context = {"a": 1, "b": 2, "c": 3}
|
||||
ms, ops = time_it(lambda s=spec, c=context: build_call_args(s, c), iterations=2000, warmup=100)
|
||||
results.append(("fn(2-deps)", 2000, ms, ops))
|
||||
|
||||
# Context 标注
|
||||
from pyflowx.task import Context
|
||||
|
||||
def ctx_fn(ctx: Context) -> int:
|
||||
return sum(ctx.values())
|
||||
|
||||
spec_ctx = TaskSpec("ctx", fn=ctx_fn, depends_on=("a", "b"))
|
||||
ms, ops = time_it(lambda s=spec_ctx, c=context: build_call_args(s, c), iterations=2000, warmup=100)
|
||||
results.append(("fn(Context-annotated)", 2000, ms, ops))
|
||||
|
||||
# **kwargs
|
||||
def kwargs_fn(**kwargs: int) -> int:
|
||||
return sum(kwargs.values())
|
||||
|
||||
spec_kw = TaskSpec("kw", fn=kwargs_fn, depends_on=("a", "b"))
|
||||
ms, ops = time_it(lambda s=spec_kw, c=context: build_call_args(s, c), iterations=2000, warmup=100)
|
||||
results.append(("fn(**kwargs)", 2000, ms, ops))
|
||||
|
||||
print_results("上下文注入 (有依赖)", results)
|
||||
|
||||
|
||||
def run_context() -> None:
|
||||
"""运行全部上下文注入基准。"""
|
||||
bench_context_no_deps()
|
||||
bench_context_with_deps()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:状态后端
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_storage() -> None:
|
||||
"""状态后端 save/load 基准。"""
|
||||
results = []
|
||||
|
||||
# MemoryBackend
|
||||
mem = MemoryBackend()
|
||||
ms, ops = time_it(lambda: mem.save("key", "value"), iterations=1000, warmup=50)
|
||||
results.append(("MemoryBackend.save", 1000, ms, ops))
|
||||
|
||||
ms, ops = time_it(mem.load, iterations=1000, warmup=50)
|
||||
results.append(("MemoryBackend.load", 1000, ms, ops))
|
||||
|
||||
# JSONBackend(batch 模式)
|
||||
tmp_dir = tempfile.mkdtemp()
|
||||
json_path = str(Path(tmp_dir) / "state.json")
|
||||
|
||||
json_backend = JSONBackend(json_path)
|
||||
with json_backend.batch():
|
||||
for i in range(100):
|
||||
json_backend.save(f"task_{i}", f"result_{i}")
|
||||
|
||||
def _json_save() -> None:
|
||||
jb = JSONBackend(json_path)
|
||||
with jb.batch():
|
||||
for i in range(10):
|
||||
jb.save(f"bench_{i}", f"val_{i}")
|
||||
|
||||
ms, ops = time_it(_json_save, iterations=50, warmup=5)
|
||||
results.append(("JSONBackend.save(batch=10)", 50, ms, ops))
|
||||
|
||||
# 复杂 value(嵌套 dict)—— 展示 batch 模式延迟验证的优化效果
|
||||
complex_value = {
|
||||
"output": {"path": "/data/result.json", "size": 1024, "checksum": "abc123"},
|
||||
"metrics": {"accuracy": 0.95, "latency_ms": 120, "samples": 1000},
|
||||
"artifacts": [f"artifact_{j}.bin" for j in range(10)],
|
||||
"metadata": {"version": "1.0", "tags": ["trained", "validated"], "created_at": "2026-07-07"},
|
||||
}
|
||||
|
||||
def _json_save_complex() -> None:
|
||||
jb = JSONBackend(json_path)
|
||||
with jb.batch():
|
||||
for i in range(10):
|
||||
jb.save(f"bench_complex_{i}", complex_value)
|
||||
|
||||
ms, ops = time_it(_json_save_complex, iterations=50, warmup=5)
|
||||
results.append(("JSONBackend.save(batch=10,complex)", 50, ms, ops))
|
||||
|
||||
ms, ops = time_it(json_backend.load, iterations=200, warmup=10)
|
||||
results.append(("JSONBackend.load", 200, ms, ops))
|
||||
|
||||
# SQLiteBackend
|
||||
db_path = str(Path(tmp_dir) / "state.db")
|
||||
|
||||
sqlite_backend = SQLiteBackend(db_path)
|
||||
with sqlite_backend.batch():
|
||||
for i in range(100):
|
||||
sqlite_backend.save(f"task_{i}", f"result_{i}")
|
||||
|
||||
def _sqlite_save() -> None:
|
||||
sb = SQLiteBackend(db_path)
|
||||
with sb.batch():
|
||||
for i in range(10):
|
||||
sb.save(f"bench_{i}", f"val_{i}")
|
||||
|
||||
ms, ops = time_it(_sqlite_save, iterations=50, warmup=5)
|
||||
results.append(("SQLiteBackend.save(batch=10)", 50, ms, ops))
|
||||
|
||||
ms, ops = time_it(sqlite_backend.load, iterations=200, warmup=10)
|
||||
results.append(("SQLiteBackend.load", 200, ms, ops))
|
||||
|
||||
print_results("状态后端 (save/load)", results)
|
||||
|
||||
# 序列化基准:_json 抽象层 dumps/loads
|
||||
ser_results = []
|
||||
report_like = {
|
||||
"run_id": "abc12345",
|
||||
"success": True,
|
||||
"results": [
|
||||
{
|
||||
"name": f"task_{i}",
|
||||
"status": "success",
|
||||
"attempts": 1,
|
||||
"duration_seconds": 0.123 + i * 0.001,
|
||||
"value": {"output": [i, i + 1, i + 2], "meta": {"tag": "api"}},
|
||||
}
|
||||
for i in range(100)
|
||||
],
|
||||
}
|
||||
serialized = _dumps(report_like)
|
||||
|
||||
ms, ops = time_it(lambda: _dumps(report_like), iterations=200, warmup=10)
|
||||
ser_results.append(("dumps(report-100)", 200, ms, ops))
|
||||
|
||||
ms, ops = time_it(lambda: _loads(serialized), iterations=200, warmup=10)
|
||||
ser_results.append(("loads(report-100)", 200, ms, ops))
|
||||
|
||||
backend_name = "orjson" if _HAS_ORJSON else "stdlib"
|
||||
print_results(f"JSON 序列化 (后端={backend_name})", ser_results)
|
||||
|
||||
# 清理临时目录
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
|
||||
|
||||
def run_storage() -> None:
|
||||
"""运行全部状态后端基准。"""
|
||||
bench_storage()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 主入口
|
||||
# ============================================================================
|
||||
|
||||
|
||||
BENCH_MODULES: dict[str, Callable[[], None]] = {
|
||||
"graph": run_graph,
|
||||
"execution": run_execution,
|
||||
"context": run_context,
|
||||
"storage": run_storage,
|
||||
"advanced": run_advanced,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""CLI 入口。"""
|
||||
args = argv if argv is not None else sys.argv[1:]
|
||||
console = Console()
|
||||
console.print("[bold cyan]PyFlowX 性能基准套件[/bold cyan]\n")
|
||||
|
||||
if not args or args[0] in ("--all", "-a"):
|
||||
for name, fn in BENCH_MODULES.items():
|
||||
console.print(f"[bold]运行: {name}[/bold]")
|
||||
fn()
|
||||
elif args[0] in BENCH_MODULES:
|
||||
BENCH_MODULES[args[0]]()
|
||||
elif args[0] in ("--help", "-h"):
|
||||
console.print("用法: python -m benchmarks [graph|execution|context|storage]")
|
||||
console.print(" 无参数 = 运行全部基准")
|
||||
else:
|
||||
console.print(f"[red]未知基准模块: {args[0]}[/red]")
|
||||
console.print(f"可用: {', '.join(BENCH_MODULES)}")
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,309 +0,0 @@
|
||||
"""高级基准:条件评估/YAML 加载/通知器/run_iter/子图/取消/CPU+I/O 密集型任务.
|
||||
|
||||
这些基准覆盖核心调度引擎的扩展路径,与 bench_graph/bench_execution/bench_context/
|
||||
bench_storage 互补,构成完整的性能度量体系。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pyflowx as px
|
||||
from benchmarks import print_results, time_it
|
||||
from pyflowx import Graph, TaskSpec
|
||||
from pyflowx.cancellation import CancelToken
|
||||
from pyflowx.conditions import BuiltinConditions
|
||||
from pyflowx.notification import CallbackNotifier, NotificationLevel
|
||||
|
||||
# ============================================================================
|
||||
# 基准:条件评估
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_conditions() -> None:
|
||||
"""条件评估基准(静态/上下文/复合)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
# 静态条件(IS_LINUX)
|
||||
cond_static = BuiltinConditions.IS_LINUX()
|
||||
ctx: dict[str, Any] = {}
|
||||
ms, ops = time_it(lambda c=cond_static, x=ctx: c(x), iterations=10000, warmup=500)
|
||||
results.append(("static(IS_LINUX)", 10000, ms, ops))
|
||||
|
||||
# 上下文条件(DEP_EQUALS)
|
||||
cond_dep = BuiltinConditions.DEP_EQUALS("a", 1)
|
||||
ctx_dep = {"a": 1, "b": 2}
|
||||
ms, ops = time_it(lambda c=cond_dep, x=ctx_dep: c(x), iterations=10000, warmup=500)
|
||||
results.append(("DEP_EQUALS", 10000, ms, ops))
|
||||
|
||||
# 复合条件(AND(OR(NOT, DEP_TRUTHY), DEP_PRESENT))
|
||||
cond_complex = BuiltinConditions.AND(
|
||||
BuiltinConditions.OR(
|
||||
BuiltinConditions.NOT(BuiltinConditions.IS_WINDOWS()),
|
||||
BuiltinConditions.DEP_TRUTHY("a"),
|
||||
),
|
||||
BuiltinConditions.DEP_PRESENT("b"),
|
||||
)
|
||||
ms, ops = time_it(lambda c=cond_complex, x=ctx_dep: c(x), iterations=10000, warmup=500)
|
||||
results.append(("AND(OR(NOT,DEP_TRUTHY),DEP_PRESENT)", 10000, ms, ops))
|
||||
|
||||
print_results("条件评估", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:YAML 加载
|
||||
# ============================================================================
|
||||
|
||||
|
||||
_YAML_TEMPLATE = """\
|
||||
jobs:
|
||||
{jobs}
|
||||
defaults:
|
||||
retry:
|
||||
max_attempts: 2
|
||||
backoff: 0.0
|
||||
"""
|
||||
|
||||
|
||||
def _make_yaml(n: int) -> str:
|
||||
"""生成 n 个任务的 YAML(链式依赖)."""
|
||||
lines = []
|
||||
for i in range(n):
|
||||
deps = f"needs: [t{i - 1}]" if i > 0 else ""
|
||||
lines.append(f" t{i}:\n cmd: ['true']\n {deps}".rstrip())
|
||||
return _YAML_TEMPLATE.format(jobs="\n".join(lines))
|
||||
|
||||
|
||||
def bench_yaml_load() -> None:
|
||||
"""YAML 加载基准(解析 + Graph 构建)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
tmp_dir = tempfile.mkdtemp()
|
||||
|
||||
for n in (10, 50, 100):
|
||||
yaml_text = _make_yaml(n)
|
||||
yaml_path = Path(tmp_dir) / f"bench_{n}.yaml"
|
||||
yaml_path.write_text(yaml_text, encoding="utf-8")
|
||||
ms, _ops = time_it(lambda p=yaml_path: Graph.from_yaml(p), iterations=20, warmup=2)
|
||||
results.append((f"yaml_load({n})", 20, ms, 1.0 / (ms / 1000) if ms > 0 else 0))
|
||||
|
||||
print_results("YAML 加载 (from_yaml)", results)
|
||||
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:通知器
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_notifiers() -> None:
|
||||
"""通知器 notify 调用开销基准."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
# 构造一个真实 TaskEvent
|
||||
from pyflowx.task import TaskEvent, TaskStatus
|
||||
|
||||
event = TaskEvent(task="t0", status=TaskStatus.SUCCESS, attempts=1)
|
||||
|
||||
# CallbackNotifier(无级别过滤)
|
||||
calls: list[int] = [0]
|
||||
|
||||
def _cb(_e: Any) -> None:
|
||||
calls[0] += 1
|
||||
|
||||
notifier = CallbackNotifier(_cb)
|
||||
ms, ops = time_it(lambda e=event, n=notifier: n.notify(e), iterations=50000, warmup=1000)
|
||||
results.append(("CallbackNotifier.notify", 50000, ms, ops))
|
||||
|
||||
# 级别过滤(仅 SUCCESS)
|
||||
notifier_filtered = CallbackNotifier(_cb, levels={NotificationLevel.SUCCESS})
|
||||
ms, ops = time_it(lambda e=event, n=notifier_filtered: n.notify(e), iterations=50000, warmup=1000)
|
||||
results.append(("CallbackNotifier(filtered).notify", 50000, ms, ops))
|
||||
|
||||
print_results("通知器 notify", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:run_iter 流式 API
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_run_iter() -> None:
|
||||
"""run_iter 流式执行基准."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
def noop() -> None:
|
||||
pass
|
||||
|
||||
for n in (50, 200, 500):
|
||||
specs = [TaskSpec(f"t{i}", fn=noop) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
|
||||
def _run_iter(g: Graph = graph) -> None:
|
||||
list(px.run_iter(g, strategy="sequential"))
|
||||
|
||||
ms, ops = time_it(_run_iter, iterations=10, warmup=2)
|
||||
results.append((f"run_iter(sequential,{n})", 10, ms, ops))
|
||||
|
||||
print_results("run_iter 流式执行", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:子图过滤
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_subgraph() -> None:
|
||||
"""子图过滤基准(subgraph_with_deps 传递闭包计算)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
for n in (100, 500, 1000):
|
||||
specs = []
|
||||
for i in range(n):
|
||||
deps = (f"t{i - 1}",) if i > 0 else ()
|
||||
specs.append(TaskSpec(f"t{i}", cmd=["true"], depends_on=deps))
|
||||
graph = Graph.from_specs(specs)
|
||||
# 取中点任务,触发前半部分传递闭包
|
||||
target = f"t{n // 2}"
|
||||
ms, ops = time_it(
|
||||
lambda g=graph, t=target: g.subgraph_with_deps([t]),
|
||||
iterations=50,
|
||||
warmup=5,
|
||||
)
|
||||
results.append((f"subgraph_with_deps({n})", 50, ms, ops))
|
||||
|
||||
print_results("子图过滤 (subgraph_with_deps)", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:取消机制
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_cancellation() -> None:
|
||||
"""取消机制基准(cancel_event 触发到返回)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
def slow() -> None:
|
||||
time.sleep(0.5)
|
||||
|
||||
for n in (50, 200):
|
||||
specs = [TaskSpec(f"t{i}", fn=slow) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
|
||||
def _cancel_after(g: Graph = graph) -> None:
|
||||
token = CancelToken()
|
||||
# 立即取消,让所有任务变 SKIPPED
|
||||
token.cancel()
|
||||
with contextlib.suppress(Exception):
|
||||
px.run(g, strategy="sequential", cancel_event=token)
|
||||
|
||||
ms, ops = time_it(_cancel_after, iterations=5, warmup=1)
|
||||
results.append((f"cancel(immediate,{n})", 5, ms, ops))
|
||||
|
||||
print_results("取消机制 (cancel_event)", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:CPU 密集型任务
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _fib(n: int) -> int:
|
||||
"""递归斐波那契(CPU 密集型)."""
|
||||
if n < 2:
|
||||
return n
|
||||
return _fib(n - 1) + _fib(n - 2)
|
||||
|
||||
|
||||
def bench_cpu_intensive() -> None:
|
||||
"""CPU 密集型任务基准(递归斐波那契)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
for n_tasks, fib_n in ((10, 20), (20, 20), (10, 25)):
|
||||
specs = [TaskSpec(f"t{i}", fn=_fib, args=(fib_n,)) for i in range(n_tasks)]
|
||||
graph = Graph.from_specs(specs)
|
||||
|
||||
# sequential
|
||||
ms, ops = time_it(
|
||||
lambda g=graph: px.run(g, strategy="sequential"),
|
||||
iterations=3,
|
||||
warmup=1,
|
||||
)
|
||||
results.append((f"cpu-seq({n_tasks}x fib{fib_n})", 3, ms, ops))
|
||||
|
||||
# thread(CPU 密集型受 GIL 限制,验证是否退化)
|
||||
ms, ops = time_it(
|
||||
lambda g=graph: px.run(g, strategy="thread", max_workers=4),
|
||||
iterations=3,
|
||||
warmup=1,
|
||||
)
|
||||
results.append((f"cpu-thread({n_tasks}x fib{fib_n})", 3, ms, ops))
|
||||
|
||||
print_results("CPU 密集型 (递归斐波那契)", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 基准:I/O 密集型任务
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bench_io_intensive() -> None:
|
||||
"""I/O 密集型任务基准(sleep 模拟)."""
|
||||
results: list[tuple[str, int, float, float]] = []
|
||||
|
||||
def io_sleep() -> None:
|
||||
time.sleep(0.01) # 10ms 模拟 I/O
|
||||
|
||||
for n in (20, 50):
|
||||
specs = [TaskSpec(f"t{i}", fn=io_sleep) for i in range(n)]
|
||||
graph = Graph.from_specs(specs)
|
||||
|
||||
# sequential(总耗时 ≈ n * 10ms)
|
||||
ms, ops = time_it(
|
||||
lambda g=graph: px.run(g, strategy="sequential"),
|
||||
iterations=3,
|
||||
warmup=1,
|
||||
)
|
||||
results.append((f"io-seq({n})", 3, ms, ops))
|
||||
|
||||
# thread(I/O 密集型应能并行,总耗时 ≈ 10ms)
|
||||
ms, ops = time_it(
|
||||
lambda g=graph: px.run(g, strategy="thread", max_workers=8),
|
||||
iterations=3,
|
||||
warmup=1,
|
||||
)
|
||||
results.append((f"io-thread({n})", 3, ms, ops))
|
||||
|
||||
# async(I/O 密集型应能并行)
|
||||
ms, ops = time_it(
|
||||
lambda g=graph: px.run(g, strategy="async"),
|
||||
iterations=3,
|
||||
warmup=1,
|
||||
)
|
||||
results.append((f"io-async({n})", 3, ms, ops))
|
||||
|
||||
print_results("I/O 密集型 (sleep 10ms)", results)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 主入口
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def run_advanced() -> None:
|
||||
"""运行全部高级基准."""
|
||||
bench_conditions()
|
||||
bench_yaml_load()
|
||||
bench_notifiers()
|
||||
bench_run_iter()
|
||||
bench_subgraph()
|
||||
bench_cancellation()
|
||||
bench_cpu_intensive()
|
||||
bench_io_intensive()
|
||||
-102
@@ -1,102 +0,0 @@
|
||||
API 参考
|
||||
========
|
||||
|
||||
任务描述
|
||||
--------
|
||||
|
||||
.. autoclass:: pyflowx.TaskSpec
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
:exclude-members: args, kwargs
|
||||
|
||||
.. autoclass:: pyflowx.RetryPolicy
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autoclass:: pyflowx.TaskHooks
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autoclass:: pyflowx.TaskStatus
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
图构建
|
||||
------
|
||||
|
||||
.. autoclass:: pyflowx.Graph
|
||||
:members:
|
||||
:undoc-members:
|
||||
:exclude-members: from_specs, from_yaml
|
||||
|
||||
.. autoclass:: pyflowx.GraphDefaults
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autofunction:: pyflowx.compose
|
||||
.. autofunction:: pyflowx.task_template
|
||||
|
||||
执行
|
||||
----
|
||||
|
||||
.. autofunction:: pyflowx.run
|
||||
|
||||
.. autoclass:: pyflowx.RunReport
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autoclass:: pyflowx.TaskResult
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
@px.tool 工具
|
||||
-------------
|
||||
|
||||
.. autoclass:: pyflowx.ToolSpec
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autofunction:: pyflowx.tool
|
||||
.. autofunction:: pyflowx.run_tool
|
||||
.. autofunction:: pyflowx.list_tools
|
||||
.. autofunction:: pyflowx.list_subcommands
|
||||
|
||||
命令执行
|
||||
--------
|
||||
|
||||
.. autofunction:: pyflowx.run_command
|
||||
|
||||
CLI 运行器
|
||||
----------
|
||||
|
||||
.. autoclass:: pyflowx.CliRunner
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
状态后端
|
||||
--------
|
||||
|
||||
.. autoclass:: pyflowx.StateBackend
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autoclass:: pyflowx.MemoryBackend
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
.. autoclass:: pyflowx.JSONBackend
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
错误家族
|
||||
--------
|
||||
|
||||
.. autoexception:: pyflowx.PyFlowXError
|
||||
.. autoexception:: pyflowx.DuplicateTaskError
|
||||
.. autoexception:: pyflowx.MissingDependencyError
|
||||
.. autoexception:: pyflowx.CycleError
|
||||
.. autoexception:: pyflowx.TaskFailedError
|
||||
.. autoexception:: pyflowx.TaskTimeoutError
|
||||
.. autoexception:: pyflowx.InjectionError
|
||||
.. autoexception:: pyflowx.StorageError
|
||||
@@ -1,45 +0,0 @@
|
||||
变更日志
|
||||
========
|
||||
|
||||
0.4.5
|
||||
-----
|
||||
|
||||
CLI 重构
|
||||
~~~~~~~~
|
||||
|
||||
- 新增 ``pf`` 统一入口:通过 ``pf <tool> [command] [options]`` 调用所有工具
|
||||
- 13 个工具迁移到 YAML 配置(filedate/filelevel/folderback/folderzip/screenshot/sshcopyid/lscalc/bumpversion/autofmt/piptool/packtool/pdftool/gittool)
|
||||
- YAML 配置支持 ``cli:`` 段声明命令行参数 schema,由 ``build_cli_parser`` 自动生成 argparse
|
||||
- 删除 13 个冗余 ``.py`` 入口脚本,统一通过 ``pf`` 调用
|
||||
- ``run()`` 在 ``verbose=True`` 时自动把 verbose 标记应用到所有 spec
|
||||
- 全局选项 ``--verbose`` 改为 ``--quiet``(默认显示执行过程)
|
||||
- ``cmd`` 任务成功时打印 stdout(此前被静默丢弃)
|
||||
- ``gittool`` 用 ``CLEAN_EXCLUDES`` 数组变量配置 ``git clean -e`` 参数
|
||||
|
||||
YAML 任务编排
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
- 支持 ``variables`` 变量定义,``${VAR}`` 在 cmd/env/cwd 中替换
|
||||
- 列表变量展开为 cmd 数组多个元素
|
||||
- ``cli:`` 段支持 subcommands/positional/options 三级 schema
|
||||
- 支持 ``type: path`` 自动转为 ``pathlib.Path``
|
||||
|
||||
文档
|
||||
~~~~
|
||||
|
||||
- 搭建 Sphinx 文档,发布到 ReadTheDocs
|
||||
- 更新 README:CLI 示例改为 ``pf`` 统一入口,模块结构表补全
|
||||
|
||||
0.3.x
|
||||
-----
|
||||
|
||||
- 新增 YAML 任务编排(GitHub Actions 风格 schema)
|
||||
- 新增 ``fn:`` 函数引用与 ``register_fn`` / ``get_fn`` 注册中心
|
||||
- 新增 ``compose`` / ``GraphComposer`` 多图组合
|
||||
- 新增 ``task_template`` 任务模板工厂
|
||||
- 新增 ``concurrency_key`` + ``concurrency_limits`` 并发限制
|
||||
- 新增 ``JSONBackend`` 断点续跑与 ``batch()`` 批量落盘
|
||||
- 新增 ``cache_key`` 缓存键函数
|
||||
- 新增条件执行(``IS_WINDOWS`` / ``HAS_INSTALLED`` / ``ENV_VAR_EQUALS`` 等)
|
||||
- 四种执行策略:``sequential`` / ``thread`` / ``async`` / ``dependency``
|
||||
- 参数名即依赖的上下文注入机制
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Sphinx 配置.
|
||||
|
||||
ReadTheDocs 构建 PyFlowX 文档站。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 确保 src/ 在 sys.path 中, autodoc 能导入 pyflowx
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from pyflowx import __version__
|
||||
|
||||
# -- 项目信息 --------------------------------------------------------------
|
||||
project = "PyFlowX"
|
||||
author = "pyflowx"
|
||||
copyright = "2024, pyflowx"
|
||||
release = __version__
|
||||
version = __version__
|
||||
|
||||
# -- Sphinx 配置 -----------------------------------------------------------
|
||||
extensions = [
|
||||
"sphinx.ext.autodoc",
|
||||
"sphinx.ext.napoleon",
|
||||
"sphinx.ext.viewcode",
|
||||
"sphinx.ext.intersphinx",
|
||||
"myst_parser",
|
||||
]
|
||||
|
||||
# -- 主题 ------------------------------------------------------------------
|
||||
html_theme = "sphinx_rtd_theme"
|
||||
html_static_path = ["_static"]
|
||||
|
||||
# -- autodoc 配置 ----------------------------------------------------------
|
||||
autodoc_default_options = {
|
||||
"members": True,
|
||||
"undoc-members": True,
|
||||
"show-inheritance": True,
|
||||
"member-order": "bysource",
|
||||
}
|
||||
autodoc_type_hints = "description"
|
||||
autodoc_typehints_format = "short"
|
||||
|
||||
# -- napoleon 配置 (Google/NumPy docstring 兼容) --------------------------
|
||||
napoleon_google_docstring = True
|
||||
napoleon_numpy_docstring = True
|
||||
napoleon_include_init_with_doc = False
|
||||
napoleon_include_private_with_doc = False
|
||||
napoleon_include_special_with_doc = True
|
||||
|
||||
# -- intersphinx -----------------------------------------------------------
|
||||
intersphinx_mapping = {
|
||||
"python": ("https://docs.python.org/3", None),
|
||||
}
|
||||
|
||||
# -- 全局选项 ---------------------------------------------------------------
|
||||
language = "zh_CN"
|
||||
master_doc = "index"
|
||||
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
|
||||
source_suffix = {
|
||||
".rst": "restructuredtext",
|
||||
".md": "markdown",
|
||||
}
|
||||
@@ -1,158 +0,0 @@
|
||||
pf 统一 CLI 入口
|
||||
================
|
||||
|
||||
所有工具通过 ``pf <tool> [command] [options]`` 调用。工具定义在 ``cli/configs/`` 目录下的 YAML 文件中。
|
||||
|
||||
基本用法
|
||||
--------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pf # 列出所有可用工具
|
||||
pf filedate # 查看 filedate 工具帮助
|
||||
pf filedate add a.txt # 调用 filedate 的 add 子命令
|
||||
pf gitt c # 调用 gittool 的 c 子命令
|
||||
pf pymake b # 调用 pymake 的 b 别名
|
||||
|
||||
全局选项
|
||||
--------
|
||||
|
||||
所有 YAML 工具支持以下全局选项:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 75
|
||||
|
||||
* - 选项
|
||||
- 说明
|
||||
* - ``--dry-run``
|
||||
- 仅打印执行计划,不执行
|
||||
* - ``--quiet`` / ``-q``
|
||||
- 减少输出,不显示执行过程
|
||||
* - ``--strategy``
|
||||
- 执行策略(``sequential`` / ``thread`` / ``async`` / ``dependency``)
|
||||
* - ``--list``
|
||||
- 列出所有任务名后退出
|
||||
|
||||
默认 ``verbose`` 开启,显示执行过程(任务开始/命令/返回码/任务成功)。``--quiet`` 关闭。
|
||||
|
||||
YAML 配置工具
|
||||
--------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 15 65
|
||||
|
||||
* - 工具
|
||||
- 别名
|
||||
- 说明
|
||||
* - ``filedate``
|
||||
- ``fd``
|
||||
- 文件日期处理
|
||||
* - ``filelevel``
|
||||
- ``fl``
|
||||
- 文件等级重命名
|
||||
* - ``folderback``
|
||||
- ``fb``
|
||||
- 文件夹备份
|
||||
* - ``folderzip``
|
||||
- ``fz``
|
||||
- 文件夹压缩
|
||||
* - ``gittool``
|
||||
- ``gitt``
|
||||
- Git 执行工具
|
||||
* - ``lscalc``
|
||||
- ``ls``
|
||||
- LS-DYNA 计算工具
|
||||
* - ``packtool``
|
||||
- ``pack``
|
||||
- Python 打包工具
|
||||
* - ``pdftool``
|
||||
- ``pdf``
|
||||
- PDF 文件工具集
|
||||
* - ``piptool``
|
||||
- ``pip``
|
||||
- pip 包管理工具
|
||||
* - ``screenshot``
|
||||
- ``ss``
|
||||
- 截图工具
|
||||
* - ``sshcopyid``
|
||||
- ``ssh``
|
||||
- SSH 密钥部署工具
|
||||
* - ``autofmt``
|
||||
- ``af``
|
||||
- 自动格式化工具
|
||||
* - ``bumpversion``
|
||||
- ``bump``
|
||||
- 版本号自动管理工具
|
||||
|
||||
传统工具
|
||||
--------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 80
|
||||
|
||||
* - 工具
|
||||
- 说明
|
||||
* - ``pymake``
|
||||
- 构建工具(替代 Makefile),如 ``pf pymake b`` 构建
|
||||
* - ``yamlrun``
|
||||
- YAML pipeline 执行器,``pf yamlrun pipeline.yaml``
|
||||
* - ``profiler``
|
||||
- 性能分析
|
||||
* - ``emlman``
|
||||
- 邮件管理
|
||||
* - ``reseticon``
|
||||
- 重置图标缓存
|
||||
|
||||
自定义工具
|
||||
----------
|
||||
|
||||
在 ``cli/configs/`` 目录新建 ``<tool>.yaml`` 即可被 ``pf`` 自动发现:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# cli/configs/mytool.yaml
|
||||
strategy: sequential
|
||||
variables:
|
||||
MSG: "hello"
|
||||
cli:
|
||||
description: "我的工具"
|
||||
usage: "pf mytool [command]"
|
||||
subcommands:
|
||||
greet:
|
||||
help: "打招呼"
|
||||
jobs:
|
||||
greet:
|
||||
cmd: ["echo", "${MSG}"]
|
||||
|
||||
执行::
|
||||
|
||||
pf mytool greet
|
||||
|
||||
CliRunner(编程式)
|
||||
-------------------
|
||||
|
||||
``CliRunner`` 把多个 Graph 映射为命令行子命令,适合构建项目专属构建工具:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
runner = px.CliRunner(
|
||||
strategy="sequential",
|
||||
description="My Build Tool",
|
||||
graphs={
|
||||
"clean": clean_graph,
|
||||
"build": build_graph,
|
||||
"test": test_graph,
|
||||
},
|
||||
)
|
||||
runner.run_cli() # 解析 sys.argv 并执行
|
||||
|
||||
命令行::
|
||||
|
||||
pf pymake clean
|
||||
pf pymake build --strategy thread
|
||||
pf pymake test --dry-run
|
||||
pf pymake --list
|
||||
pf pymake --quiet
|
||||
@@ -1,93 +0,0 @@
|
||||
执行策略与 run()
|
||||
=================
|
||||
|
||||
``run()`` 是执行入口,支持四种策略:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
report = px.run(
|
||||
graph,
|
||||
strategy="async", # sequential | thread | async | dependency
|
||||
max_workers=8, # thread 策略的线程池大小
|
||||
concurrency_limits={"db": 2}, # 按 concurrency_key 限流
|
||||
dry_run=False, # True = 仅打印计划
|
||||
verbose=True, # True = 打印执行过程
|
||||
on_event=callback, # 状态转换回调
|
||||
state=px.JSONBackend("state.json"), # 断点续跑后端
|
||||
continue_on_error=False, # True = 单任务失败不中断整体
|
||||
)
|
||||
|
||||
策略对比
|
||||
--------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 18 18 30 16 18
|
||||
|
||||
* - 策略
|
||||
- 并发模型
|
||||
- 适用场景
|
||||
- 同步任务
|
||||
- 异步任务
|
||||
* - ``sequential``
|
||||
- 串行
|
||||
- 调试、CPU 密集
|
||||
- 直接调用
|
||||
- 事件循环
|
||||
* - ``thread``
|
||||
- 线程池
|
||||
- I/O 密集同步
|
||||
- 线程池
|
||||
- 不支持
|
||||
* - ``async``
|
||||
- 事件循环
|
||||
- I/O 密集异步
|
||||
- 卸载到线程池
|
||||
- 事件循环
|
||||
* - ``dependency``
|
||||
- 依赖驱动
|
||||
- 最大化并行度
|
||||
- 卸载到线程池
|
||||
- 事件循环
|
||||
|
||||
所有策略都遵循 ``RetryPolicy``、``timeout``、上下文注入、状态后端、``concurrency_limits``,
|
||||
并发出 ``TaskEvent``(RUNNING/SUCCESS/FAILED/SKIPPED)。``dependency`` 策略无层屏障:
|
||||
任务在其所有硬依赖完成后立即启动。
|
||||
|
||||
上下文注入规则
|
||||
--------------
|
||||
|
||||
按顺序求值:
|
||||
|
||||
1. **标注为 ``Context``** 的参数 → 接收完整上游结果映射
|
||||
2. **名称匹配依赖** 的参数 → 接收该依赖的结果(含软依赖,缺失时注入默认值)
|
||||
3. **``**kwargs``** 参数 → 接收所有依赖结果(dict)
|
||||
4. **``TaskSpec.args`` / ``kwargs``** → 为非依赖参数提供静态值
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
def aggregate(ctx: px.Context) -> Dict[str, Any]:
|
||||
"""ctx 包含所有 depends_on 任务的返回值。"""
|
||||
return dict(ctx)
|
||||
|
||||
def merge(fetch_a: str, fetch_b: str) -> str:
|
||||
"""fetch_a / fetch_b 自动注入。"""
|
||||
return fetch_a + fetch_b
|
||||
|
||||
断点续跑
|
||||
--------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pyflowx import JSONBackend
|
||||
|
||||
backend = JSONBackend("state.json", ttl=3600)
|
||||
report = px.run(graph, strategy="sequential", state=backend)
|
||||
|
||||
``run()`` 内部以 ``backend.batch()`` 包裹整个执行:所有 ``save`` 延迟到运行结束时统一落盘一次。
|
||||
|
||||
缓存键:默认存储键为任务名。配置 ``cache_key`` 函数后,键为 ``"name:cache_key_value"``。
|
||||
|
||||
完整 API 说明详见 :doc:`/api`。
|
||||
@@ -1,50 +0,0 @@
|
||||
Graph —— DAG 构建
|
||||
=================
|
||||
|
||||
``Graph`` 管理任务集合,提供建构建、校验、分层、可视化能力。
|
||||
|
||||
构建方式
|
||||
--------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# 图级默认值:TaskSpec 字段为 None 时回退
|
||||
defaults = px.GraphDefaults(retry=px.RetryPolicy(max_attempts=2), timeout=60.0)
|
||||
|
||||
graph = px.Graph.from_specs([...], defaults=defaults) # 整批校验(推荐)
|
||||
|
||||
# 或增量构建
|
||||
graph = px.Graph(defaults=defaults)
|
||||
graph.add(px.TaskSpec("a", fn_a))
|
||||
graph.add(px.TaskSpec("b", fn_b, ("a",)))
|
||||
|
||||
常用方法
|
||||
--------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
graph.validate() # 显式校验(环检测)
|
||||
graph.layers() # 拓扑分层(Kahn 算法)
|
||||
graph.to_mermaid() # Mermaid 可视化
|
||||
graph.describe() # 人类可读摘要
|
||||
graph.subgraph(("api",)) # 按标签切片
|
||||
graph.subgraph_by_names(("a", "b")) # 按名称切片
|
||||
graph.map("fetch", [1, 2, 3], lambda i: TaskSpec(f"fetch_{i}", ...)) # 批量 fan-out
|
||||
|
||||
图组合
|
||||
------
|
||||
|
||||
``compose`` / ``GraphComposer`` 把带字符串引用的多个图展开为纯 ``Graph``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
graphs = {
|
||||
"build": px.Graph.from_specs([px.TaskSpec("b", cmd=["echo", "b"])]),
|
||||
"all": px.Graph.from_specs(["build", px.TaskSpec("t", cmd=["echo", "t"])]),
|
||||
}
|
||||
resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
|
||||
|
||||
引用格式:``"command_name"``(整个图)或 ``"command_name.task_name"``(特定任务)。
|
||||
``CliRunner`` 内部自动调用 ``compose``。
|
||||
|
||||
完整方法说明详见 :doc:`/api`。
|
||||
@@ -1,89 +0,0 @@
|
||||
TaskSpec —— 任务描述
|
||||
=====================
|
||||
|
||||
``TaskSpec`` 是不可变的任务描述符(``Generic[T]``,返回类型一路传到 ``RunReport``),是唯一需要配置的东西。
|
||||
|
||||
主要参数说明:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
px.TaskSpec(
|
||||
name="fetch_user", # 唯一标识
|
||||
fn=fetch_user, # 同步或异步函数
|
||||
cmd=["curl", "..."], # 或: 执行命令(覆盖 fn)
|
||||
depends_on=("auth",), # 硬依赖(参与拓扑分层)
|
||||
soft_depends_on=("cache",), # 软依赖(仅注入,不参与分层)
|
||||
args=(uid,), # 静态位置参数(追加在注入参数后)
|
||||
kwargs={"timeout": 30}, # 静态关键字参数
|
||||
retry=px.RetryPolicy(max_attempts=3, delay=1.0, backoff=2.0),
|
||||
timeout=30.0, # 超时秒数(None = 不限制)
|
||||
tags=("api", "user"), # 自由标签,用于子图过滤
|
||||
conditions=(is_prod,), # 条件函数列表(全部为 True 才执行)
|
||||
priority=10, # 同层内优先级(高优先执行,默认 0)
|
||||
concurrency_key="db", # 并发分组键(配合 concurrency_limits 限流)
|
||||
cache_key=lambda ctx: str(ctx.get("uid")), # 缓存键函数
|
||||
hooks=px.TaskHooks(pre_run=..., post_run=..., on_failure=...),
|
||||
cwd=Path("/tmp"), # 命令工作目录(仅 cmd 模式)
|
||||
env={"DEBUG": "1"}, # 环境变量覆盖
|
||||
verbose=True, # 打印命令输出(仅 cmd 模式)
|
||||
skip_if_missing=True, # 命令不存在时自动跳过(仅 list[str] cmd)
|
||||
allow_upstream_skip=False, # 上游 SKIPPED/FAILED 时是否仍执行
|
||||
continue_on_error=False, # 本任务失败是否不中断整体
|
||||
)
|
||||
|
||||
两种任务形态
|
||||
------------
|
||||
|
||||
- **函数任务**(``fn``):普通 Python 函数,参数名驱动自动注入
|
||||
- **命令任务**(``cmd``):执行外部命令,支持 ``list[str]``、``str``(shell)、``Callable`` 三种形态
|
||||
|
||||
``skip_if_missing=True`` 时,``list[str]`` 类型的 ``cmd`` 会通过 ``shutil.which`` 检查命令是否存在,不存在则跳过任务(标记为 ``SKIPPED``)而非失败。
|
||||
|
||||
重试策略
|
||||
--------
|
||||
|
||||
``RetryPolicy`` 配置重试次数、延迟、退避:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
retry = px.RetryPolicy(
|
||||
max_attempts=3, # 最大尝试次数
|
||||
delay=1.0, # 初始延迟秒数
|
||||
backoff=2.0, # 退避倍数
|
||||
jitter=0.1, # 随机抖动(避免惊群)
|
||||
retry_on=(ConnectionError,), # 仅对这些异常重试
|
||||
)
|
||||
|
||||
任务钩子
|
||||
--------
|
||||
|
||||
``TaskHooks`` 在任务生命周期触发(异常仅记录,不影响任务状态):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
hooks = px.TaskHooks(
|
||||
pre_run=lambda spec: print(f"start {spec.name}"),
|
||||
post_run=lambda spec, value: print(f"done {spec.name}"),
|
||||
on_failure=lambda spec, exc: alert(spec.name, exc),
|
||||
)
|
||||
px.TaskSpec("task", fn=work, hooks=hooks)
|
||||
|
||||
任务模板
|
||||
--------
|
||||
|
||||
``task_template`` 工厂批量生成相似 TaskSpec:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
fetch = px.task_template(
|
||||
fn=fetch_url,
|
||||
retry=px.RetryPolicy(max_attempts=5),
|
||||
timeout=30.0,
|
||||
tags=("api",),
|
||||
)
|
||||
graph = px.Graph.from_specs([
|
||||
fetch("users", url="https://api.example.com/users"),
|
||||
fetch("posts", url="https://api.example.com/posts"),
|
||||
])
|
||||
|
||||
完整字段说明详见 :doc:`/api`。
|
||||
@@ -1,164 +0,0 @@
|
||||
YAML 任务编排
|
||||
=============
|
||||
|
||||
PyFlowX 支持 GitHub Actions 风格的声明式 YAML 任务编排,从 YAML 文件直接加载任务图。
|
||||
|
||||
编程式 API
|
||||
----------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
# 从 YAML 文件加载任务图
|
||||
graph = px.Graph.from_yaml("pipeline.yaml")
|
||||
report = px.run(graph, strategy="thread")
|
||||
|
||||
# 或用函数式 API
|
||||
graph = px.load_yaml("pipeline.yaml")
|
||||
|
||||
# 从字符串解析
|
||||
graph = px.parse_yaml_string("""
|
||||
jobs:
|
||||
hello:
|
||||
cmd: ["echo", "hello"]
|
||||
""")
|
||||
|
||||
YAML Schema
|
||||
-----------
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
strategy: thread # 图级默认策略
|
||||
defaults: # 图级默认值
|
||||
retry: {max_attempts: 3}
|
||||
verbose: true
|
||||
env: {CI: "true"}
|
||||
|
||||
variables: # 变量定义 (可在 cmd/env 中 ${VAR} 引用)
|
||||
OUTPUT: "dist"
|
||||
|
||||
jobs:
|
||||
setup:
|
||||
cmd: ["git", "clone", "..."]
|
||||
runs-on: linux
|
||||
|
||||
build:
|
||||
needs: [setup] # 依赖列表
|
||||
cmd: ["python", "-m", "build"]
|
||||
timeout: 300
|
||||
retry: {max_attempts: 2, delay: 1.0}
|
||||
|
||||
test:
|
||||
needs: [build]
|
||||
cmd: ["python${{ matrix.version }}", "-m", "pytest"]
|
||||
strategy:
|
||||
matrix: # 笛卡尔积展开为 6 个任务
|
||||
version: ["3.8", "3.9", "3.10"]
|
||||
os: ["linux", "macos"]
|
||||
if: "env.CI" # 条件: 环境变量存在
|
||||
|
||||
lint:
|
||||
needs: [build]
|
||||
cmd: ["ruff", "check"]
|
||||
if: "env.CI == 'true'"
|
||||
|
||||
deploy:
|
||||
needs: [test, lint] # 矩阵依赖自动展开
|
||||
cmd: ["twine", "upload"]
|
||||
if: "env.DEPLOY_TOKEN != ''"
|
||||
allow-upstream-skip: true
|
||||
concurrency-key: deploy_lock
|
||||
|
||||
字段映射
|
||||
--------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 30 40
|
||||
|
||||
* - YAML 字段
|
||||
- TaskSpec 字段
|
||||
- 说明
|
||||
* - ``jobs.<id>``
|
||||
- ``name``
|
||||
- job ID 作为任务名
|
||||
* - ``cmd`` / ``run``
|
||||
- ``cmd``
|
||||
- ``cmd`` 为列表形式,``run`` 为 shell 字符串
|
||||
* - ``needs``
|
||||
- ``depends_on``
|
||||
- 依赖列表(矩阵任务自动展开)
|
||||
* - ``if``
|
||||
- ``conditions``
|
||||
- ``success()`` / ``always()`` / ``env.VAR`` / ``env.VAR == 'x'``
|
||||
* - ``strategy.matrix``
|
||||
- 矩阵扇出
|
||||
- 笛卡尔积展开为多个任务
|
||||
* - ``${{ matrix.key }}``
|
||||
- 占位符
|
||||
- 在 cmd/run/cwd/env 中替换
|
||||
* - ``timeout``
|
||||
- ``timeout``
|
||||
- 超时秒数
|
||||
* - ``retry``
|
||||
- ``retry``
|
||||
- ``{max_attempts, delay, backoff, jitter}``
|
||||
* - ``cwd``
|
||||
- ``cwd``
|
||||
- 工作目录
|
||||
* - ``env``
|
||||
- ``env``
|
||||
- 环境变量
|
||||
* - ``verbose``
|
||||
- ``verbose``
|
||||
- 详细输出
|
||||
* - ``continue-on-error``
|
||||
- ``continue_on_error``
|
||||
- 失败不中止整图
|
||||
* - ``skip-if-missing``
|
||||
- ``skip_if_missing``
|
||||
- 命令不存在时跳过
|
||||
* - ``allow-upstream-skip``
|
||||
- ``allow_upstream_skip``
|
||||
- 上游跳过时仍执行
|
||||
* - ``priority``
|
||||
- ``priority``
|
||||
- 同层优先级
|
||||
* - ``concurrency-key``
|
||||
- ``concurrency_key``
|
||||
- 并发限制键
|
||||
* - ``tags``
|
||||
- ``tags``
|
||||
- 自由标签
|
||||
* - ``runs-on``
|
||||
- ``tags``(追加)
|
||||
- 运行环境标签
|
||||
|
||||
CLI 配置段(``cli:``)
|
||||
----------------------
|
||||
|
||||
工具 YAML 还可定义 ``cli:`` 段,声明命令行参数 schema,由 ``pf`` 自动解析:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
cli:
|
||||
description: "FileDate - 文件日期处理工具"
|
||||
usage: "pf filedate <command> [files...]"
|
||||
subcommands:
|
||||
add:
|
||||
help: "添加日期前缀"
|
||||
positional:
|
||||
- name: FILES
|
||||
nargs: "+"
|
||||
type: path
|
||||
help: "文件路径"
|
||||
options:
|
||||
- name: CLEAR
|
||||
flag: "--clear"
|
||||
action: store_true
|
||||
help: "清除已有日期前缀"
|
||||
|
||||
支持的 ``type``:``str`` / ``int`` / ``float`` / ``path``。
|
||||
|
||||
完整 API 说明详见 :doc:`/api`。
|
||||
@@ -1,54 +0,0 @@
|
||||
PyFlowX 文档
|
||||
============
|
||||
|
||||
PyFlowX 是一个轻量、类型安全的 DAG 任务调度器:**参数名就是依赖声明**。
|
||||
无需装饰器、无需样板包装器,写一个普通函数,框架按参数名自动注入上游结果。
|
||||
|
||||
特性
|
||||
----
|
||||
|
||||
- **零样板** —— 参数名即依赖,框架自动注入上游结果
|
||||
- **四种执行策略** —— sequential(串行)、thread(线程池)、async(事件循环)、dependency(依赖驱动,最大化并行)
|
||||
- **类型安全** —— ``TaskSpec[T]`` 把返回类型一路传到 ``RunReport``
|
||||
- **DAG 校验** —— 构建时即时校验重名、缺失依赖、环
|
||||
- **自动分层** —— Kahn 算法分组,同层任务可并行
|
||||
- **重试与超时** —— 每个任务独立配置 ``RetryPolicy`` 与 ``timeout``
|
||||
- **并发限制** —— ``concurrency_key`` + ``concurrency_limits`` 按组限流
|
||||
- **断点续跑** —— ``MemoryBackend`` / ``JSONBackend``,成功结果可缓存复用
|
||||
- **命令任务** —— ``cmd`` 参数直接执行外部命令
|
||||
- **条件执行** —— ``conditions`` 按平台、环境变量等条件跳过任务
|
||||
- **pf 统一 CLI** —— ``pf <tool> [command]`` 调用所有工具
|
||||
- **最小依赖** —— ``rich`` + ``typer`` + ``typing-extensions``(3.13 以下)
|
||||
|
||||
文档导航
|
||||
--------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:caption: 入门
|
||||
|
||||
installation
|
||||
quickstart
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:caption: 用户指南
|
||||
|
||||
guide/task
|
||||
guide/graph
|
||||
guide/execution
|
||||
guide/cli
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:caption: 参考
|
||||
|
||||
api
|
||||
changelog
|
||||
|
||||
索引
|
||||
----
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`modindex`
|
||||
* :ref:`search`
|
||||
@@ -1,51 +0,0 @@
|
||||
安装
|
||||
====
|
||||
|
||||
PyFlowX 支持 Python 3.10+,运行时依赖 ``rich``、``typer``、``pyyaml`` 与 ``typing-extensions``(3.13 以下)。
|
||||
|
||||
pip 安装
|
||||
--------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pyflowx
|
||||
|
||||
uv 安装
|
||||
-------
|
||||
|
||||
推荐使用 `uv <https://docs.astral.sh/uv/>`_:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv add pyflowx
|
||||
|
||||
可选依赖
|
||||
--------
|
||||
|
||||
``office`` —— PDF/图片处理(pdftool、screenshot 等工具需要):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pyflowx[office]
|
||||
|
||||
``dev`` —— 开发工具链(ruff、pyrefly、pytest、tox 等):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pyflowx[dev]
|
||||
|
||||
验证安装
|
||||
--------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pf --version
|
||||
|
||||
输出示例::
|
||||
|
||||
PyFlowX 0.4.5
|
||||
|
||||
下一步
|
||||
------
|
||||
|
||||
前往 :doc:`quickstart` 开始使用。
|
||||
@@ -1,87 +0,0 @@
|
||||
快速上手
|
||||
========
|
||||
|
||||
核心思想:**参数名即依赖**。写一个普通函数,参数名匹配上游任务名,框架自动注入结果。
|
||||
|
||||
最小示例
|
||||
--------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
def extract() -> list[int]:
|
||||
return [1, 2, 3]
|
||||
|
||||
# 参数名 extract 自动匹配上游任务名 → 自动注入
|
||||
def double(extract: list[int]) -> list[int]:
|
||||
return [x * 2 for x in extract]
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("extract", extract),
|
||||
px.TaskSpec("double", double, ("extract",)),
|
||||
])
|
||||
|
||||
report = px.run(graph, strategy="sequential")
|
||||
print(report["double"]) # [2, 4, 6]
|
||||
|
||||
三种任务形态
|
||||
------------
|
||||
|
||||
1. **函数任务**(``fn``):普通 Python 函数,参数名驱动自动注入
|
||||
2. **命令任务**(``cmd``):执行外部命令,支持 ``list[str]`` / ``str``(shell)/ ``Callable``
|
||||
3. **YAML 声明式**:从 YAML 文件加载任务图
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("list", cmd=["ls", "-la"]),
|
||||
px.TaskSpec("greet", fn=lambda: "hello"),
|
||||
])
|
||||
|
||||
执行策略
|
||||
--------
|
||||
|
||||
PyFlowX 提供四种执行策略:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 20 60
|
||||
|
||||
* - 策略
|
||||
- 并发模型
|
||||
- 适用场景
|
||||
* - ``sequential``
|
||||
- 串行
|
||||
- 调试、CPU 密集
|
||||
* - ``thread``
|
||||
- 线程池
|
||||
- I/O 密集同步
|
||||
* - ``async``
|
||||
- 事件循环
|
||||
- I/O 密集异步
|
||||
* - ``dependency``
|
||||
- 依赖驱动
|
||||
- 最大化并行度(默认推荐)
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
report = px.run(graph, strategy="dependency")
|
||||
|
||||
结果访问
|
||||
--------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
report["task_name"] # 任务返回值
|
||||
report.result_of("task_name") # 完整 TaskResult
|
||||
report.success # 整体是否成功
|
||||
report.summary() # 统计字典
|
||||
report.failed_tasks() # 失败任务名列表
|
||||
|
||||
下一步
|
||||
------
|
||||
|
||||
- :doc:`guide/task` —— TaskSpec 详细配置
|
||||
- :doc:`guide/yaml` —— YAML 声明式任务编排
|
||||
- :doc:`guide/cli` —— ``pf`` 统一 CLI 入口
|
||||
+37
-18
@@ -7,26 +7,49 @@ classifiers = [
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Programming Language :: Python :: 3.14",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Topic :: Software Development :: Libraries :: Application Frameworks",
|
||||
]
|
||||
dependencies = [
|
||||
"pyyaml>=6.0.1",
|
||||
"rich>=13.7.0",
|
||||
"typer>=0.24.0",
|
||||
"typing-extensions>=4.13.2; python_version < '3.13'",
|
||||
"graphlib_backport >= 1.0.0; python_version < '3.9'",
|
||||
"typing-extensions>=4.13.2; python_version < '3.10'",
|
||||
]
|
||||
description = "Lightweight, type-safe DAG task scheduler with multi-strategy execution."
|
||||
keywords = ["async", "dag", "scheduler", "task", "workflow"]
|
||||
license = { text = "MIT" }
|
||||
name = "pyflowx"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
version = "0.4.8"
|
||||
requires-python = ">=3.8"
|
||||
version = "0.3.2"
|
||||
|
||||
[project.scripts]
|
||||
emlman = "pyflowx.cli.legacy.emlmanager:main"
|
||||
pf = "pyflowx.cli.pf:main"
|
||||
pxp = "pyflowx.cli.legacy.profiler:main"
|
||||
autofmt = "pyflowx.cli.autofmt:main"
|
||||
bumpversion = "pyflowx.cli.bumpversion:main"
|
||||
dockercmd = "pyflowx.cli.dev.dockercmd:main"
|
||||
emlman = "pyflowx.cli.emlmanager:main"
|
||||
filedate = "pyflowx.cli.filedate:main"
|
||||
filelvl = "pyflowx.cli.filelevel:main"
|
||||
foldback = "pyflowx.cli.folderback:main"
|
||||
foldzip = "pyflowx.cli.folderzip:main"
|
||||
gitt = "pyflowx.cli.gittool:main"
|
||||
lscalc = "pyflowx.cli.lscalc:main"
|
||||
msdown = "pyflowx.cli.llm.msdownload:main"
|
||||
packtool = "pyflowx.cli.packtool:main"
|
||||
pdftool = "pyflowx.cli.pdftool:main"
|
||||
piptool = "pyflowx.cli.piptool:main"
|
||||
pxp = "pyflowx.cli.profiler:main"
|
||||
pymake = "pyflowx.cli.pymake:main"
|
||||
reseticon = "pyflowx.cli.reseticoncache:main"
|
||||
scrcap = "pyflowx.cli.screenshot:main"
|
||||
sglang = "pyflowx.cli.llm.sglang:main"
|
||||
sshcopy = "pyflowx.cli.sshcopyid:main"
|
||||
# dev
|
||||
envdev = "pyflowx.cli.dev.envdev:main"
|
||||
# system
|
||||
clr = "pyflowx.cli.system.clearscreen:main"
|
||||
taskk = "pyflowx.cli.system.taskkill:main"
|
||||
wch = "pyflowx.cli.system.which:main"
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
@@ -44,8 +67,6 @@ dev = [
|
||||
"tox-uv>=1.13.1",
|
||||
"tox>=4.25.0",
|
||||
]
|
||||
docs = ["myst-parser>=3.0", "sphinx-rtd-theme>=2.0", "sphinx>=7.0"]
|
||||
fast = ["orjson>=3.10.0"]
|
||||
office = [
|
||||
"pillow>=10.4.0",
|
||||
"pymupdf>=1.24.11",
|
||||
@@ -74,12 +95,12 @@ packages = ["src/pyflowx"]
|
||||
pyflowx = { workspace = true }
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pyflowx[dev,docs,fast,office]"]
|
||||
dev = ["pyflowx[dev,office]"]
|
||||
|
||||
[tool.coverage.run]
|
||||
branch = true
|
||||
concurrency = ["thread"]
|
||||
omit = ["src/pyflowx/cli/*", "tests/*"]
|
||||
omit = ["src/pyflowx/cli/*", "src/pyflowx/examples/*", "tests/*"]
|
||||
source = ["pyflowx"]
|
||||
|
||||
[tool.coverage.report]
|
||||
@@ -99,7 +120,7 @@ markers = ["slow: marks tests as slow (deselect with
|
||||
# Ruff 配置 - 与 .pre-commit-config.yaml 保持一致
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py310"
|
||||
target-version = "py38"
|
||||
|
||||
[tool.ruff.lint]
|
||||
ignore = [
|
||||
@@ -132,11 +153,9 @@ select = [
|
||||
]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"**/tests/**" = ["ARG001", "ARG002"]
|
||||
"src/pyflowx/tools.py" = ["PLR0911"]
|
||||
"**/tests/**" = ["ARG001", "ARG002"]
|
||||
|
||||
[tool.pyrefly]
|
||||
preset = "strict"
|
||||
project-includes = ["**/*.ipynb", "**/*.py*"]
|
||||
python-version = "3.10"
|
||||
search-path = ["."]
|
||||
python-version = "3.8"
|
||||
|
||||
+13
-58
@@ -13,7 +13,7 @@
|
||||
* :class:`GraphDefaults` —— 图级默认值。
|
||||
* :func:`compose` —— 编程式组合多图。
|
||||
* :func:`task_template` —— 批量生成相似 TaskSpec 的工厂。
|
||||
* 状态后端::class:`StateBackend`、:class:`MemoryBackend`、:class:`JSONBackend`、:class:`SQLiteBackend`。
|
||||
* 状态后端::class:`StateBackend`、:class:`MemoryBackend`、:class:`JSONBackend`。
|
||||
|
||||
快速上手
|
||||
--------
|
||||
@@ -58,12 +58,18 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .cancellation import CancelToken
|
||||
from .command import run_command
|
||||
from .compose import GraphComposer, compose
|
||||
from .conditions import IS_LINUX, IS_MACOS, IS_POSIX, IS_WINDOWS, BuiltinConditions, Condition, Constants
|
||||
from .conditions import (
|
||||
IS_LINUX,
|
||||
IS_MACOS,
|
||||
IS_POSIX,
|
||||
IS_WINDOWS,
|
||||
BuiltinConditions,
|
||||
Condition,
|
||||
Constants,
|
||||
)
|
||||
from .context import Context, build_call_args, describe_injection
|
||||
from .diagnostics import DependencyChain, DiagnosticReport, FailureCluster, diagnose
|
||||
from .errors import (
|
||||
CycleError,
|
||||
DuplicateTaskError,
|
||||
@@ -74,31 +80,14 @@ from .errors import (
|
||||
TaskFailedError,
|
||||
TaskTimeoutError,
|
||||
)
|
||||
from .executors import Strategy, run, run_iter
|
||||
from .executors import Strategy, run
|
||||
from .graph import Graph, GraphDefaults
|
||||
from .history import RunHistory
|
||||
from .monitoring import MetricsCollector, health_check, start_metrics_server
|
||||
from .notification import (
|
||||
ALL_LEVELS,
|
||||
CallbackNotifier,
|
||||
DingTalkNotifier,
|
||||
DiscordNotifier,
|
||||
FeishuNotifier,
|
||||
NotificationLevel,
|
||||
Notifier,
|
||||
SlackNotifier,
|
||||
TelegramNotifier,
|
||||
WebhookNotifier,
|
||||
WeChatNotifier,
|
||||
)
|
||||
from .profiling import ProfileReport, TaskProfile
|
||||
from .progress import ProgressCallback, RichProgressMonitor
|
||||
from .report import RunReport
|
||||
from .runner import CliExitCode, CliRunner
|
||||
from .storage import JSONBackend, MemoryBackend, SQLiteBackend, StateBackend
|
||||
from .storage import JSONBackend, MemoryBackend, StateBackend
|
||||
from .task import (
|
||||
CacheKeyFn,
|
||||
LoopSpec,
|
||||
RetryPolicy,
|
||||
TaskCmd,
|
||||
TaskEvent,
|
||||
@@ -110,54 +99,34 @@ from .task import (
|
||||
task,
|
||||
task_template,
|
||||
)
|
||||
from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool
|
||||
from .yaml_loader import load_yaml, parse_yaml_string
|
||||
|
||||
__version__ = "0.4.8"
|
||||
__version__ = "0.4.2"
|
||||
|
||||
__all__ = [
|
||||
"ALL_LEVELS",
|
||||
"IS_LINUX",
|
||||
"IS_MACOS",
|
||||
"IS_POSIX",
|
||||
"IS_WINDOWS",
|
||||
"BuiltinConditions",
|
||||
"CacheKeyFn",
|
||||
"CallbackNotifier",
|
||||
"CancelToken",
|
||||
"CliExitCode",
|
||||
"CliRunner",
|
||||
"Condition",
|
||||
"Constants",
|
||||
"Context",
|
||||
"CycleError",
|
||||
"DependencyChain",
|
||||
"DiagnosticReport",
|
||||
"DingTalkNotifier",
|
||||
"DiscordNotifier",
|
||||
"DuplicateTaskError",
|
||||
"FailureCluster",
|
||||
"FeishuNotifier",
|
||||
"Graph",
|
||||
"GraphComposer",
|
||||
"GraphDefaults",
|
||||
"InjectionError",
|
||||
"JSONBackend",
|
||||
"LoopSpec",
|
||||
"MemoryBackend",
|
||||
"MetricsCollector",
|
||||
"MissingDependencyError",
|
||||
"NotificationLevel",
|
||||
"Notifier",
|
||||
"ProfileReport",
|
||||
"ProgressCallback",
|
||||
"PyFlowXError",
|
||||
"RetryPolicy",
|
||||
"RichProgressMonitor",
|
||||
"RunHistory",
|
||||
"RunReport",
|
||||
"SQLiteBackend",
|
||||
"SlackNotifier",
|
||||
"StateBackend",
|
||||
"StorageError",
|
||||
"Strategy",
|
||||
@@ -170,26 +139,12 @@ __all__ = [
|
||||
"TaskSpec",
|
||||
"TaskStatus",
|
||||
"TaskTimeoutError",
|
||||
"TelegramNotifier",
|
||||
"ToolSpec",
|
||||
"WeChatNotifier",
|
||||
"WebhookNotifier",
|
||||
"build_call_args",
|
||||
"cmd",
|
||||
"compose",
|
||||
"describe_injection",
|
||||
"diagnose",
|
||||
"health_check",
|
||||
"list_subcommands",
|
||||
"list_tools",
|
||||
"load_yaml",
|
||||
"parse_yaml_string",
|
||||
"run",
|
||||
"run_command",
|
||||
"run_iter",
|
||||
"run_tool",
|
||||
"start_metrics_server",
|
||||
"task",
|
||||
"task_template",
|
||||
"tool",
|
||||
]
|
||||
|
||||
@@ -1,100 +0,0 @@
|
||||
"""JSON 序列化抽象层。
|
||||
|
||||
优先使用 orjson(性能更高),回退到标准库 json。对外提供与标准库
|
||||
json 兼容的 ``dumps`` / ``loads`` / ``dump`` / ``load`` / ``JSONDecodeError``
|
||||
接口,使调用方无感知切换。
|
||||
|
||||
orjson 与标准库的差异由本模块吸收:
|
||||
|
||||
- ``dumps`` 返回 ``str``(orjson 原生返回 ``bytes``,此处统一 decode)
|
||||
- ``ensure_ascii`` 参数对 orjson 无效(始终输出 UTF-8),此处忽略
|
||||
- ``indent`` 映射为 ``orjson.OPT_INDENT_2``
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["JSONDecodeError", "dump", "dumps", "load", "loads"]
|
||||
|
||||
from typing import IO, Any
|
||||
|
||||
try:
|
||||
import orjson # type: ignore[import-not-found]
|
||||
|
||||
_HAS_ORJSON = True
|
||||
except ImportError: # pragma: no cover
|
||||
_HAS_ORJSON = False
|
||||
|
||||
|
||||
if _HAS_ORJSON:
|
||||
JSONDecodeError = orjson.JSONDecodeError # type: ignore[possibly-unbound]
|
||||
_OPT_NON_STR_KEYS = orjson.OPT_NON_STR_KEYS # type: ignore[possibly-unbound]
|
||||
|
||||
def dumps(
|
||||
obj: Any,
|
||||
*,
|
||||
ensure_ascii: bool = False, # noqa: ARG001 — 兼容标准库签名
|
||||
indent: int | None = None,
|
||||
default: Any = None,
|
||||
**_kwargs: Any,
|
||||
) -> str:
|
||||
"""序列化为 JSON 字符串(orjson 后端)。"""
|
||||
opts = _OPT_NON_STR_KEYS
|
||||
if indent is not None:
|
||||
opts |= orjson.OPT_INDENT_2 # type: ignore[possibly-unbound]
|
||||
return orjson.dumps(obj, default=default, option=opts).decode("utf-8") # type: ignore[possibly-unbound]
|
||||
|
||||
def loads(s: str | bytes) -> Any:
|
||||
"""从 JSON 字符串/字节反序列化(orjson 后端)。"""
|
||||
return orjson.loads(s) # type: ignore[possibly-unbound]
|
||||
|
||||
def dump(
|
||||
obj: Any,
|
||||
fh: IO[str],
|
||||
*,
|
||||
ensure_ascii: bool = False,
|
||||
indent: int | None = None,
|
||||
default: Any = None,
|
||||
**_kwargs: Any,
|
||||
) -> None:
|
||||
"""序列化并写入文件句柄(orjson 后端)。"""
|
||||
fh.write(dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default))
|
||||
|
||||
def load(fh: IO[str]) -> Any:
|
||||
"""从文件句柄反序列化(orjson 后端)。"""
|
||||
return orjson.loads(fh.read()) # type: ignore[possibly-unbound]
|
||||
|
||||
else: # pragma: no cover
|
||||
import json as _stdlib
|
||||
|
||||
JSONDecodeError = _stdlib.JSONDecodeError
|
||||
|
||||
def dumps(
|
||||
obj: Any,
|
||||
*,
|
||||
ensure_ascii: bool = False,
|
||||
indent: int | None = None,
|
||||
default: Any = None,
|
||||
**_kwargs: Any,
|
||||
) -> str:
|
||||
"""序列化为 JSON 字符串(标准库后端)。"""
|
||||
return _stdlib.dumps(obj, ensure_ascii=ensure_ascii, indent=indent, default=default)
|
||||
|
||||
def loads(s: str | bytes) -> Any:
|
||||
"""从 JSON 字符串/字节反序列化(标准库后端)。"""
|
||||
return _stdlib.loads(s)
|
||||
|
||||
def dump(
|
||||
obj: Any,
|
||||
fh: IO[str],
|
||||
*,
|
||||
ensure_ascii: bool = False,
|
||||
indent: int | None = None,
|
||||
default: Any = None,
|
||||
**_kwargs: Any,
|
||||
) -> None:
|
||||
"""序列化并写入文件句柄(标准库后端)。"""
|
||||
_stdlib.dump(obj, fh, ensure_ascii=ensure_ascii, indent=indent, default=default)
|
||||
|
||||
def load(fh: IO[str]) -> Any:
|
||||
"""从文件句柄反序列化(标准库后端)。"""
|
||||
return _stdlib.load(fh)
|
||||
@@ -1,114 +0,0 @@
|
||||
"""任务取消与优雅停止。
|
||||
|
||||
提供 :class:`CancelToken` 作为线程安全的取消信号,可传入 :func:`pyflowx.run`
|
||||
实现外部取消或响应 ``KeyboardInterrupt``。
|
||||
|
||||
设计要点
|
||||
--------
|
||||
* 基于 :class:`threading.Event`,跨线程/协程安全。
|
||||
* 轻量:不引入额外线程或回调,仅维护一个事件标志。
|
||||
* 可组合:多个 CancelToken 可通过 :meth:`link` 联动。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
__all__ = ["CancelToken"]
|
||||
|
||||
|
||||
class CancelToken:
|
||||
"""线程安全的取消令牌。
|
||||
|
||||
用法::
|
||||
|
||||
token = CancelToken()
|
||||
# 在另一个线程中:
|
||||
token.cancel()
|
||||
|
||||
# 传入 run():
|
||||
report = px.run(graph, cancel_event=token)
|
||||
|
||||
也可直接用 :class:`threading.Event` 代替——本类仅提供语义更清晰的 API。
|
||||
"""
|
||||
|
||||
__slots__ = ("_event", "_reason")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._event = threading.Event()
|
||||
self._reason: str | None = None
|
||||
|
||||
def cancel(self, reason: str | None = None) -> None:
|
||||
"""设置取消信号。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
reason:
|
||||
可选的取消原因描述,供日志或调试使用。
|
||||
"""
|
||||
if reason is not None:
|
||||
self._reason = reason
|
||||
self._event.set()
|
||||
|
||||
@property
|
||||
def is_cancelled(self) -> bool:
|
||||
"""是否已取消。"""
|
||||
return self._event.is_set()
|
||||
|
||||
@property
|
||||
def reason(self) -> str | None:
|
||||
"""取消原因(未取消时为 None)。"""
|
||||
return self._reason
|
||||
|
||||
def wait(self, timeout: float | None = None) -> bool:
|
||||
"""等待取消信号。
|
||||
|
||||
Returns
|
||||
-------
|
||||
bool
|
||||
``True`` 表示已取消,``False`` 表示超时。
|
||||
"""
|
||||
return self._event.wait(timeout)
|
||||
|
||||
def link(self, event: threading.Event) -> None:
|
||||
"""联动一个 ``threading.Event``:当 ``event`` 被设置时,本 token 也被取消。
|
||||
|
||||
适用于将标准 ``threading.Event`` 桥接到 CancelToken。
|
||||
"""
|
||||
|
||||
def _watcher() -> None:
|
||||
event.wait()
|
||||
self.cancel()
|
||||
|
||||
t = threading.Thread(target=_watcher, daemon=True)
|
||||
t.start()
|
||||
|
||||
def __repr__(self) -> str:
|
||||
status = "cancelled" if self._event.is_set() else "active"
|
||||
reason = f", reason={self._reason!r}" if self._reason else ""
|
||||
return f"CancelToken({status}{reason})"
|
||||
|
||||
# 兼容 threading.Event 接口:run() 内部用 .is_set() 检查
|
||||
def is_set(self) -> bool:
|
||||
"""兼容 :meth:`threading.Event.is_set`。"""
|
||||
return self._event.is_set()
|
||||
|
||||
# 允许 ``CancelToken | threading.Event`` 统一类型检查
|
||||
def __bool__(self) -> bool:
|
||||
return self._event.is_set()
|
||||
|
||||
|
||||
def _is_cancelled(token: Any) -> bool:
|
||||
"""统一检查 CancelToken 或 threading.Event 是否已取消。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
token:
|
||||
``None``、``CancelToken`` 或 ``threading.Event``。
|
||||
"""
|
||||
if token is None:
|
||||
return False
|
||||
if isinstance(token, CancelToken):
|
||||
return token.is_cancelled
|
||||
return bool(token.is_set())
|
||||
@@ -1,60 +1,52 @@
|
||||
"""autofmt - 自动格式化工具.
|
||||
"""自动格式化工具模块.
|
||||
|
||||
提供格式化代码/代码检查/自动添加文档/同步配置 子命令.
|
||||
提供 Python 代码自动格式化的常用功能封装,
|
||||
支持 docstring 自动生成、pyproject.toml 配置同步等功能.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import IGNORE_PATTERNS
|
||||
|
||||
__all__ = [
|
||||
"add_docstring",
|
||||
"auto_add_docstrings",
|
||||
"fmt",
|
||||
"format_all",
|
||||
"format_with_ruff",
|
||||
"generate_module_docstring",
|
||||
"lint",
|
||||
"lint_with_ruff",
|
||||
"sync_pyproject_config",
|
||||
try:
|
||||
import tomllib # noqa: F401
|
||||
|
||||
HAS_TOMLLIB = True
|
||||
except ImportError:
|
||||
HAS_TOMLLIB = False
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
IGNORE_PATTERNS = [
|
||||
"__pycache__",
|
||||
"*.pyc",
|
||||
"*.pyo",
|
||||
".git",
|
||||
".venv",
|
||||
".idea",
|
||||
".vscode",
|
||||
"*.egg-info",
|
||||
"dist",
|
||||
"build",
|
||||
".pytest_cache",
|
||||
".tox",
|
||||
".mypy_cache",
|
||||
]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("autofmt", subcommand="fmt", help="格式化代码")
|
||||
def fmt(target: str = ".") -> None:
|
||||
"""格式化代码 (ruff format).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
target : str
|
||||
目标路径 (默认: .)
|
||||
"""
|
||||
format_with_ruff(Path(target), fix=False)
|
||||
|
||||
|
||||
@px.tool("autofmt", subcommand="lint", help="代码检查")
|
||||
def lint(target: str = ".", fix: bool = False) -> None:
|
||||
"""代码检查 (ruff check).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
target : str
|
||||
目标路径 (默认: .)
|
||||
fix : bool
|
||||
自动修复问题
|
||||
"""
|
||||
lint_with_ruff(Path(target), fix=fix)
|
||||
|
||||
|
||||
def format_with_ruff(target: Path, fix: bool = True) -> None:
|
||||
"""使用 ruff 格式化代码.
|
||||
|
||||
@@ -110,10 +102,12 @@ def add_docstring(file_path: Path, docstring: str) -> bool:
|
||||
content = file_path.read_text(encoding="utf-8")
|
||||
tree = ast.parse(content)
|
||||
|
||||
# 检查是否已有 docstring
|
||||
first_node = tree.body[0] if tree.body else None
|
||||
if first_node and isinstance(first_node, ast.Expr) and isinstance(first_node.value, ast.Constant):
|
||||
return False
|
||||
|
||||
# 添加 docstring
|
||||
lines = content.splitlines()
|
||||
doc_lines = docstring.splitlines()
|
||||
doc_lines.append("")
|
||||
@@ -144,6 +138,7 @@ def generate_module_docstring(file_path: Path) -> str:
|
||||
stem = file_path.stem
|
||||
parent = file_path.parent.name
|
||||
|
||||
# 关键词匹配
|
||||
keywords = {
|
||||
"cli": f"Command-line interface for {parent}",
|
||||
"gui": f"Graphical user interface for {parent}",
|
||||
@@ -160,14 +155,13 @@ def generate_module_docstring(file_path: Path) -> str:
|
||||
return f'"""{stem.replace("_", " ").title()} module."""'
|
||||
|
||||
|
||||
@px.tool("autofmt", subcommand="doc", help="自动添加文档字符串")
|
||||
def auto_add_docstrings(root_dir: Path = Path()) -> int:
|
||||
def auto_add_docstrings(root_dir: Path) -> int:
|
||||
"""自动为所有 Python 文件添加 docstring.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
root_dir : Path
|
||||
根目录 (默认: 当前目录)
|
||||
根目录
|
||||
|
||||
Returns
|
||||
-------
|
||||
@@ -176,6 +170,7 @@ def auto_add_docstrings(root_dir: Path = Path()) -> int:
|
||||
"""
|
||||
count = 0
|
||||
for py_file in root_dir.rglob("*.py"):
|
||||
# 跳过忽略的文件
|
||||
if any(pattern in str(py_file) for pattern in IGNORE_PATTERNS):
|
||||
continue
|
||||
|
||||
@@ -187,20 +182,20 @@ def auto_add_docstrings(root_dir: Path = Path()) -> int:
|
||||
return count
|
||||
|
||||
|
||||
@px.tool("autofmt", subcommand="sync", help="同步 pyproject 配置")
|
||||
def sync_pyproject_config(root_dir: Path = Path()) -> None:
|
||||
def sync_pyproject_config(root_dir: Path) -> None:
|
||||
"""同步 pyproject.toml 配置到子项目.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
root_dir : Path
|
||||
根目录 (默认: 当前目录)
|
||||
根目录
|
||||
"""
|
||||
main_toml = root_dir / "pyproject.toml"
|
||||
if not main_toml.exists():
|
||||
print(f"主项目配置文件不存在: {main_toml}")
|
||||
return
|
||||
|
||||
# 查找所有子项目的 pyproject.toml
|
||||
sub_tomls = [p for p in root_dir.rglob("pyproject.toml") if p != main_toml and ".venv" not in str(p)]
|
||||
|
||||
if not sub_tomls:
|
||||
@@ -209,6 +204,7 @@ def sync_pyproject_config(root_dir: Path = Path()) -> None:
|
||||
|
||||
print(f"找到 {len(sub_tomls)} 个子项目配置文件")
|
||||
|
||||
# 对每个子项目调用 ruff format
|
||||
for sub_toml in sub_tomls:
|
||||
subprocess.run(["ruff", "format", str(sub_toml)], check=False)
|
||||
|
||||
@@ -223,6 +219,64 @@ def format_all(root_dir: Path) -> None:
|
||||
root_dir : Path
|
||||
根目录
|
||||
"""
|
||||
# 使用 ruff format
|
||||
subprocess.run(["ruff", "format", str(root_dir)], check=True)
|
||||
|
||||
# 使用 ruff check
|
||||
subprocess.run(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], check=True)
|
||||
|
||||
print(f"格式化完成: {root_dir}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""自动格式化工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="AutoFmt - 自动格式化工具",
|
||||
usage="autofmt <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# ruff format 命令
|
||||
format_parser = subparsers.add_parser("fmt", help="使用 ruff 格式化代码")
|
||||
format_parser.add_argument("--target", type=str, default=".", help="目标路径")
|
||||
|
||||
# ruff check 命令
|
||||
lint_parser = subparsers.add_parser("lint", help="使用 ruff 检查代码")
|
||||
lint_parser.add_argument("--target", type=str, default=".", help="目标路径")
|
||||
lint_parser.add_argument("--fix", action="store_true", help="自动修复")
|
||||
|
||||
# 自动添加 docstring 命令
|
||||
doc_parser = subparsers.add_parser("doc", help="自动添加 docstring")
|
||||
doc_parser.add_argument("--root-dir", type=str, default=".", help="根目录")
|
||||
|
||||
# 同步配置命令
|
||||
sync_parser = subparsers.add_parser("sync", help="同步 pyproject.toml 配置")
|
||||
sync_parser.add_argument("--root-dir", type=str, default=".", help="根目录")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "fmt":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("ruff_format", cmd=["ruff", "format", args.target], verbose=True)])
|
||||
elif args.command == "lint":
|
||||
cmd = ["ruff", "check", args.target]
|
||||
if args.fix:
|
||||
cmd.extend(["--fix", "--unsafe-fixes"])
|
||||
graph = px.Graph.from_specs([px.TaskSpec("ruff_check", cmd=cmd, verbose=True)])
|
||||
elif args.command == "doc":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("auto_docstring", fn=auto_add_docstrings, args=(Path(args.root_dir),), verbose=True)
|
||||
])
|
||||
elif args.command == "sync":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("sync_config", fn=sync_pyproject_config, args=(Path(args.root_dir),), verbose=True)
|
||||
])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,263 @@
|
||||
"""版本号自动管理工具.
|
||||
|
||||
使用 TaskSpec 模式实现, 支持语义化版本管理和多文件格式的版本号更新.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Literal, get_args
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
BumpVersionType = Literal["patch", "minor", "major"]
|
||||
|
||||
# 针对不同文件类型的版本号匹配模式
|
||||
# pyproject.toml: version = "X.Y.Z" 或 version = 'X.Y.Z'
|
||||
_PYPROJECT_VERSION_PATTERN = re.compile(
|
||||
r'(?:^|\n)\s*version\s*=\s*["\']'
|
||||
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
|
||||
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
|
||||
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
|
||||
r'["\']',
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
# __init__.py: __version__ = "X.Y.Z" 或 __version__ = 'X.Y.Z'
|
||||
_INIT_VERSION_PATTERN = re.compile(
|
||||
r'(?:^|\n)\s*__version__\s*=\s*["\']'
|
||||
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
|
||||
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
|
||||
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
|
||||
r'["\']',
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
|
||||
def _get_pattern_for_file(file_name: str) -> re.Pattern[str] | None:
|
||||
"""根据文件类型获取对应的正则表达式.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
file_name : str
|
||||
文件名
|
||||
|
||||
Returns
|
||||
-------
|
||||
re.Pattern[str] | None
|
||||
对应的正则表达式,如果无法确定则返回 None
|
||||
"""
|
||||
if file_name == "pyproject.toml":
|
||||
return _PYPROJECT_VERSION_PATTERN
|
||||
if file_name == "__init__.py":
|
||||
return _INIT_VERSION_PATTERN
|
||||
return None
|
||||
|
||||
|
||||
def _calculate_new_version(major: int, minor: int, patch: int, part: BumpVersionType) -> str:
|
||||
"""计算新版本号.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
major : int
|
||||
当前主版本号
|
||||
minor : int
|
||||
当前次版本号
|
||||
patch : int
|
||||
当前补丁版本号
|
||||
part : BumpVersionType
|
||||
要更新的部分
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
新版本号
|
||||
"""
|
||||
if part == "major":
|
||||
return f"{major + 1}.0.0"
|
||||
if part == "minor":
|
||||
return f"{major}.{minor + 1}.0"
|
||||
return f"{major}.{minor}.{patch + 1}"
|
||||
|
||||
|
||||
def _build_replacement_string(original_match: str, new_version: str, file_name: str) -> str:
|
||||
"""构建替换字符串,保留原始格式.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
original_match : str
|
||||
原始匹配的字符串
|
||||
new_version : str
|
||||
新版本号
|
||||
file_name : str
|
||||
文件名
|
||||
|
||||
Returns
|
||||
-------
|
||||
str
|
||||
替换字符串
|
||||
"""
|
||||
quote_char = '"' if '"' in original_match else "'"
|
||||
|
||||
if file_name == "pyproject.toml":
|
||||
prefix_match = re.match(r'(\s*version\s*=\s*)["\']', original_match)
|
||||
prefix = prefix_match.group(1) if prefix_match else "version = "
|
||||
return f"{prefix}{quote_char}{new_version}{quote_char}"
|
||||
|
||||
if file_name == "__init__.py":
|
||||
prefix_match = re.match(r'(\s*__version__\s*=\s*)["\']', original_match)
|
||||
prefix = prefix_match.group(1) if prefix_match else "__version__ = "
|
||||
return f"{prefix}{quote_char}{new_version}{quote_char}"
|
||||
|
||||
return new_version
|
||||
|
||||
|
||||
def bump_file_version(file_path: Path, part: BumpVersionType = "patch") -> str | None:
|
||||
"""更新文件中的版本号.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
file_path : Path
|
||||
要更新的文件路径
|
||||
part : BumpVersionType
|
||||
版本部分: patch, minor, major
|
||||
|
||||
Returns
|
||||
-------
|
||||
str | None
|
||||
更新后的新版本号,如果文件中未找到版本号则返回 None
|
||||
"""
|
||||
try:
|
||||
content = file_path.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f"读取文件 {file_path} 时出错: {e}")
|
||||
raise
|
||||
|
||||
# 获取文件对应的正则表达式
|
||||
pattern = _get_pattern_for_file(file_path.name)
|
||||
|
||||
# 对于未知文件类型,尝试两种模式
|
||||
if pattern:
|
||||
match = pattern.search(content)
|
||||
else:
|
||||
match = _PYPROJECT_VERSION_PATTERN.search(content) or _INIT_VERSION_PATTERN.search(content)
|
||||
|
||||
if not match:
|
||||
print(f"文件 {file_path} 中未找到版本号模式")
|
||||
return None
|
||||
|
||||
# 提取当前版本号
|
||||
major = int(match.group("major"))
|
||||
minor = int(match.group("minor"))
|
||||
patch = int(match.group("patch"))
|
||||
|
||||
# 计算新版本号
|
||||
new_version = _calculate_new_version(major, minor, patch, part)
|
||||
|
||||
# 构建替换字符串
|
||||
original_match = match.group(0)
|
||||
replacement = _build_replacement_string(original_match, new_version, file_path.name)
|
||||
|
||||
# 更新文件内容
|
||||
content = content.replace(original_match, replacement)
|
||||
|
||||
try:
|
||||
file_path.write_text(content, encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f"更新文件 {file_path} 版本号时出错: {e}")
|
||||
raise
|
||||
|
||||
return new_version
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""版本号管理工具主函数."""
|
||||
parser = argparse.ArgumentParser(description="BumpVersion - 版本号自动管理工具")
|
||||
parser.add_argument(
|
||||
"part",
|
||||
type=str,
|
||||
nargs="?",
|
||||
default="patch",
|
||||
choices=get_args(BumpVersionType),
|
||||
help=f"版本部分: {get_args(BumpVersionType)}",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-tag",
|
||||
action="store_true",
|
||||
help="提交后不创建 git tag",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
part = args.part
|
||||
|
||||
# 搜索文件,排除常见的虚拟环境和缓存目录
|
||||
ignore_dirs = {".venv", "venv", ".git", "__pycache__", ".tox", "node_modules", "build", "dist", ".eggs"}
|
||||
all_files = set()
|
||||
|
||||
for pattern in ["__init__.py", "pyproject.toml"]:
|
||||
for file in Path.cwd().rglob(pattern):
|
||||
# 检查路径中是否包含需要忽略的目录
|
||||
if not any(ignore_dir in file.parts for ignore_dir in ignore_dirs):
|
||||
all_files.add(file)
|
||||
|
||||
if not all_files:
|
||||
print("未找到包含版本号的文件")
|
||||
return
|
||||
|
||||
print(f"找到 {len(all_files)} 个文件需要更新版本号")
|
||||
for file in sorted(all_files):
|
||||
print(f" - {file.relative_to(Path.cwd())}")
|
||||
|
||||
# 更新所有文件的版本号(使用顺序执行避免竞争条件)
|
||||
# 使用相对于 cwd 的路径作为任务名,确保唯一性
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
f"bump_{file.relative_to(Path.cwd())}".replace("\\", "_").replace("/", "_").replace(".", "_"),
|
||||
fn=bump_file_version,
|
||||
args=(file, part),
|
||||
)
|
||||
for file in all_files
|
||||
])
|
||||
report = px.run(graph, strategy="sequential")
|
||||
|
||||
# 收集新版本号(取第一个成功的结果)
|
||||
new_version = None
|
||||
for task_name in report:
|
||||
result = report[task_name]
|
||||
if result is not None:
|
||||
new_version = result
|
||||
break
|
||||
|
||||
if not new_version:
|
||||
print("未能获取新版本号")
|
||||
return
|
||||
|
||||
print(f"版本号已更新为: {new_version}")
|
||||
|
||||
# 提交修改并创建标签
|
||||
tasks = [
|
||||
px.TaskSpec("git_add", cmd=["git", "add", "."]),
|
||||
px.TaskSpec(
|
||||
"git_commit",
|
||||
cmd=["git", "commit", "-m", f"bump version to {new_version}"],
|
||||
depends_on=("git_add",),
|
||||
),
|
||||
]
|
||||
|
||||
if not args.no_tag:
|
||||
tag_name = f"v{new_version}"
|
||||
tasks.append(
|
||||
px.TaskSpec(
|
||||
"git_tag",
|
||||
cmd=["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"],
|
||||
depends_on=("git_commit",),
|
||||
)
|
||||
)
|
||||
|
||||
graph = px.Graph.from_specs(tasks)
|
||||
px.run(graph, strategy="sequential")
|
||||
|
||||
if not args.no_tag:
|
||||
print(f"已创建标签: v{new_version}")
|
||||
@@ -0,0 +1,26 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
DockerMirrorType = Literal["tencent"]
|
||||
|
||||
DOCKER_MIRROR_URLS: dict[DockerMirrorType, str] = {"tencent": "ccr.ccs.tencentyun.com"}
|
||||
|
||||
|
||||
def main():
|
||||
# parser = argparse.ArgumentParser(description="Docker 命令行工具")
|
||||
# parser.add_argument("--username", nargs="?", default="", type=str, help="Docker 用户名")
|
||||
# args = parser.parse_args()
|
||||
|
||||
tasks: list[px.TaskSpec] = [
|
||||
px.cmd(["docker", "login", "--username", "xxx", DOCKER_MIRROR_URLS["tencent"]], name="docker_login_tencent"),
|
||||
]
|
||||
|
||||
alias: dict[str, str | list[str | px.TaskSpec] | px.TaskSpec | px.Graph] = {
|
||||
"login": "docker_login_tencent",
|
||||
}
|
||||
|
||||
runner = px.CliRunner(strategy="sequential", tasks=tasks, aliases=alias)
|
||||
runner.run_cli()
|
||||
@@ -0,0 +1,357 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
from pathlib import Path
|
||||
from typing import Literal, get_args
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import BuiltinConditions
|
||||
from pyflowx.tasks.system import setenv_group, write_file
|
||||
|
||||
# ============================================================================
|
||||
# Mirror 配置
|
||||
# ============================================================================
|
||||
DOWNLOAD_MIRROR_SCRIPT: str = "curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh"
|
||||
INSTALL_MIRROR_SCRIPT: str = "sudo bash /tmp/linuxmirrors.sh"
|
||||
|
||||
# ============================================================================
|
||||
# Python 配置
|
||||
# ============================================================================
|
||||
PyMirrorType = Literal["tsinghua", "aliyun", "huaweicloud", "ustc", "zju"]
|
||||
|
||||
PIP_INDEX_URLS: dict[PyMirrorType, str] = {
|
||||
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
|
||||
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
|
||||
"huaweicloud": "https://mirrors.huaweicloud.com/repository/pypi/simple/",
|
||||
"ustc": "https://pypi.mirrors.ustc.edu.cn/simple/",
|
||||
"zju": "https://mirrors.zju.edu.cn/pypi/simple/",
|
||||
}
|
||||
|
||||
PIP_TRUSTED_HOSTS: dict[PyMirrorType, str] = {
|
||||
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
|
||||
"aliyun": "mirrors.aliyun.com",
|
||||
"huaweicloud": "mirrors.huaweicloud.com",
|
||||
"ustc": "pypi.mirrors.ustc.edu.cn",
|
||||
"zju": "mirrors.zju.edu.cn",
|
||||
}
|
||||
PIP_CONFIG_PATH = Path.home() / ".pip" / "pip.conf" if BuiltinConditions.IS_LINUX() else Path.home() / "pip" / "pip.ini"
|
||||
|
||||
UV_INDEX_URLS = PIP_INDEX_URLS
|
||||
UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
|
||||
|
||||
# ============================================================================
|
||||
# Conda 配置
|
||||
# ============================================================================
|
||||
CondaMirrorType = Literal["tsinghua", "ustc", "bsfu", "aliyun"]
|
||||
|
||||
CONDA_MIRROR_URLS: dict[CondaMirrorType, list[str]] = {
|
||||
"tsinghua": [
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"ustc": [
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/dev/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"bsfu": [
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/dev/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"aliyun": [
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/r/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/pro/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/dev/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/menpo/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/pytorch/",
|
||||
],
|
||||
}
|
||||
CONDA_CONFIG_PATH = Path.home() / ".condarc"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Qt 配置
|
||||
# ============================================================================
|
||||
|
||||
QT_LIBS: list[str] = [
|
||||
"build-essential",
|
||||
"libgl1",
|
||||
"libegl1",
|
||||
"libglib2.0-0",
|
||||
"libfontconfig1",
|
||||
"libfreetype6",
|
||||
"libxkbcommon0",
|
||||
"libdbus-1-3",
|
||||
"libxcb-xinerama0",
|
||||
"libxcb-icccm4",
|
||||
"libxcb-image0",
|
||||
"libxcb-keysyms1",
|
||||
"libxcb-randr0",
|
||||
"libxcb-render-util0",
|
||||
"libxcb-shape0",
|
||||
"libxcb-xfixes0",
|
||||
"libxcb-cursor0",
|
||||
]
|
||||
|
||||
CHINESE_FONTS: list[str] = [
|
||||
"fonts-noto-cjk",
|
||||
"fonts-wqy-microhei",
|
||||
"fonts-wqy-zenhei",
|
||||
"fonts-noto-color-emoji",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# Rust 配置
|
||||
# ============================================================================
|
||||
RustMirrorType = Literal["tsinghua", "ustc", "aliyun"]
|
||||
RustVersionType = Literal["stable", "nightly", "beta"]
|
||||
DEFAULT_RUST_VERSION: RustVersionType = "stable"
|
||||
DEFAULT_MIRROR: RustMirrorType = "tsinghua"
|
||||
|
||||
RUSTUP_MIRRORS: dict[RustMirrorType, dict[str, str]] = {
|
||||
"tsinghua": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
|
||||
},
|
||||
"aliyun": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
|
||||
},
|
||||
"ustc": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
|
||||
},
|
||||
}
|
||||
RUSTUP_DOWNLOAD_URL_LINUX = "https://mirrors.aliyun.com/repo/rust/rustup-init.sh"
|
||||
RUSTUP_DOWNLOAD_URL_WINDOWS = "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"
|
||||
RUST_CONFIG_PATH = Path.home() / ".cargo" / "config.toml"
|
||||
RUST_SCCACHE_DIR: Path = Path.home() / ".cargo" / "sccache"
|
||||
RUST_SCCACHE_CACHE_SIZE: str = "20G"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""主函数."""
|
||||
parser = argparse.ArgumentParser(description="环境开发工具")
|
||||
parser.add_argument(
|
||||
"--python-mirror",
|
||||
nargs="?",
|
||||
type=str,
|
||||
default="tsinghua",
|
||||
choices=get_args(PyMirrorType),
|
||||
help="Python 镜像源",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--conda-mirror",
|
||||
nargs="?",
|
||||
type=str,
|
||||
default="tsinghua",
|
||||
choices=get_args(CondaMirrorType),
|
||||
help="Conda 镜镜像源",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rust-mirror",
|
||||
nargs="?",
|
||||
type=str,
|
||||
default=DEFAULT_MIRROR,
|
||||
choices=get_args(RustMirrorType),
|
||||
help="Rust 镜像源",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rust-version",
|
||||
nargs="?",
|
||||
type=str,
|
||||
default=DEFAULT_RUST_VERSION,
|
||||
choices=get_args(RustVersionType),
|
||||
help=f"Rust 版本, 推荐: {get_args(RustVersionType)}",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
python_mirror = args.python_mirror
|
||||
conda_mirror_urls = CONDA_MIRROR_URLS[args.conda_mirror]
|
||||
rust_mirror = args.rust_mirror
|
||||
rust_version = args.rust_version
|
||||
|
||||
# 确保配置文件目录存在
|
||||
PIP_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
CONDA_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
RUST_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
RUST_SCCACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 使用 conditions 自动控制任务执行
|
||||
graph = px.Graph.from_specs([
|
||||
# 系统镜像配置(仅 Linux 且未配置国内镜像)
|
||||
px.TaskSpec(
|
||||
"download_mirror",
|
||||
cmd=DOWNLOAD_MIRROR_SCRIPT,
|
||||
conditions=(
|
||||
BuiltinConditions.IS_LINUX(),
|
||||
BuiltinConditions.NOT(
|
||||
BuiltinConditions.OR(
|
||||
*[
|
||||
BuiltinConditions.FILE_CONTENT_EXISTS(f, m)
|
||||
for f in [
|
||||
"/etc/apt/sources.list",
|
||||
"/etc/apt/sources.list.d/ubuntu.sources",
|
||||
]
|
||||
for m in get_args(PyMirrorType)
|
||||
],
|
||||
)
|
||||
),
|
||||
),
|
||||
verbose=True,
|
||||
),
|
||||
px.TaskSpec(
|
||||
"install_mirror",
|
||||
cmd=INSTALL_MIRROR_SCRIPT,
|
||||
depends_on=("download_mirror",),
|
||||
verbose=True,
|
||||
),
|
||||
# 安装 Qt 依赖(仅 Linux)
|
||||
px.TaskSpec(
|
||||
"install_qt_libs",
|
||||
cmd=["sudo", "apt", "install", "-y", *QT_LIBS],
|
||||
conditions=(BuiltinConditions.IS_LINUX(),),
|
||||
depends_on=("install_mirror",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
# 安装中文字体(仅 Linux)
|
||||
px.TaskSpec(
|
||||
"install_fonts",
|
||||
cmd=["sudo", "apt", "install", "-y", *CHINESE_FONTS],
|
||||
conditions=(BuiltinConditions.IS_LINUX(),),
|
||||
depends_on=("install_mirror",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
# 安装 Docker
|
||||
px.TaskSpec(
|
||||
"install_docker",
|
||||
cmd=["sudo", "apt", "install", "-y", "docker-compose-v2"],
|
||||
conditions=(BuiltinConditions.IS_LINUX(),),
|
||||
depends_on=("install_mirror",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
px.TaskSpec(
|
||||
"add_docker_group",
|
||||
cmd=["sudo", "usermod", "-aG", "docker", getpass.getuser()],
|
||||
conditions=(BuiltinConditions.IS_LINUX(),),
|
||||
depends_on=("install_docker",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
px.TaskSpec(
|
||||
"refresh_docker_group",
|
||||
cmd=["newgrp", "docker"],
|
||||
conditions=(BuiltinConditions.IS_LINUX(),),
|
||||
depends_on=("add_docker_group",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
# 设置 Python 环境变量
|
||||
*setenv_group({
|
||||
"PIP_INDEX_URL": PIP_INDEX_URLS[python_mirror],
|
||||
"PIP_TRUSTED_HOSTS": PIP_TRUSTED_HOSTS[python_mirror],
|
||||
"UV_INDEX_URL": UV_INDEX_URLS[python_mirror],
|
||||
"UV_PYTHON_INSTALL_MIRROR": UV_PYTHON_INSTALL_MIRROR,
|
||||
"UV_HTTP_TIMEOUT": "600",
|
||||
"UV_LINK_MODE": "copy",
|
||||
}),
|
||||
# 写入 Python 配置(仅当未配置)
|
||||
write_file(
|
||||
str(PIP_CONFIG_PATH),
|
||||
f"[global]\nindex-url = {PIP_INDEX_URLS[python_mirror]}\ntrusted-host = {PIP_TRUSTED_HOSTS[python_mirror]}",
|
||||
),
|
||||
# 写入 Conda 配置(仅当未配置)
|
||||
write_file(
|
||||
str(CONDA_CONFIG_PATH),
|
||||
"show_channel_urls: true\nchannels:\n - " + "\n - ".join(conda_mirror_urls) + "\n - defaults",
|
||||
),
|
||||
# 设置 Rust 镜像源
|
||||
*setenv_group({
|
||||
"RUSTUP_DIST_SERVER": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_DIST_SERVER"],
|
||||
"RUSTUP_UPDATE_ROOT": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_UPDATE_ROOT"],
|
||||
"RUST_SCCACHE_DIR": str(RUST_SCCACHE_DIR),
|
||||
"RUST_SCCACHE_CACHE_SIZE": RUST_SCCACHE_CACHE_SIZE,
|
||||
}),
|
||||
# 写入 Rust 配置(仅当未配置)
|
||||
write_file(
|
||||
str(RUST_CONFIG_PATH),
|
||||
f"""
|
||||
[source.crates-io]
|
||||
replace-with = '{rust_mirror}'
|
||||
|
||||
[source.{rust_mirror}]
|
||||
registry = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
|
||||
|
||||
[registries.{rust_mirror}]
|
||||
index = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
|
||||
""",
|
||||
),
|
||||
# 下载 Rustup 安装脚本
|
||||
px.TaskSpec(
|
||||
"download_rustup",
|
||||
cmd=["curl", "-fsSL", RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
|
||||
conditions=(BuiltinConditions.IS_LINUX(), BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup"))),
|
||||
verbose=True,
|
||||
),
|
||||
px.TaskSpec(
|
||||
"download_rustup_win",
|
||||
cmd=[
|
||||
"powershell",
|
||||
"-Command",
|
||||
"Invoke-WebRequest",
|
||||
"-Uri",
|
||||
RUSTUP_DOWNLOAD_URL_WINDOWS,
|
||||
"-OutFile",
|
||||
"rustup-init.exe",
|
||||
],
|
||||
conditions=(
|
||||
BuiltinConditions.IS_WINDOWS(),
|
||||
BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup")),
|
||||
),
|
||||
verbose=True,
|
||||
),
|
||||
# 安装 Rust 工具链
|
||||
px.TaskSpec(
|
||||
"install_rust",
|
||||
cmd=["rustup", "toolchain", "install", rust_version],
|
||||
conditions=(BuiltinConditions.HAS_INSTALLED("rustup"),),
|
||||
depends_on=("setenv_rustup_dist_server",),
|
||||
allow_upstream_skip=True,
|
||||
verbose=True,
|
||||
),
|
||||
])
|
||||
px.run(graph, strategy="thread", verbose=True)
|
||||
@@ -154,7 +154,7 @@ class EmailDatabase:
|
||||
cursor.execute(query, (f"%{keyword}%", f"%{keyword}%", f"%{keyword}%", f"%{keyword}%", limit, offset))
|
||||
|
||||
columns = [description[0] for description in cursor.description]
|
||||
return [dict(zip(columns, row, strict=False)) for row in cursor.fetchall()]
|
||||
return [dict(zip(columns, row)) for row in cursor.fetchall()]
|
||||
|
||||
def get_grouped_emails(self) -> dict[str, list[dict[str, Any]]]:
|
||||
"""获取按主题分组的邮件."""
|
||||
@@ -165,7 +165,7 @@ class EmailDatabase:
|
||||
cursor.execute(f"SELECT * FROM {TABLE_NAME} ORDER BY subject, date_parsed DESC")
|
||||
|
||||
columns = [description[0] for description in cursor.description]
|
||||
emails = [dict(zip(columns, row, strict=False)) for row in cursor.fetchall()]
|
||||
emails = [dict(zip(columns, row)) for row in cursor.fetchall()]
|
||||
|
||||
# 按主题分组
|
||||
grouped: dict[str, list[dict[str, Any]]] = {}
|
||||
@@ -567,15 +567,13 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
|
||||
|
||||
emails = self.db.search_emails(keyword, field, limit, offset)
|
||||
total_count = self.db.get_email_count()
|
||||
self._send_json_response(
|
||||
{
|
||||
"emails": emails,
|
||||
"count": len(emails),
|
||||
"total": total_count,
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
)
|
||||
self._send_json_response({
|
||||
"emails": emails,
|
||||
"count": len(emails),
|
||||
"total": total_count,
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
})
|
||||
|
||||
def _api_get_email(self, query_params: dict[str, list[str]]) -> None:
|
||||
"""API: 获取单个邮件详情."""
|
||||
@@ -602,7 +600,7 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
|
||||
self._send_json_response({"error": "邮件不存在"}, 404)
|
||||
return
|
||||
|
||||
email_data = dict(zip(columns, row, strict=False))
|
||||
email_data = dict(zip(columns, row))
|
||||
self._send_json_response({"email": email_data})
|
||||
|
||||
def _api_get_grouped_emails(self) -> None:
|
||||
@@ -1,26 +1,18 @@
|
||||
"""filedate - 文件日期处理工具.
|
||||
"""文件日期处理工具.
|
||||
|
||||
提供添加日期前缀/清除日期前缀 子命令.
|
||||
自动检测文件名的日期前缀,
|
||||
并根据文件的实际创建或修改时间重命名文件.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"DATE_PATTERN",
|
||||
"SEP",
|
||||
"add_date_prefix",
|
||||
"get_file_timestamp",
|
||||
"process_file_date",
|
||||
"process_files_date",
|
||||
"remove_date_prefix",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
@@ -30,7 +22,7 @@ SEP = "_"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@@ -77,34 +69,11 @@ def process_file_date(filepath: Path, clear: bool = False) -> None:
|
||||
if clear:
|
||||
remove_date_prefix(filepath)
|
||||
else:
|
||||
# 先移除旧日期前缀,再添加新日期前缀
|
||||
new_path = remove_date_prefix(filepath)
|
||||
add_date_prefix(new_path)
|
||||
|
||||
|
||||
@px.tool("filedate", subcommand="add", help="添加日期前缀")
|
||||
def process_files_date_add(files: list[Path]) -> None:
|
||||
"""添加日期前缀.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
files : list[Path]
|
||||
文件路径列表
|
||||
"""
|
||||
process_files_date(files, clear=False)
|
||||
|
||||
|
||||
@px.tool("filedate", subcommand="clear", help="清除日期前缀")
|
||||
def process_files_date_clear(files: list[Path]) -> None:
|
||||
"""清除日期前缀.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
files : list[Path]
|
||||
文件路径列表
|
||||
"""
|
||||
process_files_date(files, clear=True)
|
||||
|
||||
|
||||
def process_files_date(targets: list[Path], clear: bool = False) -> None:
|
||||
"""批量处理文件日期前缀.
|
||||
|
||||
@@ -118,3 +87,51 @@ def process_files_date(targets: list[Path], clear: bool = False) -> None:
|
||||
for target in targets:
|
||||
if target.exists() and not target.name.startswith("."):
|
||||
process_file_date(target, clear)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""文件日期处理工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="FileDate - 文件日期处理工具",
|
||||
usage="filedate <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 添加日期前缀命令
|
||||
add_parser = subparsers.add_parser("add", help="添加日期前缀")
|
||||
add_parser.add_argument("files", nargs="+", help="文件路径")
|
||||
|
||||
# 清除日期前缀命令
|
||||
clear_parser = subparsers.add_parser("clear", help="清除日期前缀")
|
||||
clear_parser.add_argument("files", nargs="+", help="文件路径")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "add":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"process_files_date",
|
||||
fn=process_files_date,
|
||||
args=([Path(f) for f in args.files],),
|
||||
kwargs={"clear": False},
|
||||
)
|
||||
])
|
||||
elif args.command == "clear":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"process_files_date",
|
||||
fn=process_files_date,
|
||||
args=([Path(f) for f in args.files],),
|
||||
kwargs={"clear": True},
|
||||
)
|
||||
])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -1,22 +1,16 @@
|
||||
"""filelevel - 文件等级重命名工具.
|
||||
"""文件等级重命名工具.
|
||||
|
||||
提供设置文件等级 子命令.
|
||||
根据文件等级配置自动重命名文件,
|
||||
支持多种等级标识和括号格式.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"BRACKETS",
|
||||
"LEVELS",
|
||||
"process_file_level",
|
||||
"process_files_level",
|
||||
"remove_marks",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
@@ -33,7 +27,7 @@ BRACKETS: tuple[str, str] = (" ([_(【-", " )]_)】")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@@ -75,34 +69,72 @@ def process_file_level(filepath: Path, level: int = 0) -> None:
|
||||
filestem = filepath.stem
|
||||
original_stem = filestem
|
||||
|
||||
# 移除所有等级标记
|
||||
for level_names in LEVELS.values():
|
||||
if level_names:
|
||||
filestem = remove_marks(filestem, level_names.split(","))
|
||||
|
||||
# 移除数字标记
|
||||
for digit in map(str, range(1, 10)):
|
||||
filestem = remove_marks(filestem, [digit])
|
||||
|
||||
# 添加等级标记
|
||||
if level > 0:
|
||||
levelstr = LEVELS.get(str(level), "").split(",")[0]
|
||||
if levelstr:
|
||||
filestem = f"{filestem}({levelstr})"
|
||||
|
||||
# 重命名文件
|
||||
if filestem != original_stem:
|
||||
new_path = filepath.with_name(filestem + filepath.suffix)
|
||||
filepath.rename(new_path)
|
||||
print(f"重命名: {filepath} -> {new_path}")
|
||||
|
||||
|
||||
@px.tool("filelevel", subcommand="set", help="设置文件等级")
|
||||
def process_files_level(files: list[Path], level: int = 0) -> None:
|
||||
def process_files_level(targets: list[Path], level: int = 0) -> None:
|
||||
"""批量处理文件等级标记.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
files : list[Path]
|
||||
targets : list[Path]
|
||||
文件路径列表
|
||||
level : int
|
||||
文件等级 (0-4)
|
||||
"""
|
||||
for target in files:
|
||||
for target in targets:
|
||||
process_file_level(target, level)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""文件等级重命名工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="FileLevel - 文件等级重命名工具",
|
||||
usage="filelevel <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 设置等级命令
|
||||
level_parser = subparsers.add_parser("set", help="设置文件等级")
|
||||
level_parser.add_argument("files", nargs="+", help="文件路径")
|
||||
level_parser.add_argument("--level", type=int, choices=[0, 1, 2, 3, 4], required=True, help="文件等级 (0-4)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "set":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"process_files_level", fn=process_files_level, args=([Path(f) for f in args.files], args.level)
|
||||
)
|
||||
]
|
||||
)
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -1,6 +1,7 @@
|
||||
"""folderback - 文件夹备份工具.
|
||||
"""文件夹备份工具.
|
||||
|
||||
备份当前文件夹到指定目录, 自动清理旧备份.
|
||||
备份文件和文件夹为 zip 文件,
|
||||
自动删除超过最大数量的旧备份文件.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -11,15 +12,8 @@ from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"backup_folder",
|
||||
"remove_dump",
|
||||
"zip_target",
|
||||
]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@@ -46,18 +40,17 @@ def zip_target(src: Path, dst: Path, max_zip: int) -> None:
|
||||
print(f"备份完成: {target_path}")
|
||||
|
||||
|
||||
@px.tool("folderback", help="备份文件夹")
|
||||
def backup_folder(src: str = ".", dst: str = "./backup", max_zip: int = 5) -> None:
|
||||
def backup_folder(src: str, dst: str, max_zip: int = 5) -> None:
|
||||
"""备份文件夹.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
src : str
|
||||
源文件夹路径 (默认: 当前目录)
|
||||
源文件夹路径
|
||||
dst : str
|
||||
目标文件夹路径 (默认: ./backup)
|
||||
目标文件夹路径
|
||||
max_zip : int
|
||||
最大备份数量 (默认: 5)
|
||||
最大备份数量
|
||||
"""
|
||||
src_path = Path(src)
|
||||
dst_path = Path(dst)
|
||||
@@ -71,3 +64,22 @@ def backup_folder(src: str = ".", dst: str = "./backup", max_zip: int = 5) -> No
|
||||
print(f"创建目标文件夹: {dst_path}")
|
||||
|
||||
zip_target(src_path, dst_path, max_zip)
|
||||
|
||||
|
||||
@px.task
|
||||
def folderback_default() -> None:
|
||||
"""备份当前目录到 ./backup."""
|
||||
backup_folder(".", "./backup", 5)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""文件夹备份工具主函数."""
|
||||
runner = px.CliRunner(
|
||||
strategy="thread",
|
||||
description="FolderBack - 文件夹备份工具",
|
||||
aliases={
|
||||
# 备份当前目录到 ./backup
|
||||
"b": folderback_default,
|
||||
},
|
||||
)
|
||||
runner.run_cli()
|
||||
@@ -1,6 +1,7 @@
|
||||
"""folderzip - 文件夹压缩工具.
|
||||
"""文件夹压缩工具.
|
||||
|
||||
压缩当前目录下的所有子文件夹为 zip 文件.
|
||||
压缩目录下的所有文件/文件夹为 zip 文件,
|
||||
默认压缩当前目录下的所有子文件夹.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -10,25 +11,18 @@ from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"IGNORE_DIRS",
|
||||
"IGNORE_EXT",
|
||||
"IGNORE_FILES",
|
||||
"archive_folder",
|
||||
"zip_folders",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
IGNORE_DIRS: list[str] = [".git", ".idea", ".vscode", "__pycache__"]
|
||||
IGNORE_FILES: list[str] = [".gitignore"]
|
||||
IGNORE: list[str] = [*IGNORE_DIRS, *IGNORE_FILES]
|
||||
IGNORE_EXT: list[str] = [".zip", ".rar", ".7z", ".tar", ".gz"]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@@ -42,14 +36,13 @@ def archive_folder(folder: Path) -> None:
|
||||
print(f"压缩完成: {folder.name}.zip")
|
||||
|
||||
|
||||
@px.tool("folderzip", help="压缩文件夹")
|
||||
def zip_folders(cwd: str = ".") -> None:
|
||||
"""压缩目录下的所有文件夹.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
cwd : str
|
||||
工作目录 (默认: 当前目录)
|
||||
工作目录
|
||||
"""
|
||||
cwd_path = Path(cwd)
|
||||
if not cwd_path.exists():
|
||||
@@ -62,3 +55,22 @@ def zip_folders(cwd: str = ".") -> None:
|
||||
|
||||
for dir_path in dirs:
|
||||
archive_folder(dir_path)
|
||||
|
||||
|
||||
@px.task
|
||||
def folderzip_default() -> None:
|
||||
"""压缩当前目录下的所有文件夹."""
|
||||
zip_folders(".")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""文件夹压缩工具主函数."""
|
||||
runner = px.CliRunner(
|
||||
strategy="thread",
|
||||
description="FolderZip - 文件夹压缩工具",
|
||||
aliases={
|
||||
# 压缩当前目录下的所有文件夹
|
||||
"z": folderzip_default,
|
||||
},
|
||||
)
|
||||
runner.run_cli()
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Git 工具模块.
|
||||
|
||||
提供 Git 仓库管理的常用操作封装,
|
||||
支持初始化、提交、清理、推送等功能.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
EXCLUDE_DIRS = [
|
||||
# 编辑器相关目录
|
||||
".vscode",
|
||||
".idea",
|
||||
".editorconfig",
|
||||
".trae",
|
||||
".qoder",
|
||||
# 项目相关目录
|
||||
".venv",
|
||||
".git",
|
||||
".tox",
|
||||
".pytest_cache",
|
||||
"node_modules",
|
||||
".ruff_cache",
|
||||
]
|
||||
EXCLUDE_CMDS = [arg for d in EXCLUDE_DIRS for arg in ["-e", d]]
|
||||
|
||||
|
||||
def init_sub_dirs() -> None:
|
||||
"""初始化子目录的Git仓库."""
|
||||
sub_dirs = [subdir for subdir in Path.cwd().iterdir() if subdir.is_dir()]
|
||||
for subdir in sub_dirs:
|
||||
px.run(
|
||||
px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"init",
|
||||
cmd=["git", "init"],
|
||||
conditions=(lambda _: not_has_git_repo(),),
|
||||
cwd=subdir,
|
||||
),
|
||||
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",)),
|
||||
px.TaskSpec("commit", cmd=["git", "commit", "-m", "init commit"], depends_on=("add",)),
|
||||
]),
|
||||
)
|
||||
|
||||
|
||||
@px.task(name="isub")
|
||||
def isub() -> None:
|
||||
"""初始化子目录的Git仓库."""
|
||||
init_sub_dirs()
|
||||
|
||||
|
||||
push: px.TaskSpec = px.TaskSpec("push", cmd=["git", "push"])
|
||||
pull: px.TaskSpec = px.TaskSpec("pull", cmd=["git", "pull"])
|
||||
kill_tgit: px.TaskSpec = px.TaskSpec("task_kill", cmd=["taskkill", "/f", "/t", "/im", "tgitcache.exe"])
|
||||
|
||||
|
||||
def not_has_git_repo() -> bool:
|
||||
"""检查当前目录没有Git仓库."""
|
||||
return not Path.cwd().exists() or not (Path.cwd() / ".git").is_dir()
|
||||
|
||||
|
||||
def has_files() -> bool:
|
||||
"""检查当前目录是否有文件."""
|
||||
return bool(list(Path.cwd().glob("*")))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Git工具主函数."""
|
||||
runner = px.CliRunner(
|
||||
strategy="thread",
|
||||
description="Gittool - Git 执行工具.",
|
||||
aliases={
|
||||
# 添加并提交
|
||||
"a": px.Graph.from_specs([
|
||||
px.TaskSpec("add", cmd=["git", "add", "."], conditions=(lambda _: has_files(),)),
|
||||
px.TaskSpec("commit", cmd=["git", "commit", "-m", "chore: update"], depends_on=("add",)),
|
||||
]),
|
||||
# 清理(chain: clean → status)
|
||||
"c": px.Graph().chain(
|
||||
px.TaskSpec("clean", cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS]),
|
||||
px.TaskSpec("status", cmd=["git", "status", "--porcelain"]),
|
||||
),
|
||||
# 初始化、添加并提交
|
||||
"i": px.Graph.from_specs([
|
||||
px.TaskSpec("init", cmd=["git", "init"], conditions=(lambda _: not_has_git_repo(),)),
|
||||
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",), conditions=(lambda _: has_files(),)),
|
||||
px.TaskSpec(
|
||||
"commit",
|
||||
cmd=["git", "commit", "-m", "init commit"],
|
||||
depends_on=("add",),
|
||||
conditions=(lambda _: has_files(),),
|
||||
),
|
||||
]),
|
||||
# 初始化子目录
|
||||
"isub": isub,
|
||||
# 推送
|
||||
"p": push,
|
||||
# 拉取
|
||||
"pl": pull,
|
||||
# 重启TGit缓存
|
||||
"r": kill_tgit,
|
||||
},
|
||||
)
|
||||
runner.run_cli()
|
||||
@@ -1,15 +0,0 @@
|
||||
"""legacy 子包 — 未迁移到 @px.tool 的独立工具.
|
||||
|
||||
每个子模块有自己的 ``main()`` 函数, 由 ``pf`` 入口通过
|
||||
``_LEGACY_TOOLS`` 路由调用. 这些工具因逻辑复杂 (web 应用、runpy 注入等)
|
||||
暂未用 ``@px.tool`` 装饰器重写.
|
||||
|
||||
子模块
|
||||
------
|
||||
- :mod:`emlmanager` —— EML 邮件管理 Web 应用
|
||||
- :mod:`profiler` —— pxp 性能分析器 (monkey-patch + runpy)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Download from ModelScopeHub."""
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from typing import Literal, get_args
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
DownloadType = Literal["model", "dataset", "space"]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Download a model from ModelScopeHub.")
|
||||
parser.add_argument("name", help="Target name.")
|
||||
parser.add_argument("--type", "-t", nargs="?", default="model", choices=get_args(DownloadType), help="Target type.")
|
||||
parser.add_argument("--dir", default=None, help="Download directory.")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.name:
|
||||
parser.error("name is required")
|
||||
|
||||
download_dir: Path = Path(args.dir) if args.dir else Path.home() / ".models" / args.name.split("/")[-1]
|
||||
download_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
name="download",
|
||||
cmd=[
|
||||
"uvx",
|
||||
"modelscope",
|
||||
"download",
|
||||
f"--{args.type}",
|
||||
args.name,
|
||||
"--local_dir",
|
||||
str(download_dir),
|
||||
],
|
||||
verbose=True,
|
||||
),
|
||||
])
|
||||
|
||||
px.run(graph, strategy="thread", verbose=True)
|
||||
@@ -0,0 +1,63 @@
|
||||
"""使用 SGLang 运行本地模型."""
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import BuiltinConditions, Constants
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="启动 SGLang 服务")
|
||||
parser.add_argument("--model", default="~/.models/Qwen2.5-Coder-32B-Instruct-AWQ", help="模型路径")
|
||||
parser.add_argument("--port", type=int, default=8000, help="服务端口")
|
||||
parser.add_argument("--ctx-len", type=int, default=28672, help="最大上下文长度")
|
||||
parser.add_argument("--mem", type=float, default=0.75, help="显存占比 (0-1)")
|
||||
parser.add_argument("--host", default="0.0.0.0", help="主机地址")
|
||||
parser.add_argument("--log-level", default="info", help="日志级别")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.model:
|
||||
parser.error("model is required")
|
||||
|
||||
model_dir = Path(args.model).expanduser()
|
||||
if not model_dir.exists():
|
||||
parser.error(f"Model directory {model_dir} does not exist.")
|
||||
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
name="download",
|
||||
cmd=[
|
||||
"uv",
|
||||
"install",
|
||||
"sglang[all]",
|
||||
],
|
||||
conditions=(BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("sglang")),),
|
||||
verbose=True,
|
||||
),
|
||||
px.TaskSpec(
|
||||
name="run",
|
||||
cmd=[
|
||||
"python" if Constants.IS_WINDOWS else "python3",
|
||||
"-m",
|
||||
"sglang.launch_server",
|
||||
"--model-path",
|
||||
str(model_dir),
|
||||
"--host",
|
||||
str(args.host),
|
||||
"--port",
|
||||
"8000",
|
||||
"--mem-fraction-static",
|
||||
str(args.mem),
|
||||
"--context-length",
|
||||
"32768",
|
||||
"--tool-call-parser",
|
||||
"qwen",
|
||||
"--log-level",
|
||||
str(args.log_level),
|
||||
],
|
||||
verbose=True,
|
||||
),
|
||||
])
|
||||
|
||||
px.run(graph, strategy="sequential", verbose=True)
|
||||
@@ -0,0 +1,174 @@
|
||||
"""LS-DYNA 计算工具.
|
||||
|
||||
用于管理 LS-DYNA 仿真计算任务,
|
||||
支持启动、监控和管理计算进程.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
LS_DYNA_COMMANDS: dict[str, list[str]] = {
|
||||
"windows": ["ls-dyna_mpp", "i=input.k", "ncpu=4"],
|
||||
"linux": ["ls-dyna_mpp", "i=input.k", "ncpu=8"],
|
||||
"macos": ["ls-dyna_mpp", "i=input.k", "ncpu=4"],
|
||||
}
|
||||
|
||||
DEFAULT_INPUT_FILE: str = "input.k"
|
||||
DEFAULT_NCPU: int = 4
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def get_ls_dyna_command(input_file: str, ncpu: int) -> list[str]:
|
||||
"""获取 LS-DYNA 命令.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[str]
|
||||
LS-DYNA 命令列表
|
||||
"""
|
||||
if Constants.IS_WINDOWS or Constants.IS_MACOS:
|
||||
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
|
||||
else:
|
||||
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
|
||||
|
||||
|
||||
def run_ls_dyna(input_file: str, ncpu: int = DEFAULT_NCPU) -> None:
|
||||
"""运行 LS-DYNA 计算.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
"""
|
||||
input_path = Path(input_file)
|
||||
if not input_path.exists():
|
||||
print(f"输入文件不存在: {input_path}")
|
||||
return
|
||||
|
||||
cmd = get_ls_dyna_command(input_file, ncpu)
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"LS-DYNA 计算完成: {input_file}")
|
||||
except FileNotFoundError:
|
||||
print("未找到 ls-dyna_mpp 命令")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"LS-DYNA 计算失败: {e}")
|
||||
|
||||
|
||||
def run_ls_dyna_mpi(input_file: str, ncpu: int = DEFAULT_NCPU) -> None:
|
||||
"""运行 LS-DYNA MPI 计算.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
"""
|
||||
input_path = Path(input_file)
|
||||
if not input_path.exists():
|
||||
print(f"输入文件不存在: {input_path}")
|
||||
return
|
||||
|
||||
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"LS-DYNA MPI 计算完成: {input_file}")
|
||||
except FileNotFoundError:
|
||||
print("未找到 mpirun 或 ls-dyna_mpp 命令")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"LS-DYNA MPI 计算失败: {e}")
|
||||
|
||||
|
||||
def check_ls_dyna_status() -> None:
|
||||
"""检查 LS-DYNA 进程状态."""
|
||||
try:
|
||||
if Constants.IS_WINDOWS:
|
||||
result = subprocess.run(
|
||||
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
print(result.stdout)
|
||||
else:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-f", "ls-dyna"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.stdout.strip():
|
||||
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
|
||||
else:
|
||||
print("没有运行中的 LS-DYNA 进程")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"检查进程状态失败: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""LS-DYNA 计算工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="LSCalc - LS-DYNA 计算工具",
|
||||
usage="lscalc <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 运行计算命令
|
||||
run_parser = subparsers.add_parser("run", help="运行 LS-DYNA 计算")
|
||||
run_parser.add_argument("input_file", help="输入文件路径")
|
||||
run_parser.add_argument("--ncpu", type=int, default=DEFAULT_NCPU, help="CPU 核心数")
|
||||
|
||||
# 运行 MPI 计算命令
|
||||
mpi_parser = subparsers.add_parser("mpi", help="运行 LS-DYNA MPI 计算")
|
||||
mpi_parser.add_argument("input_file", help="输入文件路径")
|
||||
mpi_parser.add_argument("--ncpu", type=int, default=DEFAULT_NCPU, help="CPU 核心数")
|
||||
|
||||
# 检查进程状态命令
|
||||
subparsers.add_parser("status", help="检查 LS-DYNA 进程状态")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "run":
|
||||
graph = px.Graph.from_specs(
|
||||
[px.TaskSpec("run_ls_dyna", fn=run_ls_dyna, args=(args.input_file,), kwargs={"ncpu": args.ncpu})]
|
||||
)
|
||||
elif args.command == "mpi":
|
||||
graph = px.Graph.from_specs(
|
||||
[px.TaskSpec("run_ls_dyna_mpi", fn=run_ls_dyna_mpi, args=(args.input_file,), kwargs={"ncpu": args.ncpu})]
|
||||
)
|
||||
elif args.command == "status":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("check_ls_dyna_status", fn=check_ls_dyna_status)])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,349 @@
|
||||
"""Python 打包工具模块.
|
||||
|
||||
提供 Python 项目打包的常用功能封装,
|
||||
支持源码打包、依赖打包、嵌入式 Python 安装等功能.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import shutil
|
||||
import subprocess
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
DEFAULT_BUILD_DIR = ".pypack"
|
||||
DEFAULT_DIST_DIR = "dist"
|
||||
DEFAULT_LIB_DIR = "libs"
|
||||
DEFAULT_CACHE_DIR = ".cache/pypack"
|
||||
|
||||
IGNORE_PATTERNS = [
|
||||
"__pycache__",
|
||||
"*.pyc",
|
||||
"*.pyo",
|
||||
".git",
|
||||
".venv",
|
||||
".idea",
|
||||
".vscode",
|
||||
"*.egg-info",
|
||||
"dist",
|
||||
"build",
|
||||
".pytest_cache",
|
||||
".tox",
|
||||
".mypy_cache",
|
||||
]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def pack_source(project_dir: Path, output_dir: Path) -> None:
|
||||
"""打包项目源码.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
project_dir : Path
|
||||
项目目录
|
||||
output_dir : Path
|
||||
输出目录
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 检测项目名称
|
||||
pyproject_file = project_dir / "pyproject.toml"
|
||||
project_name = project_dir.name
|
||||
|
||||
if pyproject_file.exists():
|
||||
try:
|
||||
import tomllib
|
||||
|
||||
content = pyproject_file.read_text(encoding="utf-8")
|
||||
data = tomllib.loads(content)
|
||||
project_name = data.get("project", {}).get("name", project_name)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# 打包源码
|
||||
source_dir = output_dir / "src" / project_name
|
||||
source_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 复制文件
|
||||
src_subdir = project_dir / "src"
|
||||
if src_subdir.exists():
|
||||
shutil.copytree(
|
||||
src_subdir,
|
||||
source_dir / "src",
|
||||
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
else:
|
||||
for item in project_dir.iterdir():
|
||||
if item.name in IGNORE_PATTERNS or item.name.startswith("."):
|
||||
continue
|
||||
dst_item = source_dir / item.name
|
||||
if item.is_dir():
|
||||
shutil.copytree(
|
||||
item,
|
||||
dst_item,
|
||||
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
else:
|
||||
shutil.copy2(item, dst_item)
|
||||
|
||||
print(f"源码打包完成: {source_dir}")
|
||||
|
||||
|
||||
def pack_dependencies(lib_dir: Path, dependencies: list[str]) -> None:
|
||||
"""打包项目依赖.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lib_dir : Path
|
||||
依赖库目录
|
||||
dependencies : list[str]
|
||||
依赖列表
|
||||
"""
|
||||
lib_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not dependencies:
|
||||
print("没有依赖需要打包")
|
||||
return
|
||||
|
||||
# 使用 pip 安装依赖到目标目录
|
||||
cmd = [
|
||||
"pip",
|
||||
"install",
|
||||
"--target",
|
||||
str(lib_dir),
|
||||
"--no-compile",
|
||||
"--no-warn-script-location",
|
||||
]
|
||||
cmd.extend(dependencies)
|
||||
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"依赖打包完成: {lib_dir}")
|
||||
|
||||
|
||||
def pack_wheel(project_dir: Path, output_dir: Path) -> None:
|
||||
"""打包项目为 wheel 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
project_dir : Path
|
||||
项目目录
|
||||
output_dir : Path
|
||||
输出目录
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 使用 pip wheel 打包
|
||||
cmd = [
|
||||
"pip",
|
||||
"wheel",
|
||||
"--no-deps",
|
||||
"--wheel-dir",
|
||||
str(output_dir),
|
||||
str(project_dir),
|
||||
]
|
||||
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"Wheel 打包完成: {output_dir}")
|
||||
|
||||
|
||||
def install_embed_python(version: str, output_dir: Path) -> None:
|
||||
"""安装嵌入式 Python.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
version : str
|
||||
Python 版本 (如: 3.10, 3.11)
|
||||
output_dir : Path
|
||||
输出目录
|
||||
"""
|
||||
import platform
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 构建下载 URL
|
||||
arch = platform.machine().lower()
|
||||
if arch in ["x86_64", "amd64"]:
|
||||
arch = "amd64"
|
||||
elif arch in ["arm64", "aarch64"]:
|
||||
arch = "arm64"
|
||||
|
||||
# 解析完整版本号
|
||||
version_map = {
|
||||
"3.8": "3.8.10",
|
||||
"3.9": "3.9.13",
|
||||
"3.10": "3.10.11",
|
||||
"3.11": "3.11.9",
|
||||
"3.12": "3.12.4",
|
||||
}
|
||||
full_version = version_map.get(version, f"{version}.0")
|
||||
|
||||
# Windows 嵌入式 Python 下载 URL
|
||||
url = f"https://www.python.org/ftp/python/{full_version}/python-{full_version}-embed-{arch}.zip"
|
||||
|
||||
# 下载并解压
|
||||
cache_file = Path(DEFAULT_CACHE_DIR) / f"python-{full_version}-embed-{arch}.zip"
|
||||
cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not cache_file.exists():
|
||||
print(f"正在下载嵌入式 Python {full_version}...")
|
||||
import urllib.request
|
||||
|
||||
urllib.request.urlretrieve(url, cache_file)
|
||||
print(f"下载完成: {cache_file}")
|
||||
|
||||
# 解压
|
||||
with zipfile.ZipFile(cache_file, "r") as zf:
|
||||
zf.extractall(output_dir)
|
||||
|
||||
print(f"嵌入式 Python 安装完成: {output_dir}")
|
||||
|
||||
|
||||
def create_zip_package(source_dir: Path, output_file: Path) -> None:
|
||||
"""创建 ZIP 打包文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
source_dir : Path
|
||||
源目录
|
||||
output_file : Path
|
||||
输出文件
|
||||
"""
|
||||
output_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with zipfile.ZipFile(output_file, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for file in source_dir.rglob("*"):
|
||||
if file.is_file():
|
||||
arcname = file.relative_to(source_dir)
|
||||
zf.write(file, arcname)
|
||||
|
||||
print(f"ZIP 打包完成: {output_file}")
|
||||
|
||||
|
||||
def clean_build_dir(build_dir: Path) -> None:
|
||||
"""清理构建目录.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
build_dir : Path
|
||||
构建目录
|
||||
"""
|
||||
if build_dir.exists():
|
||||
shutil.rmtree(build_dir)
|
||||
print(f"清理完成: {build_dir}")
|
||||
else:
|
||||
print(f"目录不存在: {build_dir}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Python 打包工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="PackTool - Python 打包工具",
|
||||
usage="packtool <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 源码打包命令
|
||||
src_parser = subparsers.add_parser("src", help="打包项目源码")
|
||||
src_parser.add_argument("--project-dir", type=str, default=".", help="项目目录")
|
||||
src_parser.add_argument("--output-dir", type=str, default=DEFAULT_BUILD_DIR, help="输出目录")
|
||||
|
||||
# 依赖打包命令
|
||||
deps_parser = subparsers.add_parser("deps", help="打包项目依赖")
|
||||
deps_parser.add_argument("--lib-dir", type=str, default=DEFAULT_LIB_DIR, help="依赖库目录")
|
||||
deps_parser.add_argument("dependencies", nargs="*", help="依赖列表")
|
||||
|
||||
# Wheel 打包命令
|
||||
wheel_parser = subparsers.add_parser("wheel", help="打包项目为 wheel 文件")
|
||||
wheel_parser.add_argument("--project-dir", type=str, default=".", help="项目目录")
|
||||
wheel_parser.add_argument("--output-dir", type=str, default=DEFAULT_DIST_DIR, help="输出目录")
|
||||
|
||||
# 嵌入式 Python 安装命令
|
||||
embed_parser = subparsers.add_parser("embed", help="安装嵌入式 Python")
|
||||
embed_parser.add_argument("--version", type=str, default="3.10", help="Python 版本")
|
||||
embed_parser.add_argument("--output-dir", type=str, default="python", help="输出目录")
|
||||
|
||||
# ZIP 打包命令
|
||||
zip_parser = subparsers.add_parser("zip", help="创建 ZIP 打包文件")
|
||||
zip_parser.add_argument("--source-dir", type=str, default=".", help="源目录")
|
||||
zip_parser.add_argument("--output-file", type=str, default="package.zip", help="输出文件")
|
||||
|
||||
# 清理命令
|
||||
subparsers.add_parser("clean", help="清理构建目录")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "src":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"pack_source",
|
||||
fn=pack_source,
|
||||
args=(Path(args.project_dir), Path(args.output_dir)),
|
||||
)
|
||||
]
|
||||
)
|
||||
elif args.command == "deps":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"pack_deps",
|
||||
fn=pack_dependencies,
|
||||
args=(Path(args.lib_dir), args.dependencies),
|
||||
)
|
||||
]
|
||||
)
|
||||
elif args.command == "wheel":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"pack_wheel",
|
||||
fn=pack_wheel,
|
||||
args=(Path(args.project_dir), Path(args.output_dir)),
|
||||
)
|
||||
]
|
||||
)
|
||||
elif args.command == "embed":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"install_embed",
|
||||
fn=install_embed_python,
|
||||
args=(args.version, Path(args.output_dir)),
|
||||
)
|
||||
]
|
||||
)
|
||||
elif args.command == "zip":
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"create_zip",
|
||||
fn=create_zip_package,
|
||||
args=(Path(args.source_dir), Path(args.output_file)),
|
||||
)
|
||||
]
|
||||
)
|
||||
elif args.command == "clean":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("clean_build", fn=clean_build_dir, args=(Path(DEFAULT_BUILD_DIR),))])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,523 @@
|
||||
"""PDF 工具模块.
|
||||
|
||||
提供 PDF 文件操作的常用功能封装,
|
||||
支持合并、拆分、压缩、加密、水印、OCR等功能.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
try:
|
||||
import fitz # PyMuPDF
|
||||
|
||||
HAS_PYMUPDF = True
|
||||
except ImportError:
|
||||
HAS_PYMUPDF = False
|
||||
|
||||
try:
|
||||
import pypdf
|
||||
|
||||
HAS_PYPDF = True
|
||||
except ImportError:
|
||||
HAS_PYPDF = False
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
PDF_SUFFIX = ".pdf"
|
||||
DEFAULT_QUALITY = 75
|
||||
DEFAULT_PASSWORD = ""
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def pdf_merge(input_paths: list[Path], output_path: Path) -> None:
|
||||
"""合并多个 PDF 文件."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库,请安装: pip install pypdf")
|
||||
return
|
||||
|
||||
writer = pypdf.PdfWriter()
|
||||
for input_path in input_paths:
|
||||
if input_path.exists():
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"合并完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_split(input_path: Path, output_dir: Path) -> None:
|
||||
"""拆分 PDF 文件为单页."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库,请安装: pip install pypdf")
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for i, page in enumerate(reader.pages):
|
||||
writer = pypdf.PdfWriter()
|
||||
writer.add_page(page)
|
||||
output_file = output_dir / f"{input_path.stem}_page_{i + 1}.pdf"
|
||||
with open(output_file, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"拆分完成: {output_dir}")
|
||||
|
||||
|
||||
def pdf_compress(input_path: Path, output_path: Path) -> None:
|
||||
"""压缩 PDF 文件."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
|
||||
doc.close()
|
||||
|
||||
original_size = input_path.stat().st_size
|
||||
new_size = output_path.stat().st_size
|
||||
ratio = (1 - new_size / original_size) * 100
|
||||
print(f"压缩完成: {output_path} (缩小 {ratio:.1f}%)")
|
||||
|
||||
|
||||
def pdf_encrypt(input_path: Path, output_path: Path, password: str) -> None:
|
||||
"""加密 PDF 文件."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库,请安装: pip install pypdf")
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
writer = pypdf.PdfWriter()
|
||||
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
writer.encrypt(user_password=password, owner_password=password, use_128bit=True)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"加密完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_decrypt(input_path: Path, output_path: Path, password: str) -> None:
|
||||
"""解密 PDF 文件."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库,请安装: pip install pypdf")
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
if reader.is_encrypted:
|
||||
reader.decrypt(password)
|
||||
|
||||
writer = pypdf.PdfWriter()
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"解密完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_extract_text(input_path: Path, output_path: Path) -> None:
|
||||
"""提取 PDF 文本."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
text = ""
|
||||
for page in doc:
|
||||
text += str(page.get_text()) + "\n\n"
|
||||
doc.close()
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(text, encoding="utf-8")
|
||||
print(f"文本提取完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_extract_images(input_path: Path, output_dir: Path) -> None:
|
||||
"""提取 PDF 图片."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
image_count = 0
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
for page_num, page in enumerate(doc):
|
||||
images = page.get_images(full=True)
|
||||
for img_idx, img in enumerate(images):
|
||||
xref = img[0]
|
||||
base_image = doc.extract_image(xref)
|
||||
image_data = base_image["image"]
|
||||
image_ext = base_image["ext"]
|
||||
image_path = output_dir / f"page_{page_num + 1}_img_{img_idx + 1}.{image_ext}"
|
||||
image_path.write_bytes(image_data)
|
||||
image_count += 1
|
||||
|
||||
doc.close()
|
||||
print(f"图片提取完成: {output_dir} (共 {image_count} 张)")
|
||||
|
||||
|
||||
def pdf_add_watermark(input_path: Path, output_path: Path, text: str = "CONFIDENTIAL") -> None:
|
||||
"""添加 PDF 水印."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
for page in doc:
|
||||
rect = page.rect
|
||||
text_width = fitz.get_text_length(text, fontsize=48)
|
||||
x = (rect.width - text_width) / 2
|
||||
y = rect.height / 2
|
||||
page.insert_text((x, y), text, fontsize=48, rotate=45, color=(0, 0, 0))
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"水印添加完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_rotate(input_path: Path, output_path: Path, rotation: int = 90) -> None:
|
||||
"""旋转 PDF 页面."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
for page in doc:
|
||||
page.set_rotation(rotation)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"旋转完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_crop(input_path: Path, output_path: Path, margins: tuple[int, int, int, int]) -> None:
|
||||
"""裁剪 PDF 页面."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
left, top, right, bottom = margins
|
||||
|
||||
for page in doc:
|
||||
rect = page.rect
|
||||
new_rect = fitz.Rect(
|
||||
rect.x0 + left,
|
||||
rect.y0 + top,
|
||||
rect.x1 - right,
|
||||
rect.y1 - bottom,
|
||||
)
|
||||
page.set_cropbox(new_rect)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"裁剪完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_info(input_path: Path) -> None:
|
||||
"""显示 PDF 信息."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
print(f"文件: {input_path}")
|
||||
print(f"页数: {doc.page_count}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"标题: {doc.metadata.get('title', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"作者: {doc.metadata.get('author', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"创建日期: {doc.metadata.get('creationDate', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"修改日期: {doc.metadata.get('modDate', 'N/A')}")
|
||||
print(f"文件大小: {input_path.stat().st_size / 1024:.1f} KB")
|
||||
doc.close()
|
||||
|
||||
|
||||
def pdf_ocr(input_path: Path, output_path: Path, lang: str = "chi_sim+eng") -> None:
|
||||
"""PDF OCR 识别."""
|
||||
try:
|
||||
import pytesseract
|
||||
from PIL import Image
|
||||
except ImportError:
|
||||
print("未安装 OCR 相关库,请安装: pip install pytesseract pillow")
|
||||
return
|
||||
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
new_doc = fitz.open()
|
||||
|
||||
for page in doc:
|
||||
pix = page.get_pixmap()
|
||||
img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
|
||||
ocr_text = pytesseract.image_to_string(img, lang=lang)
|
||||
|
||||
new_page = new_doc.new_page(width=page.rect.width, height=page.rect.height)
|
||||
new_page.insert_image(new_page.rect, pixmap=pix)
|
||||
text_rect = fitz.Rect(0, 0, page.rect.width, page.rect.height)
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
new_page.insert_textbox(text_rect, ocr_text, fontname="china-ss", fontsize=11)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
new_doc.save(str(output_path))
|
||||
new_doc.close()
|
||||
doc.close()
|
||||
print(f"OCR 识别完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_reorder(input_path: Path, output_path: Path, order: list[int]) -> None:
|
||||
"""重排 PDF 页面顺序."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库,请安装: pip install pypdf")
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
writer = pypdf.PdfWriter()
|
||||
|
||||
for page_num in order:
|
||||
if 0 <= page_num < len(reader.pages):
|
||||
writer.add_page(reader.pages[page_num])
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"重排完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_to_images(input_path: Path, output_dir: Path, dpi: int = 300) -> None:
|
||||
"""PDF 转图片."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
for page_num, page in enumerate(doc):
|
||||
pix = page.get_pixmap(dpi=dpi)
|
||||
image_path = output_dir / f"{input_path.stem}_page_{page_num + 1}.png"
|
||||
pix.save(str(image_path))
|
||||
|
||||
doc.close()
|
||||
print(f"转换完成: {output_dir}")
|
||||
|
||||
|
||||
def pdf_repair(input_path: Path, output_path: Path) -> None:
|
||||
"""修复 PDF 文件."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库,请安装: pip install PyMuPDF")
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
|
||||
doc.close()
|
||||
print(f"修复完成: {output_path}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None: # noqa: PLR0912
|
||||
"""PDF 工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="PDFTool - PDF 文件工具集",
|
||||
usage="pdftool <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 合并 PDF 命令
|
||||
merge_parser = subparsers.add_parser("m", help="合并 PDF 文件")
|
||||
merge_parser.add_argument("inputs", nargs="+", help="输入 PDF 文件路径")
|
||||
merge_parser.add_argument("--output", type=str, default="merged.pdf", help="输出文件路径")
|
||||
|
||||
# 拆分 PDF 命令
|
||||
split_parser = subparsers.add_parser("s", help="拆分 PDF 文件为单页")
|
||||
split_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
split_parser.add_argument("--output-dir", type=str, default="split", help="输出目录")
|
||||
|
||||
# 压缩 PDF 命令
|
||||
compress_parser = subparsers.add_parser("c", help="压缩 PDF 文件")
|
||||
compress_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
compress_parser.add_argument("--output", type=str, default="compressed.pdf", help="输出文件路径")
|
||||
|
||||
# 加密 PDF 命令
|
||||
encrypt_parser = subparsers.add_parser("e", help="加密 PDF 文件")
|
||||
encrypt_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
encrypt_parser.add_argument("--output", type=str, default="encrypted.pdf", help="输出文件路径")
|
||||
encrypt_parser.add_argument("--password", type=str, required=True, help="密码")
|
||||
|
||||
# 解密 PDF 命令
|
||||
decrypt_parser = subparsers.add_parser("d", help="解密 PDF 文件")
|
||||
decrypt_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
decrypt_parser.add_argument("--output", type=str, default="decrypted.pdf", help="输出文件路径")
|
||||
decrypt_parser.add_argument("--password", type=str, required=True, help="密码")
|
||||
|
||||
# 提取文本命令
|
||||
extract_text_parser = subparsers.add_parser("xt", help="提取 PDF 文本")
|
||||
extract_text_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
extract_text_parser.add_argument("--output", type=str, default="output.txt", help="输出文件路径")
|
||||
|
||||
# 提取图片命令
|
||||
extract_images_parser = subparsers.add_parser("xi", help="提取 PDF 图片")
|
||||
extract_images_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
extract_images_parser.add_argument("--output-dir", type=str, default="images", help="输出目录")
|
||||
|
||||
# 添加水印命令
|
||||
watermark_parser = subparsers.add_parser("w", help="添加 PDF 水印")
|
||||
watermark_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
watermark_parser.add_argument("--output", type=str, default="watermarked.pdf", help="输出文件路径")
|
||||
watermark_parser.add_argument("--text", type=str, default="CONFIDENTIAL", help="水印文本")
|
||||
|
||||
# 旋转 PDF 命令
|
||||
rotate_parser = subparsers.add_parser("r", help="旋转 PDF 页面")
|
||||
rotate_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
rotate_parser.add_argument("--output", type=str, default="rotated.pdf", help="输出文件路径")
|
||||
rotate_parser.add_argument("--rotation", type=int, default=90, help="旋转角度 (90, 180, 270)")
|
||||
|
||||
# 裁剪 PDF 命令
|
||||
crop_parser = subparsers.add_parser("crop", help="裁剪 PDF 页面")
|
||||
crop_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
crop_parser.add_argument("--output", type=str, default="cropped.pdf", help="输出文件路径")
|
||||
crop_parser.add_argument("--left", type=int, default=10, help="左边裁剪")
|
||||
crop_parser.add_argument("--top", type=int, default=10, help="顶部裁剪")
|
||||
crop_parser.add_argument("--right", type=int, default=10, help="右边裁剪")
|
||||
crop_parser.add_argument("--bottom", type=int, default=10, help="底部裁剪")
|
||||
|
||||
# 显示信息命令
|
||||
info_parser = subparsers.add_parser("i", help="显示 PDF 信息")
|
||||
info_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
|
||||
# OCR 识别命令
|
||||
ocr_parser = subparsers.add_parser("ocr", help="PDF OCR 识别")
|
||||
ocr_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
ocr_parser.add_argument("--output", type=str, default="ocr.pdf", help="输出文件路径")
|
||||
ocr_parser.add_argument("--lang", type=str, default="chi_sim+eng", help="OCR 语言")
|
||||
|
||||
# 转换图片命令
|
||||
to_images_parser = subparsers.add_parser("img", help="PDF 转图片")
|
||||
to_images_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
to_images_parser.add_argument("--output-dir", type=str, default="images", help="输出目录")
|
||||
to_images_parser.add_argument("--dpi", type=int, default=300, help="图片 DPI")
|
||||
|
||||
# 修复 PDF 命令
|
||||
repair_parser = subparsers.add_parser("repair", help="修复 PDF 文件")
|
||||
repair_parser.add_argument("input", help="输入 PDF 文件路径")
|
||||
repair_parser.add_argument("--output", type=str, default="repaired.pdf", help="输出文件路径")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "m":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_merge", fn=pdf_merge, args=([Path(p) for p in args.inputs], Path(args.output)))
|
||||
])
|
||||
elif args.command == "s":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_split", fn=pdf_split, args=(Path(args.input), Path(args.output_dir)))
|
||||
])
|
||||
elif args.command == "c":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_compress", fn=pdf_compress, args=(Path(args.input), Path(args.output)))
|
||||
])
|
||||
elif args.command == "e":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_encrypt", fn=pdf_encrypt, args=(Path(args.input), Path(args.output), args.password))
|
||||
])
|
||||
elif args.command == "d":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_decrypt", fn=pdf_decrypt, args=(Path(args.input), Path(args.output), args.password))
|
||||
])
|
||||
elif args.command == "xt":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_extract_text", fn=pdf_extract_text, args=(Path(args.input), Path(args.output)))
|
||||
])
|
||||
elif args.command == "xi":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_extract_images", fn=pdf_extract_images, args=(Path(args.input), Path(args.output_dir)))
|
||||
])
|
||||
elif args.command == "w":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pdf_watermark",
|
||||
fn=pdf_add_watermark,
|
||||
args=(Path(args.input), Path(args.output)),
|
||||
kwargs={"text": args.text},
|
||||
)
|
||||
])
|
||||
elif args.command == "r":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pdf_rotate",
|
||||
fn=pdf_rotate,
|
||||
args=(Path(args.input), Path(args.output)),
|
||||
kwargs={"rotation": args.rotation},
|
||||
)
|
||||
])
|
||||
elif args.command == "crop":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pdf_crop",
|
||||
fn=pdf_crop,
|
||||
args=(Path(args.input), Path(args.output)),
|
||||
kwargs={"margins": (args.left, args.top, args.right, args.bottom)},
|
||||
)
|
||||
])
|
||||
elif args.command == "i":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("pdf_info", fn=pdf_info, args=(Path(args.input),))])
|
||||
elif args.command == "ocr":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_ocr", fn=pdf_ocr, args=(Path(args.input), Path(args.output)), kwargs={"lang": args.lang})
|
||||
])
|
||||
elif args.command == "img":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pdf_to_images",
|
||||
fn=pdf_to_images,
|
||||
args=(Path(args.input), Path(args.output_dir)),
|
||||
kwargs={"dpi": args.dpi},
|
||||
)
|
||||
])
|
||||
elif args.command == "repair":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pdf_repair", fn=pdf_repair, args=(Path(args.input), Path(args.output)))
|
||||
])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -1,657 +0,0 @@
|
||||
"""PyFlowX 统一 CLI 入口.
|
||||
|
||||
通过 ``pf <tool> [command] [options]`` 调用所有工具,
|
||||
工具定义在 ``pyflowx.ops`` 子包中, 每个模块用 ``@px.tool`` 装饰器注册.
|
||||
|
||||
用法
|
||||
----
|
||||
pf # 列出所有可用工具
|
||||
pf filedate # 查看 filedate 工具帮助
|
||||
pf filedate add a.txt # 调用 filedate 的 add 子命令
|
||||
pf pymake b # 调用 pymake 的 b 别名
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import difflib
|
||||
import importlib
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
from rich.text import Text
|
||||
from rich.tree import Tree
|
||||
|
||||
from pyflowx import __version__
|
||||
from pyflowx.tools import _TOOL_REGISTRY, run_tool
|
||||
|
||||
|
||||
class PfApp:
|
||||
"""pf 统一入口应用.
|
||||
|
||||
路由 ``pf <tool> [command]`` 到 ``@px.tool`` 注册的工具或传统 Python 工具.
|
||||
"""
|
||||
|
||||
# 工具名到 ops 模块名的映射 (支持短别名)
|
||||
_TOOL_ALIASES: dict[str, str] = {
|
||||
"autofmt": "autofmt",
|
||||
"af": "autofmt",
|
||||
"bump": "bumpversion",
|
||||
"bumpversion": "bumpversion",
|
||||
"bv": "bumpversion",
|
||||
"clr": "clr",
|
||||
"clearscreen": "clr",
|
||||
"dockercmd": "dockercmd",
|
||||
"docker": "dockercmd",
|
||||
"envdev": "envdev",
|
||||
"env": "envdev",
|
||||
"filedate": "filedate",
|
||||
"fd": "filedate",
|
||||
"filelevel": "filelevel",
|
||||
"fl": "filelevel",
|
||||
"folderback": "folderback",
|
||||
"foldback": "folderback",
|
||||
"fb": "folderback",
|
||||
"folderzip": "folderzip",
|
||||
"foldzip": "folderzip",
|
||||
"fz": "folderzip",
|
||||
"git": "gittool",
|
||||
"gitt": "gittool",
|
||||
"gittool": "gittool",
|
||||
"gt": "gittool",
|
||||
"ls": "lscalc",
|
||||
"lscalc": "lscalc",
|
||||
"msdown": "msdownload",
|
||||
"msdownload": "msdownload",
|
||||
"msd": "msdownload",
|
||||
"pack": "packtool",
|
||||
"packtool": "packtool",
|
||||
"pk": "packtool",
|
||||
"pdf": "pdftool",
|
||||
"pdftool": "pdftool",
|
||||
"pt": "pdftool",
|
||||
"pip": "piptool",
|
||||
"pymake": "pymake",
|
||||
"piptool": "piptool",
|
||||
"pp": "piptool",
|
||||
"reseticon": "reseticoncache",
|
||||
"reseticoncache": "reseticoncache",
|
||||
"ric": "reseticoncache",
|
||||
"screenshot": "screenshot",
|
||||
"scrcap": "screenshot",
|
||||
"ss": "screenshot",
|
||||
"se": "setenv",
|
||||
"setenv": "setenv",
|
||||
"sglang": "sglang",
|
||||
"sg": "sglang",
|
||||
"ssh": "sshcopyid",
|
||||
"sshcopy": "sshcopyid",
|
||||
"sshcopyid": "sshcopyid",
|
||||
"sc": "sshcopyid",
|
||||
"taskk": "taskkill",
|
||||
"taskkill": "taskkill",
|
||||
"tk": "taskkill",
|
||||
"wch": "which",
|
||||
"which": "which",
|
||||
"wf": "writefile",
|
||||
"writefile": "writefile",
|
||||
}
|
||||
|
||||
# 传统工具: 有自己的 main() 函数 (无法 @px.tool 化的复杂逻辑)
|
||||
_LEGACY_TOOLS: dict[str, str] = {
|
||||
"emlman": "pyflowx.cli.legacy.emlmanager:main",
|
||||
"profiler": "pyflowx.cli.legacy.profiler:main",
|
||||
"pxp": "pyflowx.cli.legacy.profiler:main",
|
||||
}
|
||||
|
||||
# 规范工具名 → 完整模块路径 (ops/ 按功能分组后的动态导入映射)
|
||||
_TOOL_MODULES: dict[str, str] = {
|
||||
"autofmt": "pyflowx.ops.dev.autofmt",
|
||||
"bumpversion": "pyflowx.ops.dev.bumpversion",
|
||||
"clr": "pyflowx.ops.system.clr",
|
||||
"dockercmd": "pyflowx.ops.infra.dockercmd",
|
||||
"envdev": "pyflowx.ops.infra.envdev",
|
||||
"filedate": "pyflowx.ops.files.filedate",
|
||||
"filelevel": "pyflowx.ops.files.filelevel",
|
||||
"folderback": "pyflowx.ops.files.folderback",
|
||||
"folderzip": "pyflowx.ops.files.folderzip",
|
||||
"gittool": "pyflowx.ops.dev.gittool",
|
||||
"lscalc": "pyflowx.ops.dev.lscalc",
|
||||
"msdownload": "pyflowx.ops.infra.msdownload",
|
||||
"packtool": "pyflowx.ops.dev.packtool",
|
||||
"pdftool": "pyflowx.ops.files.pdftool",
|
||||
"piptool": "pyflowx.ops.dev.piptool",
|
||||
"pymake": "pyflowx.ops.dev.pymake",
|
||||
"reseticoncache": "pyflowx.ops.system.reseticoncache",
|
||||
"screenshot": "pyflowx.ops.files.screenshot",
|
||||
"setenv": "pyflowx.ops.system.setenv",
|
||||
"sglang": "pyflowx.ops.infra.sglang",
|
||||
"sshcopyid": "pyflowx.ops.infra.sshcopyid",
|
||||
"taskkill": "pyflowx.ops.system.taskkill",
|
||||
"which": "pyflowx.ops.system.which",
|
||||
"writefile": "pyflowx.ops.system.writefile",
|
||||
}
|
||||
|
||||
def __init__(self, argv: Sequence[str] | None = None) -> None:
|
||||
self._argv = list(argv) if argv is not None else sys.argv[1:]
|
||||
self._console = Console()
|
||||
self._err = Console(stderr=True)
|
||||
|
||||
def run(self) -> int:
|
||||
"""主入口, 返回退出码."""
|
||||
if not self._argv or self._argv[0] in ("--help", "-h"):
|
||||
self._list_tools()
|
||||
return 0
|
||||
|
||||
first = self._argv[0]
|
||||
if first in ("--version", "-V"):
|
||||
self._console.print(f"PyFlowX [bold cyan]{__version__}[/bold cyan]")
|
||||
return 0
|
||||
|
||||
# 内建子命令(与 @px.tool 注册的 ops 工具同级)
|
||||
builtin = {
|
||||
"yamlrun": self._run_yaml,
|
||||
"graph": self._run_graph,
|
||||
"info": self._run_info,
|
||||
"completion": self._run_completion,
|
||||
}
|
||||
if first in builtin:
|
||||
return builtin[first](self._argv[1:])
|
||||
|
||||
rest = self._argv[1:]
|
||||
resolved = self._resolve_tool(first)
|
||||
if resolved is None:
|
||||
self._print_unknown_tool(first)
|
||||
return 1
|
||||
|
||||
kind, target = resolved
|
||||
if kind == "legacy":
|
||||
return self._run_legacy(target, rest)
|
||||
return self._run_tool(target, rest)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 工具列表 (rich)
|
||||
# ------------------------------------------------------------------ #
|
||||
def _list_tools(self) -> None:
|
||||
"""rich 表格列出所有可用工具."""
|
||||
self._console.print(
|
||||
Panel(
|
||||
Text(f"PyFlowX v{__version__}", style="bold cyan", justify="center"),
|
||||
subtitle="[dim]pf <tool> [command] [options][/dim]",
|
||||
)
|
||||
)
|
||||
|
||||
table = Table(title="@px.tool 工具", show_header=True, header_style="bold", show_lines=False)
|
||||
table.add_column("命令", style="cyan", no_wrap=True)
|
||||
table.add_column("别名", style="dim", no_wrap=True)
|
||||
table.add_column("说明")
|
||||
|
||||
for tool in sorted(set(self._TOOL_ALIASES.values())):
|
||||
aliases = self._aliases_for(tool)
|
||||
table.add_row(f"pf {tool}", ", ".join(aliases), self._tool_description(tool))
|
||||
|
||||
self._console.print(table)
|
||||
|
||||
if self._LEGACY_TOOLS:
|
||||
legacy = Table(title="传统工具", show_header=True, header_style="bold", show_lines=False)
|
||||
legacy.add_column("命令", style="cyan", no_wrap=True)
|
||||
for tool in sorted(self._LEGACY_TOOLS):
|
||||
legacy.add_row(f"pf {tool}")
|
||||
self._console.print(legacy)
|
||||
|
||||
self._console.print("\n[bold]示例:[/bold]")
|
||||
self._console.print(" [cyan]pf filedate add a.txt[/cyan] # 给文件添加日期前缀")
|
||||
self._console.print(" [cyan]pf pymake b[/cyan] # 构建 Python 包")
|
||||
self._console.print(" [cyan]pf gitt c[/cyan] # 清理并查看 git 状态")
|
||||
self._console.print(" [cyan]pf yamlrun pipeline.yaml[/cyan] # 执行 YAML 任务图")
|
||||
|
||||
def _aliases_for(self, canonical: str) -> list[str]:
|
||||
"""获取工具的别名 (不含规范名本身)."""
|
||||
return sorted(a for a, t in self._TOOL_ALIASES.items() if t == canonical and a != canonical)
|
||||
|
||||
def _tool_description(self, tool_name: str) -> str:
|
||||
"""获取工具描述 (从 _TOOL_REGISTRY 中已注册 ToolSpec 的 description/help)."""
|
||||
with contextlib.suppress(ImportError, KeyError):
|
||||
importlib.import_module(self._TOOL_MODULES[tool_name])
|
||||
|
||||
if tool_name not in _TOOL_REGISTRY:
|
||||
return ""
|
||||
|
||||
subs = _TOOL_REGISTRY[tool_name]
|
||||
for spec in subs.values():
|
||||
if spec.description:
|
||||
return spec.description
|
||||
for spec in subs.values():
|
||||
if not spec.hidden and spec.help:
|
||||
return spec.help
|
||||
return ""
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 路由
|
||||
# ------------------------------------------------------------------ #
|
||||
def _resolve_tool(self, name: str) -> tuple[str, str] | None:
|
||||
"""解析工具名, 返回 (类型, 目标)."""
|
||||
if name in self._TOOL_ALIASES:
|
||||
return ("tool", self._TOOL_ALIASES[name])
|
||||
if name in self._LEGACY_TOOLS:
|
||||
return ("legacy", self._LEGACY_TOOLS[name])
|
||||
return None
|
||||
|
||||
def _print_unknown_tool(self, name: str) -> None:
|
||||
"""打印未知工具错误 + 模糊匹配建议."""
|
||||
self._err.print(f"[red]错误:[/red] 未知工具 [yellow]{name!r}[/yellow]")
|
||||
suggestions = difflib.get_close_matches(name, list(self._TOOL_ALIASES), n=3, cutoff=0.5)
|
||||
if suggestions:
|
||||
self._err.print(f"[dim]是否想用: {', '.join(suggestions)}[/dim]")
|
||||
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
|
||||
|
||||
def _run_legacy(self, module_path: str, argv: list[str]) -> int:
|
||||
"""运行传统工具的 main() 函数."""
|
||||
module_name, func_name = module_path.split(":", 1)
|
||||
module = importlib.import_module(module_name)
|
||||
func = getattr(module, func_name)
|
||||
|
||||
original_argv = sys.argv
|
||||
sys.argv = [f"pf {module_name.split('.')[-1]}", *argv]
|
||||
try:
|
||||
func()
|
||||
return 0
|
||||
except SystemExit as e:
|
||||
return int(e.code) if e.code is not None else 0
|
||||
finally:
|
||||
sys.argv = original_argv
|
||||
|
||||
def _run_yaml(self, argv: list[str]) -> int:
|
||||
"""执行 yamlrun 命令: 从 YAML 文件加载任务图并执行."""
|
||||
import argparse
|
||||
from typing import get_args
|
||||
|
||||
from pyflowx import Graph, run
|
||||
from pyflowx.errors import PyFlowXError, TaskFailedError
|
||||
from pyflowx.executors import Strategy
|
||||
from pyflowx.runner import CliExitCode
|
||||
|
||||
parser = argparse.ArgumentParser(prog="pf yamlrun", description="执行 YAML 任务图")
|
||||
_ = parser.add_argument("file", help="YAML 文件路径")
|
||||
_ = parser.add_argument(
|
||||
"--strategy",
|
||||
choices=list(get_args(Strategy)),
|
||||
default="dependency",
|
||||
help="执行策略 (默认: %(default)s)",
|
||||
)
|
||||
_ = parser.add_argument("--dry-run", action="store_true", help="仅打印执行计划")
|
||||
_ = parser.add_argument("--list", action="store_true", help="列出所有任务名")
|
||||
_ = parser.add_argument("--list-tag", help="--list 模式下只显示含此标签的任务")
|
||||
_ = parser.add_argument(
|
||||
"--list-name",
|
||||
help="--list 模式下按子串过滤任务名(大小写不敏感)",
|
||||
)
|
||||
_ = parser.add_argument("--quiet", action="store_true", help="静默模式")
|
||||
_ = parser.add_argument("--progress", action="store_true", help="显示 rich 进度条")
|
||||
_ = parser.add_argument("--only", help="只运行指定任务(逗号分隔)及其依赖")
|
||||
_ = parser.add_argument("--tags", help="只运行匹配标签的任务(逗号分隔)及其依赖")
|
||||
|
||||
parsed = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
graph = Graph.from_yaml(parsed.file)
|
||||
except (OSError, ValueError, PyFlowXError) as e:
|
||||
self._err.print(f"[red]错误:[/red] 加载 YAML 失败: {e}")
|
||||
return CliExitCode.FAILURE.value
|
||||
|
||||
if parsed.list:
|
||||
self._print_task_list(
|
||||
graph,
|
||||
tag_filter=parsed.list_tag,
|
||||
name_filter=parsed.list_name,
|
||||
)
|
||||
return CliExitCode.SUCCESS.value
|
||||
|
||||
if parsed.dry_run:
|
||||
self._console.print(graph.describe())
|
||||
return CliExitCode.SUCCESS.value
|
||||
|
||||
only = parsed.only.split(",") if parsed.only else None
|
||||
tags = parsed.tags.split(",") if parsed.tags else None
|
||||
progress = parsed.progress and not parsed.quiet
|
||||
|
||||
try:
|
||||
report = run(
|
||||
graph,
|
||||
strategy=parsed.strategy,
|
||||
verbose=not parsed.quiet,
|
||||
only=only,
|
||||
tags=tags,
|
||||
progress=progress,
|
||||
)
|
||||
if not report.success and not parsed.quiet:
|
||||
self._print_diagnostics(report)
|
||||
return CliExitCode.SUCCESS.value if report.success else CliExitCode.FAILURE.value
|
||||
except KeyboardInterrupt:
|
||||
print("\n操作已取消", file=sys.stderr)
|
||||
return CliExitCode.INTERRUPTED.value
|
||||
except PyFlowXError as e:
|
||||
self._err.print(f"[red]错误:[/red] {e}")
|
||||
# TaskFailedError 携带 report 时打印诊断摘要
|
||||
if isinstance(e, TaskFailedError) and e.report is not None and not parsed.quiet:
|
||||
self._print_diagnostics(e.report)
|
||||
return CliExitCode.FAILURE.value
|
||||
|
||||
def _print_diagnostics(self, report: Any) -> None:
|
||||
"""打印失败诊断摘要到 stderr。"""
|
||||
diag = report.diagnose()
|
||||
if diag is not None:
|
||||
self._err.print()
|
||||
self._err.print(diag.describe(), style="red")
|
||||
|
||||
def _print_task_list(
|
||||
self,
|
||||
graph: Any,
|
||||
tag_filter: str | None,
|
||||
name_filter: str | None,
|
||||
) -> None:
|
||||
"""打印任务列表,可按标签/名称子串过滤。"""
|
||||
name_lower = name_filter.lower() if name_filter else None
|
||||
for name in graph.names:
|
||||
if name_lower is not None and name_filter is not None and name_lower not in name.lower():
|
||||
continue
|
||||
if tag_filter is not None:
|
||||
spec = graph.spec(name)
|
||||
if tag_filter not in spec.tags:
|
||||
continue
|
||||
self._console.print(name)
|
||||
|
||||
def _run_graph(self, argv: list[str]) -> int:
|
||||
"""执行 graph 命令: 终端可视化 YAML 任务图 DAG."""
|
||||
import argparse
|
||||
|
||||
from pyflowx import Graph
|
||||
from pyflowx.errors import PyFlowXError
|
||||
from pyflowx.runner import CliExitCode
|
||||
|
||||
parser = argparse.ArgumentParser(prog="pf graph", description="可视化 YAML 任务图 DAG")
|
||||
_ = parser.add_argument("file", help="YAML 文件路径")
|
||||
_ = parser.add_argument(
|
||||
"--format",
|
||||
choices=["ascii", "mermaid", "list", "deps"],
|
||||
default="ascii",
|
||||
help="输出格式 (默认: %(default)s)",
|
||||
)
|
||||
_ = parser.add_argument(
|
||||
"--orientation",
|
||||
default="TD",
|
||||
help="Mermaid 方向: TD/TB/BT/LR/RL (默认: %(default)s)",
|
||||
)
|
||||
_ = parser.add_argument(
|
||||
"--color-by",
|
||||
choices=["tag", "none"],
|
||||
default="tag",
|
||||
help="ASCII 格式节点着色策略: tag(按首标签着色) / none(单色) (默认: %(default)s)",
|
||||
)
|
||||
|
||||
parsed = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
graph = Graph.from_yaml(parsed.file)
|
||||
except (OSError, ValueError, PyFlowXError) as e:
|
||||
self._err.print(f"[red]错误:[/red] 加载 YAML 失败: {e}")
|
||||
return CliExitCode.FAILURE.value
|
||||
|
||||
fmt = parsed.format
|
||||
try:
|
||||
if fmt == "mermaid":
|
||||
# Mermaid 输出不加颜色,便于复制粘贴到 mermaid.live
|
||||
self._console.print(graph.to_mermaid(parsed.orientation), markup=False)
|
||||
elif fmt == "list":
|
||||
for name in graph.names:
|
||||
self._console.print(name)
|
||||
elif fmt == "deps":
|
||||
self._render_deps_table(graph)
|
||||
else:
|
||||
self._render_ascii(graph, color_by=parsed.color_by)
|
||||
except ValueError as e:
|
||||
self._err.print(f"[red]错误:[/red] {e}")
|
||||
return CliExitCode.FAILURE.value
|
||||
return CliExitCode.SUCCESS.value
|
||||
|
||||
def _render_ascii(self, graph: Any, color_by: str = "tag") -> None:
|
||||
"""用 rich 渲染分层 DAG 视图与依赖关系表。
|
||||
|
||||
``color_by="tag"`` 时按任务首标签选择 rich 颜色,便于视觉分组;
|
||||
``color_by="none"`` 时统一使用 cyan。
|
||||
"""
|
||||
layers = graph.layers()
|
||||
if not layers:
|
||||
self._console.print("[dim]空图(无任务)[/dim]")
|
||||
return
|
||||
|
||||
palette = ("cyan", "magenta", "yellow", "green", "blue", "red", "bright_cyan")
|
||||
tag_colors: dict[str, str] = {}
|
||||
counter = [0]
|
||||
|
||||
def _color_for(name: str) -> str:
|
||||
if color_by == "none":
|
||||
return "cyan"
|
||||
spec = graph.spec(name)
|
||||
if not spec.tags:
|
||||
return "cyan"
|
||||
tag = spec.tags[0]
|
||||
if tag not in tag_colors:
|
||||
tag_colors[tag] = palette[counter[0] % len(palette)]
|
||||
counter[0] += 1
|
||||
return tag_colors[tag]
|
||||
|
||||
tree = Tree(f"[bold cyan]Graph[/bold cyan] ({len(graph)} 任务, {len(layers)} 层)")
|
||||
for i, layer in enumerate(layers, 1):
|
||||
names = ", ".join(f"[{_color_for(n)}]{n}[/{_color_for(n)}]" for n in layer)
|
||||
tree.add(f"[bold]Layer {i}[/bold] ({len(layer)}): {names}")
|
||||
self._console.print(tree)
|
||||
|
||||
if tag_colors and color_by == "tag":
|
||||
legend = " ".join(f"[{c}]{t}[/{c}]" for t, c in tag_colors.items())
|
||||
self._console.print(f"[dim]标签颜色:[/dim] {legend}")
|
||||
|
||||
self._console.print()
|
||||
self._render_deps_table(graph)
|
||||
|
||||
def _render_deps_table(self, graph: Any) -> None:
|
||||
"""渲染任务依赖关系表."""
|
||||
table = Table(title="任务依赖关系", show_header=True, header_style="bold")
|
||||
table.add_column("任务", style="cyan", no_wrap=True)
|
||||
table.add_column("硬依赖", style="yellow")
|
||||
table.add_column("软依赖", style="dim")
|
||||
table.add_column("标签", style="green")
|
||||
|
||||
for name in graph.names:
|
||||
spec = graph.spec(name)
|
||||
hard = ", ".join(spec.depends_on) if spec.depends_on else "-"
|
||||
soft = ", ".join(spec.soft_depends_on) if spec.soft_depends_on else "-"
|
||||
tags = ", ".join(spec.tags) if spec.tags else "-"
|
||||
table.add_row(name, hard, soft, tags)
|
||||
self._console.print(table)
|
||||
|
||||
def _run_tool(self, target: str, argv: list[str]) -> int:
|
||||
"""运行 @px.tool 工具.
|
||||
|
||||
导入 ``pyflowx.ops.<group>.<target>`` 触发 ``@px.tool`` 注册, 然后调用 ``run_tool``.
|
||||
"""
|
||||
with contextlib.suppress(ImportError, KeyError):
|
||||
importlib.import_module(self._TOOL_MODULES[target])
|
||||
|
||||
if target not in _TOOL_REGISTRY:
|
||||
self._err.print(f"[red]错误:[/red] 未找到工具 [yellow]{target!r}[/yellow]")
|
||||
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
|
||||
return 1
|
||||
|
||||
return run_tool(target, argv)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# info: 工具详情
|
||||
# ------------------------------------------------------------------ #
|
||||
def _run_info(self, argv: list[str]) -> int:
|
||||
"""显示工具详情: ``pf info [tool]``。
|
||||
|
||||
无参数时列出所有工具的简表(含子命令数、描述);
|
||||
指定工具时显示该工具的所有子命令详情。
|
||||
"""
|
||||
if not argv:
|
||||
self._print_all_tools_detail()
|
||||
return 0
|
||||
|
||||
name = argv[0]
|
||||
resolved = self._resolve_tool(name)
|
||||
if resolved is None or resolved[0] != "tool":
|
||||
self._err.print(f"[red]错误:[/red] 未知工具 [yellow]{name!r}[/yellow]")
|
||||
self._err.print("[dim]运行 'pf' 查看可用工具列表[/dim]")
|
||||
return 1
|
||||
|
||||
canonical = resolved[1]
|
||||
self._print_tool_detail(canonical)
|
||||
return 0
|
||||
|
||||
def _print_all_tools_detail(self) -> None:
|
||||
"""打印所有工具的详情表。"""
|
||||
# 触发工具模块导入以填充 _TOOL_REGISTRY
|
||||
for mod_path in self._TOOL_MODULES.values():
|
||||
with contextlib.suppress(ImportError):
|
||||
importlib.import_module(mod_path)
|
||||
|
||||
table = Table(title="工具详情", show_header=True, header_style="bold")
|
||||
table.add_column("工具", style="cyan", no_wrap=True)
|
||||
table.add_column("别名", style="dim")
|
||||
table.add_column("子命令数", justify="right")
|
||||
table.add_column("描述")
|
||||
|
||||
for canonical in sorted(set(self._TOOL_ALIASES.values())):
|
||||
aliases = ", ".join(self._aliases_for(canonical))
|
||||
subs = _TOOL_REGISTRY.get(canonical, {})
|
||||
desc = self._tool_description(canonical)
|
||||
table.add_row(canonical, aliases or "-", str(len(subs)), desc or "-")
|
||||
|
||||
self._console.print(table)
|
||||
|
||||
def _print_tool_detail(self, canonical: str) -> None:
|
||||
"""打印指定工具的子命令详情。"""
|
||||
with contextlib.suppress(ImportError, KeyError):
|
||||
importlib.import_module(self._TOOL_MODULES[canonical])
|
||||
|
||||
subs = _TOOL_REGISTRY.get(canonical, {})
|
||||
aliases = self._aliases_for(canonical)
|
||||
self._console.print(f"[bold cyan]{canonical}[/bold cyan]", end="")
|
||||
if aliases:
|
||||
self._console.print(f" [dim]别名: {', '.join(aliases)}[/dim]")
|
||||
else:
|
||||
self._console.print()
|
||||
|
||||
desc = self._tool_description(canonical)
|
||||
if desc:
|
||||
self._console.print(f"[dim]描述: {desc}[/dim]")
|
||||
|
||||
if not subs:
|
||||
self._console.print("[dim]无子命令(直接调用)[/dim]")
|
||||
return
|
||||
|
||||
table = Table(title=f"{canonical} 子命令", show_header=True, header_style="bold")
|
||||
table.add_column("子命令", style="cyan", no_wrap=True)
|
||||
table.add_column("描述")
|
||||
table.add_column("隐藏", justify="center")
|
||||
|
||||
for name in sorted(n for n in subs if n is not None):
|
||||
spec = subs[name]
|
||||
table.add_row(
|
||||
name,
|
||||
spec.description or spec.help or "-",
|
||||
"是" if spec.hidden else "-",
|
||||
)
|
||||
self._console.print(table)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# completion: shell 补全脚本生成
|
||||
# ------------------------------------------------------------------ #
|
||||
def _run_completion(self, argv: list[str]) -> int:
|
||||
"""生成 shell 补全脚本: ``pf completion bash|zsh|fish``。"""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(prog="pf completion", description="生成 shell 补全脚本")
|
||||
_ = parser.add_argument(
|
||||
"shell",
|
||||
choices=["bash", "zsh", "fish"],
|
||||
help="目标 shell",
|
||||
)
|
||||
parsed = parser.parse_args(argv)
|
||||
|
||||
tools = sorted(set(self._TOOL_ALIASES.values()))
|
||||
builtin_cmds = ["yamlrun", "graph", "info", "completion"]
|
||||
all_cmds = builtin_cmds + tools
|
||||
|
||||
if parsed.shell == "bash":
|
||||
self._console.print(self._bash_completion(all_cmds), markup=False)
|
||||
elif parsed.shell == "zsh":
|
||||
self._console.print(self._zsh_completion(all_cmds), markup=False)
|
||||
else:
|
||||
self._console.print(self._fish_completion(all_cmds), markup=False)
|
||||
return 0
|
||||
|
||||
@staticmethod
|
||||
def _bash_completion(cmds: list[str]) -> str:
|
||||
"""生成 bash 补全脚本。"""
|
||||
words = " ".join(cmds)
|
||||
return f"""# pf bash 补全 —— 添加到 ~/.bashrc 或保存到 /etc/bash_completion.d/pf
|
||||
_pf_complete() {{
|
||||
local cur prev
|
||||
cur="${{COMP_WORDS[COMP_CWORD]}}"
|
||||
prev="${{COMP_WORDS[COMP_CWORD-1]}}"
|
||||
|
||||
if [ "$COMP_CWORD" -eq 1 ]; then
|
||||
COMPREPLY=($(compgen -W "{words}" -- "$cur"))
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 二级补全:交给具体子命令(pf <tool> --help 自描述)
|
||||
return 0
|
||||
}}
|
||||
complete -F _pf_complete pf
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _zsh_completion(cmds: list[str]) -> str:
|
||||
"""生成 zsh 补全脚本。"""
|
||||
words = " ".join(cmds)
|
||||
return f"""# pf zsh 补全 —— 添加到 ~/.zshrc
|
||||
#compdef pf
|
||||
|
||||
_pf() {{
|
||||
local -a cmds
|
||||
cmds=({words})
|
||||
|
||||
if [ "$CURRENT" -eq 2 ]; then
|
||||
_describe 'command' cmds
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 0
|
||||
}}
|
||||
|
||||
_pf "$@"
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _fish_completion(cmds: list[str]) -> str:
|
||||
"""生成 fish 补全脚本。"""
|
||||
lines = ["# pf fish 补全 —— 保存到 ~/.config/fish/completions/pf.fish"]
|
||||
for c in cmds:
|
||||
lines.append(f'complete -c pf -n "__fish_use_subcommand" -a "{c}"')
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""pf 统一入口主函数."""
|
||||
sys.exit(PfApp().run())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,195 @@
|
||||
"""pip 包管理工具模块.
|
||||
|
||||
提供 pip 包管理操作的封装,
|
||||
支持安装、卸载、下载等功能.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import fnmatch
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
PACKAGE_DIR = "packages"
|
||||
REQUIREMENTS_FILE = "requirements.txt"
|
||||
|
||||
# 受保护的包名集合
|
||||
_PROTECTED_PACKAGES: frozenset[str] = frozenset({
|
||||
"pyflowx",
|
||||
"bitool",
|
||||
})
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _get_installed_packages() -> list[str]:
|
||||
"""获取当前环境中所有已安装的包名."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pip", "list", "--format=freeze"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
packages: list[str] = []
|
||||
for line in result.stdout.strip().split("\n"):
|
||||
if line and "==" in line:
|
||||
pkg_name = line.split("==")[0].strip()
|
||||
packages.append(pkg_name)
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return []
|
||||
return packages
|
||||
|
||||
|
||||
def _expand_wildcard_packages(pattern: str) -> list[str]:
|
||||
"""展开通配符模式为实际的包名列表."""
|
||||
if not any(char in pattern for char in ["*", "?", "[", "]"]):
|
||||
return [pattern]
|
||||
|
||||
installed_packages = _get_installed_packages()
|
||||
matched = [pkg for pkg in installed_packages if fnmatch.fnmatchcase(pkg.lower(), pattern.lower())]
|
||||
return matched
|
||||
|
||||
|
||||
def _filter_protected_packages(packages: list[str]) -> list[str]:
|
||||
"""过滤掉受保护的包名."""
|
||||
safe = [p for p in packages if p.lower() not in {p.lower() for p in _PROTECTED_PACKAGES}]
|
||||
filtered = [p for p in packages if p.lower() in {p.lower() for p in _PROTECTED_PACKAGES}]
|
||||
if filtered:
|
||||
print(f"跳过受保护的包: {', '.join(filtered)}")
|
||||
return safe
|
||||
|
||||
|
||||
def pip_uninstall(pkg_names: list[str]) -> None:
|
||||
"""卸载包."""
|
||||
packages_to_uninstall: list[str] = []
|
||||
for pattern in pkg_names:
|
||||
packages_to_uninstall.extend(_expand_wildcard_packages(pattern))
|
||||
|
||||
packages_to_uninstall = _filter_protected_packages(packages_to_uninstall)
|
||||
|
||||
if not packages_to_uninstall:
|
||||
return
|
||||
|
||||
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True)
|
||||
|
||||
|
||||
def pip_reinstall(pkg_names: list[str], offline: bool = False) -> None:
|
||||
"""重新安装包."""
|
||||
safe_pkgs = _filter_protected_packages(pkg_names)
|
||||
if not safe_pkgs:
|
||||
print("所有指定的包均为受保护包, 跳过重装")
|
||||
return
|
||||
|
||||
subprocess.run(["pip", "uninstall", "-y", *safe_pkgs], check=True)
|
||||
|
||||
options = ["--no-index", "--find-links", "."] if offline else []
|
||||
subprocess.run(["pip", "install", *options, *safe_pkgs], check=True)
|
||||
|
||||
|
||||
def pip_download(pkg_names: list[str], offline: bool = False) -> None:
|
||||
"""下载包."""
|
||||
options = ["--no-index", "--find-links", "."] if offline else []
|
||||
subprocess.run(
|
||||
["pip", "download", *pkg_names, *options, "-d", PACKAGE_DIR],
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
def pip_freeze() -> None:
|
||||
"""冻结依赖."""
|
||||
result = subprocess.run(
|
||||
["pip", "freeze", "--exclude-editable"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
Path(REQUIREMENTS_FILE).write_text(result.stdout)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""pip 工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="PipTool - pip 包管理工具",
|
||||
usage="piptool <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 安装命令
|
||||
install_parser = subparsers.add_parser("i", help="安装包")
|
||||
install_parser.add_argument("packages", nargs="+", help="要安装的包名")
|
||||
|
||||
# 卸载命令
|
||||
uninstall_parser = subparsers.add_parser("u", help="卸载包")
|
||||
uninstall_parser.add_argument("packages", nargs="+", help="要卸载的包名 (支持通配符)")
|
||||
|
||||
# 重装命令
|
||||
reinstall_parser = subparsers.add_parser("r", help="重新安装包")
|
||||
reinstall_parser.add_argument("packages", nargs="+", help="要重装的包名")
|
||||
reinstall_parser.add_argument("--offline", action="store_true", help="使用离线模式")
|
||||
|
||||
# 下载命令
|
||||
download_parser = subparsers.add_parser("d", help="下载包")
|
||||
download_parser.add_argument("packages", nargs="+", help="要下载的包名")
|
||||
download_parser.add_argument("--offline", action="store_true", help="使用离线模式")
|
||||
|
||||
# 升级 pip 命令
|
||||
subparsers.add_parser("up", help="升级 pip")
|
||||
|
||||
# 冻结依赖命令
|
||||
subparsers.add_parser("f", help="冻结依赖到 requirements.txt")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "i":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("pip_install", cmd=["pip", "install", *args.packages], verbose=True)])
|
||||
elif args.command == "u":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pip_uninstall", fn=pip_uninstall, args=(args.packages,), verbose=True)
|
||||
])
|
||||
elif args.command == "r":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pip_reinstall",
|
||||
fn=pip_reinstall,
|
||||
args=(args.packages,),
|
||||
kwargs={"offline": args.offline},
|
||||
verbose=True,
|
||||
)
|
||||
])
|
||||
elif args.command == "d":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec(
|
||||
"pip_download",
|
||||
fn=pip_download,
|
||||
args=(args.packages,),
|
||||
kwargs={"offline": args.offline},
|
||||
verbose=True,
|
||||
)
|
||||
])
|
||||
elif args.command == "up":
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("pip_upgrade", cmd=["python", "-m", "pip", "install", "--upgrade", "pip"], verbose=True)
|
||||
])
|
||||
elif args.command == "f":
|
||||
graph = px.Graph.from_specs([px.TaskSpec("pip_freeze", fn=pip_freeze, verbose=True)])
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -40,10 +40,10 @@ import webbrowser
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ... import executors as _executors
|
||||
from ... import runner as _runner
|
||||
from ...profiling import ProfileReport
|
||||
from ...report import RunReport
|
||||
from .. import executors as _executors
|
||||
from .. import runner as _runner
|
||||
from ..profiling import ProfileReport
|
||||
from ..report import RunReport
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Python 构建工具模块.
|
||||
|
||||
完全替代传统的 Makefile,
|
||||
提供更好的跨平台兼容性和 Python 生态集成.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
# 项目根目录(pymake.py 在 src/pyflowx/cli,向上四层到达根目录)
|
||||
ROOT_DIR = Path(__file__).parent.parent.parent.parent
|
||||
|
||||
MATURIN_BUILD_COMMAND = ["maturin", "build", "-r"]
|
||||
if Constants.IS_WINDOWS:
|
||||
MATURIN_BUILD_COMMAND.extend(["--target", "x86_64-win7-windows-msvc", "-Zbuild-std", "-i", "python3.8"])
|
||||
|
||||
# 扁平注册所有任务(px.cmd 自动从命令前两段推导 name)
|
||||
# 所有任务指定 cwd=ROOT_DIR,确保在项目根目录执行
|
||||
tasks: list[px.TaskSpec] = [
|
||||
px.cmd(["uv", "build"], cwd=ROOT_DIR),
|
||||
px.cmd(MATURIN_BUILD_COMMAND, cwd=ROOT_DIR),
|
||||
px.cmd(["uv", "sync"], cwd=ROOT_DIR),
|
||||
px.cmd(["gitt", "c"], name="git_clean", cwd=ROOT_DIR),
|
||||
px.cmd(
|
||||
["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"],
|
||||
name="test",
|
||||
cwd=ROOT_DIR,
|
||||
),
|
||||
px.cmd(
|
||||
["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"],
|
||||
name="test_fast",
|
||||
cwd=ROOT_DIR,
|
||||
),
|
||||
px.cmd(
|
||||
["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
|
||||
name="test_coverage",
|
||||
cwd=ROOT_DIR,
|
||||
),
|
||||
px.cmd(["pyrefly", "check", "."], cwd=ROOT_DIR),
|
||||
px.cmd(["git", "add", "-A"], name="git_add_all", cwd=ROOT_DIR),
|
||||
px.cmd(["bumpversion"], cwd=ROOT_DIR),
|
||||
px.cmd(["bumpversion", "minor"], cwd=ROOT_DIR),
|
||||
px.cmd(["git", "push"], cwd=ROOT_DIR),
|
||||
px.cmd(["git", "push", "--tags"], name="git_push_tags", cwd=ROOT_DIR),
|
||||
px.cmd(["hatch", "publish"], name="publish_python", cwd=ROOT_DIR),
|
||||
px.cmd(["twine", "upload", "--disable-progress-bar"], name="twine_publish", cwd=ROOT_DIR),
|
||||
]
|
||||
|
||||
# 单任务别名(alias 名与任务名相同):直接内联 TaskSpec,避免 str 自引用
|
||||
aliases: dict[str, str | list[str | px.TaskSpec] | px.TaskSpec | px.Graph] = {
|
||||
# 构建命令
|
||||
"b": "uv_build",
|
||||
"bc": "maturin_build",
|
||||
"ba": ["b", "bc"],
|
||||
# 安装命令
|
||||
"sync": "uv_sync",
|
||||
# 清理命令
|
||||
"c": "git_clean",
|
||||
# 开发工具
|
||||
"bump": ["c", "tc", "git_add_all", "bumpversion"],
|
||||
"bumpmi": "bumpversion_minor",
|
||||
"cov": ["git_clean", "test_coverage"],
|
||||
"doc": px.cmd(["sphinx-build", "-b", "html", "docs", "docs/_build"], name="doc", cwd=ROOT_DIR),
|
||||
"lint": px.cmd(["ruff", "check", "--fix", "--unsafe-fixes"], name="lint", cwd=ROOT_DIR),
|
||||
"pb": ["twine_publish", "publish_python"],
|
||||
"t": "test",
|
||||
"tf": "test_fast",
|
||||
"tc": ["pyrefly_check", "lint"],
|
||||
"tox": px.cmd(["tox", "-p", "auto"], name="tox", cwd=ROOT_DIR),
|
||||
# 发布命令
|
||||
"p": ["git_clean", "git_push", "git_push_tags"],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""pymake 构建工具.
|
||||
|
||||
🔨 构建命令:
|
||||
pymake b - 构建 Python 主包 (uv build)
|
||||
pymake bc - 构建 Rust 核心模块 (maturin build)
|
||||
pymake ba - 构建所有包 (先 Python 后 Rust)
|
||||
|
||||
📦 安装命令 (开发模式):
|
||||
pymake sync - 安装依赖包 (uv sync)
|
||||
|
||||
🧹 清理命令:
|
||||
pymake c - 清理所有构建产物 (gitt c)
|
||||
|
||||
🛠️ 开发工具:
|
||||
pymake t - 运行测试 (pytest)
|
||||
pymake tc - 运行测试并生成覆盖率报告
|
||||
pymake tf - 运行快速测试 (pytest -m not slow)
|
||||
pymake lint - 代码格式化与检查 (ruff)
|
||||
pymake type - 类型检查 (mypy, ty)
|
||||
pymake doc - 构建文档 (sphinx)
|
||||
|
||||
🔬 多版本测试:
|
||||
pymake tox - 多版本 Python 测试 (tox -p auto)
|
||||
|
||||
📦 发布命令:
|
||||
pymake pb - 发布到 PyPI (twine + hatch)
|
||||
|
||||
🔖 版本管理:
|
||||
pymake bump - 自动升级版本号并提交修改 (清理 + 检查 + 格式化 + git add + bumpversion)
|
||||
|
||||
💡 常用工作流:
|
||||
1. 日常开发: pymake lint && pymake t
|
||||
2. 构建发布包: pymake ba
|
||||
3. 多版本兼容性测试: pymake tox
|
||||
4. 发布到 PyPI: pymake pb
|
||||
|
||||
📝 示例:
|
||||
pymake ba # 构建所有包
|
||||
pymake sync # 安装依赖
|
||||
pymake t # 运行测试
|
||||
pymake tox # 多版本兼容性测试
|
||||
pymake lint # 格式化代码
|
||||
pymake type # 类型检查
|
||||
"""
|
||||
runner = px.CliRunner(strategy="sequential", description="PyMake - Python 构建工具", tasks=tasks, aliases=aliases)
|
||||
runner.run_cli()
|
||||
@@ -0,0 +1,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.tasks.system import reset_icon_cache
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""重启图标缓存工具主函数."""
|
||||
graph = px.Graph.from_specs(reset_icon_cache())
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -1,10 +1,11 @@
|
||||
"""screenshot - 截图工具.
|
||||
"""截图工具.
|
||||
|
||||
跨平台截图: Windows 用 PowerShell, macOS 用 screencapture, Linux 用 gnome-screenshot/scrot.
|
||||
跨平台截图工具, 支持全屏截图和区域截图.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
@@ -12,6 +13,10 @@ from pathlib import Path
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def get_screenshot_path(filename: str | None = None) -> Path:
|
||||
"""获取截图保存路径.
|
||||
@@ -35,7 +40,6 @@ def get_screenshot_path(filename: str | None = None) -> Path:
|
||||
return screenshots_dir / filename
|
||||
|
||||
|
||||
@px.tool("screenshot", subcommand="full", help="全屏截图")
|
||||
def take_screenshot_full(filename: str | None = None) -> None:
|
||||
"""全屏截图.
|
||||
|
||||
@@ -47,6 +51,7 @@ def take_screenshot_full(filename: str | None = None) -> None:
|
||||
output_path = get_screenshot_path(filename)
|
||||
|
||||
if Constants.IS_WINDOWS:
|
||||
# Windows: 使用 PowerShell 截图
|
||||
ps_script = f"""
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
@@ -61,8 +66,10 @@ $bitmap.Dispose()
|
||||
"""
|
||||
subprocess.run(["powershell", "-Command", ps_script], check=True)
|
||||
elif Constants.IS_MACOS:
|
||||
# macOS: 使用 screencapture
|
||||
subprocess.run(["screencapture", "-x", str(output_path)], check=True)
|
||||
else:
|
||||
# Linux: 使用 gnome-screenshot 或 scrot
|
||||
try:
|
||||
subprocess.run(["gnome-screenshot", "-f", str(output_path)], check=True)
|
||||
except FileNotFoundError:
|
||||
@@ -71,7 +78,6 @@ $bitmap.Dispose()
|
||||
print(f"截图已保存: {output_path}")
|
||||
|
||||
|
||||
@px.tool("screenshot", subcommand="area", help="区域截图")
|
||||
def take_screenshot_area(filename: str | None = None) -> None:
|
||||
"""区域截图.
|
||||
|
||||
@@ -83,6 +89,7 @@ def take_screenshot_area(filename: str | None = None) -> None:
|
||||
output_path = get_screenshot_path(filename)
|
||||
|
||||
if Constants.IS_WINDOWS:
|
||||
# Windows: 使用 PowerShell 截图 (需要用户选择区域)
|
||||
ps_script = f"""
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
@@ -106,11 +113,51 @@ $bitmap.Dispose()
|
||||
"""
|
||||
subprocess.run(["powershell", "-Command", ps_script], check=True)
|
||||
elif Constants.IS_MACOS:
|
||||
# macOS: 使用 screencapture 交互模式
|
||||
subprocess.run(["screencapture", "-i", str(output_path)], check=True)
|
||||
else:
|
||||
# Linux: 使用 gnome-screenshot 交互模式
|
||||
try:
|
||||
subprocess.run(["gnome-screenshot", "-a", "-f", str(output_path)], check=True)
|
||||
except FileNotFoundError:
|
||||
subprocess.run(["scrot", "-s", str(output_path)], check=True)
|
||||
|
||||
print(f"截图已保存: {output_path}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""截图工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Screenshot - 截图工具",
|
||||
usage="screenshot <command> [options]",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", help="可用命令")
|
||||
|
||||
# 全屏截图命令
|
||||
full_parser = subparsers.add_parser("full", help="全屏截图")
|
||||
full_parser.add_argument("--filename", type=str, help="文件名")
|
||||
|
||||
# 区域截图命令
|
||||
area_parser = subparsers.add_parser("area", help="区域截图")
|
||||
area_parser.add_argument("--filename", type=str, help="文件名")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "full":
|
||||
graph = px.Graph.from_specs(
|
||||
[px.TaskSpec("screenshot_full", fn=take_screenshot_full, kwargs={"filename": args.filename})]
|
||||
)
|
||||
elif args.command == "area":
|
||||
graph = px.Graph.from_specs(
|
||||
[px.TaskSpec("screenshot_area", fn=take_screenshot_area, kwargs={"filename": args.filename})]
|
||||
)
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -1,20 +1,23 @@
|
||||
"""sshcopyid - SSH 密钥部署工具.
|
||||
"""SSH 密钥部署工具.
|
||||
|
||||
将本地 SSH 公钥部署到远程服务器.
|
||||
类似 ssh-copy-id, 自动将 SSH 公钥部署到远程服务器,
|
||||
支持密码认证和密钥认证两种方式.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = ["ssh_copy_id"]
|
||||
# ============================================================================
|
||||
# 辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器")
|
||||
def ssh_copy_id(
|
||||
hostname: str,
|
||||
username: str,
|
||||
@@ -40,6 +43,7 @@ def ssh_copy_id(
|
||||
timeout : int
|
||||
SSH 操作超时秒数, 默认 30
|
||||
"""
|
||||
# 读取公钥
|
||||
pub_key_path = Path(keypath).expanduser()
|
||||
if not pub_key_path.exists():
|
||||
print(f"公钥文件不存在: {pub_key_path}")
|
||||
@@ -47,10 +51,12 @@ def ssh_copy_id(
|
||||
|
||||
pub_key = pub_key_path.read_text().strip()
|
||||
|
||||
# 构建部署脚本
|
||||
script = f"""mkdir -p ~/.ssh && chmod 700 ~/.ssh
|
||||
cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys
|
||||
grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys"""
|
||||
|
||||
# 使用 sshpass 执行
|
||||
try:
|
||||
subprocess.run(
|
||||
[
|
||||
@@ -74,7 +80,7 @@ grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}'
|
||||
)
|
||||
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}")
|
||||
except FileNotFoundError:
|
||||
print(f"未找到 sshpass 工具, 请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
|
||||
print(f"未找到 sshpass 工具,请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
|
||||
sys.exit(1)
|
||||
except subprocess.TimeoutExpired:
|
||||
print("SSH 连接超时")
|
||||
@@ -82,3 +88,35 @@ grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}'
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"SSH 执行失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI Runner
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""SSH 密钥部署工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="SSHCopyID - SSH 密钥部署工具",
|
||||
usage="sshcopyid <hostname> <username> <password> [--port PORT] [--keypath KEYPATH]",
|
||||
)
|
||||
parser.add_argument("hostname", type=str, help="远程服务器主机名或 IP 地址")
|
||||
parser.add_argument("username", type=str, help="远程服务器用户名")
|
||||
parser.add_argument("password", type=str, help="远程服务器密码")
|
||||
parser.add_argument("--port", type=int, default=22, help="SSH 端口 (默认: 22)")
|
||||
parser.add_argument("--keypath", type=str, default="~/.ssh/id_rsa.pub", help="公钥文件路径")
|
||||
parser.add_argument("--timeout", type=int, default=30, help="SSH 操作超时秒数 (默认: 30)")
|
||||
args = parser.parse_args()
|
||||
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(
|
||||
"ssh_deploy",
|
||||
fn=ssh_copy_id,
|
||||
args=(args.hostname, args.username, args.password),
|
||||
kwargs={"port": args.port, "keypath": args.keypath, "timeout": args.timeout},
|
||||
)
|
||||
]
|
||||
)
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,15 @@
|
||||
"""清屏工具.
|
||||
|
||||
跨平台清屏工具, 支持终端和控制台清屏.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.tasks.system import clr
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""清屏工具主函数."""
|
||||
graph = px.Graph.from_specs([clr()])
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,40 @@
|
||||
"""进程终止工具.
|
||||
|
||||
跨平台进程终止工具, 支持按名称终止进程.
|
||||
用法: taskkill proc_name [proc_name ...]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""进程终止工具主函数."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="TaskKill - 进程终止工具",
|
||||
usage="taskkill <process_name> [process_name ...]",
|
||||
)
|
||||
parser.add_argument(
|
||||
"process_names",
|
||||
type=str,
|
||||
nargs="+",
|
||||
help="进程名称 (如: chrome.exe python node)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if Constants.IS_WINDOWS:
|
||||
cmd = ["taskkill", "/f", "/im"]
|
||||
else:
|
||||
cmd = ["pkill", "-f"]
|
||||
|
||||
graph = px.Graph.from_specs(
|
||||
[
|
||||
px.TaskSpec(f"kill_{proc_name}", cmd=[*cmd, f"{proc_name}*"], verbose=True)
|
||||
for proc_name in args.process_names
|
||||
],
|
||||
)
|
||||
px.run(graph, strategy="thread")
|
||||
@@ -0,0 +1,21 @@
|
||||
"""命令查找工具.
|
||||
|
||||
跨平台查找可执行命令路径, 类似 Unix 的 which 命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.tasks.system import which
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""命令查找工具主函数."""
|
||||
parser = argparse.ArgumentParser(description="Which - 命令查找工具")
|
||||
parser.add_argument("commands", nargs="+", help="要查找的命令名称, 如: python ls ps gcc...")
|
||||
args = parser.parse_args()
|
||||
|
||||
graph = px.Graph.from_specs([which(cmd) for cmd in args.commands])
|
||||
px.run(graph, strategy="thread")
|
||||
+7
-14
@@ -10,16 +10,12 @@ from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from typing import Any, cast
|
||||
|
||||
from rich.console import Console
|
||||
from typing import Any, List, Union, cast
|
||||
|
||||
from .task import TaskSpec
|
||||
|
||||
__all__ = ["run_command"]
|
||||
|
||||
_cmd_console = Console()
|
||||
|
||||
|
||||
def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
|
||||
"""执行 ``spec.cmd`` 指定的命令(list / shell 字符串 / 可调用对象)。
|
||||
@@ -43,9 +39,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
|
||||
if callable(cmd) and not isinstance(cmd, (list, str)):
|
||||
name = getattr(cmd, "__name__", "callable")
|
||||
if verbose:
|
||||
_cmd_console.print(f"[cyan]▸[/cyan] 执行可调用命令: [bold]{name}[/bold]")
|
||||
print(f"[verbose] 执行可调用命令: {name}", flush=True)
|
||||
if cwd is not None:
|
||||
_cmd_console.print(f" [dim]工作目录: {cwd}[/dim]")
|
||||
print(f"[verbose] 工作目录: {cwd}", flush=True)
|
||||
try:
|
||||
return cmd()
|
||||
except Exception as e:
|
||||
@@ -62,9 +58,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
|
||||
label = "Shell 命令"
|
||||
|
||||
if verbose:
|
||||
_cmd_console.print(f"[cyan]▸[/cyan] {verb}: [bold]{cmd_str}[/bold]")
|
||||
print(f"[verbose] {verb}: {cmd_str}", flush=True)
|
||||
if cwd is not None:
|
||||
_cmd_console.print(f" [dim]工作目录: {cwd}[/dim]")
|
||||
print(f"[verbose] 工作目录: {cwd}", flush=True)
|
||||
|
||||
# 合并环境变量
|
||||
run_env: dict[str, str] | None = None
|
||||
@@ -74,7 +70,7 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cast(str | list[str], cmd),
|
||||
cast(Union[str, List[str]], cmd),
|
||||
shell=not is_list,
|
||||
cwd=cwd,
|
||||
env=run_env,
|
||||
@@ -91,12 +87,9 @@ def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
|
||||
raise RuntimeError(f"{label}执行异常: {cmd_str}: {e}") from e
|
||||
|
||||
if verbose:
|
||||
style = "green" if result.returncode == 0 else "red"
|
||||
_cmd_console.print(f"[{style}]返回码: {result.returncode}[/{style}]")
|
||||
print(f"[verbose] 返回码: {result.returncode}", flush=True)
|
||||
|
||||
if result.returncode == 0:
|
||||
if not verbose and result.stdout:
|
||||
print(result.stdout, end="", flush=True)
|
||||
return None
|
||||
|
||||
err_msg = f"{label}执行失败: `{cmd_str}`, 返回码: {result.returncode}"
|
||||
|
||||
@@ -16,9 +16,8 @@ import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, Callable
|
||||
|
||||
from .task import Condition, Context
|
||||
|
||||
|
||||
+1
-50
@@ -16,9 +16,8 @@ DAG 库中泛滥的样板包装器。
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from collections.abc import Mapping
|
||||
from functools import lru_cache
|
||||
from typing import Any
|
||||
from typing import Any, Mapping
|
||||
|
||||
from .errors import InjectionError
|
||||
from .task import Context, TaskSpec
|
||||
@@ -44,28 +43,6 @@ def _signature(fn: Any) -> inspect.Signature:
|
||||
return inspect.signature(fn)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1024)
|
||||
def _fn_no_dep_injection(fn: Any) -> tuple[tuple[str, ...] | None, str | None]:
|
||||
"""预计算 fn 在无依赖/无静态参数时的注入计划。
|
||||
|
||||
返回 ``(context_params, error_param)``:
|
||||
- ``context_params``: Context 标注参数名元组;``None`` 表示存在必填参数需走慢路径。
|
||||
- ``error_param``: 第一个无默认值且非 Context 标注的参数名(用于错误信息);``None`` 表示无此类参数。
|
||||
"""
|
||||
sig = inspect.signature(fn)
|
||||
context_params: list[str] = []
|
||||
for pname, param in sig.parameters.items():
|
||||
if _is_context_annotation(param.annotation):
|
||||
context_params.append(pname)
|
||||
continue
|
||||
if param.default is inspect.Parameter.empty and param.kind not in (
|
||||
inspect.Parameter.VAR_POSITIONAL,
|
||||
inspect.Parameter.VAR_KEYWORD,
|
||||
):
|
||||
return None, pname
|
||||
return tuple(context_params), None
|
||||
|
||||
|
||||
def _is_context_annotation(annotation: Any) -> bool:
|
||||
"""判断参数标注是否为(或指向)``Context``。"""
|
||||
if annotation is Context:
|
||||
@@ -76,28 +53,6 @@ def _is_context_annotation(annotation: Any) -> bool:
|
||||
return name in ("Context", "Mapping")
|
||||
|
||||
|
||||
def _try_fast_path(spec: TaskSpec[Any]) -> tuple[tuple[Any, ...], dict[str, Any]] | None:
|
||||
"""尝试快速路径:cmd 无参任务或 fn 无依赖任务。
|
||||
|
||||
返回 ``None`` 表示快速路径不适用,需走慢路径。
|
||||
"""
|
||||
# 快速路径 1:cmd 任务(无 fn)的 effective_fn 是无参闭包。
|
||||
if spec.fn is None and spec.cmd is not None and not spec.args and not spec.kwargs:
|
||||
return (), {}
|
||||
# 快速路径 2:fn 任务无依赖、无静态 args/kwargs 时,跳过 dep_context/collisions/leftover 构建。
|
||||
if not spec.depends_on and not spec.soft_depends_on and not spec.args and not spec.kwargs:
|
||||
fn = spec.effective_fn
|
||||
context_params, error_param = _fn_no_dep_injection(fn)
|
||||
if error_param is not None:
|
||||
raise InjectionError(
|
||||
spec.name,
|
||||
f"parameter {error_param!r} has no dependency, static value, or default.",
|
||||
)
|
||||
assert context_params is not None # error_param is None ⟺ context_params 非 None
|
||||
return (), {p: {} for p in context_params}
|
||||
return None
|
||||
|
||||
|
||||
def build_call_args(
|
||||
spec: TaskSpec[Any],
|
||||
context: Mapping[str, Any],
|
||||
@@ -107,10 +62,6 @@ def build_call_args(
|
||||
``context`` 必须已包含所有硬依赖与软依赖的结果(软依赖被跳过时由
|
||||
执行器填入 :attr:`TaskSpec.defaults` 中的默认值)。
|
||||
"""
|
||||
fast = _try_fast_path(spec)
|
||||
if fast is not None:
|
||||
return fast
|
||||
|
||||
fn = spec.effective_fn
|
||||
sig = _signature(fn)
|
||||
params = sig.parameters
|
||||
|
||||
@@ -1,324 +0,0 @@
|
||||
"""失败诊断:从 :class:`RunReport` 提取结构化的根因分析。
|
||||
|
||||
本模块不依赖 :class:`Graph`,仅通过 ``TaskResult.spec.depends_on`` 重建
|
||||
依赖关系,因此可对反序列化后的报告进行诊断。
|
||||
|
||||
核心概念
|
||||
--------
|
||||
|
||||
* **根因任务(root cause)**:FAILED 状态且其所有硬依赖都是 SUCCESS
|
||||
(或无依赖)的任务。它是依赖链中"最先出问题"的环节。
|
||||
* **依赖链(dependency chain)**:从根因到某个失败任务的路径。
|
||||
* **相似失败聚类**:按 ``(异常类型, 消息前缀)`` 聚类,发现批量失败模式。
|
||||
* **根因提示**:识别常见异常类型(FileNotFoundError/ImportError 等),
|
||||
给出可操作建议。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .report import RunReport
|
||||
from .task import TaskStatus
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FailureCluster:
|
||||
"""相似失败的聚类。
|
||||
|
||||
属性
|
||||
----
|
||||
pattern:
|
||||
异常类型名 + 消息前缀(如 ``"ValueError: invalid input"``)。
|
||||
tasks:
|
||||
同类失败的任务名列表。
|
||||
sample_error:
|
||||
代表性错误消息(``repr(error)``)。
|
||||
"""
|
||||
|
||||
pattern: str
|
||||
tasks: list[str]
|
||||
sample_error: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DependencyChain:
|
||||
"""失败任务的依赖链分析。
|
||||
|
||||
属性
|
||||
----
|
||||
failed_task:
|
||||
失败的任务名。
|
||||
root_cause:
|
||||
依赖链中最先失败的任务(无失败依赖的失败任务)。
|
||||
``failed_task`` 自身即为根因时与 ``failed_task`` 相同。
|
||||
chain:
|
||||
从 ``root_cause`` 到 ``failed_task`` 的任务名路径(含两端)。
|
||||
broken_at:
|
||||
链中第一个非 SUCCESS 的任务(通常等于 ``root_cause``)。
|
||||
"""
|
||||
|
||||
failed_task: str
|
||||
root_cause: str | None
|
||||
chain: list[str]
|
||||
broken_at: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiagnosticReport:
|
||||
"""运行失败的诊断报告。
|
||||
|
||||
属性
|
||||
----
|
||||
failed_tasks:
|
||||
所有 FAILED 状态的任务名。
|
||||
skipped_due_to_failure:
|
||||
因上游失败而 SKIPPED 的任务名(reason 含"失败"或"取消")。
|
||||
root_causes:
|
||||
根因任务列表(FAILED 且所有硬依赖 SUCCESS)。
|
||||
dependency_chains:
|
||||
每个失败任务的依赖链分析。
|
||||
failure_clusters:
|
||||
相似失败聚类。
|
||||
hints:
|
||||
针对根因异常的可操作建议。
|
||||
"""
|
||||
|
||||
failed_tasks: list[str]
|
||||
skipped_due_to_failure: list[str]
|
||||
root_causes: list[str]
|
||||
dependency_chains: list[DependencyChain]
|
||||
failure_clusters: list[FailureCluster]
|
||||
hints: list[str] = field(default_factory=list)
|
||||
|
||||
def describe(self) -> str:
|
||||
"""人类可读的多行诊断报告。"""
|
||||
lines: list[str] = ["=== 失败诊断报告 ==="]
|
||||
|
||||
# 根因
|
||||
if self.root_causes:
|
||||
lines.append(f"\n根因任务 ({len(self.root_causes)}):")
|
||||
for rc in self.root_causes:
|
||||
lines.append(f" - {rc}")
|
||||
else:
|
||||
lines.append("\n根因任务: 无法识别(可能所有失败任务的依赖均未成功)")
|
||||
|
||||
# 依赖链
|
||||
if self.dependency_chains:
|
||||
lines.append(f"\n依赖链 ({len(self.dependency_chains)}):")
|
||||
for dc in self.dependency_chains:
|
||||
chain_str = " -> ".join(dc.chain) if dc.chain else dc.failed_task
|
||||
lines.append(f" {dc.failed_task}: {chain_str}")
|
||||
|
||||
# 相似失败聚类
|
||||
if self.failure_clusters:
|
||||
lines.append(f"\n相似失败聚类 ({len(self.failure_clusters)}):")
|
||||
for cluster in self.failure_clusters:
|
||||
tasks_str = ", ".join(cluster.tasks)
|
||||
lines.append(f" [{cluster.pattern}] ({len(cluster.tasks)} 任务): {tasks_str}")
|
||||
|
||||
# 跳过的任务
|
||||
if self.skipped_due_to_failure:
|
||||
lines.append(f"\n因失败而跳过的任务 ({len(self.skipped_due_to_failure)}):")
|
||||
for name in self.skipped_due_to_failure:
|
||||
lines.append(f" - {name}")
|
||||
|
||||
# 提示
|
||||
if self.hints:
|
||||
lines.append("\n建议:")
|
||||
for hint in self.hints:
|
||||
lines.append(f" * {hint}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 内部辅助
|
||||
# ---------------------------------------------------------------------- #
|
||||
_HINT_MAP: dict[str, str] = {
|
||||
"FileNotFoundError": "文件不存在,检查路径配置与工作目录",
|
||||
"ImportError": "模块导入失败,检查依赖是否安装(pip install / uv add)",
|
||||
"ModuleNotFoundError": "模块不存在,检查依赖安装或 PYTHONPATH",
|
||||
"TimeoutError": "任务超时,考虑增加 timeout 配置或优化任务性能",
|
||||
"TaskTimeoutError": "任务超时,考虑增加 timeout 配置或优化任务性能",
|
||||
"ConnectionError": "网络连接失败,检查目标地址可达性与防火墙设置",
|
||||
"PermissionError": "权限不足,检查文件/目录权限或以管理员身份运行",
|
||||
"KeyError": "键缺失,检查上下文注入参数名是否匹配任务签名",
|
||||
"InjectionError": "上下文注入失败,检查任务签名参数与上游任务名",
|
||||
"StorageError": "状态后端错误,检查存储路径权限与磁盘空间",
|
||||
"CycleError": "依赖图存在环,检查 depends_on 是否形成循环",
|
||||
"MemoryError": "内存不足,考虑分批处理或减少并发数",
|
||||
"OSError": "系统错误,检查文件描述符/磁盘空间/权限",
|
||||
}
|
||||
|
||||
|
||||
def _classify_error(error: BaseException | None) -> tuple[str, str]:
|
||||
"""返回 ``(异常类型名, 消息前缀)`` 用于聚类。"""
|
||||
if error is None:
|
||||
return ("Unknown", "")
|
||||
type_name = type(error).__name__
|
||||
msg = str(error)
|
||||
prefix = msg[:50] if len(msg) > 50 else msg
|
||||
return (type_name, prefix)
|
||||
|
||||
|
||||
def _find_root_cause(
|
||||
failed_task: str,
|
||||
report: RunReport,
|
||||
) -> tuple[str | None, list[str]]:
|
||||
"""回溯 ``failed_task`` 的依赖链,返回 (根因任务, 路径)。
|
||||
|
||||
路径从根因到 ``failed_task``(含两端)。根因为 FAILED 且所有硬依赖
|
||||
非 FAILED 的任务;若 ``failed_task`` 自身满足条件,则为其自身。
|
||||
"""
|
||||
# BFS 反向遍历,找到链中"最深处"的根因
|
||||
visited: set[str] = set()
|
||||
|
||||
# 用 DFS 记录路径:path 始终以当前 task 开头,向根因方向延伸
|
||||
def _trace(task: str, path: list[str]) -> tuple[str | None, list[str]]:
|
||||
if task in visited: # 防止循环依赖(diagnose 不依赖 Graph 校验,results 可能有环)
|
||||
return (None, path)
|
||||
visited.add(task)
|
||||
result = report.results[task] # _trace 只接收 failed_tasks 或 failed_deps,均保证在 results 中
|
||||
|
||||
# 检查此任务是否为根因:FAILED 且所有硬依赖非 FAILED
|
||||
deps = result.spec.depends_on
|
||||
failed_deps = [d for d in deps if d in report.results and report.results[d].status == TaskStatus.FAILED]
|
||||
|
||||
if result.status == TaskStatus.FAILED and not failed_deps:
|
||||
return (task, path)
|
||||
|
||||
# 递归向上找
|
||||
for dep in failed_deps:
|
||||
root, new_path = _trace(dep, [dep, *path])
|
||||
if root is not None:
|
||||
return (root, new_path)
|
||||
|
||||
# 递归未找到明确根因时,当前任务作为断点返回
|
||||
return (task, path)
|
||||
|
||||
return _trace(failed_task, [failed_task])
|
||||
|
||||
|
||||
def _generate_hints(report: RunReport, root_causes: list[str]) -> list[str]:
|
||||
"""针对根因异常生成可操作建议。"""
|
||||
hints: list[str] = []
|
||||
seen_types: set[str] = set()
|
||||
for rc_name in root_causes:
|
||||
result = report.results.get(rc_name)
|
||||
if result is None or result.error is None:
|
||||
continue
|
||||
type_name = type(result.error).__name__
|
||||
if type_name in seen_types:
|
||||
continue
|
||||
seen_types.add(type_name)
|
||||
# 精确匹配
|
||||
if type_name in _HINT_MAP:
|
||||
hints.append(f"{rc_name}: {_HINT_MAP[type_name]}")
|
||||
continue
|
||||
# 父类匹配
|
||||
for parent in type(result.error).__mro__:
|
||||
parent_name = parent.__name__
|
||||
if parent_name in _HINT_MAP:
|
||||
hints.append(f"{rc_name}: {_HINT_MAP[parent_name]}")
|
||||
break
|
||||
return hints
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 公共 API
|
||||
# ---------------------------------------------------------------------- #
|
||||
def diagnose(report: RunReport) -> DiagnosticReport | None:
|
||||
"""生成失败诊断报告。
|
||||
|
||||
运行成功(``report.success`` 为 ``True``)时返回 ``None``;
|
||||
失败时返回 :class:`DiagnosticReport`,包含根因分析、依赖链回溯、
|
||||
相似失败聚类与可操作建议。
|
||||
|
||||
本函数仅依赖 ``report.results`` 中的 ``spec.depends_on`` 重建依赖
|
||||
关系,不依赖 :class:`Graph` 对象,因此可对反序列化后的报告进行诊断。
|
||||
"""
|
||||
if report.success:
|
||||
return None
|
||||
|
||||
failed_tasks = report.failed_tasks()
|
||||
if not failed_tasks:
|
||||
# success=False 但无 FAILED 任务(可能是用户中断等情况)
|
||||
return DiagnosticReport(
|
||||
failed_tasks=[],
|
||||
skipped_due_to_failure=report.skipped_tasks(),
|
||||
root_causes=[],
|
||||
dependency_chains=[],
|
||||
failure_clusters=[],
|
||||
hints=["运行未成功但无失败任务,可能是被取消或中断"],
|
||||
)
|
||||
|
||||
# 识别根因:FAILED 且所有硬依赖非 FAILED
|
||||
root_causes: list[str] = []
|
||||
for name in failed_tasks:
|
||||
result = report.results[name]
|
||||
failed_deps = [
|
||||
d for d in result.spec.depends_on if d in report.results and report.results[d].status == TaskStatus.FAILED
|
||||
]
|
||||
if not failed_deps:
|
||||
root_causes.append(name)
|
||||
|
||||
# 依赖链回溯
|
||||
chains: list[DependencyChain] = []
|
||||
for failed in failed_tasks:
|
||||
root, path = _find_root_cause(failed, report)
|
||||
# broken_at 是路径中第一个非 SUCCESS 的任务
|
||||
broken_at: str | None = None
|
||||
for task in path:
|
||||
r = report.results.get(task)
|
||||
if r is not None and r.status != TaskStatus.SUCCESS:
|
||||
broken_at = task
|
||||
break
|
||||
chains.append(
|
||||
DependencyChain(
|
||||
failed_task=failed,
|
||||
root_cause=root,
|
||||
chain=path,
|
||||
broken_at=broken_at,
|
||||
)
|
||||
)
|
||||
|
||||
# 相似失败聚类
|
||||
clusters_map: dict[tuple[str, str], list[str]] = defaultdict(list)
|
||||
sample_errors: dict[tuple[str, str], str] = {}
|
||||
for name in failed_tasks:
|
||||
result = report.results[name]
|
||||
key = _classify_error(result.error)
|
||||
clusters_map[key].append(name)
|
||||
if key not in sample_errors:
|
||||
sample_errors[key] = repr(result.error) if result.error else ""
|
||||
|
||||
clusters: list[FailureCluster] = [
|
||||
FailureCluster(
|
||||
pattern=f"{type_name}: {prefix}" if prefix else type_name,
|
||||
tasks=sorted(tasks),
|
||||
sample_error=sample_errors.get((type_name, prefix), ""),
|
||||
)
|
||||
for (type_name, prefix), tasks in sorted(clusters_map.items(), key=lambda x: -len(x[1]))
|
||||
]
|
||||
|
||||
# 因失败而跳过的任务(reason 包含"失败"或"取消"或"skip")
|
||||
skipped_due_to_failure: list[str] = []
|
||||
for name, result in report.results.items():
|
||||
if result.status == TaskStatus.SKIPPED:
|
||||
reason = (result.reason or "").lower()
|
||||
if any(kw in reason for kw in ("失败", "取消", "fail", "cancel", "abort")):
|
||||
skipped_due_to_failure.append(name)
|
||||
|
||||
# 根因提示
|
||||
hints = _generate_hints(report, root_causes)
|
||||
|
||||
return DiagnosticReport(
|
||||
failed_tasks=failed_tasks,
|
||||
skipped_due_to_failure=skipped_due_to_failure,
|
||||
root_causes=root_causes,
|
||||
dependency_chains=chains,
|
||||
failure_clusters=clusters,
|
||||
hints=hints,
|
||||
)
|
||||
@@ -6,11 +6,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .report import RunReport
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
class PyFlowXError(Exception):
|
||||
@@ -51,8 +47,7 @@ class TaskFailedError(PyFlowXError):
|
||||
"""任务耗尽所有重试后仍失败时抛出。
|
||||
|
||||
原始异常保留在 :attr:`__cause__` 上,同时通过 :attr:`cause` 暴露,
|
||||
便于用户代码访问。``report`` 字段(可选)携带失败时的 :class:`RunReport`,
|
||||
便于调用方生成诊断报告。
|
||||
便于用户代码访问。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -61,7 +56,6 @@ class TaskFailedError(PyFlowXError):
|
||||
cause: BaseException,
|
||||
attempts: int,
|
||||
layer: int | None = None,
|
||||
report: RunReport | None = None,
|
||||
) -> None:
|
||||
location = f" (layer {layer})" if layer is not None else ""
|
||||
super().__init__(f"Task '{task}' failed after {attempts} attempt(s){location}: {cause}")
|
||||
@@ -69,7 +63,6 @@ class TaskFailedError(PyFlowXError):
|
||||
self.cause = cause
|
||||
self.attempts = attempts
|
||||
self.layer = layer
|
||||
self.report = report
|
||||
|
||||
|
||||
class TaskTimeoutError(PyFlowXError):
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Example 3: async aggregation with static args and Context injection.
|
||||
|
||||
Shows:
|
||||
* async task functions executed with strategy="async".
|
||||
* static positional args (TaskSpec.args) for parameterised tasks.
|
||||
* Context annotation to receive the full upstream result mapping.
|
||||
* on_event callback for real-time progress.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import Any
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
|
||||
async def fetch_user(uid: int) -> dict[str, Any]:
|
||||
await asyncio.sleep(0.2)
|
||||
return {"id": uid, "name": f"User{uid}"}
|
||||
|
||||
|
||||
async def fetch_posts(uid: int) -> list[int]:
|
||||
await asyncio.sleep(0.2)
|
||||
return [uid, uid + 1]
|
||||
|
||||
|
||||
# Context annotation → receives the full mapping of upstream results.
|
||||
def aggregate(ctx: px.Context) -> dict[str, Any]:
|
||||
return dict(ctx)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
graph = px.Graph.from_specs([
|
||||
# Static positional args parameterise the same function twice.
|
||||
px.TaskSpec("fetch_user", fetch_user, args=(1,)),
|
||||
px.TaskSpec("fetch_posts", fetch_posts, args=(1,)),
|
||||
px.TaskSpec("aggregate", aggregate, depends_on=("fetch_user", "fetch_posts")),
|
||||
])
|
||||
|
||||
print("=== Dry run ===")
|
||||
_ = px.run(graph, strategy="async", dry_run=True)
|
||||
|
||||
events: list[px.TaskEvent] = []
|
||||
print("\n=== Async execution ===")
|
||||
report = px.run(graph, strategy="async", on_event=events.append)
|
||||
|
||||
for ev in events:
|
||||
print(f" event: {ev.task} -> {ev.status.value}")
|
||||
|
||||
print(f"\naggregate = {report['aggregate']}")
|
||||
print(report.describe())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,77 @@
|
||||
"""Example 1: ETL pipeline (sequential strategy).
|
||||
|
||||
Demonstrates the core PyFlowX workflow:
|
||||
* Define tasks as plain functions.
|
||||
* Declare the DAG with a list of TaskSpec.
|
||||
* Parameter names == dependency names → automatic context injection,
|
||||
no wrappers needed (contrast with flowweaver's get_task_result boilerplate).
|
||||
* dry_run to preview, then execute and read typed results from RunReport.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
# --- task functions: pure, testable, no framework coupling ------------- #
|
||||
|
||||
|
||||
def extract_customers() -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"id": "C001", "name": "Alice"},
|
||||
{"id": "C002", "name": "Bob"},
|
||||
]
|
||||
|
||||
|
||||
def extract_orders() -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"id": "O001", "customer_id": "C001", "amount": 150.0},
|
||||
{"id": "O002", "customer_id": "C002", "amount": 200.5},
|
||||
]
|
||||
|
||||
|
||||
# Parameter names match dependency names → automatic injection.
|
||||
def transform(
|
||||
extract_customers: list[dict[str, Any]],
|
||||
extract_orders: list[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
cmap = {c["id"]: c for c in extract_customers}
|
||||
return [{**o, "customer_name": cmap[o["customer_id"]]["name"]} for o in extract_orders if o["customer_id"] in cmap]
|
||||
|
||||
|
||||
def load(transform: list[dict[str, Any]]) -> int:
|
||||
print(f" loaded {len(transform)} records")
|
||||
return len(transform)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("extract_customers", extract_customers, tags=("extract",)),
|
||||
px.TaskSpec("extract_orders", extract_orders, tags=("extract",)),
|
||||
px.TaskSpec(
|
||||
"transform",
|
||||
transform,
|
||||
depends_on=("extract_customers", "extract_orders"),
|
||||
tags=("transform",),
|
||||
),
|
||||
px.TaskSpec(
|
||||
"load", load, depends_on=("transform",), retry=px.RetryPolicy(max_attempts=1, delay=1.0), tags=("load",)
|
||||
),
|
||||
])
|
||||
|
||||
print("=== Execution plan ===")
|
||||
print(graph.describe())
|
||||
|
||||
print("\n=== Dry run (no execution) ===")
|
||||
_ = px.run(graph, strategy="sequential", dry_run=True)
|
||||
|
||||
print("\n=== Sequential execution ===")
|
||||
report = px.run(graph, strategy="sequential")
|
||||
print(report.describe())
|
||||
print(f"\nload result = {report['load']}")
|
||||
print(f"summary = {report.summary()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Example 2: parallel execution (thread strategy).
|
||||
|
||||
Same DAG run with sequential vs. thread strategy to show layer-internal
|
||||
parallelism. Tasks within a layer run concurrently; layers are barriers.
|
||||
|
||||
Layer 1: [fetch_a, fetch_b] (parallel)
|
||||
Layer 2: [merge] (waits for both)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
|
||||
def fetch_a() -> str:
|
||||
time.sleep(0.5)
|
||||
return "a"
|
||||
|
||||
|
||||
def fetch_b() -> str:
|
||||
time.sleep(0.5)
|
||||
return "b"
|
||||
|
||||
|
||||
def merge(fetch_a: str, fetch_b: str) -> str:
|
||||
return fetch_a + fetch_b
|
||||
|
||||
|
||||
def main() -> None:
|
||||
graph = px.Graph.from_specs([
|
||||
px.TaskSpec("fetch_a", fetch_a),
|
||||
px.TaskSpec("fetch_b", fetch_b),
|
||||
px.TaskSpec("merge", merge, depends_on=("fetch_a", "fetch_b")),
|
||||
])
|
||||
|
||||
print("=== Mermaid diagram ===")
|
||||
print(graph.to_mermaid("LR"))
|
||||
|
||||
print("\n=== Sequential (expect ~1.0s) ===")
|
||||
start = time.time()
|
||||
report_seq = px.run(graph, strategy="sequential")
|
||||
t_seq = time.time() - start
|
||||
print(f" result={report_seq['merge']} time={t_seq:.2f}s")
|
||||
|
||||
print("\n=== Threaded (expect ~0.5s) ===")
|
||||
start = time.time()
|
||||
report_thr = px.run(graph, strategy="thread", max_workers=2)
|
||||
t_thr = time.time() - start
|
||||
print(f" result={report_thr['merge']} time={t_thr:.2f}s")
|
||||
|
||||
print(f"\nspeedup = {t_seq / t_thr:.2f}x")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+65
-547
@@ -46,32 +46,20 @@ import concurrent.futures
|
||||
import contextlib
|
||||
import inspect
|
||||
import logging
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping
|
||||
from dataclasses import replace as dc_replace
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, cast
|
||||
from typing import Any, Awaitable, Callable, Literal, Mapping, cast
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
from .cancellation import CancelToken, _is_cancelled
|
||||
from .context import build_call_args, describe_injection
|
||||
from .errors import TaskFailedError, TaskTimeoutError
|
||||
from .graph import Graph
|
||||
from .notification import Notifier
|
||||
from .progress import ProgressCallback, RichProgressMonitor
|
||||
from .report import RunReport
|
||||
from .storage import StateBackend, resolve_backend
|
||||
from .task import TaskEvent, TaskHooks, TaskResult, TaskSpec, TaskStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# verbose 模式输出用的 Console (不抢占用户 stdout, 仅格式化)
|
||||
_verbose_console = Console()
|
||||
|
||||
# 进程池复用:同一次 run() 内的 process 任务共享一个 ProcessPoolExecutor。
|
||||
# 模块级缓存避免每次任务都创建/销毁进程池的开销。
|
||||
# run() 结束后通过 _shutdown_process_pool() 关闭(shutdown(wait=False) +
|
||||
@@ -227,11 +215,7 @@ def _apply_cached(
|
||||
result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中")
|
||||
report.results[name] = result
|
||||
_emit(on_event, result)
|
||||
logger.info(
|
||||
"task %r skipped (cached)",
|
||||
name,
|
||||
extra={"run_id": report.run_id, "task_name": name, "status": TaskStatus.SKIPPED.value},
|
||||
)
|
||||
logger.info("task %r skipped (cached)", name)
|
||||
return True
|
||||
|
||||
|
||||
@@ -291,17 +275,7 @@ def _prepare_for_execution(
|
||||
reason=skip_reason,
|
||||
)
|
||||
_emit(on_event, result)
|
||||
logger.info(
|
||||
"task %r skipped (%s)",
|
||||
spec.name,
|
||||
skip_reason,
|
||||
extra={
|
||||
"run_id": report.run_id if report is not None else "-",
|
||||
"task_name": spec.name,
|
||||
"status": TaskStatus.SKIPPED.value,
|
||||
"reason": skip_reason,
|
||||
},
|
||||
)
|
||||
logger.info("task %r skipped (%s)", spec.name, skip_reason)
|
||||
return result
|
||||
|
||||
|
||||
@@ -323,30 +297,15 @@ def _finalize_failure(
|
||||
layer_idx: int | None,
|
||||
on_event: EventCallback | None,
|
||||
continue_on_error: bool,
|
||||
name: str,
|
||||
report: RunReport | None,
|
||||
) -> None:
|
||||
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。
|
||||
|
||||
失败结果在抛出前写入 ``report.results[name]``,使流式 API
|
||||
(:func:`run_iter`) 能在 re-raise 前yield 该结果。
|
||||
"""
|
||||
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。"""
|
||||
result.status = TaskStatus.FAILED
|
||||
result.finished_at = datetime.now()
|
||||
if report is not None:
|
||||
report.results[name] = result
|
||||
_emit(on_event, result)
|
||||
if continue_on_error:
|
||||
logger.warning(
|
||||
"task %r failed but continue_on_error=True; continuing.",
|
||||
result.spec.name,
|
||||
extra={
|
||||
"run_id": report.run_id if report is not None else "-",
|
||||
"task_name": result.spec.name,
|
||||
"status": TaskStatus.FAILED.value,
|
||||
"attempts": result.attempts,
|
||||
"error_type": type(result.error).__name__ if result.error else "Unknown",
|
||||
},
|
||||
)
|
||||
return
|
||||
raise TaskFailedError(
|
||||
@@ -354,7 +313,6 @@ def _finalize_failure(
|
||||
cause=result.error if result.error is not None else RuntimeError("unknown"),
|
||||
attempts=result.attempts,
|
||||
layer=layer_idx,
|
||||
report=report,
|
||||
)
|
||||
|
||||
|
||||
@@ -364,8 +322,6 @@ def _handle_failure(
|
||||
exc: BaseException,
|
||||
layer_idx: int | None,
|
||||
on_event: EventCallback | None,
|
||||
name: str,
|
||||
report: RunReport | None,
|
||||
) -> bool:
|
||||
"""统一处理失败:超时转换、重试决策、finalize。
|
||||
|
||||
@@ -375,7 +331,6 @@ def _handle_failure(
|
||||
``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。
|
||||
"""
|
||||
# asyncio.TimeoutError → TaskTimeoutError(统一异常类型)
|
||||
run_id = report.run_id if report is not None else "-"
|
||||
if isinstance(exc, asyncio.TimeoutError):
|
||||
exc = TaskTimeoutError(spec.name, spec.timeout or 0.0)
|
||||
logger.warning(
|
||||
@@ -383,13 +338,6 @@ def _handle_failure(
|
||||
spec.name,
|
||||
result.attempts,
|
||||
spec.retry.max_attempts,
|
||||
extra={
|
||||
"run_id": run_id,
|
||||
"task_name": spec.name,
|
||||
"status": TaskStatus.FAILED.value,
|
||||
"attempts": result.attempts,
|
||||
"error_type": "TaskTimeoutError",
|
||||
},
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
@@ -398,19 +346,12 @@ def _handle_failure(
|
||||
result.attempts,
|
||||
spec.retry.max_attempts,
|
||||
exc,
|
||||
extra={
|
||||
"run_id": run_id,
|
||||
"task_name": spec.name,
|
||||
"status": TaskStatus.FAILED.value,
|
||||
"attempts": result.attempts,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
)
|
||||
result.error = exc
|
||||
if _should_retry(spec, result.attempts, exc):
|
||||
return False
|
||||
_run_hooks(spec.hooks, "on_failure", spec, exc)
|
||||
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error, name, report)
|
||||
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error)
|
||||
return True
|
||||
|
||||
|
||||
@@ -447,7 +388,7 @@ class SyncTaskRunner:
|
||||
_mark_success(spec, result, value)
|
||||
return result
|
||||
except Exception as exc:
|
||||
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
|
||||
if _handle_failure(spec, result, exc, layer_idx, on_event):
|
||||
return result
|
||||
wait = spec.retry.wait_seconds(result.attempts)
|
||||
if wait > 0:
|
||||
@@ -474,7 +415,7 @@ class AsyncTaskRunner:
|
||||
result: TaskResult[Any] = TaskResult(spec=spec)
|
||||
result.started_at = datetime.now()
|
||||
args, kwargs = build_call_args(spec, context)
|
||||
loop = asyncio.get_running_loop()
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
_run_hooks(spec.hooks, "pre_run", spec)
|
||||
_emit_running(on_event, spec)
|
||||
@@ -486,7 +427,7 @@ class AsyncTaskRunner:
|
||||
_mark_success(spec, result, value)
|
||||
return result
|
||||
except Exception as exc:
|
||||
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
|
||||
if _handle_failure(spec, result, exc, layer_idx, on_event):
|
||||
return result
|
||||
wait = spec.retry.wait_seconds(result.attempts)
|
||||
if wait > 0:
|
||||
@@ -572,10 +513,6 @@ def _filter_and_sort(
|
||||
for name in layer:
|
||||
spec = graph.resolved_spec(name)
|
||||
specs[name] = spec
|
||||
# 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过
|
||||
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
|
||||
context[name] = report.results[name].value
|
||||
continue
|
||||
if not _apply_cached(name, spec, context, report, backend, on_event):
|
||||
to_run.append(name)
|
||||
return _sort_by_priority(to_run, specs)
|
||||
@@ -662,7 +599,7 @@ class ThreadedLayerRunner:
|
||||
backend: StateBackend,
|
||||
layer_idx: int,
|
||||
on_event: EventCallback | None,
|
||||
pool: concurrent.futures.ThreadPoolExecutor,
|
||||
max_workers: int,
|
||||
concurrency_limits: Mapping[str, int],
|
||||
) -> None:
|
||||
to_run = _filter_and_sort(layer, graph, context, report, backend, on_event)
|
||||
@@ -684,18 +621,21 @@ class ThreadedLayerRunner:
|
||||
if sem is not None:
|
||||
sem.release()
|
||||
|
||||
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
|
||||
pool.submit(_run_threaded_task, name): name for name in to_run
|
||||
}
|
||||
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
|
||||
try:
|
||||
for fut in concurrent.futures.as_completed(future_to_name):
|
||||
name = future_to_name[fut]
|
||||
completed[name] = fut.result()
|
||||
finally:
|
||||
with lock:
|
||||
for name, (task_ctx, result) in completed.items():
|
||||
_store_result(name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event)
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
|
||||
future_to_name: dict[concurrent.futures.Future[tuple[dict[str, Any], TaskResult[Any]]], str] = {
|
||||
pool.submit(_run_threaded_task, name): name for name in to_run
|
||||
}
|
||||
completed: dict[str, tuple[dict[str, Any], TaskResult[Any]]] = {}
|
||||
try:
|
||||
for fut in concurrent.futures.as_completed(future_to_name):
|
||||
name = future_to_name[fut]
|
||||
completed[name] = fut.result()
|
||||
finally:
|
||||
with lock:
|
||||
for name, (task_ctx, result) in completed.items():
|
||||
_store_result(
|
||||
name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event
|
||||
)
|
||||
|
||||
|
||||
class AsyncLayerRunner:
|
||||
@@ -726,30 +666,10 @@ class AsyncLayerRunner:
|
||||
return task_ctx, result
|
||||
|
||||
results = await asyncio.gather(*[_run_async_task(name) for name in to_run])
|
||||
for name, (task_ctx, result) in zip(to_run, results, strict=True):
|
||||
for name, (task_ctx, result) in zip(to_run, results):
|
||||
_store_result(name, result, graph.resolved_spec(name), task_ctx, context, report, backend, on_event)
|
||||
|
||||
|
||||
def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
|
||||
"""从 ``dynamic`` 任务的返回值中提取新 specs。
|
||||
|
||||
仅当 ``spec.dynamic`` 为 ``True`` 且任务成功时提取。返回值可以是单个
|
||||
:class:`TaskSpec` 或其列表。提取后,原结果的 ``value`` 被替换为生成的
|
||||
任务名列表(便于查询),specs 列表返回给调用方注册调度。
|
||||
"""
|
||||
if not spec.dynamic or result.status != TaskStatus.SUCCESS:
|
||||
return []
|
||||
value = result.value
|
||||
spawned: list[TaskSpec[Any]] = []
|
||||
if isinstance(value, TaskSpec):
|
||||
spawned = [value]
|
||||
elif isinstance(value, list) and value and all(isinstance(x, TaskSpec) for x in value):
|
||||
spawned = value
|
||||
if not spawned:
|
||||
return []
|
||||
return spawned
|
||||
|
||||
|
||||
class DependencyRunner:
|
||||
"""依赖驱动调度:任务在硬/软依赖完成后立即启动,无层屏障。
|
||||
|
||||
@@ -768,17 +688,13 @@ class DependencyRunner:
|
||||
backend: StateBackend,
|
||||
on_event: EventCallback | None,
|
||||
concurrency_limits: Mapping[str, int],
|
||||
cancel_event: threading.Event | CancelToken | None = None,
|
||||
) -> None:
|
||||
all_names = list(graph.all_specs().keys())
|
||||
semaphores = _build_semaphores(all_names, graph, asyncio.Semaphore, concurrency_limits)
|
||||
futures: dict[str, asyncio.Task[TaskResult[Any]]] = {}
|
||||
futures: dict[str, asyncio.Future[TaskResult[Any]]] = {}
|
||||
|
||||
async def _run_task(name: str) -> TaskResult[Any]:
|
||||
spec = graph.resolved_spec(name)
|
||||
# 检查点恢复:已在 report 中的 SUCCESS 任务直接返回
|
||||
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
|
||||
return report.results[name]
|
||||
# 等待所有硬依赖完成
|
||||
for dep in spec.depends_on:
|
||||
if dep in futures:
|
||||
@@ -788,244 +704,48 @@ class DependencyRunner:
|
||||
if dep in futures:
|
||||
await futures[dep]
|
||||
|
||||
# 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED
|
||||
if _is_cancelled(cancel_event) and name not in report.results:
|
||||
result: TaskResult[Any] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SKIPPED,
|
||||
finished_at=datetime.now(),
|
||||
reason="执行被取消",
|
||||
)
|
||||
report.results[name] = result
|
||||
_emit(on_event, result)
|
||||
return result
|
||||
|
||||
task_ctx = _build_context(spec, context, report)
|
||||
if _apply_cached(name, spec, context, report, backend, on_event):
|
||||
return report.results[name]
|
||||
|
||||
sem = _get_sem(semaphores, spec)
|
||||
result = await AsyncTaskRunner.run(spec, task_ctx, None, on_event, report, sem)
|
||||
|
||||
# 动态任务生成:提取 specs,替换 result.value 为生成任务名列表
|
||||
spawned = _extract_dynamic_specs(result, spec)
|
||||
if spawned:
|
||||
spawned_names = [s.name for s in spawned]
|
||||
result = dc_replace(result, value=spawned_names)
|
||||
# 重新存储已修改的 result
|
||||
context[name] = result.value
|
||||
report.results[name] = result
|
||||
_emit(on_event, result)
|
||||
else:
|
||||
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
|
||||
|
||||
# 注册并调度动态生成的 specs
|
||||
for raw_spec in spawned:
|
||||
# 确保新 spec 依赖生成方(保证顺序与上下文可见性)
|
||||
final_spec = raw_spec
|
||||
if name not in raw_spec.depends_on and name not in raw_spec.soft_depends_on:
|
||||
final_spec = dc_replace(raw_spec, depends_on=(*raw_spec.depends_on, name))
|
||||
graph._register_single(final_spec)
|
||||
# 为新 spec 的 concurrency_key 创建信号量
|
||||
if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores:
|
||||
limit = concurrency_limits.get(final_spec.concurrency_key, 1)
|
||||
semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit)
|
||||
futures[final_spec.name] = loop.create_task(_run_task(final_spec.name))
|
||||
|
||||
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
|
||||
return result
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
loop = asyncio.get_event_loop()
|
||||
for name in all_names:
|
||||
futures[name] = loop.create_task(_run_task(name))
|
||||
|
||||
# 循环等待所有 futures(含动态生成的),直到全部完成。
|
||||
# asyncio.wait 不像 gather 自动传播异常,需手动检查并 re-raise,
|
||||
# 匹配 gather 的 fail-fast 行为(首个异常即取消剩余任务并抛出)。
|
||||
while futures:
|
||||
await asyncio.wait(futures.values(), return_when=asyncio.FIRST_COMPLETED)
|
||||
completed_names = [n for n, f in futures.items() if f.done()]
|
||||
for done_name in completed_names:
|
||||
f = futures[done_name]
|
||||
exc = f.exception()
|
||||
if exc is not None:
|
||||
for remaining in futures.values():
|
||||
if not remaining.done():
|
||||
remaining.cancel()
|
||||
raise exc
|
||||
del futures[done_name]
|
||||
await asyncio.gather(*futures.values())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 公共 API
|
||||
# ---------------------------------------------------------------------- #
|
||||
def _compose_callbacks(*callbacks: EventCallback | None) -> EventCallback | None:
|
||||
"""组合多个回调为一个,按顺序调用;全为 None 时返回 None。"""
|
||||
fns = [c for c in callbacks if c is not None]
|
||||
if not fns:
|
||||
return None
|
||||
|
||||
def _composed(event: TaskEvent) -> None:
|
||||
for fn in fns:
|
||||
fn(event)
|
||||
|
||||
return _composed
|
||||
|
||||
|
||||
def _finish_monitoring(monitor: ProgressCallback | None, notifier_list: list[Notifier]) -> None:
|
||||
"""执行结束时清理监控器与通知器资源。"""
|
||||
if monitor is not None:
|
||||
monitor.finish()
|
||||
for n in notifier_list:
|
||||
n.close()
|
||||
|
||||
|
||||
def _resolve_progress(
|
||||
progress: bool | ProgressCallback,
|
||||
verbose: bool,
|
||||
on_event: EventCallback | None,
|
||||
notifiers: Notifier | list[Notifier] | None,
|
||||
) -> tuple[ProgressCallback | None, list[Notifier], EventCallback | None]:
|
||||
"""解析 ``progress``/``notifiers`` 参数,返回 (monitor, notifier_list, effective_callback)。
|
||||
|
||||
将 progress/verbose/on_event/notifiers 的回调组合逻辑从 :func:`run` 中提取出来,
|
||||
控制 ``run`` 的分支数在 ruff PLR0912 阈值内。
|
||||
"""
|
||||
monitor: ProgressCallback | None = None
|
||||
if progress is True:
|
||||
monitor = RichProgressMonitor()
|
||||
elif progress is not False:
|
||||
monitor = progress
|
||||
|
||||
notifier_list: list[Notifier] = []
|
||||
if notifiers is not None:
|
||||
notifier_list = notifiers if isinstance(notifiers, list) else [notifiers]
|
||||
|
||||
callbacks: list[EventCallback] = []
|
||||
if verbose:
|
||||
callbacks.append(_make_verbose_callback(None))
|
||||
if monitor is not None:
|
||||
callbacks.append(monitor.on_event)
|
||||
if on_event is not None:
|
||||
callbacks.append(on_event)
|
||||
for n in notifier_list:
|
||||
callbacks.append(n.notify)
|
||||
effective_callback = _compose_callbacks(*callbacks) if callbacks else None
|
||||
return monitor, notifier_list, effective_callback
|
||||
|
||||
|
||||
def _apply_subgraph_filter(
|
||||
graph: Graph,
|
||||
only: Iterable[str] | None,
|
||||
tags: Iterable[str] | None,
|
||||
) -> Graph:
|
||||
"""根据 ``only``/``tags`` 过滤图,返回包含传递依赖的子图。
|
||||
|
||||
``only`` 与 ``tags`` 取并集:匹配任一条件的任务及其所有传递依赖
|
||||
(硬依赖 + 软依赖)都会被包含在子图中,使子图可独立执行。
|
||||
"""
|
||||
names: set[str] = set()
|
||||
if only is not None:
|
||||
names.update(only)
|
||||
if tags is not None:
|
||||
tag_set = set(tags)
|
||||
for name, spec in graph.all_specs().items():
|
||||
if tag_set & set(spec.tags):
|
||||
names.add(name)
|
||||
if not names:
|
||||
return Graph(defaults=graph.defaults)
|
||||
return graph.subgraph_with_deps(names)
|
||||
|
||||
|
||||
def _dispatch_strategy(
|
||||
strategy: Strategy,
|
||||
graph: Graph,
|
||||
context: dict[str, Any],
|
||||
report: RunReport,
|
||||
backend: StateBackend,
|
||||
effective_callback: EventCallback | None,
|
||||
max_workers: int | None,
|
||||
limits: Mapping[str, int],
|
||||
cancel_event: threading.Event | CancelToken | None,
|
||||
) -> None:
|
||||
"""按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。
|
||||
|
||||
``dynamic=True`` 任务仅在 ``dependency`` 策略下支持(需运行时扩展图)。
|
||||
"""
|
||||
has_dynamic = any(s.dynamic for s in graph.all_specs().values())
|
||||
if has_dynamic and strategy != "dependency":
|
||||
raise ValueError(
|
||||
f"dynamic=True 任务仅支持 strategy='dependency',当前策略: {strategy!r}。"
|
||||
"动态任务生成需要运行时扩展图,层屏障模型(sequential/thread/async)不支持。"
|
||||
)
|
||||
if strategy == "sequential":
|
||||
layers = graph.layers()
|
||||
_drive_sequential(graph, layers, context, report, backend, effective_callback, cancel_event)
|
||||
elif strategy == "thread":
|
||||
layers = graph.layers()
|
||||
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits, cancel_event)
|
||||
elif strategy == "async":
|
||||
layers = graph.layers()
|
||||
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits, cancel_event))
|
||||
elif strategy == "dependency":
|
||||
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits, cancel_event))
|
||||
else: # pragma: no cover - Strategy Literal 已穷尽所有取值
|
||||
raise ValueError(f"Unknown strategy: {strategy!r}")
|
||||
|
||||
|
||||
def _make_verbose_callback(on_event: EventCallback | None) -> EventCallback:
|
||||
"""包装 on_event 回调, 在 verbose 模式下用 rich 打印任务生命周期。"""
|
||||
"""包装 on_event 回调, 在 verbose 模式下打印任务生命周期。"""
|
||||
|
||||
def _verbose_callback(event: TaskEvent) -> None:
|
||||
dur = f" ({event.duration:.3f}s)" if event.duration is not None else ""
|
||||
if event.status == TaskStatus.RUNNING:
|
||||
_verbose_console.print(f"[cyan]▸[/cyan] [bold]{event.task!r}[/bold] 开始执行...")
|
||||
print(f"[verbose] 任务 {event.task!r} 开始执行...", flush=True)
|
||||
elif event.status == TaskStatus.SUCCESS:
|
||||
_verbose_console.print(f"[green]✓[/green] [bold]{event.task!r}[/bold] 成功[dim]{dur}[/dim]")
|
||||
print(f"[verbose] 任务 {event.task!r} 成功{dur}", flush=True)
|
||||
elif event.status == TaskStatus.FAILED:
|
||||
err = f": {event.error}" if event.error else ""
|
||||
_verbose_console.print(
|
||||
f"[red]✗[/red] [bold]{event.task!r}[/bold] 失败[dim]{dur} (尝试 {event.attempts} 次)[/dim][red]{err}[/red]"
|
||||
print(
|
||||
f"[verbose] 任务 {event.task!r} 失败{dur} (尝试 {event.attempts} 次){err}",
|
||||
flush=True,
|
||||
)
|
||||
elif event.status == TaskStatus.SKIPPED:
|
||||
reason = f" ({event.reason})" if event.reason else ""
|
||||
_verbose_console.print(f"[yellow]○[/yellow] [bold]{event.task!r}[/bold] 跳过[dim]{reason}[/dim]")
|
||||
print(f"[verbose] 任务 {event.task!r} 跳过{reason}", flush=True)
|
||||
if on_event is not None:
|
||||
on_event(event)
|
||||
|
||||
return _verbose_callback
|
||||
|
||||
|
||||
def _load_resume_report(source: RunReport | str | Path) -> RunReport:
|
||||
"""从 RunReport 或 JSON 文件路径加载前次运行报告。"""
|
||||
if isinstance(source, RunReport):
|
||||
return source
|
||||
return RunReport.from_json(Path(source).read_text())
|
||||
|
||||
|
||||
def _apply_resume(
|
||||
resume_from: RunReport | str | Path | None,
|
||||
graph: Graph,
|
||||
report: RunReport,
|
||||
context: dict[str, Any],
|
||||
) -> None:
|
||||
"""从前次报告中预填充 SUCCESS 任务的 result 和 context。
|
||||
|
||||
执行器在遇到已存在 result 时自动跳过(_filter_and_sort / _run_task)。
|
||||
"""
|
||||
if resume_from is None:
|
||||
return
|
||||
prev_report = _load_resume_report(resume_from)
|
||||
all_specs = graph.all_specs()
|
||||
restored = 0
|
||||
for name, result in prev_report.results.items():
|
||||
if result.status == TaskStatus.SUCCESS and name in all_specs:
|
||||
report.results[name] = dc_replace(result, reason="从检查点恢复")
|
||||
context[name] = result.value
|
||||
restored += 1
|
||||
if restored:
|
||||
logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored})
|
||||
|
||||
|
||||
def run(
|
||||
graph: Graph,
|
||||
strategy: Strategy = "dependency",
|
||||
@@ -1036,13 +756,6 @@ def run(
|
||||
on_event: EventCallback | None = None,
|
||||
state: StateBackend | None = None,
|
||||
concurrency_limits: Mapping[str, int] | None = None,
|
||||
progress: bool | ProgressCallback = False,
|
||||
notifiers: Notifier | list[Notifier] | None = None,
|
||||
cancel_event: threading.Event | CancelToken | None = None,
|
||||
only: Iterable[str] | None = None,
|
||||
tags: Iterable[str] | None = None,
|
||||
resume_from: RunReport | str | Path | None = None,
|
||||
_report: RunReport | None = None,
|
||||
) -> RunReport:
|
||||
"""执行图并返回 :class:`RunReport`。
|
||||
|
||||
@@ -1066,28 +779,6 @@ def run(
|
||||
concurrency_limits:
|
||||
``{concurrency_key: max_concurrent}`` 映射。具有相同
|
||||
``concurrency_key`` 的任务共享信号量,限制同时运行实例数。
|
||||
progress:
|
||||
进度监控:``False``(默认)无监控;``True`` 使用
|
||||
:class:`~pyflowx.progress.RichProgressMonitor` 显示终端进度条;
|
||||
传入 :class:`~pyflowx.progress.ProgressCallback` 实例使用自定义监控器。
|
||||
与 ``on_event`` 独立且可共存。
|
||||
notifiers:
|
||||
任务通知器:``None``(默认)无通知;传入 :class:`~pyflowx.notification.Notifier`
|
||||
实例或列表。通知器的 ``notify`` 方法在每次任务事件时调用,``close`` 在
|
||||
执行结束时调用。支持飞书/钉钉/企业微信 webhook 与 Python 回调。
|
||||
cancel_event:
|
||||
取消信号:传入 :class:`~pyflowx.cancellation.CancelToken` 或
|
||||
:class:`threading.Event`,设置后待运行任务被标记为 SKIPPED,
|
||||
运行中任务等待完成后返回部分结果。``KeyboardInterrupt`` 也会
|
||||
触发相同的优雅取消流程。
|
||||
only:
|
||||
只运行指定任务名及其传递依赖。与 ``tags`` 取并集。
|
||||
tags:
|
||||
只运行匹配任意标签的任务及其传递依赖。与 ``only`` 取并集。
|
||||
resume_from:
|
||||
从之前的运行报告恢复:传入 :class:`RunReport` 或 JSON 文件路径。
|
||||
前次运行中 SUCCESS 的任务会被跳过(标记 reason="从检查点恢复"),
|
||||
FAILED/SKIPPED 的任务及其下游会被重新执行。
|
||||
|
||||
抛出
|
||||
----
|
||||
@@ -1101,190 +792,52 @@ def run(
|
||||
_print_dry_run(graph, layers)
|
||||
return RunReport(success=True)
|
||||
|
||||
# 子图过滤:only/tags 选择任务子集及其传递依赖
|
||||
if only is not None or tags is not None:
|
||||
graph = _apply_subgraph_filter(graph, only, tags)
|
||||
|
||||
# verbose 模式下, 把所有 spec 的 verbose 标记设为 True,
|
||||
# 使 execute_command 打印执行命令与返回码 (任务生命周期由 callback 打印)
|
||||
if verbose:
|
||||
from dataclasses import replace
|
||||
|
||||
graph = Graph.from_specs([replace(s, verbose=True) if not s.verbose else s for s in graph.all_specs().values()])
|
||||
|
||||
# 入口统一校验一次:所有策略共用,避免 layers() / dependency 路径
|
||||
# 各自重复调用 validate()。
|
||||
graph.validate()
|
||||
|
||||
# 组合回调链:verbose 打印 + 进度监控 + 用户回调 + 通知器
|
||||
monitor, notifier_list, effective_callback = _resolve_progress(progress, verbose, on_event, notifiers)
|
||||
|
||||
effective_callback: EventCallback | None = _make_verbose_callback(on_event) if verbose else on_event
|
||||
backend = resolve_backend(state)
|
||||
report = _report if _report is not None else RunReport()
|
||||
report = RunReport()
|
||||
context: dict[str, Any] = {}
|
||||
limits = concurrency_limits or {}
|
||||
|
||||
# 检查点恢复:预填充 SUCCESS 任务的 result 和 context,执行器自动跳过。
|
||||
_apply_resume(resume_from, graph, report, context)
|
||||
|
||||
logger.info(
|
||||
"运行开始: run_id=%s strategy=%s tasks=%d",
|
||||
report.run_id,
|
||||
strategy,
|
||||
len(graph),
|
||||
extra={"run_id": report.run_id, "strategy": strategy, "total_tasks": len(graph)},
|
||||
)
|
||||
|
||||
if monitor is not None:
|
||||
monitor.start(len(graph))
|
||||
|
||||
# 内部取消令牌:KeyboardInterrupt 时设置,驱动各策略停止启动新任务。
|
||||
internal_cancel = CancelToken()
|
||||
effective_cancel: CancelToken | threading.Event | None = cancel_event
|
||||
|
||||
# backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend);
|
||||
# MemoryBackend 为 no-op。即使中途抛出 TaskFailedError,batch 退出时
|
||||
# 仍会 flush 一次,保留已成功任务的结果以便断点续跑。
|
||||
try:
|
||||
with backend.batch():
|
||||
try:
|
||||
_dispatch_strategy(
|
||||
strategy, graph, context, report, backend, effective_callback, max_workers, limits, effective_cancel
|
||||
)
|
||||
except TaskFailedError:
|
||||
report.success = False
|
||||
raise
|
||||
except KeyboardInterrupt:
|
||||
# Ctrl+C:设置内部取消信号,驱动各策略停止启动新任务。
|
||||
internal_cancel.cancel("KeyboardInterrupt")
|
||||
report.success = False
|
||||
logger.warning(
|
||||
"执行被用户中断(KeyboardInterrupt),已取消待运行任务",
|
||||
extra={"run_id": report.run_id, "status": "cancelled"},
|
||||
)
|
||||
finally:
|
||||
# 关闭进程池:通知管理线程退出 + kill 工作进程,避免
|
||||
# threading._shutdown 等待管理线程 join 工作进程导致 ~7s 阻塞。
|
||||
_shutdown_process_pool()
|
||||
finally:
|
||||
_finish_monitoring(monitor, notifier_list)
|
||||
|
||||
# 取消(外部或 KeyboardInterrupt)后标记未完成任务为 SKIPPED 并修正报告
|
||||
if _is_cancelled(effective_cancel) or internal_cancel.is_cancelled:
|
||||
_mark_remaining_skipped(graph, report, effective_callback)
|
||||
report.success = False
|
||||
|
||||
logger.info(
|
||||
"运行结束: run_id=%s success=%s tasks=%d",
|
||||
report.run_id,
|
||||
report.success,
|
||||
len(report.results),
|
||||
extra={
|
||||
"run_id": report.run_id,
|
||||
"success": report.success,
|
||||
"total_tasks": len(report.results),
|
||||
},
|
||||
)
|
||||
return report
|
||||
|
||||
|
||||
def run_iter(
|
||||
graph: Graph,
|
||||
strategy: Strategy = "dependency",
|
||||
*,
|
||||
max_workers: int | None = None,
|
||||
verbose: bool = False,
|
||||
state: StateBackend | None = None,
|
||||
concurrency_limits: Mapping[str, int] | None = None,
|
||||
) -> Iterator[tuple[str, TaskResult[Any]]]:
|
||||
"""流式执行图,逐个 yield 完成任务的 ``(name, TaskResult)``。
|
||||
|
||||
与 :func:`run` 相同的执行语义,但以生成器形式逐个返回结果而非等待
|
||||
全部完成后返回 :class:`RunReport`。适用于:
|
||||
|
||||
* 大量任务需要逐个处理结果(如写入数据库、发送通知)
|
||||
* 内存敏感场景(不需要同时持有所有结果)
|
||||
* 需要在任务完成时即时响应
|
||||
|
||||
内部通过后台线程执行 :func:`run` + 队列传递结果实现,不改变 ``run``
|
||||
的执行逻辑。生成器耗尽后若执行过程抛异常则 re-raise。
|
||||
|
||||
参数与 :func:`run` 一致(不含 ``dry_run``/``on_event``/``progress``/
|
||||
``notifiers``/``_report``,这些由流式机制内部管理)。
|
||||
|
||||
Yields
|
||||
------
|
||||
tuple[str, TaskResult]
|
||||
``(task_name, result)``,按任务完成顺序 yield。终态事件
|
||||
(SUCCESS/FAILED/SKIPPED)均会 yield。
|
||||
|
||||
Raises
|
||||
------
|
||||
ValueError
|
||||
``strategy`` 不被识别时。
|
||||
TaskFailedError
|
||||
任何任务耗尽重试后仍失败时(在 yield 完已失败任务后 re-raise)。
|
||||
|
||||
Example
|
||||
-------
|
||||
for name, result in px.run_iter(graph):
|
||||
print(f"{name}: {result.status.value}")
|
||||
"""
|
||||
report = RunReport()
|
||||
ready: queue.Queue[str | None] = queue.Queue()
|
||||
error_box: list[BaseException | None] = [None]
|
||||
yielded: set[str] = set()
|
||||
|
||||
_terminal = {TaskStatus.SUCCESS, TaskStatus.FAILED, TaskStatus.SKIPPED}
|
||||
|
||||
def _on_event(event: TaskEvent) -> None:
|
||||
if event.status in _terminal:
|
||||
ready.put(event.task)
|
||||
|
||||
def _worker() -> None:
|
||||
with backend.batch():
|
||||
try:
|
||||
run(
|
||||
graph,
|
||||
strategy=strategy,
|
||||
max_workers=max_workers,
|
||||
verbose=verbose,
|
||||
on_event=_on_event,
|
||||
state=state,
|
||||
concurrency_limits=concurrency_limits,
|
||||
_report=report,
|
||||
)
|
||||
except BaseException as exc:
|
||||
error_box[0] = exc
|
||||
if strategy == "sequential":
|
||||
layers = graph.layers()
|
||||
_drive_sequential(graph, layers, context, report, backend, effective_callback)
|
||||
elif strategy == "thread":
|
||||
layers = graph.layers()
|
||||
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits)
|
||||
elif strategy == "async":
|
||||
layers = graph.layers()
|
||||
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits))
|
||||
elif strategy == "dependency":
|
||||
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits))
|
||||
else:
|
||||
raise ValueError(f"Unknown strategy: {strategy!r}")
|
||||
except TaskFailedError:
|
||||
report.success = False
|
||||
raise
|
||||
finally:
|
||||
ready.put(None)
|
||||
# 关闭进程池:通知管理线程退出 + kill 工作进程,避免
|
||||
# threading._shutdown 等待管理线程 join 工作进程导致 ~7s 阻塞。
|
||||
_shutdown_process_pool()
|
||||
|
||||
thread = threading.Thread(target=_worker, daemon=True)
|
||||
thread.start()
|
||||
|
||||
try:
|
||||
while True:
|
||||
name = ready.get()
|
||||
if name is None:
|
||||
break
|
||||
if name in yielded:
|
||||
continue
|
||||
yielded.add(name)
|
||||
if name in report.results:
|
||||
yield name, report.results[name]
|
||||
finally:
|
||||
thread.join(timeout=0)
|
||||
|
||||
if error_box[0] is not None:
|
||||
raise error_box[0]
|
||||
return report
|
||||
|
||||
|
||||
def _print_dry_run(graph: Graph, layers: list[list[str]]) -> None:
|
||||
"""打印执行计划但不运行任何任务。"""
|
||||
_verbose_console.print(f"[bold]Dry run:[/bold] [cyan]{len(graph)}[/cyan] tasks, [cyan]{len(layers)}[/cyan] layers")
|
||||
print(f"Dry run: {len(graph)} tasks, {len(layers)} layers")
|
||||
for idx, layer in enumerate(layers, 1):
|
||||
_verbose_console.print(f" [dim]Layer {idx}:[/dim] {layer}")
|
||||
print(f" Layer {idx}: {layer}")
|
||||
for name in layer:
|
||||
_verbose_console.print(f" [cyan]-[/cyan] {describe_injection(graph.resolved_spec(name))}")
|
||||
print(f" - {describe_injection(graph.resolved_spec(name))}")
|
||||
|
||||
|
||||
def _drive_sequential(
|
||||
@@ -1294,11 +847,8 @@ def _drive_sequential(
|
||||
report: RunReport,
|
||||
backend: StateBackend,
|
||||
on_event: EventCallback | None,
|
||||
cancel_event: threading.Event | CancelToken | None = None,
|
||||
) -> None:
|
||||
for idx, layer in enumerate(layers, 1):
|
||||
if _is_cancelled(cancel_event):
|
||||
return
|
||||
SequentialLayerRunner.execute(layer, graph, context, report, backend, idx, on_event)
|
||||
|
||||
|
||||
@@ -1311,16 +861,10 @@ def _drive_threaded(
|
||||
on_event: EventCallback | None,
|
||||
max_workers: int | None,
|
||||
concurrency_limits: Mapping[str, int],
|
||||
cancel_event: threading.Event | CancelToken | None = None,
|
||||
) -> None:
|
||||
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
|
||||
max_layer_size = max((len(layer) for layer in layers), default=1)
|
||||
pool_workers = max_workers or max(1, min(32, max_layer_size))
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
|
||||
for idx, layer in enumerate(layers, 1):
|
||||
if _is_cancelled(cancel_event):
|
||||
return
|
||||
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, pool, concurrency_limits)
|
||||
for idx, layer in enumerate(layers, 1):
|
||||
workers = max_workers or max(1, min(32, len(layer)))
|
||||
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, workers, concurrency_limits)
|
||||
|
||||
|
||||
async def _async_drive(
|
||||
@@ -1331,32 +875,6 @@ async def _async_drive(
|
||||
backend: StateBackend,
|
||||
on_event: EventCallback | None,
|
||||
concurrency_limits: Mapping[str, int],
|
||||
cancel_event: threading.Event | CancelToken | None = None,
|
||||
) -> None:
|
||||
for idx, layer in enumerate(layers, 1):
|
||||
if _is_cancelled(cancel_event):
|
||||
return
|
||||
await AsyncLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, concurrency_limits)
|
||||
|
||||
|
||||
def _mark_remaining_skipped(
|
||||
graph: Graph,
|
||||
report: RunReport,
|
||||
on_event: EventCallback | None,
|
||||
) -> None:
|
||||
"""将未执行的任务标记为 SKIPPED(取消场景)。
|
||||
|
||||
遍历图中所有任务,将尚未出现在 ``report.results`` 中的任务标记为
|
||||
SKIPPED,使最终报告包含完整的任务状态映射。
|
||||
"""
|
||||
for name, spec in graph.all_specs().items():
|
||||
if name in report.results:
|
||||
continue
|
||||
result: TaskResult[Any] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus.SKIPPED,
|
||||
finished_at=datetime.now(),
|
||||
reason="执行被取消",
|
||||
)
|
||||
report.results[name] = result
|
||||
_emit(on_event, result)
|
||||
|
||||
+30
-261
@@ -1,7 +1,7 @@
|
||||
"""DAG 构建、校验、分层与可视化。
|
||||
|
||||
使用自实现的 Kahn 算法进行拓扑排序。图以增量方式构建并即时校验,
|
||||
使配置错误在构建时(而非执行时)快速失败。
|
||||
使用标准库的 :mod:`graphlib`(3.9+)或 :mod:`graphlib_backport`(3.8)
|
||||
进行拓扑排序。图以增量方式构建并即时校验,使配置错误在构建时(而非执行时)快速失败。
|
||||
|
||||
支持:
|
||||
* 图级默认值 :class:`GraphDefaults`,TaskSpec 字段为 ``None`` 时回退。
|
||||
@@ -18,60 +18,24 @@ __all__ = [
|
||||
]
|
||||
|
||||
import inspect
|
||||
from collections.abc import Callable, Iterable, Mapping, Sequence
|
||||
import sys
|
||||
from dataclasses import dataclass, field, replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, Callable, Iterable, Mapping, Sequence
|
||||
|
||||
from .errors import CycleError, DuplicateTaskError, MissingDependencyError
|
||||
from .task import Context, RetryPolicy, TaskSpec
|
||||
|
||||
if sys.version_info >= (3, 9): # pragma: no cover
|
||||
import graphlib # pyright: ignore[reportUnreachable]
|
||||
|
||||
def _topological_layers(deps: Mapping[str, tuple[str, ...]]) -> tuple[list[list[str]], list[str] | None]:
|
||||
"""Kahn 算法分层拓扑排序。
|
||||
_TopologicalSorter = graphlib.TopologicalSorter
|
||||
else: # pragma: no cover
|
||||
import graphlib # type: ignore[import-untyped]
|
||||
|
||||
返回 ``(layers, cycle_nodes)``:无环时 ``cycle_nodes`` 为 ``None``;
|
||||
有环时为参与环的未处理节点列表(非精确环路径,仅指示存在环)。
|
||||
|
||||
直接用 dict + list 计算入度与反向邻接,在 diamond(1000) 图上约
|
||||
0.45 万 ops/s(含 Graph 构造开销)。
|
||||
"""
|
||||
# 入度(依赖数)与反向邻接表
|
||||
in_degree: dict[str, int] = {}
|
||||
dependents: dict[str, list[str]] = {}
|
||||
for name, d in deps.items():
|
||||
in_degree[name] = len(d)
|
||||
dependents.setdefault(name, [])
|
||||
for name, d in deps.items():
|
||||
for dep in d:
|
||||
if dep in dependents:
|
||||
dependents[dep].append(name)
|
||||
|
||||
# 初始层:入度为 0 的节点
|
||||
current = sorted(name for name, deg in in_degree.items() if deg == 0)
|
||||
layers: list[list[str]] = []
|
||||
processed = 0
|
||||
|
||||
while current:
|
||||
layers.append(current)
|
||||
nxt: list[str] = []
|
||||
for node in current:
|
||||
for dependent in dependents[node]:
|
||||
in_degree[dependent] -= 1
|
||||
if in_degree[dependent] == 0:
|
||||
nxt.append(dependent)
|
||||
processed += 1
|
||||
nxt.sort()
|
||||
current = nxt
|
||||
|
||||
if processed < len(deps):
|
||||
cycle_nodes = [name for name, deg in in_degree.items() if deg > 0]
|
||||
return layers, cycle_nodes
|
||||
|
||||
return layers, None
|
||||
_TopologicalSorter = graphlib.TopologicalSorter # pragma: no cover
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@dataclass
|
||||
class GraphDefaults:
|
||||
"""图级默认值。TaskSpec 对应字段为 ``None`` 时回退到此处。
|
||||
|
||||
@@ -168,7 +132,7 @@ def _make_namespaced_fn(orig_fn: Any, ns: str, dep_names: set[str]) -> Any:
|
||||
return wrapper
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@dataclass
|
||||
class Graph:
|
||||
"""校验后的有向无环任务图。
|
||||
|
||||
@@ -192,18 +156,6 @@ class Graph:
|
||||
# 在 specs / defaults 变更时失效。
|
||||
_resolved_cache: dict[str, TaskSpec[Any]] = field(default_factory=dict)
|
||||
|
||||
# layers() 缓存:避免重复 run() 调用时重算拓扑排序。
|
||||
# 在 specs 变更时失效。
|
||||
_layers_cache: list[list[str]] | None = field(default=None)
|
||||
|
||||
# loop 组映射:原 spec 名 → 展开后的实例名列表。
|
||||
# 用于依赖重写:其他任务引用 loop 原名时,自动展开为依赖所有实例。
|
||||
_loop_groups: dict[str, list[str]] = field(default_factory=dict)
|
||||
|
||||
# 任务分组映射:组名 → 组内任务名元组。
|
||||
# 用于依赖重写:引用组名等价于依赖组内所有任务。
|
||||
_groups: dict[str, tuple[str, ...]] = field(default_factory=dict)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 构建
|
||||
# ------------------------------------------------------------------ #
|
||||
@@ -235,123 +187,13 @@ class Graph:
|
||||
prev_name = current.name
|
||||
return self
|
||||
|
||||
def pipeline(self, *specs: TaskSpec[Any]) -> Graph:
|
||||
"""创建数据流水线:每个任务的结果通过上下文注入传给下一个。
|
||||
|
||||
与 :meth:`chain` 行为一致(依赖关系相同),但语义上强调数据流:
|
||||
前驱任务的返回值会自动注入后继任务的同名参数。
|
||||
|
||||
后继任务的 ``fn`` 参数名需匹配前驱任务名才能接收数据。
|
||||
``outputs`` 字段声明的命名输出可通过
|
||||
:meth:`RunReport.output_of` 查询。
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> graph = px.Graph().pipeline(extract_spec, transform_spec, load_spec)
|
||||
>>> report = px.run(graph)
|
||||
>>> report.output_of("extract", "items")
|
||||
"""
|
||||
return self.chain(*specs)
|
||||
|
||||
def group(self, name: str, tasks: Sequence[str]) -> Graph:
|
||||
"""声明任务分组,引用组名等价于依赖组内所有任务。
|
||||
|
||||
组名必须不与任何已注册任务名冲突。组内任务必须已注册(或在本批
|
||||
``from_specs`` 收集阶段先于引用方注册)。声明后,后续 ``add`` 或
|
||||
``from_specs`` 中 ``depends_on``/``soft_depends_on`` 引用组名时,
|
||||
会被重写为依赖组内全部任务。
|
||||
|
||||
参数
|
||||
----
|
||||
name:
|
||||
组名。不能与 ``self.specs`` 中任务名重复,也不能与已有组名重复。
|
||||
tasks:
|
||||
组内任务名序列。所有任务必须已在图中注册。
|
||||
|
||||
返回 ``self`` 支持链式调用。
|
||||
|
||||
Raises
|
||||
------
|
||||
ValueError
|
||||
组名与任务名冲突、组名已存在、或组内任务未注册时。
|
||||
"""
|
||||
if not name:
|
||||
raise ValueError("group.name 必须为非空字符串。")
|
||||
if name in self.specs:
|
||||
raise ValueError(f"组名 {name!r} 与已注册任务名冲突。")
|
||||
if name in self._groups:
|
||||
raise ValueError(f"组名 {name!r} 已存在。")
|
||||
missing = [t for t in tasks if t not in self.specs]
|
||||
if missing:
|
||||
raise ValueError(f"组 {name!r} 引用了未注册的任务: {missing}")
|
||||
if not tasks:
|
||||
raise ValueError(f"组 {name!r} 不能为空。")
|
||||
self._groups[name] = tuple(tasks)
|
||||
return self
|
||||
|
||||
def _register(self, spec: TaskSpec[Any]) -> None:
|
||||
"""注册单个 spec。若 ``spec.loop`` 非 ``None``,展开为多实例。"""
|
||||
if spec.loop is not None:
|
||||
for expanded in self._expand_loop_to_specs(spec):
|
||||
self._register_single(expanded)
|
||||
return
|
||||
self._register_single(self._rewrite_deps(spec))
|
||||
|
||||
def _register_single(self, spec: TaskSpec[Any]) -> None:
|
||||
"""注册单个已展开的 spec(无 loop)。"""
|
||||
if spec.name in self.specs:
|
||||
raise DuplicateTaskError(spec.name)
|
||||
self.specs[spec.name] = spec
|
||||
# 拓扑依赖仅含硬依赖;软依赖仅用于注入,不影响分层。
|
||||
self.deps[spec.name] = spec.depends_on
|
||||
self._resolved_cache.clear()
|
||||
self._layers_cache = None
|
||||
|
||||
def _expand_loop_to_specs(self, spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
|
||||
"""展开 ``spec.loop`` 为多实例列表,并记录到 ``_loop_groups``。
|
||||
|
||||
返回展开后的 spec 列表(``loop`` 字段已清空,``args`` 已追加 item)。
|
||||
不写入 ``specs``/``deps``,由调用方注册。
|
||||
"""
|
||||
loop = spec.loop
|
||||
assert loop is not None # 由 _register 保证
|
||||
expanded: list[TaskSpec[Any]] = []
|
||||
expanded_names: list[str] = []
|
||||
for i, item in enumerate(loop.items):
|
||||
key = loop.key_fn(i, item) if loop.key_fn is not None else f"{spec.name}_{i}"
|
||||
new_spec = replace(spec, name=key, args=(*spec.args, item), loop=None)
|
||||
new_spec = self._rewrite_deps(new_spec)
|
||||
expanded.append(new_spec)
|
||||
expanded_names.append(key)
|
||||
self._loop_groups[spec.name] = expanded_names
|
||||
return expanded
|
||||
|
||||
def _rewrite_deps(self, spec: TaskSpec[Any]) -> TaskSpec[Any]:
|
||||
"""重写 spec 的依赖:引用 loop 组名或 group 组名时展开为所有实例。"""
|
||||
if not self._loop_groups and not self._groups:
|
||||
return spec
|
||||
# 合并 loop 组与普通组:组名 → 展开后的任务名列表
|
||||
combined: dict[str, Sequence[str]] = {}
|
||||
combined.update(self._loop_groups)
|
||||
combined.update(self._groups)
|
||||
new_hard: list[str] = []
|
||||
new_soft: list[str] = []
|
||||
changed = False
|
||||
for dep in spec.depends_on:
|
||||
if dep in combined:
|
||||
new_hard.extend(combined[dep])
|
||||
changed = True
|
||||
else:
|
||||
new_hard.append(dep)
|
||||
for dep in spec.soft_depends_on:
|
||||
if dep in combined:
|
||||
new_soft.extend(combined[dep])
|
||||
changed = True
|
||||
else:
|
||||
new_soft.append(dep)
|
||||
if not changed:
|
||||
return spec
|
||||
return replace(spec, depends_on=tuple(new_hard), soft_depends_on=tuple(new_soft))
|
||||
|
||||
@classmethod
|
||||
def from_specs(
|
||||
@@ -377,31 +219,15 @@ class Graph:
|
||||
"""
|
||||
graph = cls(defaults=defaults or GraphDefaults(), namespace=namespace)
|
||||
pending_refs: list[str] = []
|
||||
collected: list[TaskSpec[Any]] = []
|
||||
|
||||
for spec in specs:
|
||||
if isinstance(spec, str):
|
||||
pending_refs.append(spec)
|
||||
elif isinstance(spec, TaskSpec):
|
||||
collected.append(spec)
|
||||
graph._register(spec)
|
||||
else:
|
||||
raise TypeError(f"from_specs 只接受 TaskSpec 或 str,收到: {type(spec)}")
|
||||
|
||||
# 两阶段:先展开所有 loop(记录 _loop_groups),再统一重写依赖并批量注册。
|
||||
# 批量注册跳过每次 _register 的 cache 清空(N 次清空降为 0 次);
|
||||
# cache 失效由后续 validate()/layers() 首次调用自然触发。
|
||||
expanded: list[TaskSpec[Any]] = []
|
||||
for spec in collected:
|
||||
if spec.loop is not None:
|
||||
expanded.extend(graph._expand_loop_to_specs(spec))
|
||||
else:
|
||||
expanded.append(graph._rewrite_deps(spec))
|
||||
|
||||
for spec in expanded:
|
||||
if spec.name in graph.specs:
|
||||
raise DuplicateTaskError(spec.name)
|
||||
graph.specs[spec.name] = spec
|
||||
graph.deps[spec.name] = spec.depends_on
|
||||
|
||||
if pending_refs:
|
||||
graph._pending_refs = pending_refs
|
||||
|
||||
@@ -409,30 +235,6 @@ class Graph:
|
||||
graph.validate()
|
||||
return graph
|
||||
|
||||
@classmethod
|
||||
def from_yaml(cls, path: str | Path) -> Graph:
|
||||
"""从 YAML 文件加载任务图(GitHub Actions 风格)。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
path:
|
||||
YAML 文件路径。
|
||||
|
||||
Returns
|
||||
-------
|
||||
Graph
|
||||
解析后的任务图,支持 ``jobs``/``needs``/``strategy.matrix``/
|
||||
``if``/``continue-on-error``/``env``/``defaults`` 等字段。
|
||||
|
||||
Raises
|
||||
------
|
||||
ValueError
|
||||
YAML 结构不符合 schema 时。
|
||||
"""
|
||||
from .yaml_loader import load_yaml
|
||||
|
||||
return load_yaml(path)
|
||||
|
||||
def add_subgraph(self, sub: Graph, *, namespace: str | None = None) -> Graph:
|
||||
"""将子图合并到当前图,任务名加命名空间前缀避免冲突。
|
||||
|
||||
@@ -487,16 +289,14 @@ class Graph:
|
||||
raise MissingDependencyError(name, dep)
|
||||
|
||||
def validate(self) -> None:
|
||||
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。
|
||||
|
||||
顺带填充 :attr:`_layers_cache`(无环时),使后续 :meth:`layers`
|
||||
直接命中缓存,避免 :func:`_topological_layers` 二次计算。
|
||||
"""
|
||||
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。"""
|
||||
self._validate_references()
|
||||
layers, cycle_nodes = _topological_layers(self.deps)
|
||||
if cycle_nodes is not None:
|
||||
raise CycleError(cycle_nodes)
|
||||
self._layers_cache = layers
|
||||
sorter = _TopologicalSorter(self.deps)
|
||||
try:
|
||||
sorter.prepare()
|
||||
except graphlib.CycleError as exc: # type: ignore[name-defined]
|
||||
cycle: Sequence[str] = exc.args[1] if len(exc.args) > 1 else []
|
||||
raise CycleError(list(cycle)) from exc
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 内省
|
||||
@@ -568,18 +368,19 @@ class Graph:
|
||||
同层任务无相互硬依赖,可并发执行。软依赖不参与分层。
|
||||
层按执行顺序返回。图有环时抛出 :class:`CycleError`。
|
||||
|
||||
结果按实例缓存;specs 变更时失效(:meth:`add` / :meth:`_register`)。
|
||||
|
||||
.. note::
|
||||
本方法假定图已通过 :meth:`validate` 校验(由 :func:`pyflowx.run`
|
||||
在入口统一执行一次)。若直接调用本方法,需自行先校验。
|
||||
"""
|
||||
if self._layers_cache is not None:
|
||||
return self._layers_cache
|
||||
result, cycle_nodes = _topological_layers(self.deps)
|
||||
if cycle_nodes is not None:
|
||||
raise CycleError(cycle_nodes)
|
||||
self._layers_cache = result
|
||||
sorter = _TopologicalSorter(self.deps)
|
||||
result: list[list[str]] = []
|
||||
sorter.prepare()
|
||||
while sorter.is_active():
|
||||
ready = list(sorter.get_ready())
|
||||
ready.sort()
|
||||
result.append(ready)
|
||||
for node in ready:
|
||||
sorter.done(node)
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
@@ -608,38 +409,6 @@ class Graph:
|
||||
]
|
||||
return Graph.from_specs(kept, defaults=self.defaults)
|
||||
|
||||
def subgraph_with_deps(self, names: Iterable[str]) -> Graph:
|
||||
"""返回包含 ``names`` 及其所有传递依赖的新图。
|
||||
|
||||
与 :meth:`subgraph_by_names` 不同,本方法会沿 ``depends_on`` 和
|
||||
``soft_depends_on`` 向上遍历,确保被选中的任务所需的上游全部包含在内,
|
||||
使子图可独立执行。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
names:
|
||||
需要执行的任务名。每个名称必须存在于当前图中。
|
||||
"""
|
||||
seeds: set[str] = set(names)
|
||||
for n in seeds:
|
||||
if n not in self.specs:
|
||||
raise KeyError(f"Unknown task name: {n!r}")
|
||||
# BFS 收集传递依赖(硬依赖 + 软依赖)
|
||||
closure: set[str] = set()
|
||||
queue: list[str] = list(seeds)
|
||||
while queue:
|
||||
name = queue.pop()
|
||||
if name in closure:
|
||||
continue
|
||||
closure.add(name)
|
||||
for dep in self.all_deps(name):
|
||||
if dep not in closure:
|
||||
queue.append(dep)
|
||||
kept: list[TaskSpec[Any]] = [
|
||||
_prune_deps(spec, lambda d: d in closure) for spec in self.specs.values() if spec.name in closure
|
||||
]
|
||||
return Graph.from_specs(kept, defaults=self.defaults)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Fan-out / map-reduce
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
"""运行历史管理:存储、查询与比较多次运行的 :class:`RunReport`。
|
||||
|
||||
适用于:
|
||||
* 断点续跑:从最近一次失败运行恢复,跳过已成功任务。
|
||||
* 趋势分析:对比多次运行的任务耗时、状态变化。
|
||||
* 审计追溯:按 run_id 查询历史报告。
|
||||
|
||||
用法
|
||||
----
|
||||
history = RunHistory(".pyflowx/runs")
|
||||
history.save(report)
|
||||
latest = history.latest()
|
||||
if latest is not None and not latest.success:
|
||||
# 从最近失败处恢复
|
||||
px.run(graph, resume_from=latest)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["RunHistory"]
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from .report import RunReport
|
||||
|
||||
|
||||
class RunHistory:
|
||||
"""运行历史记录管理器。
|
||||
|
||||
将 :class:`RunReport` 以 JSON 文件存储到指定目录,文件名为
|
||||
``{run_id}.json``。支持按 run_id 加载、列出全部、查询最近一次、删除。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
dir:
|
||||
存储目录。不存在时自动创建。
|
||||
"""
|
||||
|
||||
def __init__(self, dir: str | Path) -> None:
|
||||
self.dir = Path(dir)
|
||||
self.dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def save(self, report: RunReport) -> Path:
|
||||
"""保存运行报告,返回文件路径。"""
|
||||
path = self.dir / f"{report.run_id}.json"
|
||||
path.write_text(report.to_json(), encoding="utf-8")
|
||||
return path
|
||||
|
||||
def load(self, run_id: str) -> RunReport:
|
||||
"""按 run_id 加载报告。
|
||||
|
||||
Raises
|
||||
------
|
||||
FileNotFoundError
|
||||
指定 run_id 的文件不存在。
|
||||
"""
|
||||
path = self.dir / f"{run_id}.json"
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"运行报告 {run_id!r} 不存在: {path}")
|
||||
return RunReport.from_json(path.read_text(encoding="utf-8"))
|
||||
|
||||
def list_runs(self) -> list[str]:
|
||||
"""列出全部 run_id(按文件名排序)。"""
|
||||
return sorted(p.stem for p in self.dir.glob("*.json"))
|
||||
|
||||
def latest(self) -> RunReport | None:
|
||||
"""返回最近修改的报告;目录为空时返回 ``None``。"""
|
||||
runs = sorted(self.dir.glob("*.json"), key=lambda p: p.stat().st_mtime, reverse=True)
|
||||
if not runs:
|
||||
return None
|
||||
return RunReport.from_json(runs[0].read_text(encoding="utf-8"))
|
||||
|
||||
def delete(self, run_id: str) -> bool:
|
||||
"""删除指定 run_id 的报告。返回是否实际删除。"""
|
||||
path = self.dir / f"{run_id}.json"
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
return True
|
||||
return False
|
||||
|
||||
def __len__(self) -> int:
|
||||
return sum(1 for _ in self.dir.glob("*.json"))
|
||||
|
||||
def __contains__(self, run_id: object) -> bool:
|
||||
if not isinstance(run_id, str):
|
||||
return False
|
||||
return (self.dir / f"{run_id}.json").exists()
|
||||
@@ -1,217 +0,0 @@
|
||||
"""监控导出:Prometheus 指标收集与健康检查。
|
||||
|
||||
提供零依赖的轻量监控方案:
|
||||
|
||||
* :class:`MetricsCollector` —— 通过 ``on_event`` 回调收集任务指标,
|
||||
导出 Prometheus 文本格式。可配合 ``http.server`` 或 ``prometheus_client``
|
||||
暴露指标端点。
|
||||
* :func:`health_check` —— 分析 :class:`RunReport` 返回结构化健康状态。
|
||||
|
||||
用法
|
||||
----
|
||||
collector = px.MetricsCollector()
|
||||
report = px.run(graph, on_event=collector.on_event)
|
||||
print(collector.metrics_text())
|
||||
print(px.health_check(report))
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["MetricsCollector", "health_check", "start_metrics_server"]
|
||||
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
from .report import RunReport
|
||||
from .task import TaskEvent, TaskStatus
|
||||
|
||||
|
||||
class MetricsCollector:
|
||||
"""收集任务执行指标,导出 Prometheus 文本格式。
|
||||
|
||||
作为 :func:`run` 的 ``on_event`` 回调使用,收集每个任务的生命周期
|
||||
事件并聚合为 Prometheus 指标。线程安全(回调可能从多个线程调用)。
|
||||
|
||||
指标列表
|
||||
--------
|
||||
- ``pyflowx_task_total{task, status}`` — 任务计数器
|
||||
- ``pyflowx_task_duration_seconds{task}`` — 任务耗时(gauge,最近值)
|
||||
- ``pyflowx_task_duration_seconds_sum{task}`` — 任务累计耗时
|
||||
- ``pyflowx_task_retries_total{task}`` — 重试次数
|
||||
- ``pyflowx_run_total{status}`` — 运行计数器
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._task_count: dict[str, dict[str, int]] = {}
|
||||
self._task_duration_last: dict[str, float] = {}
|
||||
self._task_duration_sum: dict[str, float] = {}
|
||||
self._task_retries: dict[str, int] = {}
|
||||
self._lock_counts: dict[str, int] = {}
|
||||
|
||||
def on_event(self, event: TaskEvent) -> None:
|
||||
"""``on_event`` 回调:记录任务生命周期事件。"""
|
||||
task = event.task
|
||||
status = event.status.value
|
||||
|
||||
# 任务计数
|
||||
self._task_count.setdefault(task, {})
|
||||
self._task_count[task][status] = self._task_count[task].get(status, 0) + 1
|
||||
|
||||
# 耗时(仅在终态时记录)
|
||||
if event.duration is not None and event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED):
|
||||
self._task_duration_last[task] = event.duration
|
||||
self._task_duration_sum[task] = self._task_duration_sum.get(task, 0.0) + event.duration
|
||||
|
||||
# 重试次数
|
||||
if event.attempts > 1 and event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED):
|
||||
self._task_retries[task] = self._task_retries.get(task, 0) + event.attempts - 1
|
||||
|
||||
def record_run(self, report: RunReport) -> None:
|
||||
"""记录一次完整运行的结果。"""
|
||||
status = "success" if report.success else "failed"
|
||||
self._lock_counts[status] = self._lock_counts.get(status, 0) + 1
|
||||
|
||||
def metrics_text(self) -> str:
|
||||
"""导出 Prometheus 文本格式指标字符串。"""
|
||||
lines: list[str] = []
|
||||
|
||||
# pyflowx_task_total
|
||||
if self._task_count:
|
||||
lines.append("# HELP pyflowx_task_total Total tasks by status.")
|
||||
lines.append("# TYPE pyflowx_task_total counter")
|
||||
for task in sorted(self._task_count):
|
||||
for status in sorted(self._task_count[task]):
|
||||
count = self._task_count[task][status]
|
||||
lines.append(f'pyflowx_task_total{{task="{task}",status="{status}"}} {count}')
|
||||
|
||||
# pyflowx_task_duration_seconds (last)
|
||||
if self._task_duration_last:
|
||||
lines.append("# HELP pyflowx_task_duration_seconds Last task duration in seconds.")
|
||||
lines.append("# TYPE pyflowx_task_duration_seconds gauge")
|
||||
for task in sorted(self._task_duration_last):
|
||||
lines.append(f'pyflowx_task_duration_seconds{{task="{task}"}} {self._task_duration_last[task]}')
|
||||
|
||||
# pyflowx_task_duration_seconds_sum
|
||||
if self._task_duration_sum:
|
||||
lines.append("# HELP pyflowx_task_duration_seconds_sum Total task duration in seconds.")
|
||||
lines.append("# TYPE pyflowx_task_duration_seconds_sum counter")
|
||||
for task in sorted(self._task_duration_sum):
|
||||
lines.append(f'pyflowx_task_duration_seconds_sum{{task="{task}"}} {self._task_duration_sum[task]}')
|
||||
|
||||
# pyflowx_task_retries_total
|
||||
if self._task_retries:
|
||||
lines.append("# HELP pyflowx_task_retries_total Total task retries.")
|
||||
lines.append("# TYPE pyflowx_task_retries_total counter")
|
||||
for task in sorted(self._task_retries):
|
||||
lines.append(f'pyflowx_task_retries_total{{task="{task}"}} {self._task_retries[task]}')
|
||||
|
||||
# pyflowx_run_total
|
||||
if self._lock_counts:
|
||||
lines.append("# HELP pyflowx_run_total Total runs by status.")
|
||||
lines.append("# TYPE pyflowx_run_total counter")
|
||||
for status in sorted(self._lock_counts):
|
||||
lines.append(f'pyflowx_run_total{{status="{status}"}} {self._lock_counts[status]}')
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
def reset(self) -> None:
|
||||
"""清空所有收集的指标。"""
|
||||
self._task_count.clear()
|
||||
self._task_duration_last.clear()
|
||||
self._task_duration_sum.clear()
|
||||
self._task_retries.clear()
|
||||
self._lock_counts.clear()
|
||||
|
||||
|
||||
def health_check(report: RunReport) -> dict[str, Any]:
|
||||
"""分析运行报告,返回结构化健康状态。
|
||||
|
||||
状态判定:
|
||||
- ``healthy`` — 全部任务 SUCCESS
|
||||
- ``degraded`` — 部分(非全部)任务 FAILED
|
||||
- ``unhealthy`` — 全部任务 FAILED
|
||||
- ``unknown`` — 无任务
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict
|
||||
包含 ``status``/``total``/``success``/``failed``/``skipped``
|
||||
/``duration`` 字段。
|
||||
"""
|
||||
total = len(report.results)
|
||||
if total == 0:
|
||||
return {"status": "unknown", "total": 0, "message": "无任务结果"}
|
||||
|
||||
counts: dict[str, int] = {}
|
||||
total_duration = 0.0
|
||||
for r in report.results.values():
|
||||
status = r.status.value
|
||||
counts[status] = counts.get(status, 0) + 1
|
||||
if r.duration is not None:
|
||||
total_duration += r.duration
|
||||
|
||||
success = counts.get("success", 0)
|
||||
failed = counts.get("failed", 0)
|
||||
skipped = counts.get("skipped", 0)
|
||||
|
||||
if failed == 0:
|
||||
status = "healthy"
|
||||
elif failed < total:
|
||||
status = "degraded"
|
||||
else:
|
||||
status = "unhealthy"
|
||||
|
||||
return {
|
||||
"status": status,
|
||||
"total": total,
|
||||
"success": success,
|
||||
"failed": failed,
|
||||
"skipped": skipped,
|
||||
"duration": round(total_duration, 3),
|
||||
}
|
||||
|
||||
|
||||
def start_metrics_server(collector: MetricsCollector, port: int = 9100, host: str = "0.0.0.0") -> Callable[[], None]:
|
||||
"""启动简单 HTTP 服务器暴露 Prometheus 指标。
|
||||
|
||||
返回停止函数,调用后关闭服务器。阻塞调用方线程,建议在后台线程运行。
|
||||
|
||||
.. note::
|
||||
使用标准库 ``http.server``,零依赖。生产环境建议配合
|
||||
``prometheus_client`` 或反向代理使用。
|
||||
"""
|
||||
import sys
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
if sys.version_info >= (3, 12):
|
||||
from typing import override
|
||||
else:
|
||||
from typing_extensions import override # pragma: no cover
|
||||
|
||||
class _Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self) -> None:
|
||||
if self.path == "/metrics":
|
||||
body = collector.metrics_text().encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
else:
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
|
||||
@override
|
||||
def log_message(self, format: str, *args: object) -> None:
|
||||
pass # 静默访问日志
|
||||
|
||||
server = HTTPServer((host, port), _Handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
|
||||
def stop() -> None:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
return stop
|
||||
@@ -1,320 +0,0 @@
|
||||
"""任务通知系统。
|
||||
|
||||
在任务生命周期事件(RUNNING/SUCCESS/FAILED/SKIPPED)发生时发送通知到
|
||||
外部系统。支持两类 sink:
|
||||
|
||||
* :class:`CallbackNotifier` —— 包装 Python 回调,零外部依赖
|
||||
* :class:`WebhookNotifier` 及其子类 —— 通过 HTTP POST 发送到飞书/钉钉/企业微信
|
||||
|
||||
通知系统与 :class:`pyflowx.progress.ProgressCallback` 独立且可共存。
|
||||
``notifiers`` 参数在 :func:`pyflowx.run` 中传入,通知器的 ``notify`` 方法
|
||||
被加入回调链,``close`` 在执行结束时调用。
|
||||
|
||||
设计要点
|
||||
--------
|
||||
* 事件过滤在 Notifier 内部,通过 ``levels`` 参数控制哪些事件触发通知。
|
||||
* webhook 使用 stdlib :mod:`urllib.request`,零新依赖。
|
||||
* 通知发送失败仅记录日志,不影响任务执行。
|
||||
|
||||
快速上手
|
||||
--------
|
||||
import pyflowx as px
|
||||
|
||||
# 1. 回调通知
|
||||
notifier = px.CallbackNotifier(
|
||||
lambda e: print(f"[通知] {e.task}: {e.status.value}")
|
||||
)
|
||||
px.run(graph, notifiers=notifier)
|
||||
|
||||
# 2. 飞书 webhook(仅失败时通知)
|
||||
feishu = px.FeishuNotifier(
|
||||
url="https://open.feishu.cn/open-apis/bot/v2/hook/xxx",
|
||||
levels=(px.NotificationLevel.FAILED,),
|
||||
)
|
||||
px.run(graph, notifiers=[feishu])
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = [
|
||||
"ALL_LEVELS",
|
||||
"CallbackNotifier",
|
||||
"DingTalkNotifier",
|
||||
"DiscordNotifier",
|
||||
"FeishuNotifier",
|
||||
"NotificationLevel",
|
||||
"Notifier",
|
||||
"SlackNotifier",
|
||||
"TelegramNotifier",
|
||||
"WeChatNotifier",
|
||||
"WebhookNotifier",
|
||||
]
|
||||
|
||||
import logging
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Callable, Iterable
|
||||
from enum import Enum
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
if sys.version_info >= (3, 12):
|
||||
from typing import override
|
||||
else:
|
||||
from typing_extensions import override # pragma: no cover
|
||||
|
||||
from ._json import dumps
|
||||
from .task import TaskEvent, TaskStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NotificationLevel(Enum):
|
||||
"""通知触发的事件级别。"""
|
||||
|
||||
RUNNING = "running"
|
||||
SUCCESS = "success"
|
||||
FAILED = "failed"
|
||||
SKIPPED = "skipped"
|
||||
|
||||
|
||||
_LEVEL_STATUS_MAP: dict[NotificationLevel, TaskStatus] = {
|
||||
NotificationLevel.RUNNING: TaskStatus.RUNNING,
|
||||
NotificationLevel.SUCCESS: TaskStatus.SUCCESS,
|
||||
NotificationLevel.FAILED: TaskStatus.FAILED,
|
||||
NotificationLevel.SKIPPED: TaskStatus.SKIPPED,
|
||||
}
|
||||
|
||||
#: 所有事件级别的冻结集合。
|
||||
ALL_LEVELS: frozenset[NotificationLevel] = frozenset(_LEVEL_STATUS_MAP.keys())
|
||||
|
||||
#: 各平台消息前缀图标。
|
||||
_STATUS_ICONS: dict[TaskStatus, str] = {
|
||||
TaskStatus.RUNNING: "▶",
|
||||
TaskStatus.SUCCESS: "✓",
|
||||
TaskStatus.FAILED: "✗",
|
||||
TaskStatus.SKIPPED: "○",
|
||||
}
|
||||
|
||||
|
||||
def _status_to_level(status: TaskStatus) -> NotificationLevel | None:
|
||||
"""将 TaskStatus 映射为 NotificationLevel,无映射时返回 None。"""
|
||||
for level, st in _LEVEL_STATUS_MAP.items():
|
||||
if st == status:
|
||||
return level
|
||||
return None
|
||||
|
||||
|
||||
def _format_event_message(event: TaskEvent) -> str:
|
||||
"""格式化事件为人类可读消息文本(各平台共享)。"""
|
||||
icon = _STATUS_ICONS.get(event.status, "•")
|
||||
dur = f" ({event.duration:.3f}s)" if event.duration is not None else ""
|
||||
parts = [f"{icon} 任务 {event.task} {event.status.value}{dur}"]
|
||||
if event.error:
|
||||
parts.append(f" 错误: {event.error}")
|
||||
if event.reason:
|
||||
parts.append(f" 原因: {event.reason}")
|
||||
if event.attempts > 1:
|
||||
parts.append(f" 尝试次数: {event.attempts}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class Notifier(Protocol):
|
||||
"""通知器协议。
|
||||
|
||||
生命周期:执行期间多次 ``notify`` → 结束时 ``close``。
|
||||
实现者应确保 ``notify`` 不抛异常(或仅在不可恢复时抛),
|
||||
通知失败不应影响任务执行。
|
||||
"""
|
||||
|
||||
def notify(self, event: TaskEvent) -> None:
|
||||
"""处理单个任务事件。"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""释放资源(如关闭连接)。"""
|
||||
...
|
||||
|
||||
|
||||
class CallbackNotifier:
|
||||
"""包装 Python 回调为 :class:`Notifier`。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
callback:
|
||||
接收 :class:`TaskEvent` 的回调函数。
|
||||
levels:
|
||||
触发通知的事件级别集合。默认 :data:`ALL_LEVELS`(全生命周期)。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
callback: Callable[[TaskEvent], None],
|
||||
levels: Iterable[NotificationLevel] = ALL_LEVELS,
|
||||
) -> None:
|
||||
self._callback = callback
|
||||
self._levels = frozenset(levels)
|
||||
|
||||
def notify(self, event: TaskEvent) -> None:
|
||||
level = _status_to_level(event.status)
|
||||
if level is not None and level in self._levels:
|
||||
self._callback(event)
|
||||
|
||||
def close(self) -> None:
|
||||
"""无资源需释放。"""
|
||||
|
||||
|
||||
class WebhookNotifier:
|
||||
"""通过 HTTP POST 发送通知的基类。
|
||||
|
||||
使用 stdlib :mod:`urllib.request`,零外部依赖。子类覆盖
|
||||
:meth:`_build_payload` 以适配不同平台的消息格式。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
url:
|
||||
webhook 接收地址。
|
||||
levels:
|
||||
触发通知的事件级别集合。默认 :data:`ALL_LEVELS`。
|
||||
timeout:
|
||||
HTTP 请求超时秒数。默认 ``10``。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
levels: Iterable[NotificationLevel] = ALL_LEVELS,
|
||||
timeout: float = 10.0,
|
||||
) -> None:
|
||||
self._url = url
|
||||
self._levels = frozenset(levels)
|
||||
self._timeout = timeout
|
||||
|
||||
def notify(self, event: TaskEvent) -> None:
|
||||
level = _status_to_level(event.status)
|
||||
if level is None or level not in self._levels:
|
||||
return
|
||||
payload = self._build_payload(event)
|
||||
self._send(payload)
|
||||
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
"""构建请求体。子类覆盖以适配平台格式。"""
|
||||
return {"text": _format_event_message(event)}
|
||||
|
||||
def _send(self, payload: dict[str, Any]) -> None:
|
||||
"""发送 JSON POST 请求,失败仅记录日志。"""
|
||||
data = dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
self._url,
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
|
||||
_ = resp.read()
|
||||
except (urllib.error.URLError, OSError) as exc:
|
||||
logger.warning("webhook 通知发送失败 (%s): %s", self._url, exc)
|
||||
|
||||
def close(self) -> None:
|
||||
"""无持久连接需关闭。"""
|
||||
|
||||
|
||||
class FeishuNotifier(WebhookNotifier):
|
||||
"""飞书(Lark)自定义机器人通知器。
|
||||
|
||||
消息格式:``{"msg_type": "text", "content": {"text": ...}}``。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"msg_type": "text",
|
||||
"content": {"text": _format_event_message(event)},
|
||||
}
|
||||
|
||||
|
||||
class DingTalkNotifier(WebhookNotifier):
|
||||
"""钉钉自定义机器人通知器。
|
||||
|
||||
消息格式:``{"msgtype": "text", "text": {"content": ...}}``。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"msgtype": "text",
|
||||
"text": {"content": _format_event_message(event)},
|
||||
}
|
||||
|
||||
|
||||
class WeChatNotifier(WebhookNotifier):
|
||||
"""企业微信群机器人通知器。
|
||||
|
||||
消息格式:``{"msgtype": "text", "text": {"content": ...}}``。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"msgtype": "text",
|
||||
"text": {"content": _format_event_message(event)},
|
||||
}
|
||||
|
||||
|
||||
class SlackNotifier(WebhookNotifier):
|
||||
"""Slack Incoming Webhook 通知器。
|
||||
|
||||
消息格式:``{"text": ...}``。Slack webhook 仅需 ``text`` 字段,
|
||||
与基类格式一致,独立成类便于类型区分与后续扩展(如 channel/username)。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {"text": _format_event_message(event)}
|
||||
|
||||
|
||||
class DiscordNotifier(WebhookNotifier):
|
||||
"""Discord Webhook 通知器。
|
||||
|
||||
消息格式:``{"content": ...}``。Discord webhook 使用 ``content`` 字段
|
||||
承载消息文本。
|
||||
"""
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {"content": _format_event_message(event)}
|
||||
|
||||
|
||||
class TelegramNotifier(WebhookNotifier):
|
||||
"""Telegram Bot 通知器。
|
||||
|
||||
通过 ``sendMessage`` API 发送消息。需要在 ``__init__`` 中传入
|
||||
``chat_id``(聊天/群组 ID),并使用 bot token 拼接 webhook URL:
|
||||
``https://api.telegram.org/bot<TOKEN>/sendMessage``。
|
||||
|
||||
消息格式:``{"chat_id": ..., "text": ...}``。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
chat_id:
|
||||
接收消息的聊天或群组 ID(数字字符串或 @channelusername)。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
url: str,
|
||||
chat_id: str,
|
||||
levels: Iterable[NotificationLevel] = ALL_LEVELS,
|
||||
timeout: float = 10.0,
|
||||
) -> None:
|
||||
super().__init__(url=url, levels=levels, timeout=timeout)
|
||||
self._chat_id = chat_id
|
||||
|
||||
@override
|
||||
def _build_payload(self, event: TaskEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"chat_id": self._chat_id,
|
||||
"text": _format_event_message(event),
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
"""ops 子包 — CLI 工具模块集合 (按功能分组).
|
||||
|
||||
每个子模块使用 ``@px.tool`` 装饰器注册 CLI 工具, 由 ``pf`` 入口按需
|
||||
``importlib.import_module`` 触发注册. 子模块按功能分组到子目录.
|
||||
|
||||
子目录
|
||||
------
|
||||
- :mod:`files` —— 文件与文档操作 (filedate/filelevel/folderback/folderzip/pdftool/screenshot)
|
||||
- :mod:`dev` —— 开发与构建工具 (autofmt/bumpversion/gittool/packtool/piptool/pymake/lscalc)
|
||||
- :mod:`system` —— 系统工具 (clr/reseticoncache/taskkill/which)
|
||||
- :mod:`infra` —— 基础设施与服务部署 (dockercmd/envdev/sshcopyid/msdownload/sglang)
|
||||
|
||||
:mod:`_common` 保留在根目录, 提供跨平台命令选择、平台守卫等共享辅助.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,66 +0,0 @@
|
||||
"""ops 共享辅助: 跨平台命令选择、平台守卫、共享忽略模式."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
__all__ = ["IGNORE_PATTERNS", "ensure_platform", "platform_command"]
|
||||
|
||||
IGNORE_PATTERNS: list[str] = [
|
||||
"__pycache__",
|
||||
"*.pyc",
|
||||
"*.pyo",
|
||||
".git",
|
||||
".venv",
|
||||
".idea",
|
||||
".vscode",
|
||||
"*.egg-info",
|
||||
"dist",
|
||||
"build",
|
||||
".pytest_cache",
|
||||
".tox",
|
||||
".mypy_cache",
|
||||
]
|
||||
|
||||
|
||||
def platform_command(windows: Sequence[str], posix: Sequence[str]) -> list[str]:
|
||||
"""按当前平台返回对应命令.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
windows : Sequence[str]
|
||||
Windows 平台命令 (含参数).
|
||||
posix : Sequence[str]
|
||||
Linux/macOS 平台命令 (含参数).
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[str]
|
||||
当前平台对应的命令列表 (副本, 可安全修改).
|
||||
"""
|
||||
return list(windows) if Constants.IS_WINDOWS else list(posix)
|
||||
|
||||
|
||||
def ensure_platform(predicate: bool, tool_name: str, platform_name: str) -> bool:
|
||||
"""平台不满足时打印提示, 返回是否满足.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
predicate : bool
|
||||
平台匹配谓词 (如 ``Constants.IS_WINDOWS``).
|
||||
tool_name : str
|
||||
工具名称, 用于提示信息前缀.
|
||||
platform_name : str
|
||||
目标平台名称, 用于提示信息.
|
||||
|
||||
Returns
|
||||
-------
|
||||
bool
|
||||
平台满足时返回 True, 否则打印 ``{tool_name}: 仅在 {platform_name} 上支持`` 并返回 False.
|
||||
"""
|
||||
if not predicate:
|
||||
print(f"{tool_name}: 仅在 {platform_name} 上支持")
|
||||
return False
|
||||
return True
|
||||
@@ -1,5 +0,0 @@
|
||||
"""dev 子包 — 开发与构建工具."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,232 +0,0 @@
|
||||
"""版本号管理模块.
|
||||
|
||||
提供单文件版本号更新 (``bump_file_version``) 与项目级批量版本号同步
|
||||
(``bump_project_version``) 能力. ``bump_project_version`` 通过 ``@px.tool``
|
||||
注册为 CLI 工具, 可通过 ``pf bumpversion`` 调用.
|
||||
|
||||
设计要点
|
||||
--------
|
||||
``bump_project_version`` 采用 "先读取基准、再统一写入" 的两阶段策略:
|
||||
先扫描所有 ``__init__.py`` / ``pyproject.toml`` 文件, 读取各自的版本号,
|
||||
取最大值作为基准版本计算新版本号, 然后把新版本号统一写入所有文件,
|
||||
避免文件间版本号不同步导致的跳号问题.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"BumpVersionType",
|
||||
"bump_file_version",
|
||||
"bump_project_version",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
BumpVersionType = Literal["patch", "minor", "major"]
|
||||
|
||||
_PYPROJECT_VERSION_PATTERN = re.compile(
|
||||
r'(?:^|\n)\s*version\s*=\s*["\']'
|
||||
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
|
||||
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
|
||||
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
|
||||
r'["\']',
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
_INIT_VERSION_PATTERN = re.compile(
|
||||
r'(?:^|\n)\s*__version__\s*=\s*["\']'
|
||||
r"(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
|
||||
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
|
||||
r"(?:\+(?P<buildmetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?"
|
||||
r'["\']',
|
||||
re.MULTILINE,
|
||||
)
|
||||
|
||||
_IGNORE_DIRS = frozenset({".venv", "venv", ".git", "__pycache__", ".tox", "node_modules", "build", "dist", ".eggs"})
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 私有辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _get_pattern_for_file(file_name: str) -> re.Pattern[str] | None:
|
||||
"""根据文件类型获取对应的正则表达式."""
|
||||
if file_name == "pyproject.toml":
|
||||
return _PYPROJECT_VERSION_PATTERN
|
||||
if file_name == "__init__.py":
|
||||
return _INIT_VERSION_PATTERN
|
||||
return None
|
||||
|
||||
|
||||
def _calculate_new_version(major: int, minor: int, patch: int, part: BumpVersionType) -> str:
|
||||
"""计算新版本号."""
|
||||
if part == "major":
|
||||
return f"{major + 1}.0.0"
|
||||
if part == "minor":
|
||||
return f"{major}.{minor + 1}.0"
|
||||
return f"{major}.{minor}.{patch + 1}"
|
||||
|
||||
|
||||
def _build_replacement_string(original_match: str, new_version: str, file_name: str) -> str:
|
||||
"""构建替换字符串, 保留原始格式."""
|
||||
quote_char = '"' if '"' in original_match else "'"
|
||||
key = "__version__" if file_name == "__init__.py" else "version"
|
||||
prefix_match = re.match(rf"(\s*{key}\s*=\s*)[\"']", original_match)
|
||||
prefix = prefix_match.group(1) if prefix_match else f"{key} = "
|
||||
return f"{prefix}{quote_char}{new_version}{quote_char}"
|
||||
|
||||
|
||||
def _read_version_tuple(file_path: Path) -> tuple[int, int, int] | None:
|
||||
"""从文件中读取版本号, 返回 (major, minor, patch) 元组; 未找到返回 None.
|
||||
|
||||
读取失败时抛出 ``OSError`` / ``UnicodeDecodeError`` 由调用方处理.
|
||||
"""
|
||||
pattern = _get_pattern_for_file(file_path.name)
|
||||
if pattern is None:
|
||||
return None
|
||||
|
||||
content = file_path.read_text(encoding="utf-8")
|
||||
match = pattern.search(content)
|
||||
if not match:
|
||||
return None
|
||||
|
||||
return int(match.group("major")), int(match.group("minor")), int(match.group("patch"))
|
||||
|
||||
|
||||
def _write_version_to_file(file_path: Path, new_version: str) -> bool:
|
||||
"""把新版本号写入指定文件; 成功返回 True, 未匹配到版本号返回 False."""
|
||||
pattern = _get_pattern_for_file(file_path.name)
|
||||
if pattern is None: # pragma: no cover - 调用方已保证 pattern 不为 None
|
||||
return False
|
||||
|
||||
content = file_path.read_text(encoding="utf-8")
|
||||
match = pattern.search(content)
|
||||
if not match: # pragma: no cover - 调用方已通过 _read_version_tuple 验证
|
||||
return False
|
||||
|
||||
replacement = _build_replacement_string(match.group(0), new_version, file_path.name)
|
||||
content = content.replace(match.group(0), replacement)
|
||||
|
||||
try:
|
||||
file_path.write_text(content, encoding="utf-8")
|
||||
except OSError as e:
|
||||
print(f"更新文件 {file_path} 版本号时出错: {e}")
|
||||
raise
|
||||
|
||||
return True
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def bump_file_version(file_path: Path, part: BumpVersionType = "patch") -> str | None:
|
||||
"""更新单个文件中的版本号.
|
||||
|
||||
读取文件当前版本号, 按 ``part`` 指定的部分递增, 写回文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
file_path : Path
|
||||
要更新的文件路径 (``pyproject.toml`` 或 ``__init__.py``)
|
||||
part : BumpVersionType
|
||||
版本部分: patch, minor, major
|
||||
|
||||
Returns
|
||||
-------
|
||||
str | None
|
||||
更新后的新版本号; 文件中未找到版本号或读取失败时返回 None
|
||||
"""
|
||||
version_tuple = _read_version_tuple(file_path)
|
||||
if version_tuple is None:
|
||||
print(f"文件 {file_path} 中未找到版本号模式")
|
||||
return None
|
||||
|
||||
major, minor, patch = version_tuple
|
||||
new_version = _calculate_new_version(major, minor, patch, part)
|
||||
|
||||
if not _write_version_to_file(file_path, new_version): # pragma: no cover - _read_version_tuple 已验证
|
||||
return None
|
||||
|
||||
return new_version
|
||||
|
||||
|
||||
@px.tool("bumpversion", help="版本号自动管理")
|
||||
def bump_project_version(part: BumpVersionType = "patch", no_tag: bool = False) -> str | None:
|
||||
"""批量同步项目所有版本号文件并提交.
|
||||
|
||||
扫描当前目录下所有 ``__init__.py`` 和 ``pyproject.toml`` 文件
|
||||
(排除虚拟环境和缓存目录), 先读取每个文件的当前版本号取最大值作为基准,
|
||||
计算新版本号后统一写入所有文件, 最后执行 git add (按文件名) + commit + tag.
|
||||
|
||||
采用 "先读取基准、再统一写入" 的两阶段策略, 即使某些文件版本号不同步,
|
||||
也能在一次 bump 后重新对齐, 避免跳号.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
part : BumpVersionType
|
||||
版本部分: patch, minor, major
|
||||
no_tag : bool
|
||||
提交后不创建 git tag
|
||||
|
||||
Returns
|
||||
-------
|
||||
str | None
|
||||
更新后的新版本号; 未找到版本号文件时返回 None
|
||||
"""
|
||||
all_files: set[Path] = set()
|
||||
for pattern in ("__init__.py", "pyproject.toml"):
|
||||
for file in Path.cwd().rglob(pattern):
|
||||
if not any(ignore_dir in file.parts for ignore_dir in _IGNORE_DIRS):
|
||||
all_files.add(file)
|
||||
|
||||
if not all_files:
|
||||
print("未找到包含版本号的文件")
|
||||
return None
|
||||
|
||||
print(f"找到 {len(all_files)} 个文件需要更新版本号")
|
||||
cwd = Path.cwd()
|
||||
for file in sorted(all_files):
|
||||
print(f" - {file.relative_to(cwd)}")
|
||||
|
||||
# 阶段 1: 读取所有文件版本号, 取最大值作为基准
|
||||
versions: list[tuple[int, int, int]] = []
|
||||
for file in sorted(all_files):
|
||||
v = _read_version_tuple(file)
|
||||
if v is not None:
|
||||
versions.append(v)
|
||||
|
||||
if not versions:
|
||||
print("未能从任何文件读取版本号")
|
||||
return None
|
||||
|
||||
major, minor, patch = max(versions)
|
||||
new_version = _calculate_new_version(major, minor, patch, part)
|
||||
print(f"基准版本: {major}.{minor}.{patch} -> 新版本: {new_version}")
|
||||
|
||||
# 阶段 2: 统一写入新版本号到所有文件
|
||||
for file in sorted(all_files):
|
||||
_write_version_to_file(file, new_version)
|
||||
|
||||
# 阶段 3: git add (按文件名) + commit + tag
|
||||
relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)]
|
||||
subprocess.run(["git", "add", *relative_files], check=True)
|
||||
subprocess.run(["git", "commit", "-m", f"bump version to {new_version}"], check=True)
|
||||
|
||||
if not no_tag:
|
||||
tag_name = f"v{new_version}"
|
||||
subprocess.run(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], check=True)
|
||||
print(f"已创建标签: {tag_name}")
|
||||
|
||||
return new_version
|
||||
@@ -1,142 +0,0 @@
|
||||
"""gittool - Git 执行工具.
|
||||
|
||||
提供添加提交/清理/初始化/推送/拉取 子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
EXCLUDE_DIRS = [
|
||||
# 编辑器相关目录
|
||||
".vscode",
|
||||
".idea",
|
||||
".editorconfig",
|
||||
".trae",
|
||||
".qoder",
|
||||
# 项目相关目录
|
||||
".venv",
|
||||
".git",
|
||||
".ruff_cache",
|
||||
".tox",
|
||||
"node_modules",
|
||||
]
|
||||
|
||||
EXCLUDE_CMDS = [arg for d in EXCLUDE_DIRS for arg in ["-e", d]]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 私有辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def not_has_git_repo() -> bool:
|
||||
"""检查当前目录没有 Git 仓库."""
|
||||
return not Path.cwd().exists() or not (Path.cwd() / ".git").is_dir()
|
||||
|
||||
|
||||
def has_files() -> bool:
|
||||
"""检查当前 Git 仓库是否有未提交的更改."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "status", "--porcelain"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
return bool(result.stdout.strip())
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# fn 子命令
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("gittool", subcommand="a", help="添加并提交")
|
||||
def git_add_commit(message: str = "chore: update") -> None:
|
||||
"""执行 git add + git commit (仅当有未提交更改时).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
message : str
|
||||
提交信息 (默认: ``chore: update``)
|
||||
"""
|
||||
if not has_files():
|
||||
print("没有文件需要提交")
|
||||
return
|
||||
subprocess.run(["git", "add", "."], check=True)
|
||||
subprocess.run(["git", "commit", "-m", message], check=True)
|
||||
|
||||
|
||||
@px.tool("gittool", subcommand="i", help="初始化并提交")
|
||||
def git_init_add_commit(message: str = "init commit") -> None:
|
||||
"""执行 git init (若需) + git add + git commit (若有更改).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
message : str
|
||||
提交信息 (默认: ``init commit``)
|
||||
"""
|
||||
if not_has_git_repo():
|
||||
subprocess.run(["git", "init"], check=True)
|
||||
if has_files():
|
||||
subprocess.run(["git", "add", "."], check=True)
|
||||
subprocess.run(["git", "commit", "-m", message], check=True)
|
||||
else:
|
||||
print("没有文件需要提交")
|
||||
|
||||
|
||||
@px.tool("gittool", subcommand="isub", help="初始化子目录")
|
||||
def init_sub_dirs() -> None:
|
||||
"""初始化子目录的 Git 仓库."""
|
||||
sub_dirs = [subdir for subdir in Path.cwd().iterdir() if subdir.is_dir()]
|
||||
for subdir in sub_dirs:
|
||||
px.run(
|
||||
px.Graph().chain(
|
||||
px.cmd(["git", "init"], conditions=(lambda _: not_has_git_repo(),), cwd=subdir),
|
||||
px.cmd(["git", "add", "."]),
|
||||
px.cmd(["git", "commit", "-m", "init commit"]),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# cmd 子命令
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool(
|
||||
"gittool",
|
||||
subcommand="clean",
|
||||
help="清理未跟踪文件",
|
||||
cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS],
|
||||
hidden=True,
|
||||
)
|
||||
def clean() -> None:
|
||||
"""清理 Git 未跟踪文件 (隐藏命令, 被 c 依赖)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"gittool",
|
||||
subcommand="c",
|
||||
help="清理并查看状态",
|
||||
cmd=["git", "status", "--porcelain"],
|
||||
needs=["clean"],
|
||||
)
|
||||
def c() -> None:
|
||||
"""清理未跟踪文件并查看 Git 状态."""
|
||||
|
||||
|
||||
@px.tool("gittool", subcommand="p", help="推送", cmd=["git", "push"])
|
||||
def p() -> None:
|
||||
"""推送代码到远程仓库."""
|
||||
|
||||
|
||||
@px.tool("gittool", subcommand="pl", help="拉取", cmd=["git", "pull"])
|
||||
def pl() -> None:
|
||||
"""从远程仓库拉取代码."""
|
||||
@@ -1,111 +0,0 @@
|
||||
"""lscalc - LS-DYNA 计算工具.
|
||||
|
||||
运行 LS-DYNA 计算 (单机/MPI) 与进程状态检查.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
|
||||
_DEFAULT_NCPU: int = 4
|
||||
|
||||
|
||||
def get_ls_dyna_command(input_file: str, ncpu: int) -> list[str]:
|
||||
"""获取 LS-DYNA 命令.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[str]
|
||||
LS-DYNA 命令列表
|
||||
"""
|
||||
return ["ls-dyna_mpp", f"i={input_file}", f"ncpu={ncpu}"]
|
||||
|
||||
|
||||
@px.tool("lscalc", subcommand="run", help="运行 LS-DYNA 计算")
|
||||
def run_ls_dyna(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
|
||||
"""运行 LS-DYNA 计算.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
"""
|
||||
input_path = Path(input_file)
|
||||
if not input_path.exists():
|
||||
print(f"输入文件不存在: {input_path}")
|
||||
return
|
||||
|
||||
cmd = get_ls_dyna_command(input_file, ncpu)
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"LS-DYNA 计算完成: {input_file}")
|
||||
except FileNotFoundError:
|
||||
print("未找到 ls-dyna_mpp 命令")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"LS-DYNA 计算失败: {e}")
|
||||
|
||||
|
||||
@px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算")
|
||||
def run_ls_dyna_mpi(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
|
||||
"""运行 LS-DYNA MPI 计算.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_file : str
|
||||
输入文件路径
|
||||
ncpu : int
|
||||
CPU 核心数
|
||||
"""
|
||||
input_path = Path(input_file)
|
||||
if not input_path.exists():
|
||||
print(f"输入文件不存在: {input_path}")
|
||||
return
|
||||
|
||||
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"LS-DYNA MPI 计算完成: {input_file}")
|
||||
except FileNotFoundError:
|
||||
print("未找到 mpirun 或 ls-dyna_mpp 命令")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"LS-DYNA MPI 计算失败: {e}")
|
||||
|
||||
|
||||
@px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态")
|
||||
def check_ls_dyna_status() -> None:
|
||||
"""检查 LS-DYNA 进程状态."""
|
||||
try:
|
||||
if Constants.IS_WINDOWS:
|
||||
result = subprocess.run(
|
||||
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
print(result.stdout)
|
||||
else:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-f", "ls-dyna"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.stdout.strip():
|
||||
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
|
||||
else:
|
||||
print("没有运行中的 LS-DYNA 进程")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"检查进程状态失败: {e}")
|
||||
@@ -1,236 +0,0 @@
|
||||
"""packtool - Python 打包工具.
|
||||
|
||||
提供源码打包/依赖打包/wheel 构建/嵌入式 Python 安装/zip 包创建/清理 子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import IGNORE_PATTERNS
|
||||
|
||||
__all__ = [
|
||||
"clean_build_dir",
|
||||
"create_zip_package",
|
||||
"install_embed_python",
|
||||
"pack_dependencies",
|
||||
"pack_source",
|
||||
"pack_wheel",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
DEFAULT_BUILD_DIR = ".pypack"
|
||||
DEFAULT_DIST_DIR = "dist"
|
||||
DEFAULT_LIB_DIR = "libs"
|
||||
DEFAULT_CACHE_DIR = ".cache/pypack"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="src", help="打包源码")
|
||||
def pack_source(project_dir: Path = Path(), output_dir: Path = Path(".pypack")) -> None:
|
||||
"""打包项目源码.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
project_dir : Path
|
||||
项目目录 (默认: 当前目录)
|
||||
output_dir : Path
|
||||
输出目录 (默认: .pypack)
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
pyproject_file = project_dir / "pyproject.toml"
|
||||
project_name = project_dir.name
|
||||
|
||||
if pyproject_file.exists():
|
||||
try:
|
||||
import tomllib
|
||||
|
||||
content = pyproject_file.read_text(encoding="utf-8")
|
||||
data = tomllib.loads(content)
|
||||
project_name = data.get("project", {}).get("name", project_name)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
source_dir = output_dir / "src" / project_name
|
||||
source_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
src_subdir = project_dir / "src"
|
||||
if src_subdir.exists():
|
||||
shutil.copytree(
|
||||
src_subdir,
|
||||
source_dir / "src",
|
||||
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
else:
|
||||
for item in project_dir.iterdir():
|
||||
if item.name in IGNORE_PATTERNS or item.name.startswith("."):
|
||||
continue
|
||||
dst_item = source_dir / item.name
|
||||
if item.is_dir():
|
||||
shutil.copytree(
|
||||
item,
|
||||
dst_item,
|
||||
ignore=shutil.ignore_patterns(*IGNORE_PATTERNS),
|
||||
dirs_exist_ok=True,
|
||||
)
|
||||
else:
|
||||
shutil.copy2(item, dst_item)
|
||||
|
||||
print(f"源码打包完成: {source_dir}")
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="deps", help="打包依赖")
|
||||
def pack_dependencies(lib_dir: Path = Path("libs"), dependencies: list[str] | None = None) -> None:
|
||||
"""打包项目依赖.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
lib_dir : Path
|
||||
依赖库目录 (默认: libs)
|
||||
dependencies : list[str] | None
|
||||
依赖列表
|
||||
"""
|
||||
if dependencies is None:
|
||||
dependencies = []
|
||||
|
||||
lib_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not dependencies:
|
||||
print("没有依赖需要打包")
|
||||
return
|
||||
|
||||
cmd = [
|
||||
"pip",
|
||||
"install",
|
||||
"--target",
|
||||
str(lib_dir),
|
||||
"--no-compile",
|
||||
"--no-warn-script-location",
|
||||
]
|
||||
cmd.extend(dependencies)
|
||||
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"依赖打包完成: {lib_dir}")
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="wheel", help="构建 wheel")
|
||||
def pack_wheel(project_dir: Path = Path(), output_dir: Path = Path("dist")) -> None:
|
||||
"""打包项目为 wheel 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
project_dir : Path
|
||||
项目目录 (默认: 当前目录)
|
||||
output_dir : Path
|
||||
输出目录 (默认: dist)
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd = [
|
||||
"pip",
|
||||
"wheel",
|
||||
"--no-deps",
|
||||
"--wheel-dir",
|
||||
str(output_dir),
|
||||
str(project_dir),
|
||||
]
|
||||
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"Wheel 打包完成: {output_dir}")
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="embed", help="安装嵌入式 Python")
|
||||
def install_embed_python(version: str = "3.10", output_dir: Path = Path("python")) -> None:
|
||||
"""安装嵌入式 Python.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
version : str
|
||||
Python 版本 (如: 3.10, 3.11), 默认 3.10
|
||||
output_dir : Path
|
||||
输出目录 (默认: python)
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
arch = platform.machine().lower()
|
||||
if arch in ["x86_64", "amd64"]:
|
||||
arch = "amd64"
|
||||
elif arch in ["arm64", "aarch64"]:
|
||||
arch = "arm64"
|
||||
|
||||
version_map = {
|
||||
"3.8": "3.8.10",
|
||||
"3.9": "3.9.13",
|
||||
"3.10": "3.10.11",
|
||||
"3.11": "3.11.9",
|
||||
"3.12": "3.12.4",
|
||||
}
|
||||
full_version = version_map.get(version, f"{version}.0")
|
||||
|
||||
url = f"https://www.python.org/ftp/python/{full_version}/python-{full_version}-embed-{arch}.zip"
|
||||
|
||||
cache_file = Path(DEFAULT_CACHE_DIR) / f"python-{full_version}-embed-{arch}.zip"
|
||||
cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not cache_file.exists():
|
||||
print(f"正在下载嵌入式 Python {full_version}...")
|
||||
urllib.request.urlretrieve(url, cache_file)
|
||||
print(f"下载完成: {cache_file}")
|
||||
|
||||
with zipfile.ZipFile(cache_file, "r") as zf:
|
||||
zf.extractall(output_dir)
|
||||
|
||||
print(f"嵌入式 Python 安装完成: {output_dir}")
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="zip", help="创建 zip 包")
|
||||
def create_zip_package(source_dir: Path = Path(), output_file: Path = Path("package.zip")) -> None:
|
||||
"""创建 ZIP 打包文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
source_dir : Path
|
||||
源目录 (默认: 当前目录)
|
||||
output_file : Path
|
||||
输出文件 (默认: package.zip)
|
||||
"""
|
||||
output_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with zipfile.ZipFile(output_file, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for file in source_dir.rglob("*"):
|
||||
if file.is_file():
|
||||
arcname = file.relative_to(source_dir)
|
||||
zf.write(file, arcname)
|
||||
|
||||
print(f"ZIP 打包完成: {output_file}")
|
||||
|
||||
|
||||
@px.tool("packtool", subcommand="clean", help="清理构建目录")
|
||||
def clean_build_dir(build_dir: Path = Path(".pypack")) -> None:
|
||||
"""清理构建目录.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
build_dir : Path
|
||||
构建目录 (默认: .pypack)
|
||||
"""
|
||||
if build_dir.exists():
|
||||
shutil.rmtree(build_dir)
|
||||
print(f"清理完成: {build_dir}")
|
||||
else:
|
||||
print(f"目录不存在: {build_dir}")
|
||||
@@ -1,177 +0,0 @@
|
||||
"""piptool - pip 包管理工具.
|
||||
|
||||
提供安装/卸载/重装/下载/升级 pip/冻结依赖 子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"pip_download",
|
||||
"pip_freeze",
|
||||
"pip_install",
|
||||
"pip_reinstall",
|
||||
"pip_uninstall",
|
||||
"pip_upgrade",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置
|
||||
# ============================================================================
|
||||
|
||||
PACKAGE_DIR = "packages"
|
||||
REQUIREMENTS_FILE = "requirements.txt"
|
||||
|
||||
_PROTECTED_PACKAGES: frozenset[str] = frozenset(
|
||||
{
|
||||
"pyflowx",
|
||||
"bitool",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 私有辅助函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _get_installed_packages() -> list[str]:
|
||||
"""获取当前环境中所有已安装的包名."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pip", "list", "--format=freeze"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
packages: list[str] = []
|
||||
for line in result.stdout.strip().split("\n"):
|
||||
if line and "==" in line:
|
||||
pkg_name = line.split("==")[0].strip()
|
||||
packages.append(pkg_name)
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return []
|
||||
return packages
|
||||
|
||||
|
||||
def _expand_wildcard_packages(pattern: str) -> list[str]:
|
||||
"""展开通配符模式为实际的包名列表."""
|
||||
if not any(char in pattern for char in ["*", "?", "[", "]"]):
|
||||
return [pattern]
|
||||
|
||||
installed_packages = _get_installed_packages()
|
||||
matched = [pkg for pkg in installed_packages if fnmatch.fnmatchcase(pkg.lower(), pattern.lower())]
|
||||
return matched
|
||||
|
||||
|
||||
def _filter_protected_packages(packages: list[str]) -> list[str]:
|
||||
"""过滤掉受保护的包名."""
|
||||
safe = [p for p in packages if p.lower() not in {p.lower() for p in _PROTECTED_PACKAGES}]
|
||||
filtered = [p for p in packages if p.lower() in {p.lower() for p in _PROTECTED_PACKAGES}]
|
||||
if filtered:
|
||||
print(f"跳过受保护的包: {', '.join(filtered)}")
|
||||
return safe
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 公共函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="i", help="安装包")
|
||||
def pip_install(packages: list[str]) -> None:
|
||||
"""安装包.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
packages : list[str]
|
||||
包名列表
|
||||
"""
|
||||
subprocess.run(["pip", "install", *packages], check=True)
|
||||
print(f"安装完成: {', '.join(packages)}")
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="u", help="卸载包")
|
||||
def pip_uninstall(packages: list[str]) -> None:
|
||||
"""卸载包.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
packages : list[str]
|
||||
包名列表
|
||||
"""
|
||||
packages_to_uninstall: list[str] = []
|
||||
for pattern in packages:
|
||||
packages_to_uninstall.extend(_expand_wildcard_packages(pattern))
|
||||
|
||||
packages_to_uninstall = _filter_protected_packages(packages_to_uninstall)
|
||||
|
||||
if not packages_to_uninstall:
|
||||
return
|
||||
|
||||
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True)
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="r", help="重装包")
|
||||
def pip_reinstall(packages: list[str], offline: bool = False) -> None:
|
||||
"""重新安装包.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
packages : list[str]
|
||||
包名列表
|
||||
offline : bool
|
||||
离线模式
|
||||
"""
|
||||
safe_ps = _filter_protected_packages(packages)
|
||||
if not safe_ps:
|
||||
print("所有指定的包均为受保护包, 跳过重装")
|
||||
return
|
||||
|
||||
subprocess.run(["pip", "uninstall", "-y", *safe_ps], check=True)
|
||||
|
||||
options = ["--no-index", "--find-links", "."] if offline else []
|
||||
subprocess.run(["pip", "install", *options, *safe_ps], check=True)
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="d", help="下载包")
|
||||
def pip_download(packages: list[str], offline: bool = False) -> None:
|
||||
"""下载包.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
packages : list[str]
|
||||
包名列表
|
||||
offline : bool
|
||||
离线模式
|
||||
"""
|
||||
options = ["--no-index", "--find-links", "."] if offline else []
|
||||
subprocess.run(
|
||||
["pip", "download", *packages, *options, "-d", PACKAGE_DIR],
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="up", help="升级 pip")
|
||||
def pip_upgrade() -> None:
|
||||
"""升级 pip 到最新版本."""
|
||||
subprocess.run(["python", "-m", "pip", "install", "--upgrade", "pip"], check=True)
|
||||
print("pip 升级完成")
|
||||
|
||||
|
||||
@px.tool("piptool", subcommand="f", help="导出依赖")
|
||||
def pip_freeze() -> None:
|
||||
"""冻结依赖到 requirements.txt."""
|
||||
result = subprocess.run(
|
||||
["pip", "freeze", "--exclude-editable"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
Path(REQUIREMENTS_FILE).write_text(result.stdout)
|
||||
print(f"依赖已导出到 {REQUIREMENTS_FILE}")
|
||||
@@ -1,243 +0,0 @@
|
||||
"""pymake - 项目构建工具.
|
||||
|
||||
提供构建/测试/清理/文档/格式化/发布等子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = [
|
||||
"b",
|
||||
"ba",
|
||||
"bc",
|
||||
"bump",
|
||||
"bumpmi",
|
||||
"bumpversion",
|
||||
"c",
|
||||
"cov",
|
||||
"doc",
|
||||
"git_add_all",
|
||||
"git_push",
|
||||
"git_push_tags",
|
||||
"lint",
|
||||
"p",
|
||||
"pb",
|
||||
"publish_python",
|
||||
"pyrefly_check",
|
||||
"sync",
|
||||
"t",
|
||||
"tc",
|
||||
"test_coverage",
|
||||
"tf",
|
||||
"tox",
|
||||
"twine_publish",
|
||||
]
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
# ============================================================================
|
||||
# 单任务别名 (cmd 任务)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="b", help="构建 Python 主包 (uv build)", cmd=["uv", "build"])
|
||||
def b(cwd: Path = Path()) -> None:
|
||||
"""构建 Python 主包."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="bc", help="构建 Rust 核心模块 (maturin build)", cmd=["maturin", "build", "-r"])
|
||||
def bc(cwd: Path = Path()) -> None:
|
||||
"""构建 Rust 核心模块."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="sync", help="同步依赖 (uv sync)", cmd=["uv", "sync"])
|
||||
def sync(cwd: Path = Path()) -> None:
|
||||
"""同步依赖."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="c", help="清理构建产物 (调用 gitt c)", cmd=["pf", "gitt", "c"])
|
||||
def c(cwd: Path = Path()) -> None:
|
||||
"""清理构建产物."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="t",
|
||||
help="运行测试",
|
||||
cmd=["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"],
|
||||
)
|
||||
def t(cwd: Path = Path()) -> None:
|
||||
"""运行测试 (含并行)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="tf",
|
||||
help="快速测试 (无 slow)",
|
||||
cmd=["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"],
|
||||
)
|
||||
def tf(cwd: Path = Path()) -> None:
|
||||
"""快速测试 (无 slow, 无并行)."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="bumpmi", help="升级次版本号 (bumpversion minor)", cmd=["pf", "bumpversion", "minor"])
|
||||
def bumpmi(cwd: Path = Path()) -> None:
|
||||
"""升级次版本号."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="doc",
|
||||
help="构建 Sphinx 文档",
|
||||
cmd=["sphinx-build", "-b", "html", "docs", "docs/_build"],
|
||||
)
|
||||
def doc(cwd: Path = Path()) -> None:
|
||||
"""构建 Sphinx 文档."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="lint", help="代码格式化与检查 (ruff)", cmd=["ruff", "check", "--fix", "--unsafe-fixes"])
|
||||
def lint(cwd: Path = Path()) -> None:
|
||||
"""代码格式化与检查."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="tox", help="多版本测试 (tox)", cmd=["tox", "-p", "auto"])
|
||||
def tox(cwd: Path = Path()) -> None:
|
||||
"""多版本测试."""
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 内部 job (hidden, 不暴露为 subcommand)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="bumpversion",
|
||||
help="升级版本号 (patch)",
|
||||
cmd=["pf", "bumpversion", "patch"],
|
||||
needs=["git_add_all"],
|
||||
hidden=True,
|
||||
)
|
||||
def bumpversion(cwd: Path = Path()) -> None:
|
||||
"""升级版本号 (patch, 内部 job)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="test_coverage",
|
||||
help="测试并生成覆盖率",
|
||||
cmd=["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
|
||||
needs=["c"],
|
||||
hidden=True,
|
||||
)
|
||||
def test_coverage(cwd: Path = Path()) -> None:
|
||||
"""测试并生成覆盖率 (内部 job)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="pyrefly_check",
|
||||
help="pyrefly 类型检查",
|
||||
cmd=["pyrefly", "check", "."],
|
||||
hidden=True,
|
||||
)
|
||||
def pyrefly_check(cwd: Path = Path()) -> None:
|
||||
"""pyrefly 类型检查 (内部 job)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="git_add_all",
|
||||
help="git add -A",
|
||||
cmd=["git", "add", "-A"],
|
||||
needs=["tc"],
|
||||
hidden=True,
|
||||
)
|
||||
def git_add_all(cwd: Path = Path()) -> None:
|
||||
"""git add -A (内部 job)."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="git_push", help="git push", cmd=["git", "push"], hidden=True)
|
||||
def git_push(cwd: Path = Path()) -> None:
|
||||
"""git push (内部 job)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="git_push_tags",
|
||||
help="git push --tags",
|
||||
cmd=["git", "push", "--tags"],
|
||||
hidden=True,
|
||||
)
|
||||
def git_push_tags(cwd: Path = Path()) -> None:
|
||||
"""git push --tags (内部 job)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="twine_publish",
|
||||
help="twine upload",
|
||||
cmd=["twine", "upload", "--disable-progress-bar"],
|
||||
hidden=True,
|
||||
)
|
||||
def twine_publish(cwd: Path = Path()) -> None:
|
||||
"""twine upload (内部 job)."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="publish_python", help="hatch publish", cmd=["hatch", "publish"], hidden=True)
|
||||
def publish_python(cwd: Path = Path()) -> None:
|
||||
"""hatch publish (内部 job)."""
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 聚合 job (有 needs 无 cmd)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="ba", help="构建所有包 (Python + Rust)", needs=["b", "bc"], strategy="thread")
|
||||
def ba(cwd: Path = Path()) -> None:
|
||||
"""构建所有包 (Python + Rust)."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="bump", help="升级版本号 (清理 + 检查 + add + bumpversion)", needs=["bumpversion"])
|
||||
def bump(cwd: Path = Path()) -> None:
|
||||
"""升级版本号 (聚合)."""
|
||||
|
||||
|
||||
@px.tool("pymake", subcommand="cov", help="测试并生成覆盖率", needs=["test_coverage"])
|
||||
def cov(cwd: Path = Path()) -> None:
|
||||
"""测试并生成覆盖率 (聚合)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="tc",
|
||||
help="类型检查 (pyrefly + ruff)",
|
||||
needs=["c", "pyrefly_check", "lint"],
|
||||
strategy="thread",
|
||||
)
|
||||
def tc(cwd: Path = Path()) -> None:
|
||||
"""类型检查 (聚合)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="p",
|
||||
help="推送代码 (清理 + push + push tags)",
|
||||
needs=["c", "git_push", "git_push_tags"],
|
||||
strategy="thread",
|
||||
)
|
||||
def p(cwd: Path = Path()) -> None:
|
||||
"""推送代码 (聚合)."""
|
||||
|
||||
|
||||
@px.tool(
|
||||
"pymake",
|
||||
subcommand="pb",
|
||||
help="发布到 PyPI (twine + hatch)",
|
||||
needs=["twine_publish", "publish_python"],
|
||||
strategy="thread",
|
||||
)
|
||||
def pb(cwd: Path = Path()) -> None:
|
||||
"""发布到 PyPI (聚合)."""
|
||||
@@ -1,5 +0,0 @@
|
||||
"""files 子包 — 文件与文档操作工具."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,512 +0,0 @@
|
||||
"""pdftool - PDF 文件工具集.
|
||||
|
||||
提供 PDF 合并/拆分/压缩/加密/解密/提取文本/提取图片/水印/旋转/裁剪/
|
||||
信息/OCR/转图片/修复 等子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = [
|
||||
"pdf_add_watermark",
|
||||
"pdf_compress",
|
||||
"pdf_crop",
|
||||
"pdf_decrypt",
|
||||
"pdf_encrypt",
|
||||
"pdf_extract_images",
|
||||
"pdf_extract_text",
|
||||
"pdf_info",
|
||||
"pdf_merge",
|
||||
"pdf_ocr",
|
||||
"pdf_reorder",
|
||||
"pdf_repair",
|
||||
"pdf_rotate",
|
||||
"pdf_split",
|
||||
"pdf_to_images",
|
||||
]
|
||||
|
||||
try:
|
||||
import fitz # PyMuPDF
|
||||
|
||||
HAS_PYMUPDF = True
|
||||
except ImportError:
|
||||
HAS_PYMUPDF = False
|
||||
|
||||
try:
|
||||
import pypdf
|
||||
|
||||
HAS_PYPDF = True
|
||||
except ImportError:
|
||||
HAS_PYPDF = False
|
||||
|
||||
|
||||
def _require_pymupdf() -> bool:
|
||||
"""PyMuPDF 未安装时打印提示, 返回是否可用."""
|
||||
if not HAS_PYMUPDF:
|
||||
print("未安装 PyMuPDF 库, 请安装: pip install PyMuPDF")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _require_pypdf() -> bool:
|
||||
"""pypdf 未安装时打印提示, 返回是否可用."""
|
||||
if not HAS_PYPDF:
|
||||
print("未安装 pypdf 库, 请安装: pip install pypdf")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="m", help="合并 PDF")
|
||||
def pdf_merge(input_paths: list[Path], output_path: Path = Path("merged.pdf")) -> None:
|
||||
"""合并多个 PDF 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_paths : list[Path]
|
||||
输入 PDF 文件列表
|
||||
output_path : Path
|
||||
输出文件 (默认: merged.pdf)
|
||||
"""
|
||||
if not _require_pypdf():
|
||||
return
|
||||
|
||||
writer = pypdf.PdfWriter()
|
||||
for input_path in input_paths:
|
||||
if input_path.exists():
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"合并完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="s", help="拆分 PDF")
|
||||
def pdf_split(input_path: Path, output_dir: Path = Path("split")) -> None:
|
||||
"""拆分 PDF 文件为单页.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_dir : Path
|
||||
输出目录 (默认: split)
|
||||
"""
|
||||
if not _require_pypdf():
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for i, page in enumerate(reader.pages):
|
||||
writer = pypdf.PdfWriter()
|
||||
writer.add_page(page)
|
||||
output_file = output_dir / f"{input_path.stem}_page_{i + 1}.pdf"
|
||||
with open(output_file, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"拆分完成: {output_dir}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="c", help="压缩 PDF")
|
||||
def pdf_compress(input_path: Path, output_path: Path = Path("compressed.pdf")) -> None:
|
||||
"""压缩 PDF 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: compressed.pdf)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
|
||||
doc.close()
|
||||
|
||||
original_size = input_path.stat().st_size
|
||||
new_size = output_path.stat().st_size
|
||||
ratio = (1 - new_size / original_size) * 100
|
||||
print(f"压缩完成: {output_path} (缩小 {ratio:.1f}%)")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="e", help="加密 PDF")
|
||||
def pdf_encrypt(input_path: Path, output_path: Path = Path("encrypted.pdf"), password: str = "") -> None:
|
||||
"""加密 PDF 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: encrypted.pdf)
|
||||
password : str
|
||||
密码 (必填)
|
||||
"""
|
||||
if not password:
|
||||
print("错误: --password 为必填参数")
|
||||
return
|
||||
if not _require_pypdf():
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
writer = pypdf.PdfWriter()
|
||||
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
writer.encrypt(user_password=password, owner_password=password, use_128bit=True)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"加密完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="d", help="解密 PDF")
|
||||
def pdf_decrypt(input_path: Path, output_path: Path = Path("decrypted.pdf"), password: str = "") -> None:
|
||||
"""解密 PDF 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: decrypted.pdf)
|
||||
password : str
|
||||
密码 (必填)
|
||||
"""
|
||||
if not password:
|
||||
print("错误: --password 为必填参数")
|
||||
return
|
||||
if not _require_pypdf():
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
if reader.is_encrypted:
|
||||
reader.decrypt(password)
|
||||
|
||||
writer = pypdf.PdfWriter()
|
||||
for page in reader.pages:
|
||||
writer.add_page(page)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"解密完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="xt", help="提取文本")
|
||||
def pdf_extract_text(input_path: Path, output_path: Path = Path("output.txt")) -> None:
|
||||
"""提取 PDF 文本.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: output.txt)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
text = ""
|
||||
for page in doc:
|
||||
text += str(page.get_text()) + "\n\n"
|
||||
doc.close()
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(text, encoding="utf-8")
|
||||
print(f"文本提取完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="xi", help="提取图片")
|
||||
def pdf_extract_images(input_path: Path, output_dir: Path = Path("images")) -> None:
|
||||
"""提取 PDF 图片.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_dir : Path
|
||||
输出目录 (默认: images)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
image_count = 0
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
for page_num, page in enumerate(doc):
|
||||
images = page.get_images(full=True)
|
||||
for img_idx, img in enumerate(images):
|
||||
xref = img[0]
|
||||
base_image = doc.extract_image(xref)
|
||||
image_data = base_image["image"]
|
||||
image_ext = base_image["ext"]
|
||||
image_path = output_dir / f"page_{page_num + 1}_img_{img_idx + 1}.{image_ext}"
|
||||
image_path.write_bytes(image_data)
|
||||
image_count += 1
|
||||
|
||||
doc.close()
|
||||
print(f"图片提取完成: {output_dir} (共 {image_count} 张)")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="w", help="添加水印")
|
||||
def pdf_add_watermark(
|
||||
input_path: Path, output_path: Path = Path("watermarked.pdf"), text: str = "CONFIDENTIAL"
|
||||
) -> None:
|
||||
"""添加 PDF 水印.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: watermarked.pdf)
|
||||
text : str
|
||||
水印文字 (默认: CONFIDENTIAL)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
for page in doc:
|
||||
rect = page.rect
|
||||
text_width = fitz.get_text_length(text, fontsize=48)
|
||||
x = (rect.width - text_width) / 2
|
||||
y = rect.height / 2
|
||||
page.insert_text((x, y), text, fontsize=48, rotate=45, color=(0, 0, 0))
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"水印添加完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="r", help="旋转 PDF")
|
||||
def pdf_rotate(input_path: Path, output_path: Path = Path("rotated.pdf"), rotation: int = 90) -> None:
|
||||
"""旋转 PDF 页面.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: rotated.pdf)
|
||||
rotation : int
|
||||
旋转角度 (默认: 90)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
for page in doc:
|
||||
page.set_rotation(rotation)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"旋转完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="crop", help="裁剪 PDF")
|
||||
def pdf_crop(
|
||||
input_path: Path,
|
||||
output_path: Path = Path("cropped.pdf"),
|
||||
margins: tuple[int, int, int, int] = (10, 10, 10, 10),
|
||||
) -> None:
|
||||
"""裁剪 PDF 页面.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: cropped.pdf)
|
||||
margins : tuple[int, int, int, int]
|
||||
边距 (左, 上, 右, 下), 默认 (10, 10, 10, 10)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
left, top, right, bottom = margins
|
||||
|
||||
for page in doc:
|
||||
rect = page.rect
|
||||
new_rect = fitz.Rect(
|
||||
rect.x0 + left,
|
||||
rect.y0 + top,
|
||||
rect.x1 - right,
|
||||
rect.y1 - bottom,
|
||||
)
|
||||
page.set_cropbox(new_rect)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path))
|
||||
doc.close()
|
||||
print(f"裁剪完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="i", help="查看 PDF 信息")
|
||||
def pdf_info(input_path: Path) -> None:
|
||||
"""显示 PDF 信息.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
print(f"文件: {input_path}")
|
||||
print(f"页数: {doc.page_count}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"标题: {doc.metadata.get('title', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"作者: {doc.metadata.get('author', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"创建日期: {doc.metadata.get('creationDate', 'N/A')}")
|
||||
# pyrefly: ignore [missing-attribute]
|
||||
print(f"修改日期: {doc.metadata.get('modDate', 'N/A')}")
|
||||
print(f"文件大小: {input_path.stat().st_size / 1024:.1f} KB")
|
||||
doc.close()
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="ocr", help="PDF OCR 识别")
|
||||
def pdf_ocr(input_path: Path, output_path: Path = Path("ocr.pdf"), lang: str = "chi_sim+eng") -> None:
|
||||
"""PDF OCR 识别.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: ocr.pdf)
|
||||
lang : str
|
||||
识别语言 (默认: chi_sim+eng)
|
||||
"""
|
||||
try:
|
||||
import pytesseract
|
||||
from PIL import Image
|
||||
except ImportError:
|
||||
print("未安装 OCR 相关库, 请安装: pip install pytesseract pillow")
|
||||
return
|
||||
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
new_doc = fitz.open()
|
||||
|
||||
for page in doc:
|
||||
pix = page.get_pixmap()
|
||||
img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
|
||||
ocr_text = pytesseract.image_to_string(img, lang=lang)
|
||||
|
||||
new_page = new_doc.new_page(width=page.rect.width, height=page.rect.height)
|
||||
new_page.insert_image(new_page.rect, pixmap=pix)
|
||||
text_rect = fitz.Rect(0, 0, page.rect.width, page.rect.height)
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
new_page.insert_textbox(text_rect, ocr_text, fontname="china-ss", fontsize=11)
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
new_doc.save(str(output_path))
|
||||
new_doc.close()
|
||||
doc.close()
|
||||
print(f"OCR 识别完成: {output_path}")
|
||||
|
||||
|
||||
def pdf_reorder(input_path: Path, output_path: Path, order: list[int]) -> None:
|
||||
"""重排 PDF 页面顺序.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件
|
||||
order : list[int]
|
||||
页面顺序列表 (0-based)
|
||||
"""
|
||||
if not _require_pypdf():
|
||||
return
|
||||
|
||||
reader = pypdf.PdfReader(str(input_path))
|
||||
writer = pypdf.PdfWriter()
|
||||
|
||||
for page_num in order:
|
||||
if 0 <= page_num < len(reader.pages):
|
||||
writer.add_page(reader.pages[page_num])
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
writer.write(f)
|
||||
|
||||
print(f"重排完成: {output_path}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="img", help="PDF 转图片")
|
||||
def pdf_to_images(input_path: Path, output_dir: Path = Path("images"), dpi: int = 300) -> None:
|
||||
"""PDF 转图片.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_dir : Path
|
||||
输出目录 (默认: images)
|
||||
dpi : int
|
||||
DPI (默认: 300)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# pyrefly: ignore [bad-argument-type]
|
||||
for page_num, page in enumerate(doc):
|
||||
pix = page.get_pixmap(dpi=dpi)
|
||||
image_path = output_dir / f"{input_path.stem}_page_{page_num + 1}.png"
|
||||
pix.save(str(image_path))
|
||||
|
||||
doc.close()
|
||||
print(f"转换完成: {output_dir}")
|
||||
|
||||
|
||||
@px.tool("pdftool", subcommand="repair", help="修复 PDF")
|
||||
def pdf_repair(input_path: Path, output_path: Path = Path("repaired.pdf")) -> None:
|
||||
"""修复 PDF 文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
input_path : Path
|
||||
输入 PDF 文件
|
||||
output_path : Path
|
||||
输出文件 (默认: repaired.pdf)
|
||||
"""
|
||||
if not _require_pymupdf():
|
||||
return
|
||||
|
||||
doc = fitz.open(str(input_path))
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
doc.save(str(output_path), garbage=4, deflate=True, clean=True)
|
||||
doc.close()
|
||||
print(f"修复完成: {output_path}")
|
||||
@@ -1,5 +0,0 @@
|
||||
"""infra 子包 — 基础设施与服务部署工具."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,37 +0,0 @@
|
||||
"""dockercmd - Docker 镜像登录工具.
|
||||
|
||||
登录腾讯云 Docker 镜像仓库.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import getpass
|
||||
import subprocess
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
_DOCKER_MIRROR_TENCENT: str = "ccr.ccs.tencentyun.com"
|
||||
|
||||
|
||||
@px.tool("dockercmd", subcommand="login", help="登录腾讯云 Docker 镜像仓库")
|
||||
def docker_login_tencent(username: str = "") -> None:
|
||||
"""登录腾讯云 Docker 镜像仓库.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
username : str
|
||||
Docker 用户名 (为空时由 docker 交互式提示输入)
|
||||
"""
|
||||
user = username or getpass.getuser()
|
||||
try:
|
||||
subprocess.run(
|
||||
["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT],
|
||||
check=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"docker login 失败 (returncode={e.returncode}): {e}")
|
||||
return
|
||||
except FileNotFoundError:
|
||||
print("docker 命令未找到,请确认 Docker 已安装并在 PATH 中")
|
||||
return
|
||||
print(f"已登录腾讯云镜像仓库 (用户: {user})")
|
||||
@@ -1,418 +0,0 @@
|
||||
"""envdev - 开发环境镜像源配置工具.
|
||||
|
||||
配置 Python / Conda / Rust 镜像源 (Linux 还会安装 Qt 库、中文字体、Docker).
|
||||
所有镜像源参数互不影响, 可单独使用.
|
||||
Linux 专用操作 (系统镜像/Qt/字体/Docker) 在非 Linux 平台上由函数内部跳过.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import getpass
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
from pyflowx.ops._common import ensure_platform
|
||||
|
||||
__all__ = [
|
||||
"download_rustup_script",
|
||||
"install_linux_docker",
|
||||
"install_linux_fonts",
|
||||
"install_linux_qt_libs",
|
||||
"install_rust_toolchain",
|
||||
"setup_conda_mirror",
|
||||
"setup_linux_system_mirror",
|
||||
"setup_python_mirror",
|
||||
"setup_rust_mirror",
|
||||
]
|
||||
|
||||
# ============================================================================
|
||||
# 配置常量
|
||||
# ============================================================================
|
||||
|
||||
PyMirrorType = Literal["tsinghua", "aliyun", "huaweicloud", "ustc", "zju"]
|
||||
CondaMirrorType = Literal["tsinghua", "ustc", "bsfu", "aliyun"]
|
||||
RustMirrorType = Literal["tsinghua", "ustc", "aliyun"]
|
||||
|
||||
_PIP_INDEX_URLS: dict[str, str] = {
|
||||
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
|
||||
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
|
||||
"huaweicloud": "https://mirrors.huaweicloud.com/repository/pypi/simple/",
|
||||
"ustc": "https://pypi.mirrors.ustc.edu.cn/simple/",
|
||||
"zju": "https://mirrors.zju.edu.cn/pypi/simple/",
|
||||
}
|
||||
|
||||
_PIP_TRUSTED_HOSTS: dict[str, str] = {
|
||||
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
|
||||
"aliyun": "mirrors.aliyun.com",
|
||||
"huaweicloud": "mirrors.huaweicloud.com",
|
||||
"ustc": "pypi.mirrors.ustc.edu.cn",
|
||||
"zju": "mirrors.zju.edu.cn",
|
||||
}
|
||||
|
||||
_UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
|
||||
|
||||
_CONDA_MIRROR_URLS: dict[str, list[str]] = {
|
||||
"tsinghua": [
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"ustc": [
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/pkgs/dev/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.ustc.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"bsfu": [
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/main/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/free/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/r/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/pro/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/dev/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/menpo/",
|
||||
"https://mirrors.bsfu.edu.cn/anaconda/cloud/pytorch/",
|
||||
],
|
||||
"aliyun": [
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/r/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/msys2/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/pro/",
|
||||
"https://mirrors.aliyun.com/anaconda/pkgs/dev/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/bioconda/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/menpo/",
|
||||
"https://mirrors.aliyun.com/anaconda/cloud/pytorch/",
|
||||
],
|
||||
}
|
||||
|
||||
_RUSTUP_MIRRORS: dict[str, dict[str, str]] = {
|
||||
"tsinghua": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
|
||||
},
|
||||
"aliyun": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
|
||||
},
|
||||
"ustc": {
|
||||
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
|
||||
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
|
||||
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
|
||||
},
|
||||
}
|
||||
|
||||
_RUST_SCCACHE_DIR: Path = Path.home() / ".cargo" / "sccache"
|
||||
_RUST_SCCACHE_CACHE_SIZE: str = "20G"
|
||||
|
||||
_QT_LIBS: list[str] = [
|
||||
"build-essential",
|
||||
"libgl1",
|
||||
"libegl1",
|
||||
"libglib2.0-0",
|
||||
"libfontconfig1",
|
||||
"libfreetype6",
|
||||
"libxkbcommon0",
|
||||
"libdbus-1-3",
|
||||
"libxcb-xinerama0",
|
||||
"libxcb-icccm4",
|
||||
"libxcb-image0",
|
||||
"libxcb-keysyms1",
|
||||
"libxcb-randr0",
|
||||
"libxcb-render-util0",
|
||||
"libxcb-shape0",
|
||||
"libxcb-xfixes0",
|
||||
"libxcb-cursor0",
|
||||
]
|
||||
|
||||
_CHINESE_FONTS: list[str] = [
|
||||
"fonts-noto-cjk",
|
||||
"fonts-wqy-microhei",
|
||||
"fonts-wqy-zenhei",
|
||||
"fonts-noto-color-emoji",
|
||||
]
|
||||
|
||||
_DOWNLOAD_MIRROR_SCRIPT: str = "curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh"
|
||||
_INSTALL_MIRROR_SCRIPT: str = "sudo bash /tmp/linuxmirrors.sh"
|
||||
|
||||
_RUSTUP_DOWNLOAD_URL_LINUX: str = "https://mirrors.aliyun.com/repo/rust/rustup-init.sh"
|
||||
_RUSTUP_DOWNLOAD_URL_WINDOWS: str = "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 私有辅助
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def _pip_config_path() -> Path:
|
||||
"""返回当前平台的 pip 配置文件路径."""
|
||||
if Constants.IS_LINUX:
|
||||
return Path.home() / ".pip" / "pip.conf"
|
||||
return Path.home() / "pip" / "pip.ini"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 镜像源配置函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("envdev", subcommand="setup-python", help="配置 Python 镜像源")
|
||||
def setup_python_mirror(mirror: str = "tsinghua") -> None:
|
||||
"""配置 Python 镜像源 (设置环境变量 + 写入 pip 配置文件).
|
||||
|
||||
设置 ``PIP_INDEX_URL`` / ``PIP_TRUSTED_HOSTS`` / ``UV_INDEX_URL`` /
|
||||
``UV_PYTHON_INSTALL_MIRROR`` 等环境变量, 并写入 pip 配置文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
mirror : str
|
||||
镜像源名称: tsinghua/aliyun/huaweicloud/ustc/zju (默认: tsinghua)
|
||||
"""
|
||||
if mirror not in _PIP_INDEX_URLS:
|
||||
print(f"未知 Python 镜像源: {mirror}")
|
||||
return
|
||||
|
||||
index_url = _PIP_INDEX_URLS[mirror]
|
||||
trusted_host = _PIP_TRUSTED_HOSTS[mirror]
|
||||
|
||||
os.environ["PIP_INDEX_URL"] = index_url
|
||||
os.environ["PIP_TRUSTED_HOSTS"] = trusted_host
|
||||
os.environ["UV_INDEX_URL"] = index_url
|
||||
os.environ["UV_PYTHON_INSTALL_MIRROR"] = _UV_PYTHON_INSTALL_MIRROR
|
||||
os.environ["UV_HTTP_TIMEOUT"] = "600"
|
||||
os.environ["UV_LINK_MODE"] = "copy"
|
||||
|
||||
config_path = _pip_config_path()
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
content = f"[global]\nindex-url = {index_url}\ntrusted-host = {trusted_host}\n"
|
||||
config_path.write_text(content, encoding="utf-8")
|
||||
print(f"Python 镜像源已配置: {mirror} -> {config_path}")
|
||||
|
||||
|
||||
@px.tool("envdev", subcommand="setup-conda", help="配置 Conda 镜像源")
|
||||
def setup_conda_mirror(mirror: str = "tsinghua") -> None:
|
||||
"""配置 Conda 镜像源 (写入 ~/.condarc).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
mirror : str
|
||||
镜像源名称: tsinghua/ustc/bsfu/aliyun (默认: tsinghua)
|
||||
"""
|
||||
if mirror not in _CONDA_MIRROR_URLS:
|
||||
print(f"未知 Conda 镜像源: {mirror}")
|
||||
return
|
||||
|
||||
urls = _CONDA_MIRROR_URLS[mirror]
|
||||
config_path = Path.home() / ".condarc"
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
content = "show_channel_urls: true\nchannels:\n - " + "\n - ".join(urls) + "\n - defaults\n"
|
||||
config_path.write_text(content, encoding="utf-8")
|
||||
print(f"Conda 镜像源已配置: {mirror} -> {config_path}")
|
||||
|
||||
|
||||
@px.tool("envdev", subcommand="setup-rust", help="配置 Rust 镜像源")
|
||||
def setup_rust_mirror(mirror: str = "tsinghua", version: str = "stable") -> None:
|
||||
"""配置 Rust 镜像源 (设置环境变量 + 写入 cargo config + 创建 sccache 目录).
|
||||
|
||||
设置 ``RUSTUP_DIST_SERVER`` / ``RUSTUP_UPDATE_ROOT`` / ``RUST_SCCACHE_DIR``
|
||||
等环境变量, 写入 ``~/.cargo/config.toml``, 并创建 sccache 缓存目录.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
mirror : str
|
||||
镜像源名称: tsinghua/ustc/aliyun (默认: tsinghua)
|
||||
version : str
|
||||
Rust 版本 (未使用, 保留以与原 envdev 参数对齐)
|
||||
"""
|
||||
del version # 兼容旧参数, 实际安装由独立 job 处理
|
||||
|
||||
if mirror not in _RUSTUP_MIRRORS:
|
||||
print(f"未知 Rust 镜像源: {mirror}")
|
||||
return
|
||||
|
||||
mirrors = _RUSTUP_MIRRORS[mirror]
|
||||
os.environ["RUSTUP_DIST_SERVER"] = mirrors["RUSTUP_DIST_SERVER"]
|
||||
os.environ["RUSTUP_UPDATE_ROOT"] = mirrors["RUSTUP_UPDATE_ROOT"]
|
||||
os.environ["RUST_SCCACHE_DIR"] = str(_RUST_SCCACHE_DIR)
|
||||
os.environ["RUST_SCCACHE_CACHE_SIZE"] = _RUST_SCCACHE_CACHE_SIZE
|
||||
|
||||
_RUST_SCCACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
config_path = Path.home() / ".cargo" / "config.toml"
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
registry = mirrors["TOML_REGISTRY"]
|
||||
content = (
|
||||
f"\n[source.crates-io]\nreplace-with = '{mirror}'\n\n"
|
||||
f'[source.{mirror}]\nregistry = "sparse+{registry}"\n\n'
|
||||
f'[registries.{mirror}]\nindex = "sparse+{registry}"\n'
|
||||
)
|
||||
config_path.write_text(content, encoding="utf-8")
|
||||
print(f"Rust 镜像源已配置: {mirror} -> {config_path}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Rust 工具链安装
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("envdev", subcommand="download-rustup", help="下载 Rustup 安装脚本")
|
||||
def download_rustup_script() -> None:
|
||||
"""下载 Rustup 安装脚本 (跨平台, 已安装 rustup 时跳过).
|
||||
|
||||
Linux 下载 ``rustup-init.sh``, Windows 下载 ``rustup-init.exe``.
|
||||
"""
|
||||
if shutil.which("rustup") is not None:
|
||||
print("rustup 已安装, 跳过下载")
|
||||
return
|
||||
|
||||
if Constants.IS_WINDOWS:
|
||||
print("下载 rustup-init.exe...")
|
||||
subprocess.run(
|
||||
[
|
||||
"powershell",
|
||||
"-Command",
|
||||
"Invoke-WebRequest",
|
||||
"-Uri",
|
||||
_RUSTUP_DOWNLOAD_URL_WINDOWS,
|
||||
"-OutFile",
|
||||
"rustup-init.exe",
|
||||
],
|
||||
check=False,
|
||||
)
|
||||
else:
|
||||
print("下载 rustup-init.sh...")
|
||||
subprocess.run(
|
||||
["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
@px.tool(
|
||||
"envdev",
|
||||
subcommand="install-rust",
|
||||
help="安装 Rust 工具链",
|
||||
needs=["setup-rust", "download-rustup"],
|
||||
allow_upstream_skip=True,
|
||||
)
|
||||
def install_rust_toolchain(version: str = "stable") -> None:
|
||||
"""安装 Rust 工具链 (rustup 未安装时跳过).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
version : str
|
||||
Rust 版本: ``stable`` / ``nightly`` / ``beta`` (默认: ``stable``)
|
||||
"""
|
||||
if shutil.which("rustup") is None:
|
||||
print("rustup 未安装, 跳过工具链安装")
|
||||
return
|
||||
|
||||
subprocess.run(["rustup", "toolchain", "install", version], check=False)
|
||||
print(f"Rust 工具链 {version} 安装完成")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Linux 专用函数
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@px.tool("envdev", subcommand="setup-linux-mirror", help="配置 Linux 系统镜像源")
|
||||
def setup_linux_system_mirror() -> None:
|
||||
"""下载并安装 Linux 系统镜像源 (仅 Linux, 已配置国内镜像时跳过).
|
||||
|
||||
检查 ``/etc/apt/sources.list`` 与 ``/etc/apt/sources.list.d/ubuntu.sources``
|
||||
是否已配置国内镜像, 已配置则跳过; 未配置则下载并执行 linuxmirrors 脚本.
|
||||
"""
|
||||
if not ensure_platform(Constants.IS_LINUX, "setup_linux_system_mirror", "Linux"):
|
||||
return
|
||||
|
||||
apt_files = ["/etc/apt/sources.list", "/etc/apt/sources.list.d/ubuntu.sources"]
|
||||
mirror_keys = list(_PIP_INDEX_URLS.keys())
|
||||
already_configured = False
|
||||
for apt_file in apt_files:
|
||||
try:
|
||||
content = Path(apt_file).read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
if any(mirror in content for mirror in mirror_keys):
|
||||
already_configured = True
|
||||
break
|
||||
|
||||
if already_configured:
|
||||
print("已配置国内镜像源, 跳过系统镜像配置")
|
||||
return
|
||||
|
||||
print("下载 linuxmirrors 脚本...")
|
||||
subprocess.run(_DOWNLOAD_MIRROR_SCRIPT, shell=True, check=False)
|
||||
print("安装 linuxmirrors...")
|
||||
subprocess.run(_INSTALL_MIRROR_SCRIPT, shell=True, check=False)
|
||||
|
||||
|
||||
@px.tool(
|
||||
"envdev",
|
||||
subcommand="install-qt-libs",
|
||||
help="安装 Qt 依赖库",
|
||||
needs=["setup-linux-mirror"],
|
||||
allow_upstream_skip=True,
|
||||
)
|
||||
def install_linux_qt_libs() -> None:
|
||||
"""安装 Qt 依赖库 (仅 Linux)."""
|
||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"):
|
||||
return
|
||||
|
||||
subprocess.run(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False)
|
||||
print("Qt 依赖库安装完成")
|
||||
|
||||
|
||||
@px.tool(
|
||||
"envdev",
|
||||
subcommand="install-fonts",
|
||||
help="安装中文字体",
|
||||
needs=["setup-linux-mirror"],
|
||||
allow_upstream_skip=True,
|
||||
)
|
||||
def install_linux_fonts() -> None:
|
||||
"""安装中文字体 (仅 Linux)."""
|
||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"):
|
||||
return
|
||||
|
||||
subprocess.run(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False)
|
||||
print("中文字体安装完成")
|
||||
|
||||
|
||||
@px.tool(
|
||||
"envdev",
|
||||
subcommand="install-docker",
|
||||
help="安装 Docker",
|
||||
needs=["setup-linux-mirror"],
|
||||
allow_upstream_skip=True,
|
||||
)
|
||||
def install_linux_docker() -> None:
|
||||
"""安装 Docker (仅 Linux)."""
|
||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"):
|
||||
return
|
||||
|
||||
subprocess.run(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False)
|
||||
subprocess.run(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False)
|
||||
print("Docker 安装完成 (需重新登录以生效 docker 用户组)")
|
||||
@@ -1,46 +0,0 @@
|
||||
"""msdownload - ModelScope 模型/数据集下载工具.
|
||||
|
||||
从 ModelScope 下载模型、数据集或空间.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
__all__ = ["msdownload_run"]
|
||||
|
||||
|
||||
@px.tool("msdownload", help="ModelScope 模型/数据集下载工具")
|
||||
def msdownload_run(name: str, target_type: str = "model", download_dir: str | None = None) -> None:
|
||||
"""从 ModelScope 下载模型/数据集/空间.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
name : str
|
||||
目标名称 (如: ``Qwen/Qwen2.5-Coder-32B-Instruct``)
|
||||
target_type : str
|
||||
目标类型: ``model`` / ``dataset`` / ``space`` (默认: ``model``)
|
||||
download_dir : str | None
|
||||
下载目录; 为 None 时默认 ``~/.models/<name 最后一段>``
|
||||
"""
|
||||
if not name:
|
||||
print("msdownload: name 不能为空")
|
||||
return
|
||||
|
||||
if download_dir:
|
||||
out_dir = Path(download_dir)
|
||||
else:
|
||||
out_dir = Path.home() / ".models" / name.rsplit("/", 1)[-1]
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)]
|
||||
print(f"下载 {target_type}: {name} -> {out_dir}")
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"下载失败 (returncode={e.returncode}): {e}")
|
||||
except FileNotFoundError:
|
||||
print("uvx 命令未找到,请确认 uv 已安装并在 PATH 中")
|
||||
@@ -1,97 +0,0 @@
|
||||
"""sglang - SGLang 本地模型服务工具.
|
||||
|
||||
提供安装与启动 SGLang 服务的子命令.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import platform_command
|
||||
|
||||
__all__ = [
|
||||
"install_sglang",
|
||||
"run_sglang",
|
||||
]
|
||||
|
||||
|
||||
@px.tool("sglang", subcommand="install", help="安装 sglang")
|
||||
def install_sglang() -> None:
|
||||
"""安装 sglang (若未安装).
|
||||
|
||||
通过 ``shutil.which`` 检测 sglang 是否已安装, 未安装时执行 ``uv install sglang[all]``.
|
||||
"""
|
||||
if shutil.which("sglang") is not None:
|
||||
print("sglang 已安装, 跳过安装步骤")
|
||||
return
|
||||
|
||||
print("正在安装 sglang[all]...")
|
||||
try:
|
||||
subprocess.run(["uv", "install", "sglang[all]"], check=True)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"安装失败 (returncode={e.returncode}): {e}")
|
||||
except FileNotFoundError:
|
||||
print("uv 命令未找到,请确认 uv 已安装并在 PATH 中")
|
||||
|
||||
|
||||
@px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"])
|
||||
def run_sglang(
|
||||
model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ",
|
||||
port: int = 8000,
|
||||
ctx_len: int = 32768,
|
||||
mem_fraction: float = 0.75,
|
||||
host: str = "0.0.0.0",
|
||||
log_level: str = "info",
|
||||
) -> None:
|
||||
"""启动 SGLang 本地模型服务.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
model : str
|
||||
模型路径 (默认: ``~/.models/Qwen2.5-Coder-32B-Instruct-AWQ``)
|
||||
port : int
|
||||
服务端口 (默认: 8000)
|
||||
ctx_len : int
|
||||
最大上下文长度 (默认: 32768)
|
||||
mem_fraction : float
|
||||
显存占比 0-1 (默认: 0.75)
|
||||
host : str
|
||||
主机地址 (默认: 0.0.0.0)
|
||||
log_level : str
|
||||
日志级别 (默认: info)
|
||||
"""
|
||||
model_dir = Path(model).expanduser()
|
||||
if not model_dir.exists():
|
||||
print(f"模型目录不存在: {model_dir}")
|
||||
return
|
||||
|
||||
python_bin = platform_command(["python"], ["python3"])[0]
|
||||
cmd = [
|
||||
python_bin,
|
||||
"-m",
|
||||
"sglang.launch_server",
|
||||
"--model-path",
|
||||
str(model_dir),
|
||||
"--host",
|
||||
host,
|
||||
"--port",
|
||||
str(port),
|
||||
"--mem-fraction-static",
|
||||
str(mem_fraction),
|
||||
"--context-length",
|
||||
str(ctx_len),
|
||||
"--tool-call-parser",
|
||||
"qwen",
|
||||
"--log-level",
|
||||
log_level,
|
||||
]
|
||||
print(f"启动 SGLang: {model_dir} (port={port}, ctx={ctx_len}, mem={mem_fraction})")
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"SGLang 启动失败 (returncode={e.returncode}): {e}")
|
||||
except FileNotFoundError:
|
||||
print(f"{python_bin} 命令未找到,请确认 Python 已安装并在 PATH 中")
|
||||
@@ -1,5 +0,0 @@
|
||||
"""system 子包 — 系统工具."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,21 +0,0 @@
|
||||
"""clr - 清屏工具.
|
||||
|
||||
跨平台清屏: Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import platform_command
|
||||
|
||||
|
||||
@px.tool("clr", help="清屏 (跨平台)")
|
||||
def clear_screen_run() -> None:
|
||||
"""清屏 (跨平台).
|
||||
|
||||
Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
|
||||
"""
|
||||
# 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍
|
||||
subprocess.run(platform_command(["cls"], ["clear"]), check=False)
|
||||
@@ -1,52 +0,0 @@
|
||||
"""reseticoncache - 重置 Windows 图标缓存工具.
|
||||
|
||||
执行流程: 杀掉 explorer → 删除 IconCache.db → 删除 iconcache* → 重启 explorer.
|
||||
仅在 Windows 上执行, 非 Windows 平台打印提示并跳过.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.conditions import Constants
|
||||
from pyflowx.ops._common import ensure_platform
|
||||
|
||||
|
||||
@px.tool("reseticoncache", help="重置 Windows 图标缓存")
|
||||
def reset_icon_cache_run() -> None:
|
||||
"""重置 Windows 图标缓存.
|
||||
|
||||
执行流程: 杀掉 explorer → 删除 IconCache.db → 删除 iconcache* → 重启 explorer.
|
||||
仅在 Windows 上执行, 非 Windows 平台打印提示并跳过.
|
||||
"""
|
||||
if not ensure_platform(Constants.IS_WINDOWS, "reset_icon_cache", "Windows"):
|
||||
return
|
||||
|
||||
local_app_data = os.environ.get("LOCALAPPDATA", "")
|
||||
if not local_app_data:
|
||||
print("reset_icon_cache: LOCALAPPDATA 环境变量未设置")
|
||||
return
|
||||
|
||||
icon_cache_db = Path(local_app_data) / "IconCache.db"
|
||||
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
|
||||
|
||||
print("正在终止 explorer 进程...")
|
||||
subprocess.run(["taskkill", "/f", "/im", "explorer.exe"], check=False)
|
||||
|
||||
if icon_cache_db.exists():
|
||||
print(f"删除图标缓存: {icon_cache_db}")
|
||||
subprocess.run(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False)
|
||||
|
||||
if explorer_cache_dir.exists():
|
||||
print(f"清理 Explorer 缓存: {explorer_cache_dir}")
|
||||
subprocess.run(
|
||||
["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
|
||||
check=False,
|
||||
)
|
||||
|
||||
print("重启 explorer...")
|
||||
subprocess.run(["cmd", "/c", "start", "explorer.exe"], check=False)
|
||||
print("图标缓存已重置")
|
||||
@@ -1,29 +0,0 @@
|
||||
"""setenv - 设置环境变量工具.
|
||||
|
||||
设置当前进程的环境变量, 支持不覆盖已有值的 ``default`` 模式.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
|
||||
@px.tool("setenv", help="设置环境变量")
|
||||
def setenv_run(name: str, value: str, default: bool = False) -> None:
|
||||
"""设置环境变量.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
name : str
|
||||
环境变量名
|
||||
value : str
|
||||
环境变量值
|
||||
default : bool
|
||||
为 True 时使用 ``setdefault`` 不覆盖已有值 (默认 False)
|
||||
"""
|
||||
if default:
|
||||
os.environ.setdefault(name, value)
|
||||
else:
|
||||
os.environ[name] = value
|
||||
@@ -1,35 +0,0 @@
|
||||
"""taskkill - 进程终止工具.
|
||||
|
||||
跨平台按名称终止进程: Windows 用 ``taskkill``, Linux/macOS 用 ``pkill``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import platform_command
|
||||
|
||||
|
||||
@px.tool("taskkill", help="按名称终止进程 (跨平台)")
|
||||
def taskkill_run(process_names: list[str]) -> None:
|
||||
"""按名称终止进程 (跨平台).
|
||||
|
||||
Windows 使用 ``taskkill /f /im <name>*``,
|
||||
Linux/macOS 使用 ``pkill -f <name>*``.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
process_names : list[str]
|
||||
进程名称列表 (如: ``["chrome.exe", "python"]``)
|
||||
"""
|
||||
cmd_prefix = platform_command(["taskkill", "/f", "/im"], ["pkill", "-f"])
|
||||
|
||||
for name in process_names:
|
||||
print(f"终止进程: {name}")
|
||||
# pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode
|
||||
result = subprocess.run([*cmd_prefix, f"{name}*"], check=False)
|
||||
if result.returncode == 0:
|
||||
print(f" 已发送终止信号: {name}")
|
||||
else:
|
||||
print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}")
|
||||
@@ -1,36 +0,0 @@
|
||||
"""which - 命令查找工具.
|
||||
|
||||
跨平台查找可执行命令路径: Windows 用 ``where``, Linux/macOS 用 ``which``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
|
||||
import pyflowx as px
|
||||
from pyflowx.ops._common import platform_command
|
||||
|
||||
|
||||
@px.tool("which", help="查找可执行命令路径 (跨平台)")
|
||||
def which_run(commands: list[str]) -> None:
|
||||
"""查找可执行命令路径 (跨平台).
|
||||
|
||||
Windows 使用 ``where``, Linux/macOS 使用 ``which``.
|
||||
对每个命令打印 ``<cmd> -> <path>`` 或 ``<cmd> -> 未找到``.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
commands : list[str]
|
||||
要查找的命令名称列表
|
||||
"""
|
||||
which_cmd = platform_command(["where"], ["which"])[0]
|
||||
|
||||
for cmd in commands:
|
||||
# where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode
|
||||
result = subprocess.run([which_cmd, cmd], capture_output=True, text=True, check=False)
|
||||
if result.returncode == 0:
|
||||
# Windows 的 where 可能返回多行, 取第一个
|
||||
path = result.stdout.strip().split("\n")[0].strip()
|
||||
print(f"{cmd} -> {path}")
|
||||
else:
|
||||
print(f"{cmd} -> 未找到")
|
||||
@@ -1,26 +0,0 @@
|
||||
"""writefile - 写入文件工具.
|
||||
|
||||
将文本内容写入指定路径的文件, 支持指定编码.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyflowx as px
|
||||
|
||||
|
||||
@px.tool("writefile", help="写入文件")
|
||||
def write_file_run(path: str, content: str, encoding: str = "utf-8") -> None:
|
||||
"""写入文件.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
path : str
|
||||
目标文件路径
|
||||
content : str
|
||||
写入内容
|
||||
encoding : str
|
||||
文件编码 (默认 ``utf-8``)
|
||||
"""
|
||||
Path(path).write_text(content, encoding=encoding)
|
||||
@@ -1,144 +0,0 @@
|
||||
"""实时进度监控。
|
||||
|
||||
提供两种监控形式:
|
||||
|
||||
* :class:`RichProgressMonitor` —— 基于 ``rich.Progress`` 的终端进度条,
|
||||
通过 ``run(graph, progress=True)`` 开箱即用。
|
||||
* :class:`ProgressCallback` —— 协议,定义 ``start`` / ``on_event`` / ``finish``
|
||||
三个生命周期方法,供程序化集成(如接入 Web 仪表盘、日志聚合系统)。
|
||||
|
||||
复用 :class:`pyflowx.task.TaskEvent` 事件机制,与现有 ``on_event`` 回调独立
|
||||
且可共存——``on_event`` 是简单逐事件函数,``ProgressCallback`` 是带生命周期的
|
||||
监控器(需要知道总任务数以渲染进度条)。
|
||||
|
||||
快速上手
|
||||
--------
|
||||
import pyflowx as px
|
||||
|
||||
# 1. 开箱即用
|
||||
report = px.run(graph, progress=True)
|
||||
|
||||
# 2. 自定义监控器
|
||||
class MyMonitor(px.ProgressCallback):
|
||||
def start(self, total: int) -> None:
|
||||
print(f"开始执行 {total} 个任务")
|
||||
def on_event(self, event: px.TaskEvent) -> None:
|
||||
print(f"{event.task}: {event.status.value}")
|
||||
def finish(self) -> None:
|
||||
print("执行完成")
|
||||
|
||||
report = px.run(graph, progress=MyMonitor())
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = [
|
||||
"ProgressCallback",
|
||||
"RichProgressMonitor",
|
||||
]
|
||||
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
from rich.console import Console
|
||||
from rich.progress import (
|
||||
BarColumn,
|
||||
MofNCompleteColumn,
|
||||
Progress,
|
||||
SpinnerColumn,
|
||||
TaskID,
|
||||
TaskProgressColumn,
|
||||
TextColumn,
|
||||
TimeElapsedColumn,
|
||||
)
|
||||
|
||||
from .task import TaskEvent, TaskStatus
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ProgressCallback(Protocol):
|
||||
"""进度监控回调协议。
|
||||
|
||||
生命周期:``start`` → 多次 ``on_event`` → ``finish``。
|
||||
实现者可在 ``start`` 中初始化资源(如打开进度条),在 ``finish`` 中清理。
|
||||
"""
|
||||
|
||||
def start(self, total: int) -> None:
|
||||
"""执行开始时调用,``total`` 为图中的任务总数。"""
|
||||
... # pragma: no cover
|
||||
|
||||
def on_event(self, event: TaskEvent) -> None:
|
||||
"""每次任务状态转换时调用。"""
|
||||
... # pragma: no cover
|
||||
|
||||
def finish(self) -> None:
|
||||
"""执行结束(无论成功/失败)时调用。"""
|
||||
... # pragma: no cover
|
||||
|
||||
|
||||
class RichProgressMonitor:
|
||||
"""基于 :class:`rich.progress.Progress` 的默认终端进度监控器。
|
||||
|
||||
显示运行中任务名、完成数/总数、进度条、百分比与耗时。
|
||||
终态事件(SUCCESS/FAILED/SKIPPED)均推进进度。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
console:
|
||||
可选的 :class:`rich.console.Console`,用于测试注入。
|
||||
默认创建新 Console。
|
||||
"""
|
||||
|
||||
def __init__(self, console: Console | None = None) -> None:
|
||||
self._progress = Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[bold blue]{task.description}"),
|
||||
BarColumn(),
|
||||
MofNCompleteColumn(),
|
||||
TaskProgressColumn(),
|
||||
TimeElapsedColumn(),
|
||||
console=console,
|
||||
)
|
||||
self._task_id: TaskID | None = None
|
||||
self._completed: int = 0
|
||||
self._total: int = 0
|
||||
self._running: set[str] = set()
|
||||
|
||||
def start(self, total: int) -> None:
|
||||
"""启动进度条。"""
|
||||
self._total = total
|
||||
self._progress.start()
|
||||
self._task_id = self._progress.add_task("工作流执行中", total=total)
|
||||
|
||||
def on_event(self, event: TaskEvent) -> None:
|
||||
"""处理任务事件,更新进度条。"""
|
||||
if event.status == TaskStatus.RUNNING:
|
||||
self._running.add(event.task)
|
||||
self._update_description()
|
||||
elif event.status in (TaskStatus.SUCCESS, TaskStatus.FAILED, TaskStatus.SKIPPED):
|
||||
self._running.discard(event.task)
|
||||
self._completed += 1
|
||||
if self._task_id is not None:
|
||||
self._progress.advance(self._task_id)
|
||||
self._update_description()
|
||||
|
||||
def finish(self) -> None:
|
||||
"""停止进度条。"""
|
||||
self._progress.stop()
|
||||
|
||||
def _update_description(self) -> None:
|
||||
"""更新进度条描述:显示运行中任务名或完成统计。"""
|
||||
if self._task_id is None:
|
||||
return
|
||||
if self._running:
|
||||
shown = sorted(self._running)[:3]
|
||||
suffix = f" (+{len(self._running) - 3})" if len(self._running) > 3 else ""
|
||||
desc = f"运行中: {', '.join(shown)}{suffix}"
|
||||
else:
|
||||
desc = f"已完成 {self._completed}/{self._total}"
|
||||
self._progress.update(self._task_id, description=desc)
|
||||
|
||||
def __enter__(self) -> RichProgressMonitor:
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc: Any) -> None:
|
||||
self.finish()
|
||||
+4
-408
@@ -6,64 +6,10 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import html
|
||||
import io
|
||||
from collections.abc import Callable, Iterator
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import uuid4
|
||||
from typing import Any, Iterator
|
||||
|
||||
from ._json import dumps, loads
|
||||
from .task import TaskResult, TaskSpec, TaskStatus
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .graph import Graph
|
||||
from .profiling import ProfileReport
|
||||
|
||||
|
||||
def _noop_fn() -> None:
|
||||
"""反序列化重建报告时的占位 ``fn``。"""
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_output_path(result: Any, path: str) -> Any:
|
||||
"""从任务结果中按点分路径提取命名输出值。
|
||||
|
||||
路径语法:
|
||||
- ``"$"`` → 整个结果
|
||||
- ``"key"`` → ``result["key"]``
|
||||
- ``"a.b.c"`` → ``result["a"]["b"]["c"]``
|
||||
|
||||
适用于 dict 结果;非 dict 结果仅支持 ``"$"``。
|
||||
"""
|
||||
if path == "$":
|
||||
return result
|
||||
value = result
|
||||
for key in path.split("."):
|
||||
value = value[key]
|
||||
return value
|
||||
|
||||
|
||||
def _serialize_value(
|
||||
value: Any,
|
||||
value_serializer: Callable[[Any], Any] | None = None,
|
||||
) -> Any:
|
||||
"""序列化任务返回值为 JSON 兼容形式。
|
||||
|
||||
``None`` 直接返回;提供 ``value_serializer`` 时由其转换;否则尝试
|
||||
JSON 序列化,失败则回退到 ``repr(value)``,保证任意值都能落盘。
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if value_serializer is not None:
|
||||
return value_serializer(value)
|
||||
try:
|
||||
_ = dumps(value)
|
||||
except (TypeError, ValueError):
|
||||
return repr(value)
|
||||
return value
|
||||
from .task import TaskResult, TaskStatus
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -76,13 +22,10 @@ class RunReport:
|
||||
任务名 -> :class:`TaskResult` 的映射。插入顺序与任务完成顺序一致。
|
||||
success:
|
||||
当且仅当所有非跳过任务都以 ``SUCCESS`` 结束时为 ``True``。
|
||||
run_id:
|
||||
8 字符十六进制运行 ID,自动生成,用于日志追踪与跨系统关联。
|
||||
"""
|
||||
|
||||
results: dict[str, TaskResult[Any]] = field(default_factory=dict)
|
||||
success: bool = True
|
||||
run_id: str = field(default_factory=lambda: uuid4().hex[:8])
|
||||
|
||||
# ---- 类型化访问 --------------------------------------------------- #
|
||||
def __getitem__(self, name: str) -> Any:
|
||||
@@ -97,34 +40,6 @@ class RunReport:
|
||||
"""返回 ``name`` 的完整 :class:`TaskResult`。"""
|
||||
return self.results[name]
|
||||
|
||||
def output_of(self, task_name: str, output_name: str) -> Any:
|
||||
"""获取任务的命名输出。
|
||||
|
||||
根据 :attr:`TaskSpec.outputs` 中声明的路径从任务结果中提取。
|
||||
例如 ``outputs={"artifact": "path"}`` 声明后,
|
||||
``report.output_of("build", "artifact")`` 返回 ``result["path"]``。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
task_name:
|
||||
任务名。
|
||||
output_name:
|
||||
``outputs`` 映射中的键名。
|
||||
|
||||
Raises
|
||||
------
|
||||
KeyError
|
||||
任务不存在、任务未声明 outputs、或 output_name 未声明时。
|
||||
"""
|
||||
result = self.results.get(task_name)
|
||||
if result is None:
|
||||
raise KeyError(f"任务 {task_name!r} 不在报告中。")
|
||||
spec = result.spec
|
||||
if spec.outputs is None or output_name not in spec.outputs:
|
||||
raise KeyError(f"任务 {task_name!r} 未声明输出 {output_name!r}。")
|
||||
path = spec.outputs[output_name]
|
||||
return _resolve_output_path(result.value, path)
|
||||
|
||||
def __contains__(self, name: Any) -> bool:
|
||||
return name in self.results
|
||||
|
||||
@@ -144,7 +59,6 @@ class RunReport:
|
||||
if r.duration is not None:
|
||||
total_duration += r.duration
|
||||
return {
|
||||
"run_id": self.run_id,
|
||||
"success": self.success,
|
||||
"total_tasks": len(self.results),
|
||||
"by_status": counts,
|
||||
@@ -172,328 +86,10 @@ class RunReport:
|
||||
return {name: (r.duration or 0.0) for name, r in self.results.items()}
|
||||
|
||||
def describe(self) -> str:
|
||||
"""用于调试的人类可读多行报告。
|
||||
|
||||
失败时(``success=False``)在末尾附诊断摘要,便于快速定位根因。
|
||||
"""
|
||||
lines: list[str] = [f"RunReport(run_id={self.run_id}, success={self.success})"]
|
||||
"""用于调试的人类可读多行报告。"""
|
||||
lines: list[str] = [f"RunReport(success={self.success})"]
|
||||
for name, r in self.results.items():
|
||||
dur = f"{r.duration:.3f}s" if r.duration is not None else "-"
|
||||
err = f" error={r.error!r}" if r.error else ""
|
||||
lines.append(f" {name}: {r.status.value} ({dur} attempts={r.attempts}){err}")
|
||||
if not self.success:
|
||||
diag = self.diagnose()
|
||||
if diag is not None:
|
||||
lines.append("")
|
||||
lines.append(diag.describe())
|
||||
return "\n".join(lines)
|
||||
|
||||
# ---- 诊断 --------------------------------------------------------- #
|
||||
def diagnose(self) -> Any:
|
||||
"""生成失败诊断报告。
|
||||
|
||||
运行成功(``success`` 为 ``True``)时返回 ``None``;失败时返回
|
||||
:class:`pyflowx.diagnostics.DiagnosticReport`,包含根因分析、
|
||||
依赖链回溯、相似失败聚类与可操作建议。
|
||||
|
||||
返回类型标注为 ``Any`` 以避免 ``report`` 与 ``diagnostics`` 模块
|
||||
循环导入;运行时返回 ``DiagnosticReport | None``。
|
||||
"""
|
||||
from .diagnostics import diagnose as _diagnose
|
||||
|
||||
return _diagnose(self)
|
||||
|
||||
# ---- 性能剖面 ----------------------------------------------------- #
|
||||
def profile(self, graph: Graph) -> ProfileReport:
|
||||
"""生成性能剖面报告。
|
||||
|
||||
基于 ``started_at``/``finished_at`` 时间戳进行离线分析,零运行时
|
||||
开销。需要传入执行时的 :class:`Graph` 对象以计算依赖关系与关键路径。
|
||||
|
||||
参数
|
||||
----
|
||||
graph:
|
||||
执行 ``self`` 时使用的图。
|
||||
|
||||
返回
|
||||
----
|
||||
:class:`pyflowx.profiling.ProfileReport`
|
||||
|
||||
示例
|
||||
----
|
||||
report = px.run(graph)
|
||||
profile = report.profile(graph)
|
||||
print(profile.describe())
|
||||
"""
|
||||
from .profiling import ProfileReport
|
||||
|
||||
return ProfileReport.from_report(self, graph)
|
||||
|
||||
# ---- 序列化 ------------------------------------------------------- #
|
||||
def to_dict(
|
||||
self,
|
||||
value_serializer: Callable[[Any], Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""转换为 JSON 兼容的字典。
|
||||
|
||||
参数
|
||||
----
|
||||
value_serializer:
|
||||
自定义任务值序列化函数。``None`` 时尝试 JSON 兼容转换,
|
||||
失败回退到 ``repr()``。
|
||||
|
||||
返回结构包含顶层 ``run_id``、``success``、``summary`` 与 ``results``
|
||||
列表(按完成顺序保序),每个结果项含 name/status/value/error/
|
||||
attempts/started_at/finished_at/duration_seconds/reason/tags/
|
||||
depends_on 字段。
|
||||
"""
|
||||
results_list: list[dict[str, Any]] = []
|
||||
for name, r in self.results.items():
|
||||
results_list.append(
|
||||
{
|
||||
"name": name,
|
||||
"status": r.status.value,
|
||||
"value": _serialize_value(r.value, value_serializer),
|
||||
"error": repr(r.error) if r.error else None,
|
||||
"attempts": r.attempts,
|
||||
"started_at": r.started_at.isoformat() if r.started_at else None,
|
||||
"finished_at": r.finished_at.isoformat() if r.finished_at else None,
|
||||
"duration_seconds": r.duration,
|
||||
"reason": r.reason,
|
||||
"tags": list(r.spec.tags),
|
||||
"depends_on": list(r.spec.depends_on),
|
||||
"outputs": dict(r.spec.outputs) if r.spec.outputs else None,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"run_id": self.run_id,
|
||||
"success": self.success,
|
||||
"summary": self.summary(),
|
||||
"results": results_list,
|
||||
}
|
||||
|
||||
def to_json(
|
||||
self,
|
||||
indent: int = 2,
|
||||
value_serializer: Callable[[Any], Any] | None = None,
|
||||
) -> str:
|
||||
"""序列化为 JSON 字符串。
|
||||
|
||||
参数
|
||||
----
|
||||
indent:
|
||||
JSON 缩进空格数,``0`` 表示紧凑单行。
|
||||
value_serializer:
|
||||
自定义任务值序列化函数,透传给 :meth:`to_dict`。
|
||||
"""
|
||||
return dumps(
|
||||
self.to_dict(value_serializer),
|
||||
ensure_ascii=False,
|
||||
indent=indent if indent > 0 else None,
|
||||
)
|
||||
|
||||
def to_csv(self, include_value: bool = True) -> str:
|
||||
"""导出为 CSV 字符串。
|
||||
|
||||
列顺序:``name, status, attempts, duration_seconds, started_at,
|
||||
finished_at, error, reason`` + 可选 ``value``。
|
||||
|
||||
参数
|
||||
----
|
||||
include_value:
|
||||
是否包含 value 列。任务返回值可能含逗号/换行符干扰解析,
|
||||
不需要值时设为 ``False`` 得到更干净输出。
|
||||
"""
|
||||
buf = io.StringIO()
|
||||
writer = csv.writer(buf)
|
||||
header = [
|
||||
"name",
|
||||
"status",
|
||||
"attempts",
|
||||
"duration_seconds",
|
||||
"started_at",
|
||||
"finished_at",
|
||||
"error",
|
||||
"reason",
|
||||
]
|
||||
if include_value:
|
||||
header.append("value")
|
||||
writer.writerow(header)
|
||||
for name, r in self.results.items():
|
||||
row = [
|
||||
name,
|
||||
r.status.value,
|
||||
r.attempts,
|
||||
f"{r.duration:.6f}" if r.duration is not None else "",
|
||||
r.started_at.isoformat() if r.started_at else "",
|
||||
r.finished_at.isoformat() if r.finished_at else "",
|
||||
repr(r.error) if r.error else "",
|
||||
r.reason or "",
|
||||
]
|
||||
if include_value:
|
||||
row.append(_serialize_value(r.value))
|
||||
writer.writerow(row)
|
||||
return buf.getvalue()
|
||||
|
||||
def to_html(
|
||||
self,
|
||||
include_value: bool = True,
|
||||
value_serializer: Callable[[Any], Any] | None = None,
|
||||
) -> str:
|
||||
"""导出为自包含的 HTML 报告字符串。
|
||||
|
||||
生成完整的 HTML5 文档,内联 CSS(零外部依赖),适合直接在浏览器
|
||||
打开或邮件附件分享。所有用户内容(任务名、错误、原因、值)均经
|
||||
:func:`html.escape` 转义,避免 XSS。
|
||||
|
||||
参数
|
||||
----
|
||||
include_value:
|
||||
是否在表格中展示任务返回值。值可能较大或含特殊字符,
|
||||
不需要时设为 ``False`` 得到更紧凑报告。
|
||||
value_serializer:
|
||||
自定义任务值序列化函数,透传给 :func:`_serialize_value`。
|
||||
``None`` 时尝试 JSON 兼容转换,失败回退到 ``repr()``。
|
||||
"""
|
||||
summary = self.summary()
|
||||
status_badge_class = {
|
||||
TaskStatus.SUCCESS: "status-success",
|
||||
TaskStatus.FAILED: "status-failed",
|
||||
TaskStatus.SKIPPED: "status-skipped",
|
||||
TaskStatus.RUNNING: "status-running",
|
||||
TaskStatus.PENDING: "status-pending",
|
||||
}
|
||||
parts: list[str] = [
|
||||
"<!DOCTYPE html>",
|
||||
'<html lang="zh-CN">',
|
||||
"<head>",
|
||||
'<meta charset="utf-8">',
|
||||
'<meta name="viewport" content="width=device-width, initial-scale=1">',
|
||||
f"<title>PyFlowX 运行报告 {html.escape(self.run_id)}</title>",
|
||||
"<style>",
|
||||
"body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;"
|
||||
"margin:0;padding:20px;background:#f5f5f5;color:#222;}",
|
||||
".container{max-width:1100px;margin:0 auto;}",
|
||||
".summary{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));"
|
||||
"gap:12px;margin-bottom:24px;}",
|
||||
".card{background:#fff;border-radius:8px;padding:16px;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
|
||||
".card h3{margin:0 0 8px;font-size:13px;color:#666;font-weight:600;text-transform:uppercase;}",
|
||||
".card .value{font-size:22px;font-weight:700;color:#222;}",
|
||||
".card.success .value{color:#16a34a;}",
|
||||
".card.failure .value{color:#dc2626;}",
|
||||
"table{width:100%;border-collapse:collapse;background:#fff;border-radius:8px;"
|
||||
"overflow:hidden;box-shadow:0 1px 3px rgba(0,0,0,.1);}",
|
||||
"th,td{padding:10px 12px;text-align:left;border-bottom:1px solid #eee;font-size:13px;}",
|
||||
"th{background:#fafafa;font-weight:600;color:#555;}",
|
||||
"tr:last-child td{border-bottom:none;}",
|
||||
"td.name{font-family:monospace;font-weight:600;}",
|
||||
".badge{display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;}",
|
||||
".status-success{background:#dcfce7;color:#16a34a;}",
|
||||
".status-failed{background:#fee2e2;color:#dc2626;}",
|
||||
".status-skipped{background:#fef9c3;color:#ca8a04;}",
|
||||
".status-running{background:#dbeafe;color:#2563eb;}",
|
||||
".status-pending{background:#f3f4f6;color:#6b7280;}",
|
||||
".error{color:#dc2626;font-family:monospace;font-size:12px;}",
|
||||
".reason{color:#ca8a04;font-size:12px;}",
|
||||
".value-cell{font-family:monospace;font-size:12px;max-width:300px;"
|
||||
"overflow:hidden;text-overflow:ellipsis;white-space:nowrap;}",
|
||||
".empty{padding:32px;text-align:center;color:#999;}",
|
||||
"</style>",
|
||||
"</head>",
|
||||
"<body>",
|
||||
'<div class="container">',
|
||||
f"<h1>PyFlowX 运行报告 <code>{html.escape(self.run_id)}</code></h1>",
|
||||
'<div class="summary">',
|
||||
self._render_summary_card(
|
||||
"运行状态", "成功" if self.success else "失败", "success" if self.success else "failure"
|
||||
),
|
||||
self._render_summary_card("任务总数", str(summary["total_tasks"])),
|
||||
self._render_summary_card("总耗时", f"{summary['total_duration_seconds']:.3f}s"),
|
||||
self._render_summary_card("成功", str(summary["by_status"].get("success", 0))),
|
||||
self._render_summary_card("失败", str(summary["by_status"].get("failed", 0))),
|
||||
self._render_summary_card("跳过", str(summary["by_status"].get("skipped", 0))),
|
||||
"</div>", # .summary
|
||||
]
|
||||
|
||||
if not self.results:
|
||||
parts.append('<div class="empty">本次运行无任务结果。</div>')
|
||||
else:
|
||||
header_cells = [
|
||||
"<th>任务名</th>",
|
||||
"<th>状态</th>",
|
||||
"<th>尝试次数</th>",
|
||||
"<th>耗时(s)</th>",
|
||||
"<th>开始时间</th>",
|
||||
"<th>结束时间</th>",
|
||||
"<th>错误</th>",
|
||||
"<th>原因</th>",
|
||||
]
|
||||
if include_value:
|
||||
header_cells.append("<th>返回值</th>")
|
||||
parts.append("<table>")
|
||||
parts.append("<thead><tr>" + "".join(header_cells) + "</tr></thead>")
|
||||
parts.append("<tbody>")
|
||||
for name, r in self.results.items():
|
||||
badge_cls = status_badge_class.get(r.status, "status-pending")
|
||||
cells = [
|
||||
f'<td class="name">{html.escape(name)}</td>',
|
||||
f'<td><span class="badge {badge_cls}">{html.escape(r.status.value)}</span></td>',
|
||||
f"<td>{r.attempts}</td>",
|
||||
f"<td>{r.duration:.6f}</td>" if r.duration is not None else "<td>-</td>",
|
||||
f"<td>{html.escape(r.started_at.isoformat())}</td>" if r.started_at else "<td>-</td>",
|
||||
f"<td>{html.escape(r.finished_at.isoformat())}</td>" if r.finished_at else "<td>-</td>",
|
||||
f'<td class="error">{html.escape(repr(r.error))}</td>' if r.error else "<td>-</td>",
|
||||
f'<td class="reason">{html.escape(r.reason)}</td>' if r.reason else "<td>-</td>",
|
||||
]
|
||||
if include_value:
|
||||
val = _serialize_value(r.value, value_serializer)
|
||||
val_str = dumps(val, ensure_ascii=False, default=str) if val is not None else "-"
|
||||
cells.append(f'<td class="value-cell">{html.escape(val_str)}</td>')
|
||||
parts.append("<tr>" + "".join(cells) + "</tr>")
|
||||
parts.append("</tbody></table>")
|
||||
|
||||
parts.extend(["</div>", "</body>", "</html>"])
|
||||
return "\n".join(parts)
|
||||
|
||||
@staticmethod
|
||||
def _render_summary_card(label: str, value: str, modifier: str = "") -> str:
|
||||
"""渲染单个汇总卡片 HTML 片段。"""
|
||||
cls = f"card {modifier}" if modifier else "card"
|
||||
return f'<div class="{cls}"><h3>{html.escape(label)}</h3><div class="value">{html.escape(value)}</div></div>'
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, text: str) -> RunReport:
|
||||
"""从 JSON 字符串重建 :class:`RunReport`。
|
||||
|
||||
注意
|
||||
----
|
||||
只能重建可序列化字段;:class:`TaskSpec` 的 ``fn``/``cmd`` 等
|
||||
不可序列化字段无法恢复,重建的 spec 仅含 ``name``/``tags``/
|
||||
``depends_on``。重建后的 report 仅供查询/分析,**不能用于重新执行**。
|
||||
"""
|
||||
data = loads(text)
|
||||
report = cls(
|
||||
success=data.get("success", True),
|
||||
run_id=data.get("run_id", uuid4().hex[:8]),
|
||||
)
|
||||
for entry in data.get("results", []):
|
||||
spec: TaskSpec[Any] = TaskSpec(
|
||||
name=entry["name"],
|
||||
fn=_noop_fn,
|
||||
tags=tuple(entry.get("tags", ())),
|
||||
depends_on=tuple(entry.get("depends_on", ())),
|
||||
outputs=dict(entry["outputs"]) if entry.get("outputs") else None,
|
||||
)
|
||||
started = entry.get("started_at")
|
||||
finished = entry.get("finished_at")
|
||||
result: TaskResult[Any] = TaskResult(
|
||||
spec=spec,
|
||||
status=TaskStatus(entry["status"]),
|
||||
value=entry.get("value"),
|
||||
attempts=entry.get("attempts", 0),
|
||||
started_at=datetime.fromisoformat(started) if started else None,
|
||||
finished_at=datetime.fromisoformat(finished) if finished else None,
|
||||
reason=entry.get("reason"),
|
||||
)
|
||||
report.results[spec.name] = result
|
||||
return report
|
||||
|
||||
+40
-4
@@ -14,10 +14,9 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import enum
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass, field, replace
|
||||
from pathlib import Path
|
||||
from typing import Any, get_args
|
||||
from typing import Any, Sequence, get_args
|
||||
|
||||
from .compose import GraphComposer
|
||||
from .errors import PyFlowXError
|
||||
@@ -36,6 +35,39 @@ class CliExitCode(enum.IntEnum):
|
||||
INTERRUPTED = 130 # 与 POSIX 信号中断一致
|
||||
|
||||
|
||||
def _apply_verbose_to_graph(graph: Graph, verbose: bool) -> Graph:
|
||||
"""创建新图, 其中所有 TaskSpec 的 verbose 字段被设置为指定值.
|
||||
|
||||
使用 ``dataclasses.replace`` 在不可变的 TaskSpec 上创建带 verbose 标记的副本.
|
||||
依赖关系、标签等元数据全部保留.
|
||||
|
||||
Note
|
||||
-----
|
||||
自 ``_wrap_cmd`` 不再闭包捕获 ``verbose`` 后,此函数不再是必需的——
|
||||
直接翻转 ``spec.verbose`` 即可生效。保留是为了向后兼容现有调用与测试。
|
||||
TaskSpec 仍是 frozen dataclass,故仍用 ``replace`` 创建副本。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
graph : Graph
|
||||
原始图.
|
||||
verbose : bool
|
||||
要设置的 verbose 值.
|
||||
|
||||
Returns
|
||||
-------
|
||||
Graph
|
||||
所有 spec 的 verbose 字段已更新的新图.
|
||||
"""
|
||||
new_specs: list[TaskSpec[Any]] = []
|
||||
for spec in graph.all_specs().values():
|
||||
if spec.verbose == verbose:
|
||||
new_specs.append(spec)
|
||||
else:
|
||||
new_specs.append(replace(spec, verbose=verbose))
|
||||
return Graph.from_specs(new_specs)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CliRunner:
|
||||
"""命令行运行器: 根据用户输入执行对应的任务流图.
|
||||
@@ -264,8 +296,12 @@ class CliRunner:
|
||||
# 确定是否 verbose: --quiet 覆盖默认值
|
||||
verbose = self.verbose and not parsed.quiet
|
||||
|
||||
# 执行对应的图 (verbose 标记由 run() 统一应用到各 spec)
|
||||
# 对图应用 verbose 设置 (重建带 verbose 标记的 spec)
|
||||
graph = self.graphs[parsed.command]
|
||||
if verbose:
|
||||
graph = _apply_verbose_to_graph(graph, verbose=True)
|
||||
|
||||
# 执行对应的图
|
||||
try:
|
||||
report = run(
|
||||
graph,
|
||||
|
||||
+15
-218
@@ -13,23 +13,20 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from abc import ABC, abstractmethod
|
||||
from collections import OrderedDict
|
||||
from collections.abc import Iterator, Mapping
|
||||
from contextlib import AbstractContextManager, contextmanager, nullcontext
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager, nullcontext
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, ContextManager, Mapping
|
||||
|
||||
if sys.version_info >= (3, 12):
|
||||
from typing import override
|
||||
else:
|
||||
from typing_extensions import override # pragma: no cover
|
||||
|
||||
from ._json import JSONDecodeError, dump, dumps, load, loads
|
||||
from .errors import StorageError
|
||||
|
||||
|
||||
@@ -59,14 +56,6 @@ class StateBackend(ABC):
|
||||
def clear(self) -> None:
|
||||
"""清除所有存储状态。"""
|
||||
|
||||
@abstractmethod
|
||||
def invalidate(self, key: str) -> bool:
|
||||
"""删除单个键的存储结果。
|
||||
|
||||
返回 ``True`` 表示键存在且已删除,``False`` 表示键不存在。
|
||||
与 :meth:`clear` 不同,仅删除指定键而非全部。
|
||||
"""
|
||||
|
||||
def flush(self) -> None: # noqa: B027
|
||||
"""将内存中暂存的状态持久化到外部介质。
|
||||
|
||||
@@ -74,7 +63,7 @@ class StateBackend(ABC):
|
||||
:class:`JSONBackend` 在 :meth:`batch` 期间会延迟落盘,需在退出时调用。
|
||||
"""
|
||||
|
||||
def batch(self) -> AbstractContextManager[None]:
|
||||
def batch(self) -> ContextManager[None]:
|
||||
"""返回一个上下文管理器,期间 :meth:`save` 可延迟 :meth:`flush`。
|
||||
|
||||
默认实现为 no-op(如 :class:`MemoryBackend`)。:class:`JSONBackend`
|
||||
@@ -115,10 +104,6 @@ class _TTLStateBackendMixin(StateBackend):
|
||||
def _clear_raw(self) -> None:
|
||||
"""清空所有记录。"""
|
||||
|
||||
@abstractmethod
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
"""删除单条记录。返回 ``True`` 表示存在且已删除,``False`` 表示不存在。"""
|
||||
|
||||
# ---- 共享实现 ---- #
|
||||
def _now(self) -> float:
|
||||
"""当前时间戳,默认为 wall-clock 秒。"""
|
||||
@@ -154,10 +139,6 @@ class _TTLStateBackendMixin(StateBackend):
|
||||
def clear(self) -> None:
|
||||
self._clear_raw()
|
||||
|
||||
@override
|
||||
def invalidate(self, key: str) -> bool:
|
||||
return self._delete_raw(key)
|
||||
|
||||
|
||||
class MemoryBackend(_TTLStateBackendMixin):
|
||||
"""进程内 dict 后端。进程退出即丢失。
|
||||
@@ -167,20 +148,11 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
ttl:
|
||||
条目存活秒数。``None`` 表示永不过期。``has`` 在条目超过 ttl 后
|
||||
返回 ``False``(但不主动删除,下次 ``save`` 覆盖)。
|
||||
maxsize:
|
||||
最大条目数。``None`` 表示不限。超出时按 LRU(最近最少使用)
|
||||
策略驱逐最旧条目。每次 ``get``/``has`` 命中会将条目移到末尾
|
||||
(最近使用),``save`` 新增条目时若超限则驱逐头部条目。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ttl: float | None = None,
|
||||
maxsize: int | None = None,
|
||||
) -> None:
|
||||
self._store: OrderedDict[str, tuple[Any, float]] = OrderedDict()
|
||||
def __init__(self, ttl: float | None = None) -> None:
|
||||
self._store: dict[str, tuple[Any, float]] = {}
|
||||
self._ttl = ttl
|
||||
self._maxsize = maxsize
|
||||
|
||||
@override
|
||||
def _now(self) -> float:
|
||||
@@ -188,23 +160,11 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
|
||||
@override
|
||||
def _get_raw(self, key: str) -> tuple[Any, float] | None:
|
||||
entry = self._store.get(key)
|
||||
if entry is None:
|
||||
return None
|
||||
# LRU:命中时移到末尾(最近使用)
|
||||
self._store.move_to_end(key)
|
||||
return entry
|
||||
return self._store.get(key)
|
||||
|
||||
@override
|
||||
def _put_raw(self, key: str, value: Any, ts: float) -> None:
|
||||
if key in self._store:
|
||||
# 已存在:更新并移到末尾
|
||||
self._store.move_to_end(key)
|
||||
self._store[key] = (value, ts)
|
||||
# LRU 驱逐:超限时移除头部(最旧)
|
||||
if self._maxsize is not None:
|
||||
while len(self._store) > self._maxsize:
|
||||
self._store.popitem(last=False)
|
||||
|
||||
@override
|
||||
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
|
||||
@@ -215,13 +175,6 @@ class MemoryBackend(_TTLStateBackendMixin):
|
||||
def _clear_raw(self) -> None:
|
||||
self._store.clear()
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
if key in self._store:
|
||||
del self._store[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class JSONBackend(_TTLStateBackendMixin):
|
||||
"""基于文件的 JSON 存储,用于跨进程续跑。
|
||||
@@ -249,7 +202,7 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
return
|
||||
try:
|
||||
with open(self._path, encoding="utf-8") as fh:
|
||||
data: Any = load(fh)
|
||||
data: Any = json.load(fh)
|
||||
if isinstance(data, dict):
|
||||
# 兼容纯值格式与带元数据格式
|
||||
self._store = {}
|
||||
@@ -258,14 +211,14 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
self._store[k] = v
|
||||
else:
|
||||
self._store[k] = {"value": v, "ts": time.time()}
|
||||
except (OSError, JSONDecodeError) as exc:
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise StorageError(f"cannot read state file {self._path!r}", exc) from exc
|
||||
|
||||
def _flush(self) -> None:
|
||||
tmp = self._path + ".tmp"
|
||||
try:
|
||||
with open(tmp, "w", encoding="utf-8") as fh:
|
||||
dump(self._store, fh, ensure_ascii=False, indent=2)
|
||||
json.dump(self._store, fh, ensure_ascii=False, indent=2)
|
||||
_ = Path(tmp).replace(Path(self._path))
|
||||
except (OSError, TypeError) as exc:
|
||||
raise StorageError(f"cannot write state file {self._path!r}", exc) from exc
|
||||
@@ -290,34 +243,17 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
def _clear_raw(self) -> None:
|
||||
self._store.clear()
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
if key in self._store:
|
||||
del self._store[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
@override
|
||||
def clear(self) -> None:
|
||||
super().clear()
|
||||
self._flush()
|
||||
|
||||
@override
|
||||
def invalidate(self, key: str) -> bool:
|
||||
removed = super().invalidate(key)
|
||||
if removed and not self._defer_flush:
|
||||
self._flush()
|
||||
return removed
|
||||
|
||||
@override
|
||||
def save(self, key: str, value: Any) -> None:
|
||||
# batch 模式下延迟序列化验证到 flush 时(一次性 json.dump 整个 _store),
|
||||
# 避免 N 次 save 的 N 次 json.dumps 开销;非 batch 模式仍即时验证以提供精确错误。
|
||||
if not self._defer_flush:
|
||||
try:
|
||||
_ = dumps(value)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
|
||||
try:
|
||||
_ = json.dumps(value)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
|
||||
super().save(key, value)
|
||||
if not self._defer_flush:
|
||||
self._flush()
|
||||
@@ -332,8 +268,6 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
|
||||
|
||||
将整次运行 N 个任务的 N 次全量落盘降为 1 次。
|
||||
batch 期间 ``save`` 跳过逐次 JSON 序列化验证,由 flush 时的
|
||||
``json.dump`` 统一捕获不可序列化的值。
|
||||
"""
|
||||
self._defer_flush = True
|
||||
try:
|
||||
@@ -343,143 +277,6 @@ class JSONBackend(_TTLStateBackendMixin):
|
||||
self._flush()
|
||||
|
||||
|
||||
class SQLiteBackend(_TTLStateBackendMixin):
|
||||
"""基于 SQLite 文件的状态后端。
|
||||
|
||||
相比 :class:`JSONBackend` 的全量加载/落盘,:class:`SQLiteBackend` 按键
|
||||
查询、按行写入,适合大规模任务状态(10k+ 条目)。使用标准库
|
||||
:mod:`sqlite3`,无新依赖。
|
||||
|
||||
值以 JSON 文本存入 ``value`` 列,因此与 :class:`JSONBackend` 一样要求
|
||||
结果可 JSON 序列化。``ts`` 列存储写入时间戳用于 TTL 判断。
|
||||
|
||||
线程安全:内部用 :class:`threading.RLock` 序列化所有 sqlite3 调用
|
||||
(sqlite3 连接对象本身非线程安全),支持 ``thread``/``async``/``dependency``
|
||||
策略下的并发存取。
|
||||
|
||||
Parameters
|
||||
----------
|
||||
path:
|
||||
SQLite 文件路径。``":memory:"`` 创建内存数据库(进程退出丢失,
|
||||
适合测试)。
|
||||
ttl:
|
||||
条目存活秒数。``None`` 表示永不过期。``has`` 在条目超过 ttl 后
|
||||
返回 ``False``(但不主动删除,下次 ``save`` 覆盖)。
|
||||
"""
|
||||
|
||||
def __init__(self, path: str, ttl: float | None = None) -> None:
|
||||
self._path = path
|
||||
self._ttl = ttl
|
||||
self._lock = threading.RLock()
|
||||
self._in_batch = False
|
||||
try:
|
||||
self._conn = sqlite3.connect(path, check_same_thread=False)
|
||||
with self._lock:
|
||||
self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
self._conn.execute("PRAGMA synchronous=NORMAL")
|
||||
self._conn.execute(
|
||||
"CREATE TABLE IF NOT EXISTS state (key TEXT PRIMARY KEY, value TEXT NOT NULL, ts REAL NOT NULL)"
|
||||
)
|
||||
self._conn.execute("CREATE INDEX IF NOT EXISTS idx_state_ts ON state(ts)")
|
||||
self._conn.commit()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError(f"cannot open sqlite state file {path!r}", exc) from exc
|
||||
|
||||
@override
|
||||
def _get_raw(self, key: str) -> tuple[Any, float] | None:
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute("SELECT value, ts FROM state WHERE key = ?", (key,)).fetchone()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError(f"cannot read key {key!r} from sqlite state", exc) from exc
|
||||
if row is None:
|
||||
return None
|
||||
value_text, ts = row
|
||||
try:
|
||||
value = loads(value_text)
|
||||
except JSONDecodeError as exc:
|
||||
raise StorageError(f"cannot decode value for key {key!r}", exc) from exc
|
||||
return value, float(ts)
|
||||
|
||||
@override
|
||||
def _put_raw(self, key: str, value: Any, ts: float) -> None:
|
||||
try:
|
||||
value_text = dumps(value, ensure_ascii=False)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
|
||||
with self._lock:
|
||||
try:
|
||||
self._conn.execute(
|
||||
"INSERT OR REPLACE INTO state (key, value, ts) VALUES (?, ?, ?)",
|
||||
(key, value_text, ts),
|
||||
)
|
||||
if not self._in_batch:
|
||||
self._conn.commit()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError(f"cannot write key {key!r} to sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
|
||||
with self._lock:
|
||||
try:
|
||||
rows = self._conn.execute("SELECT key, value, ts FROM state").fetchall()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError("cannot iterate sqlite state", exc) from exc
|
||||
for k, value_text, ts in rows:
|
||||
try:
|
||||
value = loads(value_text)
|
||||
except JSONDecodeError as exc:
|
||||
raise StorageError(f"cannot decode value for key {k!r}", exc) from exc
|
||||
yield k, value, float(ts)
|
||||
|
||||
@override
|
||||
def _clear_raw(self) -> None:
|
||||
with self._lock:
|
||||
try:
|
||||
self._conn.execute("DELETE FROM state")
|
||||
self._conn.commit()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError("cannot clear sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
def _delete_raw(self, key: str) -> bool:
|
||||
with self._lock:
|
||||
try:
|
||||
cursor = self._conn.execute("DELETE FROM state WHERE key = ?", (key,))
|
||||
if not self._in_batch:
|
||||
self._conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError(f"cannot delete key {key!r} from sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
def flush(self) -> None:
|
||||
with self._lock:
|
||||
try:
|
||||
self._conn.commit()
|
||||
except sqlite3.DatabaseError as exc:
|
||||
raise StorageError("cannot flush sqlite state", exc) from exc
|
||||
|
||||
@override
|
||||
@contextmanager
|
||||
def batch(self) -> Iterator[None]:
|
||||
"""进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。
|
||||
|
||||
将整次运行 N 个任务的 N 次 commit 降为 1 次。
|
||||
"""
|
||||
self._in_batch = True
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
self._in_batch = False
|
||||
self.flush()
|
||||
|
||||
def close(self) -> None:
|
||||
"""关闭数据库连接,释放文件句柄。"""
|
||||
with self._lock:
|
||||
self._conn.close()
|
||||
|
||||
|
||||
def resolve_backend(backend: StateBackend | None) -> StateBackend:
|
||||
"""返回 ``backend``;为 ``None`` 时返回新的 :class:`MemoryBackend`。"""
|
||||
return backend if backend is not None else MemoryBackend()
|
||||
|
||||
+28
-87
@@ -22,15 +22,22 @@ import os
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
from collections.abc import Callable, Coroutine, Generator, Iterable, Mapping
|
||||
from contextlib import AbstractContextManager, contextmanager
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from functools import cached_property
|
||||
from pathlib import Path
|
||||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
ContextManager,
|
||||
Coroutine,
|
||||
Generator,
|
||||
Generic,
|
||||
List,
|
||||
Mapping,
|
||||
Union,
|
||||
cast,
|
||||
)
|
||||
|
||||
@@ -42,18 +49,25 @@ else:
|
||||
T = TypeVar("T", default=Any)
|
||||
|
||||
# 任务可调用对象可以是同步或异步的。显式保留联合类型,让 mypy 理解两种形态。
|
||||
TaskFn = Callable[..., T] | Callable[..., Coroutine[Any, Any, T]]
|
||||
TaskFn = Union[
|
||||
Callable[..., T],
|
||||
Callable[..., Coroutine[Any, Any, T]],
|
||||
]
|
||||
|
||||
# 跨任务结果映射。值刻意使用 ``Any``,因为不同任务返回不同类型;
|
||||
# 单任务类型由函数签名本身保留。
|
||||
Context = Mapping[str, Any]
|
||||
|
||||
# 命令类型支持
|
||||
TaskCmd = list[str] | str | Callable[..., Any]
|
||||
TaskCmd = Union[
|
||||
List[str], # 命令列表, 如 ["ls", "-la"]
|
||||
str, # shell 命令字符串
|
||||
Callable[..., Any], # Python 函数
|
||||
]
|
||||
|
||||
# 执行策略:sequential/thread/async 为层屏障模型,dependency 为依赖驱动模型。
|
||||
Strategy = Union[str, "StrategyKind"]
|
||||
StrategyKind = Any # 占位,避免循环;executors 模块用 Literal 约束
|
||||
Strategy = str | StrategyKind
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -64,65 +78,6 @@ Condition = Callable[[Context], bool]
|
||||
CacheKeyFn = Callable[[Context], str]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 任务循环
|
||||
# ---------------------------------------------------------------------- #
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LoopSpec:
|
||||
"""任务循环展开规格。
|
||||
|
||||
添加到 :class:`Graph` 时,``loop`` 非 ``None`` 的 :class:`TaskSpec`
|
||||
会被展开为多实例:每个 ``item`` 生成一个新 spec,``item`` 追加到
|
||||
``args`` 末尾,``loop`` 字段清空。
|
||||
|
||||
展开后的实例名默认为 ``f"{name}_{i}"``(``i`` 从 0 起的索引),
|
||||
也可通过 ``key_fn`` 自定义。其他任务引用 loop 原名时,``Graph``
|
||||
会自动重写为依赖所有展开实例。
|
||||
|
||||
参数
|
||||
----
|
||||
items:
|
||||
待迭代的项元组。
|
||||
key_fn:
|
||||
接受 ``(index, item)``,返回实例名。``None`` 时用
|
||||
``f"{name}_{i}"``。
|
||||
|
||||
Examples
|
||||
--------
|
||||
>>> from pyflowx import LoopSpec
|
||||
>>> spec = px.task(process, loop=LoopSpec.range(3))
|
||||
>>> # 等价于手动创建 process_0, process_1, process_2 三个任务
|
||||
"""
|
||||
|
||||
items: tuple[Any, ...]
|
||||
key_fn: Callable[[int, Any], str] | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.items:
|
||||
raise ValueError("LoopSpec.items 不能为空。")
|
||||
|
||||
@classmethod
|
||||
def range(cls, *args: int) -> LoopSpec:
|
||||
"""构造整数序列循环(签名与 :func:`range` 一致)。
|
||||
|
||||
支持三种形式:
|
||||
- ``range(stop)`` → ``[0, 1, ..., stop-1]``
|
||||
- ``range(start, stop)`` → ``[start, ..., stop-1]``
|
||||
- ``range(start, stop, step)`` → 按步长递增/递减
|
||||
"""
|
||||
# classmethod 内的 ``range`` 查找内置函数,不引用本类 LoopSpec.range
|
||||
return cls(items=tuple(range(*args)))
|
||||
|
||||
@classmethod
|
||||
def from_iterable(
|
||||
cls,
|
||||
items: Iterable[Any],
|
||||
key_fn: Callable[[int, Any], str] | None = None,
|
||||
) -> LoopSpec:
|
||||
"""从可迭代对象构造循环。"""
|
||||
return cls(items=tuple(items), key_fn=key_fn)
|
||||
|
||||
|
||||
def _format_skip_reason(failed_conditions: list[str]) -> str:
|
||||
"""格式化跳过原因:≤2 个全展示,>2 个仅展示前 2 个并附总数。"""
|
||||
if len(failed_conditions) <= 2:
|
||||
@@ -133,7 +88,7 @@ def _format_skip_reason(failed_conditions: list[str]) -> str:
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 重试策略
|
||||
# ---------------------------------------------------------------------- #
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@dataclass(frozen=True)
|
||||
class RetryPolicy:
|
||||
"""任务失败重试策略。
|
||||
|
||||
@@ -196,7 +151,7 @@ class RetryPolicy:
|
||||
# ---------------------------------------------------------------------- #
|
||||
# 任务钩子
|
||||
# ---------------------------------------------------------------------- #
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@dataclass(frozen=True)
|
||||
class TaskHooks:
|
||||
"""任务生命周期钩子。
|
||||
|
||||
@@ -220,7 +175,7 @@ class TaskStatus(Enum):
|
||||
SKIPPED = "skipped" # 用于断点续跑与子图过滤
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@dataclass(frozen=True)
|
||||
class TaskSpec(Generic[T]):
|
||||
"""单个 DAG 节点的不可变描述。
|
||||
|
||||
@@ -303,9 +258,6 @@ class TaskSpec(Generic[T]):
|
||||
同步任务的执行器:``"thread"``(默认,线程池)/ ``"process"``
|
||||
(进程池,绕过 GIL,适合 CPU 密集型;``fn`` 须可 pickle)/
|
||||
``"inline"``(直接在事件循环线程调用,最快但会阻塞循环)。
|
||||
outputs:
|
||||
任务声明的输出键值映射(YAML 编排场景使用)。仅作为元数据携带,
|
||||
不参与执行;可通过 ``report.result_of(name).spec.outputs`` 查询。
|
||||
"""
|
||||
|
||||
name: str
|
||||
@@ -332,9 +284,6 @@ class TaskSpec(Generic[T]):
|
||||
cache_key: CacheKeyFn | None = None
|
||||
hooks: TaskHooks = field(default_factory=TaskHooks)
|
||||
executor: str = "thread" # "thread" | "process" | "inline"
|
||||
outputs: Mapping[str, str] | None = None
|
||||
loop: LoopSpec | None = None
|
||||
dynamic: bool = False
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
@@ -351,7 +300,7 @@ class TaskSpec(Generic[T]):
|
||||
if self.fn is None and self.cmd is None:
|
||||
raise ValueError(f"TaskSpec '{self.name}': 必须提供 fn 或 cmd 参数。")
|
||||
|
||||
@property
|
||||
@cached_property
|
||||
def effective_fn(self) -> TaskFn[T]:
|
||||
"""获取有效的执行函数。
|
||||
|
||||
@@ -359,10 +308,8 @@ class TaskSpec(Generic[T]):
|
||||
包装函数在每次调用时从 ``self`` 读取 ``verbose``/``cwd``/``env``/
|
||||
``timeout``,避免闭包捕获运行期参数,使翻转字段无需重建 spec。
|
||||
|
||||
.. note::
|
||||
使用 ``@property`` 而非 ``cached_property`` 以兼容 ``slots=True``
|
||||
内存优化(10k+ 任务场景)。闭包创建极轻量(仅捕获 ``self``),
|
||||
重复访问的开销可忽略。
|
||||
结果按实例缓存(:func:`functools.cached_property`):frozen dataclass
|
||||
字段不可变,``_wrap_cmd`` 生成的闭包稳定,无需每次访问重建。
|
||||
"""
|
||||
if self.cmd is not None:
|
||||
return self._wrap_cmd()
|
||||
@@ -429,7 +376,7 @@ class TaskSpec(Generic[T]):
|
||||
return shutil.which(cmd[0]) is not None
|
||||
return True
|
||||
|
||||
def env_context(self) -> AbstractContextManager[None]:
|
||||
def env_context(self) -> ContextManager[None]:
|
||||
"""返回临时应用 ``env`` 与 ``cwd`` 的上下文管理器。
|
||||
|
||||
对 ``fn`` 任务生效。``cmd`` 任务在 :func:`_run_command` 中直接
|
||||
@@ -534,9 +481,6 @@ def task(
|
||||
continue_on_error: bool = False,
|
||||
cache_key: CacheKeyFn | None = None,
|
||||
hooks: TaskHooks | None = None,
|
||||
outputs: Mapping[str, str] | None = None,
|
||||
loop: LoopSpec | None = None,
|
||||
dynamic: bool = False,
|
||||
name: str | None = None,
|
||||
) -> Any:
|
||||
"""装饰器:将函数转为 :class:`TaskSpec`。
|
||||
@@ -578,9 +522,6 @@ def task(
|
||||
continue_on_error=continue_on_error,
|
||||
cache_key=cache_key,
|
||||
hooks=hooks if hooks is not None else TaskHooks(),
|
||||
outputs=dict(outputs) if outputs else None,
|
||||
loop=loop,
|
||||
dynamic=dynamic,
|
||||
)
|
||||
|
||||
if fn is None and cmd is None:
|
||||
@@ -656,7 +597,7 @@ def task_template(
|
||||
return _factory
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@dataclass
|
||||
class TaskResult(Generic[T]):
|
||||
"""运行期间产生的可变单任务记录。"""
|
||||
|
||||
@@ -677,7 +618,7 @@ class TaskResult(Generic[T]):
|
||||
return (self.finished_at - self.started_at).total_seconds()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@dataclass(frozen=True)
|
||||
class TaskEvent:
|
||||
"""执行期间向观察者发出的不可变事件。"""
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user