Compare commits

..

12 Commits

Author SHA1 Message Date
zhou 3b52b047d2 perf: 调度器热路径深度优化,线程池复用与 4 项快速路径,sequential +12.6%/async +10.6%/thread +7.2%/dependency +4.6%
CI / Lint, Typecheck & Test (push) Successful in 3m35s
2026-07-09 08:06:42 +08:00
zhou 529f94882c feat: 扩展 px.sh 支持 check=False 和 timeout 参数,完成 ops/ 下剩余 subprocess.run 迁移
CI / Lint, Typecheck & Test (push) Failing after 3m26s
2026-07-09 07:53:03 +08:00
zhou 51b69a47bc refactor: 迁移 ops/ 下 31 处 subprocess.run(check=True) 到 px.sh,统一错误处理
CI / Lint, Typecheck & Test (push) Failing after 32s
2026-07-08 23:25:01 +08:00
zhou 9f87f86189 bump version to 0.4.11
CI / Lint, Typecheck & Test (push) Has been cancelled
Release / Build, Publish & Release (push) Failing after 22s
2026-07-08 21:53:24 +08:00
zhou 9ac68ddb2a fix: 修复 CI 环境下 bumpversion -h 帮助输出 ANSI 转义码导致断言失败 2026-07-08 21:53:12 +08:00
zhou 2d7bbd1a1f fix: 修复 CI 环境下 bumpversion -h 帮助输出 ANSI 转义码导致断言失败
CI / Lint, Typecheck & Test (push) Has been cancelled
2026-07-08 21:52:34 +08:00
zhou 18f4ee2957 bump version to 0.4.10
Release / Build, Publish & Release (push) Successful in 1m21s
CI / Lint, Typecheck & Test (push) Has been cancelled
2026-07-08 21:37:30 +08:00
zhou e0c9b0c821 fix: 修复 py310 下 TaskSpec[Any]() 触发的 super() TypeError,CI 测试环境改为 py310,py314
CI / Lint, Typecheck & Test (push) Has been cancelled
2026-07-08 21:36:16 +08:00
zhou 8225f2fa3b fix: 移除 PLR0913 全局 ignore,重构 executors 内部函数使用 _ExecContext 数据类减少参数
CI / Lint, Typecheck & Test (push) Has been cancelled
2026-07-08 21:17:14 +08:00
zhou c5878e75ff chore: 扩展 tox 测试依赖并新增 pyflowx 脚本别名 2026-07-08 21:17:03 +08:00
zhou 5c7d27f682 chore: 版本号升级至0.4.9并优化代码
CI / Lint, Typecheck & Test (push) Has been cancelled
1. 调整storage.py里的batch方法返回类型为Generator
2. 格式化测试文件里的Graph.from_specs调用代码
2026-07-08 15:59:12 +08:00
zhou bbeba918cf test: 新增 pf pymake -h 和 pf bumpversion CLI 调度测试
CI / Lint, Typecheck & Test (push) Has been cancelled
使用 --dry-run 避免执行真实 bump;回归测试验证 bumpversion/bumpmi 的 cmd 使用 --part 选项而非位置参数。
2026-07-08 15:41:35 +08:00
54 changed files with 1140 additions and 638 deletions
+2 -2
View File
@@ -30,5 +30,5 @@ jobs:
- name: Ruff check - name: Ruff check
run: ruff check src tests run: ruff check src tests
- name: Tox test (py38, py313) - name: Tox test (py310, py314)
run: uvx tox run -e py38,py313 run: uvx tox run -e py310,py314
@@ -0,0 +1,63 @@
# 迭代 29: subprocess.run → px.sh 迁移
## 本轮目标
`ops/` 目录下所有 `subprocess.run(cmd, check=True)` 调用迁移到 `px.sh` 封装,统一错误处理与中文输出。
## 改动文件清单
### 源文件 (9 个)
| 文件 | 迁移调用数 | 保留调用数 | 保留原因 |
|------|-----------|-----------|---------|
| dev/gittool.py | 5 | 1 | `has_files()``check=False` 检查状态 |
| dev/piptool.py | 8 | 0 | 全部迁移,移除 `import subprocess` |
| dev/lscalc.py | 3 | 1 | Linux `pgrep``check=False` |
| dev/bumpversion.py | 3 | 0 | 全部迁移,移除 `import subprocess` |
| dev/packtool.py | 2 | 0 | 全部迁移,移除 `import subprocess` |
| dev/autofmt.py | 4 | 1 | `sync_pyproject_config``check=False` (失败可忽略) |
| infra/dockercmd.py | 1 | 0 | 全部迁移,移除 `import subprocess` |
| infra/msdownload.py | 1 | 0 | 全部迁移,移除 `import subprocess` |
| files/screenshot.py | 4 | 4 | Linux 回退模式用 `FileNotFoundError` 作为控制流 |
### 未迁移文件
- `infra/sshcopyid.py` — 使用 `timeout=` 参数(`px.sh` 不支持)
- `infra/envdev.py` — 全部 `check=False`
- `system/clr.py`, `system/reseticoncache.py`, `system/taskkill.py`, `system/which.py` — 全部 `check=False`
### 测试文件 (8 个)
- `test_piptool.py` — mock 目标从 `subprocess.run` 改为 `pyflowx.sh`,错误测试从 `side_effect` 改为 `return_value=None`
- `test_lscalc.py` — 迁移调用改 mock `pyflowx.sh`Linux `check=False` 调用保留 `subprocess.run`
- `test_packtool.py` — mock 目标改为 `pyflowx.sh`
- `test_autofmt.py` — 迁移调用改 mock `pyflowx.sh``sync_pyproject_config` 保留 `subprocess.run`
- `test_screenshot.py` — Win/macOS 测试改 mock `pyflowx.sh`Linux 回退测试保留 `subprocess.run`
- `test_bumpversion.py``_mock_subprocess` 改为 `_mock_sh`mock `pyflowx.sh`
- `test_envdev.py` — 更新 dockercmd 错误消息断言(`px.sh` 消息格式与原自定义消息不同)
- `test_llm.py` — 更新 msdownload 错误消息断言
## 关键决策与依据
1. **`check=False` 调用不迁移**`px.sh` 内部 `check=True`,失败时返回 `None` + 打印错误。`check=False` 调用需要检查 `returncode` / `stdout`,语义不同。
2. **screenshot Linux 回退不迁移**`try: gnome-screenshot except FileNotFoundError: scrot` 模式中 `FileNotFoundError` 是控制流(回退到备用命令),不是错误。`px.sh` 会打印误导性错误信息。
3. **sshcopyid 不迁移**:使用 `timeout=` 参数控制 SSH 超时,`px.sh` 不支持此参数。
4. **测试 mock 策略**:迁移调用的测试从 mock `subprocess.run` 改为 mock `pyflowx.sh`,测试调用方的行为(处理 `None` 返回值)而非底层 `subprocess.run` 的异常。
5. **错误消息断言放宽**`px.sh` 的错误消息格式(`"{label}失败 (returncode=N): ..."`)与原自定义消息不同,测试断言改为检查关键词(`"失败"``"命令未找到"`)而非精确消息。
## 验证结果
- 全部 1731 个测试通过
- 覆盖率 97.24%(门槛 95%
- ruff check + format 全部通过
- 迁移后源文件覆盖率均为 100%
## 遗留事项
- `system/` 目录和 `infra/envdev.py``check=False` 调用未迁移(语义不同,不应迁移)
- `infra/sshcopyid.py` 未迁移(需要 `timeout=` 支持)
- 如需统一这些调用,可考虑扩展 `px.sh` 支持 `check=False``timeout` 参数
+89
View File
@@ -0,0 +1,89 @@
# 迭代 30: px.sh 扩展 + 剩余 subprocess.run 迁移
## 本轮目标
扩展 `px.sh` 支持 `check=False``timeout` 参数,完成 `ops/` 目录下剩余 `subprocess.run` 调用的迁移,统一错误处理与中文输出。解决 iter-29 遗留的 `check=False` / `timeout=` / `FileNotFoundError` 控制流三类场景。
## 改动文件清单
### 核心扩展
| 文件 | 变更 |
|------|------|
| `shell.py` | 新增 `check: bool = True``timeout: float \| None = None` 参数;新增 `except subprocess.TimeoutExpired` 分支 |
扩展后的签名:
```python
def sh(
cmd: list[str] | str,
*,
capture: bool = False,
label: str = "命令",
check: bool = True,
timeout: float | None = None,
) -> subprocess.CompletedProcess[str] | None:
```
异常捕获范围:`CalledProcessError``FileNotFoundError``TimeoutExpired`(不捕获通用 `SubprocessError``OSError`)。
### 源文件迁移 (10 个)
| 文件 | 迁移调用数 | 迁移策略 |
|------|-----------|---------|
| `system/clr.py` | 1 | `check=False``px.sh(check=False)`,移除 `import subprocess` |
| `system/taskkill.py` | 1 | `check=False` + 三分支 returncode 检查 |
| `system/which.py` | 1 | `check=False` + `returncode == 0` 检查 |
| `system/reseticoncache.py` | 4 | 全部 `check=False`,移除 `import subprocess` |
| `dev/lscalc.py` | 1 | Linux `pgrep``check=False` + stdout 检查,移除 `import subprocess` |
| `dev/gittool.py` | 1 | `has_files()``check=False`,移除 `import subprocess` |
| `dev/autofmt.py` | 1 | `sync_pyproject_config``check=False`,移除 `import subprocess` |
| `files/screenshot.py` | 4 | Linux 回退:`try/except FileNotFoundError``if px.sh(...) is None: px.sh(...)`,移除 `import subprocess` |
| `infra/envdev.py` | 9 | 全部 `check=False`,移除 `import subprocess` |
| `infra/sshcopyid.py` | 1 | `timeout=``px.sh(timeout=...)`,三分支 returncode 检查,移除 `import subprocess` |
| `dev/piptool.py` | 0 | 仅 ruff format 格式化(frozenset 换行) |
### 测试文件 (7 个)
| 文件 | 变更 |
|------|------|
| `test_shell.py` | 新增 `TestShCheckFalse`4 个测试)和 `TestShTimeout`4 个测试) |
| `cli/test_system_run.py` | mock 目标从 `subprocess.run` 改为 `pyflowx.sh`,新增命令不可用测试 |
| `cli/test_reseticoncache.py` | mock 目标改为 `pyflowx.sh` |
| `cli/test_lscalc.py` | Linux 测试 mock 目标改为 `pyflowx.sh` |
| `cli/test_gittool.py` | `has_files` 测试 mock 目标改为 `pyflowx.sh` |
| `cli/test_autofmt.py` | `sync_pyproject_config` 测试 mock 目标改为 `pyflowx.sh` |
| `cli/test_screenshot.py` | Linux 测试 mock 目标改为 `pyflowx.sh`scrot 回退测试从 `FileNotFoundError` side_effect 改为 `None` 返回值 |
| `cli/test_gittool_tool.py` | 合并 2 个失败测试为 `test_has_files_false_when_sh_fails` |
### 未迁移文件(确认不迁移)
- `command.py` — 核心命令执行器,`px.sh` 的同层抽象
- `conditions.py` — 条件检查,`subprocess.run` 检查进程/命令存在性
- `shell.py``px.sh` 自身实现
## 关键决策与依据
1. **`check=False` 语义**`check=True`(默认)时非零返回码/命令未找到/超时返回 `None``check=False` 时任何返回码都返回 `CompletedProcess`(调用方检查 `returncode`),仅命令未找到/超时返回 `None`
2. **`FileNotFoundError` 控制流迁移**screenshot Linux 回退模式中 `try: gnome-screenshot except FileNotFoundError: scrot` 迁移为 `if px.sh(["gnome-screenshot", ...]) is None: px.sh(["scrot", ...])``px.sh` 捕获 `FileNotFoundError` 返回 `None`,语义等价且保留回退逻辑。
3. **异常捕获范围**:仅捕获 `CalledProcessError``FileNotFoundError``TimeoutExpired`。不捕获通用 `SubprocessError`(基类)或 `OSError`(非 `FileNotFoundError`),这些会向上传播,避免掩盖 bug。
4. **mock 兼容性**`px.sh` 内部调用 `subprocess.run`,因此测试中 mock `subprocess.run` 仍能拦截 `px.sh` 的调用。`test_envdev.py``test_sshcopyid.py` 不需要更新 mock 目标。
5. **测试合并**`test_gittool_tool.py``test_has_files_false_on_subprocess_error``test_has_files_false_on_oserror` 期望 `px.sh` 捕获通用 `SubprocessError`/`OSError`,但 `px.sh` 仅捕获特定异常。合并为 `test_has_files_false_when_sh_fails`mock `pyflowx.sh` 返回 `None`(模拟 `px.sh` 捕获异常后的返回值)。
## 验证结果
- 全部 **1739** 个测试通过(修复 iter-29 后的 2 个失败测试)
- 覆盖率 **97.27%**(门槛 95%
- ruff check 全部通过
- pyrefly check 0 errors
- P13 修改的源文件覆盖率均为 100%shell.py / envdev.py / sshcopyid.py / screenshot.py / gittool.py 等)
## 遗留事项
- `command.py``conditions.py` 仍使用 `subprocess.run`(核心抽象,不应迁移)
- 6 个预存文件需要 ruff formattest_cancellation.py / test_diagnostics.py / test_fileops.py / test_profiling.py / test_report.py / piptool.py 中 piptool.py 已格式化,其余 5 个非本次范围)
- iter-21-25 应在 iter-26 前归档清理(dev-workflow.md 规则),属遗留清理事项
+114
View File
@@ -0,0 +1,114 @@
# 迭代 31:调度器性能深度优化(P14)
## 本轮目标
针对调度器执行热路径中的函数调用开销进行深度优化,提升四种执行策略
sequential/thread/async/dependency)的 noop 吞吐。基于 cProfile
分析结果,消除高频调用路径中可避免的函数调用与对象创建。
## 改动文件清单
- `src/pyflowx/executors.py` — 4 项微优化 + 线程池复用
- `benchmarks/bench_runner.py` — 新增基准测试脚本(上一会话已建,本轮复用)
## 关键决策与依据
### 1. 线程池复用(dependency 策略主要优化)
**问题**`asyncio.run()` 每次创建新事件循环,默认线程池也随之重建。
dependency 策略把同步任务卸载到线程池,每次 `run()` 都重复创建/销毁
线程池,500 任务的线程池提交占 36% 耗时。
**方案**:新增模块级 `_thread_pool``ThreadPoolExecutor`+ 双重检查锁
惰性创建,`_submit_sync_task` 的 thread 分支改用 `_get_thread_pool()`
`atexit` 注册 `_shutdown_thread_pool` 兜底关闭。
**依据**:与既有 `_process_pool` 复用模式一致。线程池跨 `run()` 调用
复用后,`submit``_adjust_thread_count` 不再每轮重建线程。
### 2. `_sort_by_priority` 单任务快速路径
**问题**dependency 策略每轮就绪集常为 1 个任务,仍调用 `sorted()`
创建新列表 + 排序键函数。
**方案**:开头加 `if len(layer) <= 1: return layer`,跳过 `sorted()`
**依据**dependency 主循环 `while remaining or in_flight`
`_sort_by_priority(list(ready), all_specs)` 每轮调用,单任务就绪时
`sorted()` 的列表复制 + 键函数调用纯属开销。
### 3. `_build_context` 无依赖快速路径
**问题**:无依赖任务(最常见场景)仍创建空 dict + 遍历两个空元组。
**方案**:开头加 `if not spec.depends_on and not spec.soft_depends_on: return {}`
**依据**:noop 基准全部无依赖,每个任务都走此路径。省去空元组遍历的
迭代器创建开销。
### 4. `_prepare_for_execution` 三无快速路径
**问题**:无依赖、无条件、无 `skip_if_missing` 的任务(最常见场景)
仍调用 `_upstream_skip_reason`(遍历空依赖返回 None+ `should_execute`
(内部已有快速路径但有函数调用开销)。
**方案**:开头加
`if not spec.depends_on and not spec.conditions and not spec.skip_if_missing: return None`
**依据**:无依赖时 `_upstream_skip_reason` 必返回 None(遍历空元组),
`should_execute` 的快速路径也已返回 `(True, None)`。直接返回 None 省去
两次函数调用。
### 5. `_run_hooks` 调用点内联
**问题**`_run_hooks(spec.hooks, "pre_run", spec)` 内部用
`getattr(hooks, fn_name, None)` 查找钩子。大多数任务无钩子
`TaskHooks` 默认全 None),仍付出函数调用 + `getattr` 开销。
4 个调用点:`SyncTaskRunner.run`pre_run)、`AsyncTaskRunner._inner`
pre_run)、`_mark_success`post_run)、`_handle_failure`on_failure)。
**方案**:每个调用点前加 `if spec.hooks.xxx is not None:` 守卫,钩子为
None 时跳过 `_run_hooks` 调用(省去函数调用 + getattr)。
**依据**`TaskHooks``frozen=True, slots=True` dataclass,属性访问
为 O(1) slot 查找,比 `getattr` + 函数调用快。noop 基准无钩子,全部命中
快速路径。
## 验证结果
### 门禁
- ruff: All checks passed
- pyrefly: 0 errors
- pytest: 1739 passed
- coverage: 97.14%(≥95% 门槛)
### 基准对比(500 noop 任务)
| 策略 | 优化前 | 优化后 | 提升 |
|------|--------|--------|------|
| sequential | 5,032,712 ops/s | 5,664,117 ops/s | +12.6% |
| thread | 2,200,007 ops/s | 2,359,447 ops/s | +7.2% |
| async | 1,723,508 ops/s | 1,906,286 ops/s | +10.6% |
| dependency | 17,494 ops/s | 18,296 ops/s | +4.6% |
sequential 与 async 提升最显著(+10~12%),因为 noop 任务命中所有
快速路径,每任务节省的开销在总耗时中占比最大。thread 次之(+7%)。
dependency 受 asyncio 事件循环与线程池提交的固有开销主导,per-task
优化相对影响小(+4.6%)。
### cProfile 热路径变化
优化前 `build_call_args`500 次, 0.007s)与 `_run_hooks` 占用可见。
优化后两者均跌出 top 30,剩余瓶颈为 asyncio + 线程池桥接的固有开销
`run_in_executor` / `submit` / `_adjust_thread_count`),属架构性
开销,无法通过 per-task 微优化消除。
## 遗留事项
- dependency 策略对 noop 任务的固有并发开销仍较高(18K ops/s vs
sequential 5.6M ops/s),需真实 I/O 任务才能体现并行优势(设计预期)。
- `datetime.now()` 每任务调用 2 次(started_at + finished_at),约占
sequential 预算可观比例,但涉及 duration 正确性,未优化。
- ThreadedLayerRunnerthread 策略)每次 `run()` 创建新
`ThreadPoolExecutor`,但性能已可接受(2.3M ops/s),暂未复用。
+55 -22
View File
@@ -1,6 +1,6 @@
--- ---
name: "pyflowx-development" name: "pyflowx-development"
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策,并同步迭代 21(P9)、迭代 22(P10 图片处理)、迭代 23(P11 文件操作底层 API)、迭代 24(P12 通用 DAG 构造器)迭代 25(执行热路径性能优化)的行为变更。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化、任务编排、检查点恢复、监控导出、图片处理、文件操作底层 API、通用 DAG 构造器相关的功能时参考。" description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策,并同步迭代 21(P9)、迭代 22(P10 图片处理)、迭代 23(P11 文件操作底层 API)、迭代 24(P12 通用 DAG 构造器)迭代 25(执行热路径性能优化)与迭代 31(调度器深度性能优化)的行为变更。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化、任务编排、检查点恢复、监控导出、图片处理、文件操作底层 API、通用 DAG 构造器相关的功能时参考。"
--- ---
# PyFlowX 开发知识库 # PyFlowX 开发知识库
@@ -243,6 +243,35 @@ sequential(500) 从 764 → 930 ops/s+22%),thread(500) 从 108 → 119 op
- **适用边界**async/dependency 策略对 noop 任务的固有并发开销(事件循环 + - **适用边界**async/dependency 策略对 noop 任务的固有并发开销(事件循环 +
线程池提交)主导,per-task 优化相对影响小;真实 I/O 任务才能体现并行优势。 线程池提交)主导,per-task 优化相对影响小;真实 I/O 任务才能体现并行优势。
### 调度器深度优化 5 项(iter-31)
sequential(500) 从 5.0M → 5.7M ops/s+12.6%),async 从 1.7M → 1.9M
+10.6%),thread 从 2.2M → 2.4M+7.2%),dependency 从 17.5K → 18.3K
+4.6%)。
- **线程池复用**:新增模块级 `_thread_pool``ThreadPoolExecutor`+ 双重
检查锁惰性创建,`_submit_sync_task` 的 thread 分支改用 `_get_thread_pool()`。
`asyncio.run()` 每次创建新事件循环,默认线程池也随之重建;模块级缓存让
线程池跨 `run()` 调用复用。`atexit` 注册 `_shutdown_thread_pool` 兜底。
与既有 `_process_pool` 模式一致。
- **`_sort_by_priority` 单任务快速路径**`if len(layer) <= 1: return layer`。
dependency 策略每轮就绪集常为 1 个任务,跳过 `sorted()` 的列表复制 + 键函数。
- **`_build_context` 无依赖快速路径**
`if not spec.depends_on and not spec.soft_depends_on: return {}`,跳过两个
空元组遍历。
- **`_prepare_for_execution` 三无快速路径**
`if not spec.depends_on and not spec.conditions and not spec.skip_if_missing: return None`
省去 `_upstream_skip_reason`(无依赖必返回 None+ `should_execute` 两次
函数调用。
- **`_run_hooks` 调用点内联**4 个调用点(`SyncTaskRunner.run`/`AsyncTaskRunner._inner`
的 pre_run、`_mark_success` 的 post_run、`_handle_failure` 的 on_failure
前加 `if spec.hooks.xxx is not None:` 守卫。`TaskHooks` 默认全 Nonenoop
基准全部命中快速路径,省去 `_run_hooks` 函数调用 + `getattr` 开销。
- **优化边界**cProfile 确认优化后 `build_call_args` 与 `_run_hooks` 跌出
top 30。剩余瓶颈为 asyncio + 线程池桥接的固有开销
`run_in_executor`/`submit`/`_adjust_thread_count`),属架构性开销,无法
通过 per-task 微优化消除。
### 事件驱动优先级调度(iter-27) ### 事件驱动优先级调度(iter-27)
DependencyRunner 从"一次性创建所有 asyncio Task"重写为"事件驱动按需创建": DependencyRunner 从"一次性创建所有 asyncio Task"重写为"事件驱动按需创建":
@@ -625,38 +654,42 @@ DependencyRunner 主循环每轮调用 `_find_ready()` 扫描全部 `remaining`
### 可复用模式 ### 可复用模式
- **`subprocess.run` 的 `check` 参数分级策略**iter-16): - **`px.sh` 的 `check` 参数分级策略**iter-16 → iter-30 演进):
- **失败需感知**dockercmd/msdownload/sglang)→ `check=True` + - **失败需感知**dockercmd/msdownload/sglang)→ `px.sh(cmd, label="...")`(默认 `check=True`),失败返回 `None` + 中文打印
`try/except subprocess.CalledProcessError` 打印 rich 错误 - **失败可容忍但需检查**taskkill/which/has_files)→ `px.sh(cmd, check=False, label="...")` + 检查 `returncode`/`stdout`
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查 - **失败无关紧要**clr/reseticoncache/envdev)→ `px.sh(cmd, check=False, label="...")`
`returncode` 打印结果(pkill 返回 1 表示无匹配进程,非错误) - **超时控制**sshcopyid)→ `px.sh(cmd, timeout=N, label="...")`,超时返回 `None`
- **失败无关紧要**clr/which)→ `check=False`,加注释说明原因 - **命令回退**screenshot Linux)→ `if px.sh(cmd1, label="...") is None: px.sh(cmd2, label="...")`(替代 `try/except FileNotFoundError`
- **错误路径统一处理模式** - **px.sh 统一错误处理**
```python ```python
try: result = px.sh(cmd, label="操作")
subprocess.run(cmd, check=True, ...) if result is None:
except subprocess.CalledProcessError as e: # 命令失败/未找到/超时,px.sh 已打印中文错误
_console.print(f"[red]命令失败 (returncode={e.returncode})[/]") return
except FileNotFoundError:
_console.print(f"[red]命令未找到: {cmd[0]}[/]")
``` ```
`px.sh` 内部捕获 `CalledProcessError`/`FileNotFoundError`/`TimeoutExpired`,不捕获通用 `SubprocessError`/`OSError`(向上传播)。
### 设计决策 ### 设计决策
- **rich Console 而非 print**CLI 工具错误用 `rich.console.Console` - **px.sh 封装优于直接 subprocess.run**:统一中文错误输出 + `None` 返回值语义,
打印带颜色输出,与项目其他 CLI 工具一致 调用方无需重复 try/except。`check=False` 时返回 `CompletedProcess`,调用方检查 `returncode`
- **`check=True` + try/except 优于 `check=False` + 手动检查**:除非 - **`check=True` 默认优于 `check=False`**:除非命令返回非零是预期行为
命令返回非零是预期行为(如 taskkill pkill 无匹配、which 未找到命令), (如 taskkill pkill 无匹配、which 未找到命令),否则用默认 `check=True`。
否则用 `check=True` 让 subprocess 抛异常,统一在 except 中处理。 - **异常捕获范围刻意收窄**:仅捕获 `CalledProcessError`/`FileNotFoundError`/`TimeoutExpired`
通用 `SubprocessError`/`OSError` 向上传播,避免掩盖 bug。
### 踩坑总结 ### 踩坑总结
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch` - **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
装饰器与 `mock.patch.object` 上下文。mock subprocess 异常 装饰器与 `mock.patch.object` 上下文。mock `px.sh`
`monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)`
mock `subprocess.run` 抛异常用
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))` `monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。 —— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
- **mock 异常抛出技巧**`lambda *_, **__: (_ for _ in ()).throw(err)` - **mock 兼容性**`px.sh` 内部调用 `subprocess.run`,因此 mock `subprocess.run`
利用生成器仅在迭代时抛异常的特性,让无参 lambda 能抛出指定异常 仍能拦截 `px.sh` 的调用。但 mock `px.sh` 更直接(测试调用方行为而非底层实现)
- **`SubprocessError`/`OSError` 不被 px.sh 捕获**:测试中不应期望 `px.sh` 捕获
通用异常基类,应 mock `px.sh` 返回 `None` 模拟捕获后的返回值。
## 十三、测试与质量保障 ## 十三、测试与质量保障
+92
View File
@@ -0,0 +1,92 @@
"""PyFlowX 调度器性能基准测试.
测量 4 种执行策略(sequential/thread/async/dependency)的调度开销,
使用 noop 任务隔离调度器本身的开销。
用法:
uv run python benchmarks/bench_runner.py # 运行所有基准(默认 500 任务)
uv run python benchmarks/bench_runner.py -s 1000 # 指定任务数
uv run python benchmarks/bench_runner.py --profile # cProfile 分析 sequential
uv run python benchmarks/bench_runner.py --profile --strategy thread
"""
from __future__ import annotations
import argparse
import cProfile
import io
import pstats
import time
from typing import cast
import pyflowx as px
from pyflowx import TaskSpec
from pyflowx.executors import Strategy
_STRATEGIES: tuple[Strategy, ...] = ("sequential", "thread", "async", "dependency")
def _noop() -> None:
"""空操作任务,用于隔离调度器开销。"""
def build_graph(n: int) -> px.Graph:
"""构建 n 个无依赖 noop 任务的图。"""
specs = {f"task-{i}": TaskSpec(f"task-{i}", _noop) for i in range(n)}
return px.Graph(specs=specs)
def bench_strategy(n: int, strategy: Strategy) -> float:
"""测量指定策略的 ops/s。"""
graph = build_graph(n)
start = time.perf_counter()
px.run(graph, strategy=strategy)
elapsed = time.perf_counter() - start
return n / elapsed
def run_benchmarks(n: int) -> dict[str, float]:
"""运行所有基准测试,返回 {策略: ops/s}。"""
results: dict[str, float] = {}
for strategy in ("sequential", "thread", "async", "dependency"):
results[strategy] = bench_strategy(n, strategy)
return results
def run_profile(n: int, strategy: Strategy) -> None:
"""用 cProfile 分析指定策略的热路径。"""
profiler = cProfile.Profile()
profiler.enable()
graph = build_graph(n)
px.run(graph, strategy=strategy)
profiler.disable()
buf = io.StringIO()
stats = pstats.Stats(profiler, stream=buf).sort_stats("cumulative")
stats.print_stats(30)
print(buf.getvalue())
def main() -> None:
parser = argparse.ArgumentParser(description="PyFlowX 调度器性能基准测试")
parser.add_argument("-s", "--size", type=int, default=500, help="任务数量(默认 500")
parser.add_argument("--profile", action="store_true", help="启用 cProfile 分析")
parser.add_argument("--strategy", default="sequential", help="分析的策略(默认 sequential")
args = parser.parse_args()
if args.profile:
strategy = cast("Strategy", args.strategy)
print(f"cProfile 分析: {strategy} 策略, {args.size} 任务")
print("=" * 70)
run_profile(args.size, strategy)
return
print(f"PyFlowX 调度器性能基准测试({args.size} noop 任务)")
print("=" * 70)
results = run_benchmarks(args.size)
for strategy, ops in results.items():
print(f" {strategy:12s}: {ops:8.1f} ops/s")
if __name__ == "__main__":
main()
+5 -5
View File
@@ -21,12 +21,13 @@ license = { text = "MIT" }
name = "pyflowx" name = "pyflowx"
readme = "README.md" readme = "README.md"
requires-python = ">=3.10" requires-python = ">=3.10"
version = "0.4.9" version = "0.4.11"
[project.scripts] [project.scripts]
emlman = "pyflowx.cli.legacy.emlmanager:main" emlman = "pyflowx.cli.legacy.emlmanager:main"
pf = "pyflowx.cli.pf:main" pf = "pyflowx.cli.pf:main"
pxp = "pyflowx.cli.legacy.profiler:main" pxp = "pyflowx.cli.legacy.profiler:main"
pyflowx = "pyflowx.cli.pf:main"
[project.optional-dependencies] [project.optional-dependencies]
dev = [ dev = [
@@ -105,7 +106,6 @@ target-version = "py310"
ignore = [ ignore = [
"E501", # line too long (handled by formatter) "E501", # line too long (handled by formatter)
"PLC0415", # import should be at top-level (intentional for lazy imports) "PLC0415", # import should be at top-level (intentional for lazy imports)
"PLR0913", # too many arguments
"PLR0915", # too many statements (intentional for complex methods) "PLR0915", # too many statements (intentional for complex methods)
"PLR2004", # magic value comparison "PLR2004", # magic value comparison
"PTH119", # os.path.basename (intentional for sys.argv) "PTH119", # os.path.basename (intentional for sys.argv)
+1 -1
View File
@@ -112,7 +112,7 @@ from .task import (
from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool
from .yaml_loader import load_yaml, parse_yaml_string from .yaml_loader import load_yaml, parse_yaml_string
__version__ = "0.4.9" __version__ = "0.4.11"
def graph( def graph(
+171 -152
View File
@@ -52,6 +52,7 @@ import queue
import threading import threading
import time import time
from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping
from dataclasses import dataclass, field
from dataclasses import replace as dc_replace from dataclasses import replace as dc_replace
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
@@ -118,6 +119,34 @@ def _shutdown_process_pool() -> None:
atexit.register(_shutdown_process_pool) atexit.register(_shutdown_process_pool)
# 线程池复用:asyncio.run() 每次创建新事件循环,默认线程池也随之重建。
# 模块级缓存让线程池跨 run() 调用复用,避免重复创建/销毁线程的开销。
_thread_pool: concurrent.futures.ThreadPoolExecutor | None = None
_thread_pool_lock = threading.Lock()
def _get_thread_pool() -> concurrent.futures.ThreadPoolExecutor:
"""获取复用的线程池(惰性创建)。"""
global _thread_pool # noqa: PLW0603
if _thread_pool is None:
with _thread_pool_lock:
if _thread_pool is None:
_thread_pool = concurrent.futures.ThreadPoolExecutor()
return _thread_pool
def _shutdown_thread_pool() -> None:
"""关闭复用的线程池。"""
global _thread_pool # noqa: PLW0603
if _thread_pool is not None:
pool = _thread_pool
_thread_pool = None
pool.shutdown(wait=False)
atexit.register(_shutdown_thread_pool)
def _run_in_process(fn: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any: def _run_in_process(fn: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
"""模块级函数:在进程池中执行任务(须可 pickle)。 """模块级函数:在进程池中执行任务(须可 pickle)。
@@ -132,6 +161,24 @@ EventCallback = Callable[[TaskEvent], None]
Strategy = Literal["sequential", "thread", "async", "dependency"] Strategy = Literal["sequential", "thread", "async", "dependency"]
@dataclass(frozen=True)
class _ExecContext:
"""执行上下文:捆绑 run() 调用链中共享的状态,减少参数传递。
将 context/report/backend/on_event/concurrency_limits/cancel_event
打包为单一参数,使调用链中每个函数的参数数 ≤5(PLR0913)。
frozen=True 保证调用链中不可意外替换整体引用,但不阻止对
context/report 等可变属性的内部修改(如 ``ctx.context[name] = value``)。
"""
context: dict[str, Any]
report: RunReport
backend: StateBackend
on_event: EventCallback | None
concurrency_limits: Mapping[str, int] = field(default_factory=dict)
cancel_event: threading.Event | CancelToken | None = None
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
# 无状态公共辅助 # 无状态公共辅助
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
@@ -193,6 +240,9 @@ def _build_context(
硬依赖:若上游 SKIPPED/FAILED 则不注入(本任务通常也会被跳过)。 硬依赖:若上游 SKIPPED/FAILED 则不注入(本任务通常也会被跳过)。
软依赖:上游成功则注入其值;否则注入 ``spec.defaults`` 中的默认值(或 ``None``)。 软依赖:上游成功则注入其值;否则注入 ``spec.defaults`` 中的默认值(或 ``None``)。
""" """
# 快速路径:无依赖任务直接返回空 dict,跳过两个空元组遍历。
if not spec.depends_on and not spec.soft_depends_on:
return {}
ctx: dict[str, Any] = {} ctx: dict[str, Any] = {}
for dep in spec.depends_on: for dep in spec.depends_on:
if dep in global_context: if dep in global_context:
@@ -208,32 +258,28 @@ def _build_context(
def _apply_cached( def _apply_cached(
name: str,
spec: TaskSpec[Any], spec: TaskSpec[Any],
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
) -> bool: ) -> bool:
"""若 ``name`` 命中缓存,写入 context/report 并返回 True。 """若 ``spec.name`` 命中缓存,写入 context/report 并返回 True。
单次 ``backend.get`` + ``KeyError`` 回退,避免 ``has`` + ``get`` 双重 单次 ``backend.get`` + ``KeyError`` 回退,避免 ``has`` + ``get`` 双重
哈希查找与双重 TTL 判断。 哈希查找与双重 TTL 判断。
""" """
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。 # 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
storage_key = spec.name if spec.cache_key is None else spec.storage_key(context) storage_key = spec.name if spec.cache_key is None else spec.storage_key(ctx.context)
try: try:
cached = backend.get(storage_key) cached = ctx.backend.get(storage_key)
except KeyError: except KeyError:
return False return False
context[name] = cached ctx.context[spec.name] = cached
result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中") result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中")
report.results[name] = result ctx.report.results[spec.name] = result
_emit(on_event, result) _emit(ctx.on_event, result)
logger.info( logger.info(
"task %r skipped (cached)", "task %r skipped (cached)",
name, spec.name,
extra={"run_id": report.run_id, "task_name": name, "status": TaskStatus.SKIPPED.value}, extra={"run_id": ctx.report.run_id, "task_name": spec.name, "status": TaskStatus.SKIPPED.value},
) )
return True return True
@@ -244,6 +290,10 @@ def _sort_by_priority(layer: list[str], specs: Mapping[str, TaskSpec[Any]]) -> l
接受预构建的 ``{name: spec}`` 映射,避免在排序键函数中重复调用 接受预构建的 ``{name: spec}`` 映射,避免在排序键函数中重复调用
``graph.resolved_spec``(即便有缓存也省去 N 次字典查询)。 ``graph.resolved_spec``(即便有缓存也省去 N 次字典查询)。
""" """
# 快速路径:单任务或空层无需排序,跳过 sorted() 调用开销。
# dependency 策略每轮就绪集常为 1 个任务,此路径命中率高。
if len(layer) <= 1:
return layer
return sorted(layer, key=lambda n: -specs[n].priority) return sorted(layer, key=lambda n: -specs[n].priority)
@@ -304,6 +354,11 @@ def _prepare_for_execution(
返回 SKIPPED TaskResult 或 ``None``(继续执行)。 返回 SKIPPED TaskResult 或 ``None``(继续执行)。
条件判断委托给 :meth:`TaskSpec.should_execute`,避免重复实现。 条件判断委托给 :meth:`TaskSpec.should_execute`,避免重复实现。
""" """
# 快速路径:无依赖、无条件、无 skip_if_missing 时直接放行(最常见场景),
# 省去 _upstream_skip_reason 与 should_execute 两次函数调用开销。
# 无依赖时 _upstream_skip_reason 必返回 None(遍历空元组),故可安全跳过。
if not spec.depends_on and not spec.conditions and not spec.skip_if_missing:
return None
# 1. 上游被跳过/失败 # 1. 上游被跳过/失败
skip_reason = _upstream_skip_reason(spec, report) skip_reason = _upstream_skip_reason(spec, report)
# 2. 条件 / skip_if_missing(单一来源:TaskSpec.should_execute # 2. 条件 / skip_if_missing(单一来源:TaskSpec.should_execute
@@ -345,33 +400,31 @@ def _mark_success(spec: TaskSpec[Any], result: TaskResult[Any], value: Any) -> N
result.value = value result.value = value
result.status = TaskStatus.SUCCESS result.status = TaskStatus.SUCCESS
result.finished_at = datetime.now() result.finished_at = datetime.now()
_run_hooks(spec.hooks, "post_run", spec, value) if spec.hooks.post_run is not None:
_run_hooks(spec.hooks, "post_run", spec, value)
def _finalize_failure( def _finalize_failure(
result: TaskResult[Any], result: TaskResult[Any],
layer_idx: int | None, layer_idx: int | None,
on_event: EventCallback | None, ctx: _ExecContext,
continue_on_error: bool, continue_on_error: bool,
name: str,
report: RunReport | None,
) -> None: ) -> None:
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。 """标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。
失败结果在抛出前写入 ``report.results[name]``,使流式 API 失败结果在抛出前写入 ``ctx.report.results``,使流式 API
(:func:`run_iter`) 能在 re-raise 前yield 该结果。 (:func:`run_iter`) 能在 re-raise 前yield 该结果。
""" """
result.status = TaskStatus.FAILED result.status = TaskStatus.FAILED
result.finished_at = datetime.now() result.finished_at = datetime.now()
if report is not None: ctx.report.results[result.spec.name] = result
report.results[name] = result _emit(ctx.on_event, result)
_emit(on_event, result)
if continue_on_error: if continue_on_error:
logger.warning( logger.warning(
"task %r failed but continue_on_error=True; continuing.", "task %r failed but continue_on_error=True; continuing.",
result.spec.name, result.spec.name,
extra={ extra={
"run_id": report.run_id if report is not None else "-", "run_id": ctx.report.run_id,
"task_name": result.spec.name, "task_name": result.spec.name,
"status": TaskStatus.FAILED.value, "status": TaskStatus.FAILED.value,
"attempts": result.attempts, "attempts": result.attempts,
@@ -384,7 +437,7 @@ def _finalize_failure(
cause=result.error if result.error is not None else RuntimeError("unknown"), cause=result.error if result.error is not None else RuntimeError("unknown"),
attempts=result.attempts, attempts=result.attempts,
layer=layer_idx, layer=layer_idx,
report=report, report=ctx.report,
) )
@@ -393,9 +446,7 @@ def _handle_failure(
result: TaskResult[Any], result: TaskResult[Any],
exc: BaseException, exc: BaseException,
layer_idx: int | None, layer_idx: int | None,
on_event: EventCallback | None, ctx: _ExecContext,
name: str,
report: RunReport | None,
) -> bool: ) -> bool:
"""统一处理失败:超时转换、重试决策、finalize。 """统一处理失败:超时转换、重试决策、finalize。
@@ -405,7 +456,7 @@ def _handle_failure(
``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。 ``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。
""" """
# asyncio.TimeoutError → TaskTimeoutError(统一异常类型) # asyncio.TimeoutError → TaskTimeoutError(统一异常类型)
run_id = report.run_id if report is not None else "-" run_id = ctx.report.run_id
if isinstance(exc, asyncio.TimeoutError): if isinstance(exc, asyncio.TimeoutError):
exc = TaskTimeoutError(spec.name, spec.timeout or 0.0) exc = TaskTimeoutError(spec.name, spec.timeout or 0.0)
logger.warning( logger.warning(
@@ -439,8 +490,9 @@ def _handle_failure(
result.error = exc result.error = exc
if _should_retry(spec, result.attempts, exc): if _should_retry(spec, result.attempts, exc):
return False return False
_run_hooks(spec.hooks, "on_failure", spec, exc) if spec.hooks.on_failure is not None:
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error, name, report) _run_hooks(spec.hooks, "on_failure", spec, exc)
_finalize_failure(result, layer_idx, ctx, spec.continue_on_error)
return True return True
@@ -453,21 +505,21 @@ class SyncTaskRunner:
@staticmethod @staticmethod
def run( def run(
spec: TaskSpec[Any], spec: TaskSpec[Any],
context: Mapping[str, Any], task_ctx: Mapping[str, Any],
layer_idx: int | None, layer_idx: int | None,
on_event: EventCallback | None = None, ctx: _ExecContext,
report: RunReport | None = None,
) -> TaskResult[Any]: ) -> TaskResult[Any]:
skipped = _prepare_for_execution(spec, context, report, on_event) skipped = _prepare_for_execution(spec, task_ctx, ctx.report, ctx.on_event)
if skipped is not None: if skipped is not None:
return skipped return skipped
result: TaskResult[Any] = TaskResult(spec=spec) result: TaskResult[Any] = TaskResult(spec=spec)
result.started_at = datetime.now() result.started_at = datetime.now()
args, kwargs = build_call_args(spec, context) args, kwargs = build_call_args(spec, task_ctx)
_run_hooks(spec.hooks, "pre_run", spec) if spec.hooks.pre_run is not None:
_emit_running(on_event, spec) _run_hooks(spec.hooks, "pre_run", spec)
_emit_running(ctx.on_event, spec)
while True: while True:
result.attempts += 1 result.attempts += 1
@@ -481,7 +533,7 @@ class SyncTaskRunner:
_mark_success(spec, result, value) _mark_success(spec, result, value)
return result return result
except Exception as exc: except Exception as exc:
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report): if _handle_failure(spec, result, exc, layer_idx, ctx):
return result return result
wait = spec.retry.wait_seconds(result.attempts) wait = spec.retry.wait_seconds(result.attempts)
if wait > 0: if wait > 0:
@@ -494,24 +546,24 @@ class AsyncTaskRunner:
@staticmethod @staticmethod
async def run( async def run(
spec: TaskSpec[Any], spec: TaskSpec[Any],
context: Mapping[str, Any], task_ctx: Mapping[str, Any],
layer_idx: int | None, layer_idx: int | None,
on_event: EventCallback | None = None, ctx: _ExecContext,
report: RunReport | None = None,
semaphore: asyncio.Semaphore | None = None, semaphore: asyncio.Semaphore | None = None,
) -> TaskResult[Any]: ) -> TaskResult[Any]:
skipped = _prepare_for_execution(spec, context, report, on_event) skipped = _prepare_for_execution(spec, task_ctx, ctx.report, ctx.on_event)
if skipped is not None: if skipped is not None:
return skipped return skipped
async def _inner() -> TaskResult[Any]: async def _inner() -> TaskResult[Any]:
result: TaskResult[Any] = TaskResult(spec=spec) result: TaskResult[Any] = TaskResult(spec=spec)
result.started_at = datetime.now() result.started_at = datetime.now()
args, kwargs = build_call_args(spec, context) args, kwargs = build_call_args(spec, task_ctx)
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
_run_hooks(spec.hooks, "pre_run", spec) if spec.hooks.pre_run is not None:
_emit_running(on_event, spec) _run_hooks(spec.hooks, "pre_run", spec)
_emit_running(ctx.on_event, spec)
while True: while True:
result.attempts += 1 result.attempts += 1
@@ -520,7 +572,7 @@ class AsyncTaskRunner:
_mark_success(spec, result, value) _mark_success(spec, result, value)
return result return result
except Exception as exc: except Exception as exc:
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report): if _handle_failure(spec, result, exc, layer_idx, ctx):
return result return result
wait = spec.retry.wait_seconds(result.attempts) wait = spec.retry.wait_seconds(result.attempts)
if wait > 0: if wait > 0:
@@ -584,8 +636,8 @@ def _submit_sync_task(
proc_fn = partial(_run_in_process, spec.effective_fn, args, kwargs) proc_fn = partial(_run_in_process, spec.effective_fn, args, kwargs)
return loop.run_in_executor(pool, proc_fn) return loop.run_in_executor(pool, proc_fn)
# thread(默认):线程池执行 # thread(默认):复用模块级线程池,避免每次 asyncio.run() 创建新线程池的开销
return loop.run_in_executor(None, fn_call) return loop.run_in_executor(_get_thread_pool(), fn_call)
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
@@ -594,10 +646,7 @@ def _submit_sync_task(
def _filter_and_sort( def _filter_and_sort(
layer: list[str], layer: list[str],
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
) -> tuple[list[str], dict[str, TaskSpec[Any]]]: ) -> tuple[list[str], dict[str, TaskSpec[Any]]]:
"""过滤掉已命中缓存的任务,按优先级排序返回待运行列表与 specs 映射。 """过滤掉已命中缓存的任务,按优先级排序返回待运行列表与 specs 映射。
@@ -611,36 +660,32 @@ def _filter_and_sort(
spec = graph.resolved_spec(name) spec = graph.resolved_spec(name)
specs[name] = spec specs[name] = spec
# 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过 # 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过
if name in report.results and report.results[name].status == TaskStatus.SUCCESS: if name in ctx.report.results and ctx.report.results[name].status == TaskStatus.SUCCESS:
context[name] = report.results[name].value ctx.context[name] = ctx.report.results[name].value
continue continue
if not _apply_cached(name, spec, context, report, backend, on_event): if not _apply_cached(spec, ctx):
to_run.append(name) to_run.append(name)
return _sort_by_priority(to_run, specs), specs return _sort_by_priority(to_run, specs), specs
def _store_result( def _store_result(
name: str,
result: TaskResult[Any], result: TaskResult[Any],
spec: TaskSpec[Any], spec: TaskSpec[Any],
task_ctx: dict[str, Any], task_ctx: dict[str, Any],
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
) -> None: ) -> None:
"""存储任务结果到 context/report/backend 并触发事件。 """存储任务结果到 context/report/backend 并触发事件。
``spec`` 与 ``task_ctx`` 由调用方在执行前已构建,直接复用避免重复 ``spec`` 与 ``task_ctx`` 由调用方在执行前已构建,直接复用避免重复
``resolved_spec`` / ``_build_context`` 调用。 ``resolved_spec`` / ``_build_context`` 调用。
""" """
context[name] = result.value ctx.context[spec.name] = result.value
if result.status == TaskStatus.SUCCESS: if result.status == TaskStatus.SUCCESS:
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。 # 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
key = spec.name if spec.cache_key is None else spec.storage_key(task_ctx) key = spec.name if spec.cache_key is None else spec.storage_key(task_ctx)
backend.save(key, result.value) ctx.backend.save(key, result.value)
report.results[name] = result ctx.report.results[spec.name] = result
_emit(on_event, result) _emit(ctx.on_event, result)
def _build_semaphores( def _build_semaphores(
@@ -676,18 +721,15 @@ class SequentialLayerRunner:
def execute( def execute(
layer: list[str], layer: list[str],
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
layer_idx: int, layer_idx: int,
on_event: EventCallback | None,
) -> None: ) -> None:
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event) to_run, specs = _filter_and_sort(layer, graph, ctx)
for name in to_run: for name in to_run:
spec = specs[name] spec = specs[name]
task_ctx = _build_context(spec, context, report) task_ctx = _build_context(spec, ctx.context, ctx.report)
result = SyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report) result = SyncTaskRunner.run(spec, task_ctx, layer_idx, ctx)
_store_result(name, result, spec, task_ctx, context, report, backend, on_event) _store_result(result, spec, task_ctx, ctx)
class ThreadedLayerRunner: class ThreadedLayerRunner:
@@ -697,29 +739,25 @@ class ThreadedLayerRunner:
def execute( def execute(
layer: list[str], layer: list[str],
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
layer_idx: int, layer_idx: int,
on_event: EventCallback | None,
pool: concurrent.futures.ThreadPoolExecutor, pool: concurrent.futures.ThreadPoolExecutor,
concurrency_limits: Mapping[str, int],
) -> None: ) -> None:
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event) to_run, specs = _filter_and_sort(layer, graph, ctx)
if not to_run: if not to_run:
return return
semaphores = _build_semaphores(to_run, specs, threading.Semaphore, concurrency_limits) semaphores = _build_semaphores(to_run, specs, threading.Semaphore, ctx.concurrency_limits)
context_snapshot = dict(context) context_snapshot = dict(ctx.context)
lock = threading.Lock() lock = threading.Lock()
def _run_threaded_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]: def _run_threaded_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
spec = specs[name] spec = specs[name]
task_ctx = _build_context(spec, context_snapshot, report) task_ctx = _build_context(spec, context_snapshot, ctx.report)
sem = _get_sem(semaphores, spec) sem = _get_sem(semaphores, spec)
if sem is not None: if sem is not None:
sem.acquire() sem.acquire()
try: try:
return task_ctx, SyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report) return task_ctx, SyncTaskRunner.run(spec, task_ctx, layer_idx, ctx)
finally: finally:
if sem is not None: if sem is not None:
sem.release() sem.release()
@@ -735,7 +773,7 @@ class ThreadedLayerRunner:
finally: finally:
with lock: with lock:
for name, (task_ctx, result) in completed.items(): for name, (task_ctx, result) in completed.items():
_store_result(name, result, specs[name], task_ctx, context, report, backend, on_event) _store_result(result, specs[name], task_ctx, ctx)
class AsyncLayerRunner: class AsyncLayerRunner:
@@ -745,29 +783,25 @@ class AsyncLayerRunner:
async def execute( async def execute(
layer: list[str], layer: list[str],
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
layer_idx: int, layer_idx: int,
on_event: EventCallback | None,
concurrency_limits: Mapping[str, int],
) -> None: ) -> None:
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event) to_run, specs = _filter_and_sort(layer, graph, ctx)
if not to_run: if not to_run:
return return
semaphores = _build_semaphores(to_run, specs, asyncio.Semaphore, concurrency_limits) semaphores = _build_semaphores(to_run, specs, asyncio.Semaphore, ctx.concurrency_limits)
context_snapshot = dict(context) context_snapshot = dict(ctx.context)
async def _run_async_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]: async def _run_async_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
spec = specs[name] spec = specs[name]
task_ctx = _build_context(spec, context_snapshot, report) task_ctx = _build_context(spec, context_snapshot, ctx.report)
sem = _get_sem(semaphores, spec) sem = _get_sem(semaphores, spec)
result = await AsyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report, sem) result = await AsyncTaskRunner.run(spec, task_ctx, layer_idx, ctx, sem)
return task_ctx, result return task_ctx, result
results = await asyncio.gather(*[_run_async_task(name) for name in to_run]) results = await asyncio.gather(*[_run_async_task(name) for name in to_run])
for name, (task_ctx, result) in zip(to_run, results, strict=True): for name, (task_ctx, result) in zip(to_run, results, strict=True):
_store_result(name, result, specs[name], task_ctx, context, report, backend, on_event) _store_result(result, specs[name], task_ctx, ctx)
def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]: def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
@@ -806,16 +840,11 @@ class DependencyRunner:
@staticmethod @staticmethod
async def execute( async def execute(
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None: ) -> None:
all_names = list(graph.all_specs().keys()) all_names = list(graph.all_specs().keys())
all_specs: dict[str, TaskSpec[Any]] = {name: graph.resolved_spec(name) for name in all_names} all_specs: dict[str, TaskSpec[Any]] = {name: graph.resolved_spec(name) for name in all_names}
semaphores = _build_semaphores(all_names, all_specs, asyncio.Semaphore, concurrency_limits) semaphores = _build_semaphores(all_names, all_specs, asyncio.Semaphore, ctx.concurrency_limits)
# 事件驱动调度:跟踪 completed / in_flight / remaining。 # 事件驱动调度:跟踪 completed / in_flight / remaining。
# 仅当任务依赖全部完成时才创建 asyncio Task,按优先级降序调度。 # 仅当任务依赖全部完成时才创建 asyncio Task,按优先级降序调度。
@@ -825,10 +854,10 @@ class DependencyRunner:
# 检查点恢复:已在 report 中的 SUCCESS 任务直接标记完成,跳过调度。 # 检查点恢复:已在 report 中的 SUCCESS 任务直接标记完成,跳过调度。
for name in list(remaining): for name in list(remaining):
if name in report.results and report.results[name].status == TaskStatus.SUCCESS: if name in ctx.report.results and ctx.report.results[name].status == TaskStatus.SUCCESS:
completed.add(name) completed.add(name)
remaining.discard(name) remaining.discard(name)
context[name] = report.results[name].value ctx.context[name] = ctx.report.results[name].value
# 增量就绪集:用 in_degree 计数器 + dependents 反向邻接表替代每轮 O(N) 扫描。 # 增量就绪集:用 in_degree 计数器 + dependents 反向邻接表替代每轮 O(N) 扫描。
# 每轮调度开销从 O(N*D) 降至 O(D_out),大图(10k+ 任务)显著加速。 # 每轮调度开销从 O(N*D) 降至 O(D_out),大图(10k+ 任务)显著加速。
@@ -850,7 +879,7 @@ class DependencyRunner:
all_specs[final_spec.name] = graph.resolved_spec(final_spec.name) all_specs[final_spec.name] = graph.resolved_spec(final_spec.name)
remaining.add(final_spec.name) remaining.add(final_spec.name)
if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores: if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores:
limit = concurrency_limits.get(final_spec.concurrency_key, 1) limit = ctx.concurrency_limits.get(final_spec.concurrency_key, 1)
semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit) semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit)
# 计算新任务的 in_degreespawner 可能尚未标记 completed,计入 unsatisfied # 计算新任务的 in_degreespawner 可能尚未标记 completed,计入 unsatisfied
deps = (*final_spec.depends_on, *final_spec.soft_depends_on) deps = (*final_spec.depends_on, *final_spec.soft_depends_on)
@@ -867,34 +896,34 @@ class DependencyRunner:
spec = all_specs[name] spec = all_specs[name]
# 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED # 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED
if _is_cancelled(cancel_event) and name not in report.results: if _is_cancelled(ctx.cancel_event) and name not in ctx.report.results:
result: TaskResult[Any] = TaskResult( result: TaskResult[Any] = TaskResult(
spec=spec, spec=spec,
status=TaskStatus.SKIPPED, status=TaskStatus.SKIPPED,
finished_at=datetime.now(), finished_at=datetime.now(),
reason="执行被取消", reason="执行被取消",
) )
report.results[name] = result ctx.report.results[name] = result
_emit(on_event, result) _emit(ctx.on_event, result)
return result return result
task_ctx = _build_context(spec, context, report) task_ctx = _build_context(spec, ctx.context, ctx.report)
if _apply_cached(name, spec, context, report, backend, on_event): if _apply_cached(spec, ctx):
return report.results[name] return ctx.report.results[spec.name]
sem = _get_sem(semaphores, spec) sem = _get_sem(semaphores, spec)
result = await AsyncTaskRunner.run(spec, task_ctx, None, on_event, report, sem) result = await AsyncTaskRunner.run(spec, task_ctx, None, ctx, sem)
# 动态任务生成:提取 specs,替换 result.value 为生成任务名列表 # 动态任务生成:提取 specs,替换 result.value 为生成任务名列表
spawned = _extract_dynamic_specs(result, spec) spawned = _extract_dynamic_specs(result, spec)
if spawned: if spawned:
spawned_names = [s.name for s in spawned] spawned_names = [s.name for s in spawned]
result = dc_replace(result, value=spawned_names) result = dc_replace(result, value=spawned_names)
context[name] = result.value ctx.context[name] = result.value
report.results[name] = result ctx.report.results[name] = result
_emit(on_event, result) _emit(ctx.on_event, result)
else: else:
_store_result(name, result, spec, task_ctx, context, report, backend, on_event) _store_result(result, spec, task_ctx, ctx)
# 注册动态生成的 specs,接入增量调度结构 # 注册动态生成的 specs,接入增量调度结构
for raw_spec in spawned: for raw_spec in spawned:
@@ -1018,13 +1047,8 @@ def _apply_subgraph_filter(
def _dispatch_strategy( def _dispatch_strategy(
strategy: Strategy, strategy: Strategy,
graph: Graph, graph: Graph,
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
effective_callback: EventCallback | None,
max_workers: int | None, max_workers: int | None,
limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None,
) -> None: ) -> None:
"""按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。 """按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。
@@ -1038,15 +1062,15 @@ def _dispatch_strategy(
) )
if strategy == "sequential": if strategy == "sequential":
layers = graph.layers() layers = graph.layers()
_drive_sequential(graph, layers, context, report, backend, effective_callback, cancel_event) _drive_sequential(graph, layers, ctx)
elif strategy == "thread": elif strategy == "thread":
layers = graph.layers() layers = graph.layers()
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits, cancel_event) _drive_threaded(graph, layers, ctx, max_workers)
elif strategy == "async": elif strategy == "async":
layers = graph.layers() layers = graph.layers()
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits, cancel_event)) asyncio.run(_async_drive(graph, layers, ctx))
elif strategy == "dependency": elif strategy == "dependency":
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits, cancel_event)) asyncio.run(DependencyRunner.execute(graph, ctx))
else: # pragma: no cover - Strategy Literal 已穷尽所有取值 else: # pragma: no cover - Strategy Literal 已穷尽所有取值
raise ValueError(f"Unknown strategy: {strategy!r}") raise ValueError(f"Unknown strategy: {strategy!r}")
@@ -1105,7 +1129,7 @@ def _apply_resume(
logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored}) logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored})
def run( def run( # noqa: PLR0913
graph: Graph, graph: Graph,
strategy: Strategy = "dependency", strategy: Strategy = "dependency",
*, *,
@@ -1221,15 +1245,24 @@ def run(
internal_cancel = CancelToken() internal_cancel = CancelToken()
effective_cancel: CancelToken | threading.Event | None = cancel_event effective_cancel: CancelToken | threading.Event | None = cancel_event
# 打包执行上下文:将 context/report/backend/on_event/concurrency_limits/cancel_event
# 捆绑为单一参数传递给调用链,使每个内部函数参数数 ≤5(PLR0913)。
ctx = _ExecContext(
context=context,
report=report,
backend=backend,
on_event=effective_callback,
concurrency_limits=limits,
cancel_event=effective_cancel,
)
# backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend); # backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend);
# MemoryBackend 为 no-op。即使中途抛出 TaskFailedErrorbatch 退出时 # MemoryBackend 为 no-op。即使中途抛出 TaskFailedErrorbatch 退出时
# 仍会 flush 一次,保留已成功任务的结果以便断点续跑。 # 仍会 flush 一次,保留已成功任务的结果以便断点续跑。
try: try:
with backend.batch(): with backend.batch():
try: try:
_dispatch_strategy( _dispatch_strategy(strategy, graph, ctx, max_workers)
strategy, graph, context, report, backend, effective_callback, max_workers, limits, effective_cancel
)
except TaskFailedError: except TaskFailedError:
report.success = False report.success = False
raise raise
@@ -1267,7 +1300,7 @@ def run(
return report return report
def run_iter( def run_iter( # noqa: PLR0913
graph: Graph, graph: Graph,
strategy: Strategy = "dependency", strategy: Strategy = "dependency",
*, *,
@@ -1369,53 +1402,39 @@ def _print_dry_run(graph: Graph, layers: list[list[str]]) -> None:
def _drive_sequential( def _drive_sequential(
graph: Graph, graph: Graph,
layers: list[list[str]], layers: list[list[str]],
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
cancel_event: threading.Event | CancelToken | None = None,
) -> None: ) -> None:
for idx, layer in enumerate(layers, 1): for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event): if _is_cancelled(ctx.cancel_event):
return return
SequentialLayerRunner.execute(layer, graph, context, report, backend, idx, on_event) SequentialLayerRunner.execute(layer, graph, ctx, idx)
def _drive_threaded( def _drive_threaded(
graph: Graph, graph: Graph,
layers: list[list[str]], layers: list[list[str]],
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
max_workers: int | None, max_workers: int | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None: ) -> None:
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。 # 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
max_layer_size = max((len(layer) for layer in layers), default=1) max_layer_size = max((len(layer) for layer in layers), default=1)
pool_workers = max_workers or max(1, min(32, max_layer_size)) pool_workers = max_workers or max(1, min(32, max_layer_size))
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool: with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
for idx, layer in enumerate(layers, 1): for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event): if _is_cancelled(ctx.cancel_event):
return return
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, pool, concurrency_limits) ThreadedLayerRunner.execute(layer, graph, ctx, idx, pool)
async def _async_drive( async def _async_drive(
graph: Graph, graph: Graph,
layers: list[list[str]], layers: list[list[str]],
context: dict[str, Any], ctx: _ExecContext,
report: RunReport,
backend: StateBackend,
on_event: EventCallback | None,
concurrency_limits: Mapping[str, int],
cancel_event: threading.Event | CancelToken | None = None,
) -> None: ) -> None:
for idx, layer in enumerate(layers, 1): for idx, layer in enumerate(layers, 1):
if _is_cancelled(cancel_event): if _is_cancelled(ctx.cancel_event):
return return
await AsyncLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, concurrency_limits) await AsyncLayerRunner.execute(layer, graph, ctx, idx)
def _mark_remaining_skipped( def _mark_remaining_skipped(
+1 -1
View File
@@ -70,7 +70,7 @@ __all__ = [
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
# 查找与遍历 # 查找与遍历
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
def find( def find( # noqa: PLR0913
root: str | Path, root: str | Path,
pattern: str = "*", pattern: str = "*",
*, *,
+5 -6
View File
@@ -6,7 +6,6 @@
from __future__ import annotations from __future__ import annotations
import ast import ast
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -69,7 +68,7 @@ def format_with_ruff(target: Path, fix: bool = True) -> None:
if fix: if fix:
cmd.append("--fix") cmd.append("--fix")
subprocess.run(cmd, check=True) px.sh(cmd, label="ruff format")
print(f"ruff format 完成: {target}") print(f"ruff format 完成: {target}")
@@ -87,7 +86,7 @@ def lint_with_ruff(target: Path, fix: bool = True) -> None:
if fix: if fix:
cmd.extend(["--fix", "--unsafe-fixes"]) cmd.extend(["--fix", "--unsafe-fixes"])
subprocess.run(cmd, check=True) px.sh(cmd, label="ruff check")
print(f"ruff check 完成: {target}") print(f"ruff check 完成: {target}")
@@ -210,7 +209,7 @@ def sync_pyproject_config(root_dir: Path = Path()) -> None:
print(f"找到 {len(sub_tomls)} 个子项目配置文件") print(f"找到 {len(sub_tomls)} 个子项目配置文件")
for sub_toml in sub_tomls: for sub_toml in sub_tomls:
subprocess.run(["ruff", "format", str(sub_toml)], check=False) px.sh(["ruff", "format", str(sub_toml)], check=False, label="ruff format")
print("配置同步完成") print("配置同步完成")
@@ -223,6 +222,6 @@ def format_all(root_dir: Path) -> None:
root_dir : Path root_dir : Path
根目录 根目录
""" """
subprocess.run(["ruff", "format", str(root_dir)], check=True) px.sh(["ruff", "format", str(root_dir)], label="ruff format")
subprocess.run(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], check=True) px.sh(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], label="ruff check")
print(f"格式化完成: {root_dir}") print(f"格式化完成: {root_dir}")
+3 -4
View File
@@ -15,7 +15,6 @@
from __future__ import annotations from __future__ import annotations
import re import re
import subprocess
from pathlib import Path from pathlib import Path
from typing import Literal from typing import Literal
@@ -221,12 +220,12 @@ def bump_project_version(part: BumpVersionType = "patch", no_tag: bool = False)
# 阶段 3: git add (按文件名) + commit + tag # 阶段 3: git add (按文件名) + commit + tag
relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)] relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)]
subprocess.run(["git", "add", *relative_files], check=True) px.sh(["git", "add", *relative_files], label="git add")
subprocess.run(["git", "commit", "-m", f"bump version to {new_version}"], check=True) px.sh(["git", "commit", "-m", f"bump version to {new_version}"], label="git commit")
if not no_tag: if not no_tag:
tag_name = f"v{new_version}" tag_name = f"v{new_version}"
subprocess.run(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], check=True) px.sh(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], label="git tag")
print(f"已创建标签: {tag_name}") print(f"已创建标签: {tag_name}")
return new_version return new_version
+8 -15
View File
@@ -5,7 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -40,16 +39,10 @@ def not_has_git_repo() -> bool:
def has_files() -> bool: def has_files() -> bool:
"""检查当前 Git 仓库是否有未提交的更改.""" """检查当前 Git 仓库是否有未提交的更改."""
try: result = px.sh(["git", "status", "--porcelain"], capture=True, check=False, label="git status")
result = subprocess.run( if result is None:
["git", "status", "--porcelain"],
capture_output=True,
text=True,
check=False,
)
return bool(result.stdout.strip())
except (subprocess.SubprocessError, OSError):
return False return False
return bool(result.stdout.strip())
# ============================================================================ # ============================================================================
@@ -69,8 +62,8 @@ def git_add_commit(message: str = "chore: update") -> None:
if not has_files(): if not has_files():
print("没有文件需要提交") print("没有文件需要提交")
return return
subprocess.run(["git", "add", "."], check=True) px.sh(["git", "add", "."], label="git add")
subprocess.run(["git", "commit", "-m", message], check=True) px.sh(["git", "commit", "-m", message], label="git commit")
@px.tool("gittool", subcommand="i", help="初始化并提交") @px.tool("gittool", subcommand="i", help="初始化并提交")
@@ -83,10 +76,10 @@ def git_init_add_commit(message: str = "init commit") -> None:
提交信息 (默认: ``init commit``) 提交信息 (默认: ``init commit``)
""" """
if not_has_git_repo(): if not_has_git_repo():
subprocess.run(["git", "init"], check=True) px.sh(["git", "init"], label="git init")
if has_files(): if has_files():
subprocess.run(["git", "add", "."], check=True) px.sh(["git", "add", "."], label="git add")
subprocess.run(["git", "commit", "-m", message], check=True) px.sh(["git", "commit", "-m", message], label="git commit")
else: else:
print("没有文件需要提交") print("没有文件需要提交")
+14 -33
View File
@@ -5,7 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -49,13 +48,8 @@ def run_ls_dyna(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
return return
cmd = get_ls_dyna_command(input_file, ncpu) cmd = get_ls_dyna_command(input_file, ncpu)
try: if px.sh(cmd, label="LS-DYNA 计算") is not None:
subprocess.run(cmd, check=True)
print(f"LS-DYNA 计算完成: {input_file}") print(f"LS-DYNA 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA 计算失败: {e}")
@px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算") @px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算")
@@ -75,37 +69,24 @@ def run_ls_dyna_mpi(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
return return
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"] cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
try: if px.sh(cmd, label="LS-DYNA MPI 计算") is not None:
subprocess.run(cmd, check=True)
print(f"LS-DYNA MPI 计算完成: {input_file}") print(f"LS-DYNA MPI 计算完成: {input_file}")
except FileNotFoundError:
print("未找到 mpirun 或 ls-dyna_mpp 命令")
except subprocess.CalledProcessError as e:
print(f"LS-DYNA MPI 计算失败: {e}")
@px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态") @px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态")
def check_ls_dyna_status() -> None: def check_ls_dyna_status() -> None:
"""检查 LS-DYNA 进程状态.""" """检查 LS-DYNA 进程状态."""
try: if Constants.IS_WINDOWS:
if Constants.IS_WINDOWS: result = px.sh(
result = subprocess.run( ["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"], capture=True,
capture_output=True, label="tasklist",
text=True, )
check=True, if result is not None:
)
print(result.stdout) print(result.stdout)
else:
result = px.sh(["pgrep", "-f", "ls-dyna"], capture=True, check=False, label="pgrep")
if result is not None and result.stdout.strip():
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
else: else:
result = subprocess.run( print("没有运行中的 LS-DYNA 进程")
["pgrep", "-f", "ls-dyna"],
capture_output=True,
text=True,
check=False,
)
if result.stdout.strip():
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
else:
print("没有运行中的 LS-DYNA 进程")
except subprocess.CalledProcessError as e:
print(f"检查进程状态失败: {e}")
+2 -3
View File
@@ -7,7 +7,6 @@ from __future__ import annotations
import platform import platform
import shutil import shutil
import subprocess
import urllib.request import urllib.request
import zipfile import zipfile
from pathlib import Path from pathlib import Path
@@ -124,7 +123,7 @@ def pack_dependencies(lib_dir: Path = Path("libs"), dependencies: list[str] | No
] ]
cmd.extend(dependencies) cmd.extend(dependencies)
subprocess.run(cmd, check=True) px.sh(cmd, label="pip install")
print(f"依赖打包完成: {lib_dir}") print(f"依赖打包完成: {lib_dir}")
@@ -150,7 +149,7 @@ def pack_wheel(project_dir: Path = Path(), output_dir: Path = Path("dist")) -> N
str(project_dir), str(project_dir),
] ]
subprocess.run(cmd, check=True) px.sh(cmd, label="pip wheel")
print(f"Wheel 打包完成: {output_dir}") print(f"Wheel 打包完成: {output_dir}")
+20 -36
View File
@@ -6,7 +6,6 @@
from __future__ import annotations from __future__ import annotations
import fnmatch import fnmatch
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -27,12 +26,10 @@ __all__ = [
PACKAGE_DIR = "packages" PACKAGE_DIR = "packages"
REQUIREMENTS_FILE = "requirements.txt" REQUIREMENTS_FILE = "requirements.txt"
_PROTECTED_PACKAGES: frozenset[str] = frozenset( _PROTECTED_PACKAGES: frozenset[str] = frozenset({
{ "pyflowx",
"pyflowx", "bitool",
"bitool", })
}
)
# ============================================================================ # ============================================================================
@@ -42,20 +39,14 @@ _PROTECTED_PACKAGES: frozenset[str] = frozenset(
def _get_installed_packages() -> list[str]: def _get_installed_packages() -> list[str]:
"""获取当前环境中所有已安装的包名.""" """获取当前环境中所有已安装的包名."""
try: result = px.sh(["pip", "list", "--format=freeze"], capture=True, label="pip list")
result = subprocess.run( if result is None:
["pip", "list", "--format=freeze"],
capture_output=True,
text=True,
check=True,
)
packages: list[str] = []
for line in result.stdout.strip().split("\n"):
if line and "==" in line:
pkg_name = line.split("==")[0].strip()
packages.append(pkg_name)
except (subprocess.SubprocessError, OSError):
return [] return []
packages: list[str] = []
for line in result.stdout.strip().split("\n"):
if line and "==" in line:
pkg_name = line.split("==")[0].strip()
packages.append(pkg_name)
return packages return packages
@@ -92,7 +83,7 @@ def pip_install(packages: list[str]) -> None:
packages : list[str] packages : list[str]
包名列表 包名列表
""" """
subprocess.run(["pip", "install", *packages], check=True) px.sh(["pip", "install", *packages], label="pip install")
print(f"安装完成: {', '.join(packages)}") print(f"安装完成: {', '.join(packages)}")
@@ -114,7 +105,7 @@ def pip_uninstall(packages: list[str]) -> None:
if not packages_to_uninstall: if not packages_to_uninstall:
return return
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True) px.sh(["pip", "uninstall", "-y", *packages_to_uninstall], label="pip uninstall")
@px.tool("piptool", subcommand="r", help="重装包") @px.tool("piptool", subcommand="r", help="重装包")
@@ -133,10 +124,10 @@ def pip_reinstall(packages: list[str], offline: bool = False) -> None:
print("所有指定的包均为受保护包, 跳过重装") print("所有指定的包均为受保护包, 跳过重装")
return return
subprocess.run(["pip", "uninstall", "-y", *safe_ps], check=True) px.sh(["pip", "uninstall", "-y", *safe_ps], label="pip uninstall")
options = ["--no-index", "--find-links", "."] if offline else [] options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run(["pip", "install", *options, *safe_ps], check=True) px.sh(["pip", "install", *options, *safe_ps], label="pip install")
@px.tool("piptool", subcommand="d", help="下载包") @px.tool("piptool", subcommand="d", help="下载包")
@@ -151,27 +142,20 @@ def pip_download(packages: list[str], offline: bool = False) -> None:
离线模式 离线模式
""" """
options = ["--no-index", "--find-links", "."] if offline else [] options = ["--no-index", "--find-links", "."] if offline else []
subprocess.run( px.sh(["pip", "download", *packages, *options, "-d", PACKAGE_DIR], label="pip download")
["pip", "download", *packages, *options, "-d", PACKAGE_DIR],
check=True,
)
@px.tool("piptool", subcommand="up", help="升级 pip") @px.tool("piptool", subcommand="up", help="升级 pip")
def pip_upgrade() -> None: def pip_upgrade() -> None:
"""升级 pip 到最新版本.""" """升级 pip 到最新版本."""
subprocess.run(["python", "-m", "pip", "install", "--upgrade", "pip"], check=True) px.sh(["python", "-m", "pip", "install", "--upgrade", "pip"], label="pip 升级")
print("pip 升级完成") print("pip 升级完成")
@px.tool("piptool", subcommand="f", help="导出依赖") @px.tool("piptool", subcommand="f", help="导出依赖")
def pip_freeze() -> None: def pip_freeze() -> None:
"""冻结依赖到 requirements.txt.""" """冻结依赖到 requirements.txt."""
result = subprocess.run( result = px.sh(["pip", "freeze", "--exclude-editable"], capture=True, label="pip freeze")
["pip", "freeze", "--exclude-editable"], if result is not None:
capture_output=True, Path(REQUIREMENTS_FILE).write_text(result.stdout)
text=True,
check=True,
)
Path(REQUIREMENTS_FILE).write_text(result.stdout)
print(f"依赖已导出到 {REQUIREMENTS_FILE}") print(f"依赖已导出到 {REQUIREMENTS_FILE}")
+2 -2
View File
@@ -99,7 +99,7 @@ def image_resize(
@px.tool("imagetool", subcommand="c", help="裁剪图片") @px.tool("imagetool", subcommand="c", help="裁剪图片")
def image_crop( def image_crop( # noqa: PLR0913
input_path: Path, input_path: Path,
output_path: Path, output_path: Path,
left: int, left: int,
@@ -213,7 +213,7 @@ def image_convert(
@px.tool("imagetool", subcommand="wm", help="添加文字水印") @px.tool("imagetool", subcommand="wm", help="添加文字水印")
def image_watermark( def image_watermark( # noqa: PLR0913
input_path: Path, input_path: Path,
output_path: Path, output_path: Path,
text: str, text: str,
+8 -15
View File
@@ -5,7 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
@@ -59,14 +58,11 @@ $bitmap.Save('{output_path.as_posix()}')
$graphics.Dispose() $graphics.Dispose()
$bitmap.Dispose() $bitmap.Dispose()
""" """
subprocess.run(["powershell", "-Command", ps_script], check=True) px.sh(["powershell", "-Command", ps_script], label="截图")
elif Constants.IS_MACOS: elif Constants.IS_MACOS:
subprocess.run(["screencapture", "-x", str(output_path)], check=True) px.sh(["screencapture", "-x", str(output_path)], label="截图")
else: elif px.sh(["gnome-screenshot", "-f", str(output_path)], label="截图") is None:
try: px.sh(["scrot", str(output_path)], label="截图")
subprocess.run(["gnome-screenshot", "-f", str(output_path)], check=True)
except FileNotFoundError:
subprocess.run(["scrot", str(output_path)], check=True)
print(f"截图已保存: {output_path}") print(f"截图已保存: {output_path}")
@@ -104,13 +100,10 @@ $bitmap.Save('{output_path.as_posix()}')
$graphics.Dispose() $graphics.Dispose()
$bitmap.Dispose() $bitmap.Dispose()
""" """
subprocess.run(["powershell", "-Command", ps_script], check=True) px.sh(["powershell", "-Command", ps_script], label="截图")
elif Constants.IS_MACOS: elif Constants.IS_MACOS:
subprocess.run(["screencapture", "-i", str(output_path)], check=True) px.sh(["screencapture", "-i", str(output_path)], label="截图")
else: elif px.sh(["gnome-screenshot", "-a", "-f", str(output_path)], label="截图") is None:
try: px.sh(["scrot", "-s", str(output_path)], label="截图")
subprocess.run(["gnome-screenshot", "-a", "-f", str(output_path)], check=True)
except FileNotFoundError:
subprocess.run(["scrot", "-s", str(output_path)], check=True)
print(f"截图已保存: {output_path}") print(f"截图已保存: {output_path}")
+5 -9
View File
@@ -6,7 +6,6 @@
from __future__ import annotations from __future__ import annotations
import getpass import getpass
import subprocess
import pyflowx as px import pyflowx as px
@@ -23,15 +22,12 @@ def docker_login_tencent(username: str = "") -> None:
Docker 用户名 (为空时由 docker 交互式提示输入) Docker 用户名 (为空时由 docker 交互式提示输入)
""" """
user = username or getpass.getuser() user = username or getpass.getuser()
try: if (
subprocess.run( px.sh(
["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT], ["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT],
check=True, label="docker login",
) )
except subprocess.CalledProcessError as e: is None
print(f"docker login 失败 (returncode={e.returncode}): {e}") ):
return
except FileNotFoundError:
print("docker 命令未找到,请确认 Docker 已安装并在 PATH 中")
return return
print(f"已登录腾讯云镜像仓库 (用户: {user})") print(f"已登录腾讯云镜像仓库 (用户: {user})")
+11 -10
View File
@@ -10,7 +10,6 @@ from __future__ import annotations
import getpass import getpass
import os import os
import shutil import shutil
import subprocess
from pathlib import Path from pathlib import Path
from typing import Literal from typing import Literal
@@ -289,7 +288,7 @@ def download_rustup_script() -> None:
if Constants.IS_WINDOWS: if Constants.IS_WINDOWS:
print("下载 rustup-init.exe...") print("下载 rustup-init.exe...")
subprocess.run( px.sh(
[ [
"powershell", "powershell",
"-Command", "-Command",
@@ -300,12 +299,14 @@ def download_rustup_script() -> None:
"rustup-init.exe", "rustup-init.exe",
], ],
check=False, check=False,
label="下载 rustup",
) )
else: else:
print("下载 rustup-init.sh...") print("下载 rustup-init.sh...")
subprocess.run( px.sh(
["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"], ["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
check=False, check=False,
label="下载 rustup",
) )
@@ -328,7 +329,7 @@ def install_rust_toolchain(version: str = "stable") -> None:
print("rustup 未安装, 跳过工具链安装") print("rustup 未安装, 跳过工具链安装")
return return
subprocess.run(["rustup", "toolchain", "install", version], check=False) px.sh(["rustup", "toolchain", "install", version], check=False, label="安装 Rust 工具链")
print(f"Rust 工具链 {version} 安装完成") print(f"Rust 工具链 {version} 安装完成")
@@ -364,9 +365,9 @@ def setup_linux_system_mirror() -> None:
return return
print("下载 linuxmirrors 脚本...") print("下载 linuxmirrors 脚本...")
subprocess.run(_DOWNLOAD_MIRROR_SCRIPT, shell=True, check=False) px.sh(_DOWNLOAD_MIRROR_SCRIPT, check=False, label="下载镜像脚本")
print("安装 linuxmirrors...") print("安装 linuxmirrors...")
subprocess.run(_INSTALL_MIRROR_SCRIPT, shell=True, check=False) px.sh(_INSTALL_MIRROR_SCRIPT, check=False, label="安装镜像")
@px.tool( @px.tool(
@@ -381,7 +382,7 @@ def install_linux_qt_libs() -> None:
if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"): if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"):
return return
subprocess.run(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False) px.sh(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False, label="安装 Qt 依赖")
print("Qt 依赖库安装完成") print("Qt 依赖库安装完成")
@@ -397,7 +398,7 @@ def install_linux_fonts() -> None:
if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"): if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"):
return return
subprocess.run(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False) px.sh(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False, label="安装中文字体")
print("中文字体安装完成") print("中文字体安装完成")
@@ -413,6 +414,6 @@ def install_linux_docker() -> None:
if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"): if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"):
return return
subprocess.run(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False) px.sh(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False, label="安装 Docker")
subprocess.run(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False) px.sh(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False, label="加入 docker 组")
print("Docker 安装完成 (需重新登录以生效 docker 用户组)") print("Docker 安装完成 (需重新登录以生效 docker 用户组)")
+1 -7
View File
@@ -5,7 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -38,9 +37,4 @@ def msdownload_run(name: str, target_type: str = "model", download_dir: str | No
cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)] cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)]
print(f"下载 {target_type}: {name} -> {out_dir}") print(f"下载 {target_type}: {name} -> {out_dir}")
try: px.sh(cmd, label="下载")
subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
print(f"下载失败 (returncode={e.returncode}): {e}")
except FileNotFoundError:
print("uvx 命令未找到,请确认 uv 已安装并在 PATH 中")
+1 -1
View File
@@ -32,7 +32,7 @@ def install_sglang() -> None:
@px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"]) @px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"])
def run_sglang( def run_sglang( # noqa: PLR0913
model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ", model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ",
port: int = 8000, port: int = 8000,
ctx_len: int = 32768, ctx_len: int = 32768,
+24 -32
View File
@@ -5,7 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
import sys import sys
from pathlib import Path from pathlib import Path
@@ -15,7 +14,7 @@ __all__ = ["ssh_copy_id"]
@px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器") @px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器")
def ssh_copy_id( def ssh_copy_id( # noqa: PLR0913
hostname: str, hostname: str,
username: str, username: str,
password: str, password: str,
@@ -51,34 +50,27 @@ def ssh_copy_id(
cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys
grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys""" grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys"""
try: result = px.sh(
subprocess.run( [
[ "sshpass",
"sshpass", "-p",
"-p", password,
password, "ssh",
"ssh", "-p",
"-p", str(port),
str(port), "-o",
"-o", "StrictHostKeyChecking=no",
"StrictHostKeyChecking=no", "-o",
"-o", "UserKnownHostsFile=/dev/null",
"UserKnownHostsFile=/dev/null", "-o",
"-o", f"ConnectTimeout={timeout}",
f"ConnectTimeout={timeout}", f"{username}@{hostname}",
f"{username}@{hostname}", script,
script, ],
], timeout=timeout,
check=True, label="SSH 密钥部署",
timeout=timeout, )
) if result is None:
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}") print(f"可手动执行: ssh-copy-id -p {port} {username}@{hostname}")
except FileNotFoundError:
print(f"未找到 sshpass 工具, 请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
sys.exit(1)
except subprocess.TimeoutExpired:
print("SSH 连接超时")
sys.exit(1)
except subprocess.CalledProcessError as e:
print(f"SSH 执行失败: {e}")
sys.exit(1) sys.exit(1)
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}")
+1 -3
View File
@@ -5,8 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
import pyflowx as px import pyflowx as px
from pyflowx.ops._common import platform_command from pyflowx.ops._common import platform_command
@@ -18,4 +16,4 @@ def clear_screen_run() -> None:
Windows 调用 ``cls``, Linux/macOS 调用 ``clear``. Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
""" """
# 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍 # 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍
subprocess.run(platform_command(["cls"], ["clear"]), check=False) px.sh(platform_command(["cls"], ["clear"]), check=False, label="清屏")
+5 -5
View File
@@ -7,7 +7,6 @@
from __future__ import annotations from __future__ import annotations
import os import os
import subprocess
from pathlib import Path from pathlib import Path
import pyflowx as px import pyflowx as px
@@ -34,19 +33,20 @@ def reset_icon_cache_run() -> None:
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer" explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
print("正在终止 explorer 进程...") print("正在终止 explorer 进程...")
subprocess.run(["taskkill", "/f", "/im", "explorer.exe"], check=False) px.sh(["taskkill", "/f", "/im", "explorer.exe"], check=False, label="终止 explorer")
if icon_cache_db.exists(): if icon_cache_db.exists():
print(f"删除图标缓存: {icon_cache_db}") print(f"删除图标缓存: {icon_cache_db}")
subprocess.run(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False) px.sh(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False, label="删除图标缓存")
if explorer_cache_dir.exists(): if explorer_cache_dir.exists():
print(f"清理 Explorer 缓存: {explorer_cache_dir}") print(f"清理 Explorer 缓存: {explorer_cache_dir}")
subprocess.run( px.sh(
["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")], ["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
check=False, check=False,
label="清理 Explorer 缓存",
) )
print("重启 explorer...") print("重启 explorer...")
subprocess.run(["cmd", "/c", "start", "explorer.exe"], check=False) px.sh(["cmd", "/c", "start", "explorer.exe"], check=False, label="重启 explorer")
print("图标缓存已重置") print("图标缓存已重置")
+4 -4
View File
@@ -5,8 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
import pyflowx as px import pyflowx as px
from pyflowx.ops._common import platform_command from pyflowx.ops._common import platform_command
@@ -28,8 +26,10 @@ def taskkill_run(process_names: list[str]) -> None:
for name in process_names: for name in process_names:
print(f"终止进程: {name}") print(f"终止进程: {name}")
# pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode # pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode
result = subprocess.run([*cmd_prefix, f"{name}*"], check=False) result = px.sh([*cmd_prefix, f"{name}*"], check=False, label="终止进程")
if result.returncode == 0: if result is None:
print(f" 终止命令不可用: {name}")
elif result.returncode == 0:
print(f" 已发送终止信号: {name}") print(f" 已发送终止信号: {name}")
else: else:
print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}") print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}")
+2 -4
View File
@@ -5,8 +5,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
import pyflowx as px import pyflowx as px
from pyflowx.ops._common import platform_command from pyflowx.ops._common import platform_command
@@ -27,8 +25,8 @@ def which_run(commands: list[str]) -> None:
for cmd in commands: for cmd in commands:
# where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode # where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode
result = subprocess.run([which_cmd, cmd], capture_output=True, text=True, check=False) result = px.sh([which_cmd, cmd], capture=True, check=False, label="which")
if result.returncode == 0: if result is not None and result.returncode == 0:
# Windows 的 where 可能返回多行, 取第一个 # Windows 的 where 可能返回多行, 取第一个
path = result.stdout.strip().split("\n")[0].strip() path = result.stdout.strip().split("\n")[0].strip()
print(f"{cmd} -> {path}") print(f"{cmd} -> {path}")
+30 -6
View File
@@ -12,9 +12,14 @@
result = px.sh("echo hello", capture=True) result = px.sh("echo hello", capture=True)
if result is not None: if result is not None:
print(result.stdout.strip()) print(result.stdout.strip())
# check=False 时返回 CompletedProcess(含非零返回码),调用方检查 returncode
res = px.sh(["git", "status", "--porcelain"], capture=True, check=False)
if res is not None and res.returncode == 0:
print(res.stdout)
设计定位简单"执行 + 报错"场景的快捷方式需要 ``capture_output`` + 设计定位简单"执行 + 报错"场景的快捷方式``check=False`` 模式覆盖
手动检查返回码等精细控制时直接使用 :func:`subprocess.run` "容忍非零返回码 + 手动检查"场景仍需更精细控制时直接使用
:func:`subprocess.run`
""" """
from __future__ import annotations from __future__ import annotations
@@ -29,12 +34,17 @@ def sh(
*, *,
capture: bool = False, capture: bool = False,
label: str = "命令", label: str = "命令",
check: bool = True,
timeout: float | None = None,
) -> subprocess.CompletedProcess[str] | None: ) -> subprocess.CompletedProcess[str] | None:
"""执行 shell 命令,统一错误处理与中文输出。 """执行 shell 命令,统一错误处理与中文输出。
``list[str]`` 直接执行``str`` ``shell=True``支持管道/重定向等语法 ``list[str]`` 直接执行``str`` ``shell=True``支持管道/重定向等语法
成功返回 :class:`subprocess.CompletedProcess`失败时打印中文错误信息
并返回 ``None``不抛异常调用方用 ``result is None`` 判断是否失败 * ``check=True``默认成功返回 :class:`subprocess.CompletedProcess`
非零返回码 / 命令未找到 / 超时均打印中文错误并返回 ``None``
* ``check=False``任何返回码都返回 :class:`subprocess.CompletedProcess`
调用方检查 ``returncode``仅命令未找到 / 超时返回 ``None``
Parameters Parameters
---------- ----------
@@ -45,11 +55,18 @@ def sh(
可从返回值的 ``.stdout`` / ``.stderr`` 读取默认 ``False`` 可从返回值的 ``.stdout`` / ``.stderr`` 读取默认 ``False``
label : str label : str
命令标签用于错误信息中标识命令用途默认 ``"命令"`` 命令标签用于错误信息中标识命令用途默认 ``"命令"``
check : bool
``True`` 时非零返回码视为失败打印错误并返回 ``None``
``False`` 时总是返回 :class:`subprocess.CompletedProcess`
由调用方检查 ``returncode``默认 ``True``
timeout : float | None
命令超时秒数超时打印错误并返回 ``None``默认 ``None`` 不限时
Returns Returns
------- -------
subprocess.CompletedProcess[str] | None subprocess.CompletedProcess[str] | None
成功时返回执行结果失败返回 ``None`` ``check=True`` 失败返回 ``None````check=False`` 时仅命令未找到 /
超时返回 ``None``其余返回 :class:`subprocess.CompletedProcess`
Examples Examples
-------- --------
@@ -58,6 +75,9 @@ def sh(
>>> result = px.sh("pip --version", capture=True, label="pip") >>> result = px.sh("pip --version", capture=True, label="pip")
>>> if result is not None: >>> if result is not None:
... print(result.stdout) ... print(result.stdout)
>>> res = px.sh(["git", "status"], check=False, label="git") # 容忍非零返回码
>>> if res is not None and res.returncode == 0:
... print("git 可用")
""" """
try: try:
return subprocess.run( return subprocess.run(
@@ -65,7 +85,8 @@ def sh(
shell=isinstance(cmd, str), shell=isinstance(cmd, str),
capture_output=capture, capture_output=capture,
text=True, text=True,
check=True, check=check,
timeout=timeout,
) )
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:
print(f"{label}失败 (returncode={e.returncode}): {e}") print(f"{label}失败 (returncode={e.returncode}): {e}")
@@ -73,3 +94,6 @@ def sh(
except FileNotFoundError: except FileNotFoundError:
print(f"{label}失败: 命令未找到,请确认已安装并在 PATH 中") print(f"{label}失败: 命令未找到,请确认已安装并在 PATH 中")
return None return None
except subprocess.TimeoutExpired:
print(f"{label}超时 (timeout={timeout}s)")
return None
+3 -3
View File
@@ -19,7 +19,7 @@ import threading
import time import time
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from collections import OrderedDict from collections import OrderedDict
from collections.abc import Iterator, Mapping from collections.abc import Generator, Iterator, Mapping
from contextlib import AbstractContextManager, contextmanager, nullcontext from contextlib import AbstractContextManager, contextmanager, nullcontext
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@@ -328,7 +328,7 @@ class JSONBackend(_TTLStateBackendMixin):
@override @override
@contextmanager @contextmanager
def batch(self) -> Iterator[None]: def batch(self) -> Generator[None, None, None]:
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。 """进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
将整次运行 N 个任务的 N 次全量落盘降为 1 将整次运行 N 个任务的 N 次全量落盘降为 1
@@ -462,7 +462,7 @@ class SQLiteBackend(_TTLStateBackendMixin):
@override @override
@contextmanager @contextmanager
def batch(self) -> Iterator[None]: def batch(self) -> Generator[None, None, None]:
"""进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。 """进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。
将整次运行 N 个任务的 N commit 降为 1 将整次运行 N 个任务的 N commit 降为 1
+1 -1
View File
@@ -513,7 +513,7 @@ def _task_noop() -> None:
return None return None
def task( def task( # noqa: PLR0913
fn: TaskFn[Any] | None = None, fn: TaskFn[Any] | None = None,
*, *,
cmd: TaskCmd | None = None, cmd: TaskCmd | None = None,
+1 -1
View File
@@ -114,7 +114,7 @@ _TOOL_REGISTRY: dict[str, dict[str | None, ToolSpec]] = {}
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
# @tool 装饰器 + 注册表 # @tool 装饰器 + 注册表
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
def tool( def tool( # noqa: PLR0913
name: str, name: str,
*, *,
subcommand: str | None = None, subcommand: str | None = None,
+8 -8
View File
@@ -16,14 +16,14 @@ class TestFormatWithRuff:
def test_format_with_ruff(self, tmp_path: Path) -> None: def test_format_with_ruff(self, tmp_path: Path) -> None:
"""Should format with ruff.""" """Should format with ruff."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.format_with_ruff(tmp_path, fix=True) dev.format_with_ruff(tmp_path, fix=True)
assert mock_run.called assert mock_run.called
def test_format_with_ruff_no_fix(self, tmp_path: Path) -> None: def test_format_with_ruff_no_fix(self, tmp_path: Path) -> None:
"""Should format with ruff without fix.""" """Should format with ruff without fix."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.format_with_ruff(tmp_path, fix=False) dev.format_with_ruff(tmp_path, fix=False)
# Should not include --fix flag # Should not include --fix flag
@@ -39,14 +39,14 @@ class TestLintWithRuff:
def test_lint_with_ruff(self, tmp_path: Path) -> None: def test_lint_with_ruff(self, tmp_path: Path) -> None:
"""Should lint with ruff.""" """Should lint with ruff."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.lint_with_ruff(tmp_path, fix=True) dev.lint_with_ruff(tmp_path, fix=True)
assert mock_run.called assert mock_run.called
def test_lint_with_ruff_no_fix(self, tmp_path: Path) -> None: def test_lint_with_ruff_no_fix(self, tmp_path: Path) -> None:
"""Should lint with ruff without fix.""" """Should lint with ruff without fix."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.lint_with_ruff(tmp_path, fix=False) dev.lint_with_ruff(tmp_path, fix=False)
# Should not include --fix flag # Should not include --fix flag
@@ -176,7 +176,7 @@ class TestSyncPyprojectConfig:
sub_toml = sub_dir / "pyproject.toml" sub_toml = sub_dir / "pyproject.toml"
sub_toml.write_text("[tool.ruff]\n") sub_toml.write_text("[tool.ruff]\n")
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.sync_pyproject_config(tmp_path) dev.sync_pyproject_config(tmp_path)
assert mock_run.called assert mock_run.called
@@ -190,7 +190,7 @@ class TestSyncPyprojectConfig:
sub_toml = sub_dir / "pyproject.toml" sub_toml = sub_dir / "pyproject.toml"
sub_toml.write_text("[tool.ruff]\n") sub_toml.write_text("[tool.ruff]\n")
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.sync_pyproject_config(tmp_path) dev.sync_pyproject_config(tmp_path)
assert mock_run.called assert mock_run.called
@@ -204,14 +204,14 @@ class TestFormatAll:
def test_format_all_runs_ruff_format(self, tmp_path: Path) -> None: def test_format_all_runs_ruff_format(self, tmp_path: Path) -> None:
"""Should run ruff format.""" """Should run ruff format."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.format_all(tmp_path) dev.format_all(tmp_path)
assert mock_run.called assert mock_run.called
def test_format_all_runs_ruff_check(self, tmp_path: Path) -> None: def test_format_all_runs_ruff_check(self, tmp_path: Path) -> None:
"""Should run ruff check.""" """Should run ruff check."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.format_all(tmp_path) dev.format_all(tmp_path)
# Should call ruff format and ruff check # Should call ruff format and ruff check
+66 -8
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import re
import subprocess import subprocess
from pathlib import Path from pathlib import Path
@@ -9,6 +10,13 @@ import pytest
from pyflowx.ops.dev import bumpversion from pyflowx.ops.dev import bumpversion
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[a-zA-Z]")
def _strip_ansi(text: str) -> str:
"""移除 ANSI 转义码,使断言在 CI(富终端) 与本地环境一致."""
return _ANSI_RE.sub("", text)
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def auto_use_tmp_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def auto_use_tmp_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -104,15 +112,15 @@ class TestBumpProjectVersion:
"""Test bump_project_version function.""" """Test bump_project_version function."""
@staticmethod @staticmethod
def _mock_subprocess(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]: def _mock_sh(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
"""Mock subprocess.run, 返回调用记录列表.""" """Mock px.sh, 返回调用记录列表."""
calls: list[list[str]] = [] calls: list[list[str]] = []
def fake_run(cmd: list[str], **_kwargs: object) -> subprocess.CompletedProcess[bytes]: def fake_sh(cmd: list[str], **_kwargs: object) -> subprocess.CompletedProcess[bytes]:
calls.append(cmd) calls.append(cmd)
return subprocess.CompletedProcess(cmd, 0, b"", b"") return subprocess.CompletedProcess(cmd, 0, b"", b"")
monkeypatch.setattr(subprocess, "run", fake_run) monkeypatch.setattr("pyflowx.sh", fake_sh)
return calls return calls
def test_unsynced_files_synchronized( def test_unsynced_files_synchronized(
@@ -129,7 +137,7 @@ class TestBumpProjectVersion:
pyproj = tmp_path / "pyproject.toml" pyproj = tmp_path / "pyproject.toml"
pyproj.write_text('version = "0.3.5"', encoding="utf-8") pyproj.write_text('version = "0.3.5"', encoding="utf-8")
calls = self._mock_subprocess(monkeypatch) calls = self._mock_sh(monkeypatch)
result = bumpversion.bump_project_version("patch") result = bumpversion.bump_project_version("patch")
@@ -163,7 +171,7 @@ class TestBumpProjectVersion:
"""有文件但所有文件都无版本号返回 None.""" """有文件但所有文件都无版本号返回 None."""
f = tmp_path / "__init__.py" f = tmp_path / "__init__.py"
f.write_text("# no version here", encoding="utf-8") f.write_text("# no version here", encoding="utf-8")
self._mock_subprocess(monkeypatch) self._mock_sh(monkeypatch)
assert bumpversion.bump_project_version("patch") is None assert bumpversion.bump_project_version("patch") is None
assert "未能从任何文件读取版本号" in capsys.readouterr().out assert "未能从任何文件读取版本号" in capsys.readouterr().out
@@ -173,7 +181,7 @@ class TestBumpProjectVersion:
pyproj = tmp_path / "pyproject.toml" pyproj = tmp_path / "pyproject.toml"
pyproj.write_text('version = "1.0.0"', encoding="utf-8") pyproj.write_text('version = "1.0.0"', encoding="utf-8")
calls = self._mock_subprocess(monkeypatch) calls = self._mock_sh(monkeypatch)
assert bumpversion.bump_project_version("patch", no_tag=True) == "1.0.1" assert bumpversion.bump_project_version("patch", no_tag=True) == "1.0.1"
assert not any(c[:2] == ["git", "tag"] for c in calls) assert not any(c[:2] == ["git", "tag"] for c in calls)
@@ -186,7 +194,57 @@ class TestBumpProjectVersion:
pyproj = tmp_path / "pyproject.toml" pyproj = tmp_path / "pyproject.toml"
pyproj.write_text('version = "1.0.0"', encoding="utf-8") pyproj.write_text('version = "1.0.0"', encoding="utf-8")
self._mock_subprocess(monkeypatch) self._mock_sh(monkeypatch)
assert bumpversion.bump_project_version("patch") == "1.0.1" assert bumpversion.bump_project_version("patch") == "1.0.1"
assert venv_init.read_text(encoding="utf-8") == '__version__ = "0.1.0"' assert venv_init.read_text(encoding="utf-8") == '__version__ = "0.1.0"'
# ---------------------------------------------------------------------- #
# CLI 调度测试 (pf bumpversion -h / pf bumpversion --part X --dry-run)
# ---------------------------------------------------------------------- #
class TestBumpversionCliDispatch:
"""``pf bumpversion`` CLI 调度测试 (不执行真实 bump)."""
def test_bumpversion_h_shows_help(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf bumpversion -h 应返回 0 并显示 --part 选项."""
from pyflowx.cli.pf import PfApp
app = PfApp(["bumpversion", "-h"])
rc = app.run()
assert rc == 0
out = _strip_ansi(capsys.readouterr().out)
assert "--part" in out
assert "patch" in out
def test_bumpversion_part_patch_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf bumpversion --part patch --dry-run 应打印执行计划, 不执行 bump."""
from pyflowx.cli.pf import PfApp
app = PfApp(["bumpversion", "--part", "patch", "--dry-run"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Dry run" in out
assert "bumpversion" in out
def test_bumpversion_part_minor_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf bumpversion --part minor --dry-run 应打印执行计划."""
from pyflowx.cli.pf import PfApp
app = PfApp(["bumpversion", "--part", "minor", "--dry-run"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Dry run" in out
def test_bumpversion_default_part_is_patch(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf bumpversion --dry-run (无 --part) 默认使用 patch."""
from pyflowx.cli.pf import PfApp
app = PfApp(["bumpversion", "--dry-run"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Dry run" in out
assert "patch" in out
+3 -2
View File
@@ -178,7 +178,8 @@ class TestDockerLoginTencent:
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err)) monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))
docker_login_tencent("myuser") docker_login_tencent("myuser")
captured = capsys.readouterr() captured = capsys.readouterr()
assert "docker login 失败" in captured.out assert "docker login" in captured.out
assert "失败" in captured.out
assert "returncode=1" in captured.out assert "returncode=1" in captured.out
assert "已登录" not in captured.out assert "已登录" not in captured.out
@@ -189,7 +190,7 @@ class TestDockerLoginTencent:
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError())) monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
docker_login_tencent("myuser") docker_login_tencent("myuser")
captured = capsys.readouterr() captured = capsys.readouterr()
assert "docker 命令未找到" in captured.out assert "命令未找到" in captured.out
assert "已登录" not in captured.out assert "已登录" not in captured.out
+2 -2
View File
@@ -53,7 +53,7 @@ class TestHasFiles:
class _FakeResult: class _FakeResult:
stdout = " M test.txt\n" stdout = " M test.txt\n"
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult()) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: _FakeResult())
result = dev.has_files() result = dev.has_files()
assert result is True assert result is True
@@ -63,7 +63,7 @@ class TestHasFiles:
class _FakeResult: class _FakeResult:
stdout = "" stdout = ""
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult()) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: _FakeResult())
result = dev.has_files() result = dev.has_files()
assert result is False assert result is False
+3 -17
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from pathlib import Path from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
@@ -115,22 +114,9 @@ class TestGittoolHelpers:
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult()) monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult())
assert gittool.has_files() is True assert gittool.has_files() is True
def test_has_files_false_on_subprocess_error(self, monkeypatch: pytest.MonkeyPatch) -> None: def test_has_files_false_when_sh_fails(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""subprocess.SubprocessError 时 has_files 应返回 False.""" """px.sh 返回 None(命令失败/未找到)时 has_files 应返回 False."""
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)
def _raise(*args: object, **kwargs: object) -> None:
raise subprocess.SubprocessError("fail")
monkeypatch.setattr("subprocess.run", _raise)
assert gittool.has_files() is False
def test_has_files_false_on_oserror(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""OSError 时 has_files 应返回 False."""
def _raise(*args: object, **kwargs: object) -> None:
raise OSError("fail")
monkeypatch.setattr("subprocess.run", _raise)
assert gittool.has_files() is False assert gittool.has_files() is False
+1 -1
View File
@@ -90,7 +90,7 @@ class TestMsdownloadRun:
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError())) monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
msdownload_run("foo/bar") msdownload_run("foo/bar")
captured = capsys.readouterr() captured = capsys.readouterr()
assert "uvx 命令未找到" in captured.out assert "命令未找到" in captured.out
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
+5 -5
View File
@@ -43,7 +43,7 @@ class TestRunLsDyna:
input_file = tmp_path / "input.k" input_file = tmp_path / "input.k"
input_file.write_text("LS-DYNA input") input_file.write_text("LS-DYNA input")
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
system.run_ls_dyna(str(input_file), ncpu=4) system.run_ls_dyna(str(input_file), ncpu=4)
assert mock_run.called assert mock_run.called
@@ -60,7 +60,7 @@ class TestRunLsDyna:
input_file = tmp_path / "input.k" input_file = tmp_path / "input.k"
input_file.write_text("LS-DYNA input") input_file.write_text("LS-DYNA input")
with patch("subprocess.run", side_effect=FileNotFoundError): with patch("pyflowx.sh", return_value=None):
system.run_ls_dyna(str(input_file), ncpu=4) system.run_ls_dyna(str(input_file), ncpu=4)
# Should print error message # Should print error message
@@ -76,7 +76,7 @@ class TestRunLsDynaMpi:
input_file = tmp_path / "input.k" input_file = tmp_path / "input.k"
input_file.write_text("LS-DYNA input") input_file.write_text("LS-DYNA input")
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
system.run_ls_dyna_mpi(str(input_file), ncpu=8) system.run_ls_dyna_mpi(str(input_file), ncpu=8)
assert mock_run.called assert mock_run.called
@@ -97,14 +97,14 @@ class TestCheckLsDynaStatus:
def test_check_ls_dyna_status_windows(self) -> None: def test_check_ls_dyna_status_windows(self) -> None:
"""Should check LS-DYNA status on Windows.""" """Should check LS-DYNA status on Windows."""
with patch.object(Constants, "IS_WINDOWS", True), patch("subprocess.run") as mock_run: with patch.object(Constants, "IS_WINDOWS", True), patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(stdout="ls-dyna_mpp.exe", returncode=0) mock_run.return_value = MagicMock(stdout="ls-dyna_mpp.exe", returncode=0)
system.check_ls_dyna_status() system.check_ls_dyna_status()
assert mock_run.called assert mock_run.called
def test_check_ls_dyna_status_linux(self) -> None: def test_check_ls_dyna_status_linux(self) -> None:
"""Should check LS-DYNA status on Linux.""" """Should check LS-DYNA status on Linux."""
with patch.object(Constants, "IS_WINDOWS", False), patch("subprocess.run") as mock_run: with patch.object(Constants, "IS_WINDOWS", False), patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(stdout="1234", returncode=0) mock_run.return_value = MagicMock(stdout="1234", returncode=0)
system.check_ls_dyna_status() system.check_ls_dyna_status()
assert mock_run.called assert mock_run.called
+2 -2
View File
@@ -67,7 +67,7 @@ class TestPackDependencies:
"""Should pack dependencies.""" """Should pack dependencies."""
lib_dir = tmp_path / "libs" lib_dir = tmp_path / "libs"
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
system.pack_dependencies(lib_dir, ["numpy", "pandas"]) system.pack_dependencies(lib_dir, ["numpy", "pandas"])
assert mock_run.called assert mock_run.called
@@ -86,7 +86,7 @@ class TestPackWheel:
(project_dir / "pyproject.toml").write_text("[project]\nname = 'test'") (project_dir / "pyproject.toml").write_text("[project]\nname = 'test'")
output_dir = tmp_path / "dist" output_dir = tmp_path / "dist"
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
system.pack_wheel(project_dir, output_dir) system.pack_wheel(project_dir, output_dir)
assert mock_run.called assert mock_run.called
+12 -13
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from pathlib import Path from pathlib import Path
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
@@ -17,7 +16,7 @@ class TestGetInstalledPackages:
def test_get_installed_packages_success(self) -> None: def test_get_installed_packages_success(self) -> None:
"""Should get installed packages.""" """Should get installed packages."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0\n", returncode=0) mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0\n", returncode=0)
result = dev._get_installed_packages() result = dev._get_installed_packages()
assert "numpy" in result assert "numpy" in result
@@ -25,20 +24,20 @@ class TestGetInstalledPackages:
def test_get_installed_packages_empty(self) -> None: def test_get_installed_packages_empty(self) -> None:
"""Should handle empty output.""" """Should handle empty output."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(stdout="", returncode=0) mock_run.return_value = MagicMock(stdout="", returncode=0)
result = dev._get_installed_packages() result = dev._get_installed_packages()
assert result == [] assert result == []
def test_get_installed_packages_error(self) -> None: def test_get_installed_packages_error(self) -> None:
"""Should handle subprocess error.""" """Should handle subprocess error."""
with patch("subprocess.run", side_effect=subprocess.SubprocessError): with patch("pyflowx.sh", return_value=None):
result = dev._get_installed_packages() result = dev._get_installed_packages()
assert result == [] assert result == []
def test_get_installed_packages_oserror(self) -> None: def test_get_installed_packages_oserror(self) -> None:
"""Should handle OSError.""" """Should handle OSError."""
with patch("subprocess.run", side_effect=OSError): with patch("pyflowx.sh", return_value=None):
result = dev._get_installed_packages() result = dev._get_installed_packages()
assert result == [] assert result == []
@@ -106,14 +105,14 @@ class TestPipUninstall:
def test_pip_uninstall_single_package(self) -> None: def test_pip_uninstall_single_package(self) -> None:
"""Should uninstall single package.""" """Should uninstall single package."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_uninstall(["numpy"]) dev.pip_uninstall(["numpy"])
assert mock_run.called assert mock_run.called
def test_pip_uninstall_multiple_packages(self) -> None: def test_pip_uninstall_multiple_packages(self) -> None:
"""Should uninstall multiple packages.""" """Should uninstall multiple packages."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_uninstall(["numpy", "pandas", "scipy"]) dev.pip_uninstall(["numpy", "pandas", "scipy"])
# Should call pip uninstall # Should call pip uninstall
@@ -123,7 +122,7 @@ class TestPipUninstall:
"""Should handle wildcard in package name.""" """Should handle wildcard in package name."""
with ( with (
patch.object(dev, "_expand_wildcard_packages", return_value=["numpy", "numpy-core"]), patch.object(dev, "_expand_wildcard_packages", return_value=["numpy", "numpy-core"]),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_uninstall(["numpy*"]) dev.pip_uninstall(["numpy*"])
@@ -149,7 +148,7 @@ class TestPipReinstall:
def test_pip_reinstall_single_package(self) -> None: def test_pip_reinstall_single_package(self) -> None:
"""Should reinstall single package.""" """Should reinstall single package."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_reinstall(["numpy"]) dev.pip_reinstall(["numpy"])
# Should call pip uninstall and pip install # Should call pip uninstall and pip install
@@ -157,7 +156,7 @@ class TestPipReinstall:
def test_pip_reinstall_offline(self) -> None: def test_pip_reinstall_offline(self) -> None:
"""Should reinstall packages offline.""" """Should reinstall packages offline."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_reinstall(["numpy"], offline=True) dev.pip_reinstall(["numpy"], offline=True)
# Should call pip install with offline flags # Should call pip install with offline flags
@@ -177,14 +176,14 @@ class TestPipDownload:
def test_pip_download_single_package(self) -> None: def test_pip_download_single_package(self) -> None:
"""Should download single package.""" """Should download single package."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_download(["numpy"]) dev.pip_download(["numpy"])
assert mock_run.called assert mock_run.called
def test_pip_download_offline(self) -> None: def test_pip_download_offline(self) -> None:
"""Should download packages offline.""" """Should download packages offline."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
dev.pip_download(["numpy"], offline=True) dev.pip_download(["numpy"], offline=True)
# Should call pip download with offline flags # Should call pip download with offline flags
@@ -199,7 +198,7 @@ class TestPipFreeze:
def test_pip_freeze(self, tmp_path: Path) -> None: def test_pip_freeze(self, tmp_path: Path) -> None:
"""Should freeze dependencies.""" """Should freeze dependencies."""
with patch("subprocess.run") as mock_run: with patch("pyflowx.sh") as mock_run:
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0", returncode=0) mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0", returncode=0)
dev.pip_freeze() dev.pip_freeze()
assert mock_run.called assert mock_run.called
+3 -3
View File
@@ -27,7 +27,7 @@ def _fn() -> int:
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]: def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
return TaskSpec[Any](name, _fn, depends_on=deps) return TaskSpec(name, _fn, depends_on=deps)
def _result( def _result(
@@ -40,7 +40,7 @@ def _result(
) -> TaskResult[Any]: ) -> TaskResult[Any]:
"""构造带时间戳的 TaskResult.""" """构造带时间戳的 TaskResult."""
end = start + timedelta(seconds=duration) if duration > 0 else start end = start + timedelta(seconds=duration) if duration > 0 else start
return TaskResult[Any]( return TaskResult(
spec=_spec(name), spec=_spec(name),
status=status, status=status,
value=None, value=None,
@@ -139,7 +139,7 @@ class TestToHtml:
start = datetime(2024, 1, 1, 0, 0, 0) start = datetime(2024, 1, 1, 0, 0, 0)
report = px.RunReport() report = px.RunReport()
report.results["a"] = _result("a", start, 1.0) report.results["a"] = _result("a", start, 1.0)
report.results["b"] = TaskResult[Any]( report.results["b"] = TaskResult(
spec=_spec("b"), spec=_spec("b"),
status=TaskStatus.SKIPPED, status=TaskStatus.SKIPPED,
reason="skip", reason="skip",
+59 -1
View File
@@ -1,9 +1,12 @@
"""Tests for ops.pymake 模块 (@px.tool 注册验证).""" """Tests for ops.pymake 模块 (@px.tool 注册验证 + CLI 调度)."""
from __future__ import annotations from __future__ import annotations
import pytest
import pyflowx as px import pyflowx as px
import pyflowx.ops.dev.pymake import pyflowx.ops.dev.pymake
from pyflowx.cli.pf import PfApp
from pyflowx.tools import _TOOL_REGISTRY, get_tool from pyflowx.tools import _TOOL_REGISTRY, get_tool
@@ -106,3 +109,58 @@ class TestPymakeRegistration:
for name in ("ba", "bump", "cov", "tc", "p"): for name in ("ba", "bump", "cov", "tc", "p"):
spec = get_tool("pymake", name) spec = get_tool("pymake", name)
assert spec.cmd is None, f"{name} 应为聚合任务 (无 cmd)" assert spec.cmd is None, f"{name} 应为聚合任务 (无 cmd)"
def test_bumpversion_cmd_uses_part_option(self) -> None:
"""bumpversion 隐藏任务 cmd 应使用 --part patch (非位置参数)."""
spec = get_tool("pymake", "bumpversion")
assert spec.cmd is not None
assert "--part" in spec.cmd
assert "patch" in spec.cmd
def test_bumpmi_cmd_uses_part_option(self) -> None:
"""bumpmi 子命令 cmd 应使用 --part minor (非位置参数)."""
spec = get_tool("pymake", "bumpmi")
assert spec.cmd is not None
assert "--part" in spec.cmd
assert "minor" in spec.cmd
# ---------------------------------------------------------------------- #
# CLI 调度测试 (pf pymake -h / pf pymake bump --dry-run)
# ---------------------------------------------------------------------- #
class TestPymakeCliDispatch:
"""``pf pymake`` CLI 调度测试 (不执行真实命令)."""
def test_pymake_h_shows_help(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf pymake -h 应返回 0 并打印子命令列表."""
app = PfApp(["pymake", "-h"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Commands" in out
assert "bump" in out
def test_pymake_help_long_form(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf pymake --help 应与 -h 等价."""
app = PfApp(["pymake", "--help"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Commands" in out
def test_pymake_bump_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf pymake bump --dry-run 应打印执行计划, 不执行真实命令."""
app = PfApp(["pymake", "bump", "--dry-run"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Dry run" in out
assert "bumpversion" in out
def test_pymake_bumpmi_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
"""pf pymake bumpmi --dry-run 应打印执行计划."""
app = PfApp(["pymake", "bumpmi", "--dry-run"])
rc = app.run()
assert rc == 0
out = capsys.readouterr().out
assert "Dry run" in out
+2 -4
View File
@@ -2,8 +2,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
import pytest import pytest
from pyflowx.conditions import Constants from pyflowx.conditions import Constants
@@ -46,7 +44,7 @@ class TestResetIconCacheRun:
monkeypatch.setattr(Path, "exists", lambda _self: False) monkeypatch.setattr(Path, "exists", lambda _self: False)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd)) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd))
reset_icon_cache_run() reset_icon_cache_run()
captured = capsys.readouterr() captured = capsys.readouterr()
@@ -69,7 +67,7 @@ class TestResetIconCacheRun:
monkeypatch.setattr(Path, "exists", lambda _self: True) monkeypatch.setattr(Path, "exists", lambda _self: True)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd)) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd))
reset_icon_cache_run() reset_icon_cache_run()
captured = capsys.readouterr() captured = capsys.readouterr()
+10 -10
View File
@@ -41,7 +41,7 @@ class TestTakeScreenshotFull:
patch.object(Constants, "IS_WINDOWS", True), patch.object(Constants, "IS_WINDOWS", True),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_full() media.take_screenshot_full()
@@ -53,7 +53,7 @@ class TestTakeScreenshotFull:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", True), patch.object(Constants, "IS_MACOS", True),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_full() media.take_screenshot_full()
@@ -65,7 +65,7 @@ class TestTakeScreenshotFull:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_full() media.take_screenshot_full()
@@ -77,9 +77,9 @@ class TestTakeScreenshotFull:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.side_effect = [FileNotFoundError(), MagicMock(returncode=0)] mock_run.side_effect = [None, MagicMock(returncode=0)]
media.take_screenshot_full() media.take_screenshot_full()
assert mock_run.call_count == 2 assert mock_run.call_count == 2
@@ -96,7 +96,7 @@ class TestTakeScreenshotArea:
patch.object(Constants, "IS_WINDOWS", True), patch.object(Constants, "IS_WINDOWS", True),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_area() media.take_screenshot_area()
@@ -108,7 +108,7 @@ class TestTakeScreenshotArea:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", True), patch.object(Constants, "IS_MACOS", True),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_area() media.take_screenshot_area()
@@ -120,7 +120,7 @@ class TestTakeScreenshotArea:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.return_value = MagicMock(returncode=0) mock_run.return_value = MagicMock(returncode=0)
media.take_screenshot_area() media.take_screenshot_area()
@@ -132,8 +132,8 @@ class TestTakeScreenshotArea:
patch.object(Constants, "IS_WINDOWS", False), patch.object(Constants, "IS_WINDOWS", False),
patch.object(Constants, "IS_MACOS", False), patch.object(Constants, "IS_MACOS", False),
patch.object(Path, "home", return_value=tmp_path), patch.object(Path, "home", return_value=tmp_path),
patch("subprocess.run") as mock_run, patch("pyflowx.sh") as mock_run,
): ):
mock_run.side_effect = [FileNotFoundError(), MagicMock(returncode=0)] mock_run.side_effect = [None, MagicMock(returncode=0)]
media.take_screenshot_area() media.take_screenshot_area()
assert mock_run.call_count == 2 assert mock_run.call_count == 2
+25 -15
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import subprocess
from unittest.mock import MagicMock from unittest.mock import MagicMock
import pytest import pytest
@@ -23,7 +22,7 @@ class TestClearScreenRun:
"""Windows 上应调用 cls.""" """Windows 上应调用 cls."""
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock()) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
clear_screen_run() clear_screen_run()
assert ran_cmds == [["cls"]] assert ran_cmds == [["cls"]]
@@ -31,7 +30,7 @@ class TestClearScreenRun:
"""非 Windows 上应调用 clear.""" """非 Windows 上应调用 clear."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock()) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
clear_screen_run() clear_screen_run()
assert ran_cmds == [["clear"]] assert ran_cmds == [["clear"]]
@@ -46,7 +45,7 @@ class TestTaskkillRun:
"""Windows 上应使用 taskkill /f /im 并加通配符.""" """Windows 上应使用 taskkill /f /im 并加通配符."""
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock()) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
taskkill_run(["chrome.exe", "python.exe"]) taskkill_run(["chrome.exe", "python.exe"])
assert ran_cmds == [ assert ran_cmds == [
["taskkill", "/f", "/im", "chrome.exe*"], ["taskkill", "/f", "/im", "chrome.exe*"],
@@ -57,22 +56,22 @@ class TestTaskkillRun:
"""非 Windows 上应使用 pkill -f 并加通配符.""" """非 Windows 上应使用 pkill -f 并加通配符."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock()) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
taskkill_run(["chrome"]) taskkill_run(["chrome"])
assert ran_cmds == [["pkill", "-f", "chrome*"]] assert ran_cmds == [["pkill", "-f", "chrome*"]]
def test_empty_list_does_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None: def test_empty_list_does_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""空列表不应调用 subprocess.""" """空列表不应调用 px.sh."""
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
called: list[list[str]] = [] called: list[list[str]] = []
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: called.append(cmd) or MagicMock()) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: called.append(cmd) or MagicMock())
taskkill_run([]) taskkill_run([])
assert called == [] assert called == []
def test_prints_progress(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: def test_prints_progress(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
"""应打印每个进程的终止信息.""" """应打印每个进程的终止信息."""
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
monkeypatch.setattr(subprocess, "run", lambda *_, **__: MagicMock(returncode=0)) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: MagicMock(returncode=0))
taskkill_run(["chrome.exe"]) taskkill_run(["chrome.exe"])
captured = capsys.readouterr() captured = capsys.readouterr()
assert "chrome.exe" in captured.out assert "chrome.exe" in captured.out
@@ -84,13 +83,24 @@ class TestTaskkillRun:
"""returncode 非零时应打印未找到/失败信息.""" """returncode 非零时应打印未找到/失败信息."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
mock_result = MagicMock(returncode=1) mock_result = MagicMock(returncode=1)
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
taskkill_run(["nonexistent"]) taskkill_run(["nonexistent"])
captured = capsys.readouterr() captured = capsys.readouterr()
assert "未找到匹配进程或终止失败" in captured.out assert "未找到匹配进程或终止失败" in captured.out
assert "returncode=1" in captured.out assert "returncode=1" in captured.out
assert "已发送终止信号" not in captured.out assert "已发送终止信号" not in captured.out
def test_command_unavailable_prints_message(
self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""px.sh 返回 None(命令未找到)时应打印不可用信息."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)
taskkill_run(["chrome"])
captured = capsys.readouterr()
assert "终止命令不可用" in captured.out
assert "已发送终止信号" not in captured.out
# ---------------------------------------------------------------------- # # ---------------------------------------------------------------------- #
# which_run # which_run
@@ -103,7 +113,7 @@ class TestWhichRun:
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
mock_result = MagicMock(returncode=0, stdout="C:\\Windows\\python.exe\n") mock_result = MagicMock(returncode=0, stdout="C:\\Windows\\python.exe\n")
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
which_run(["python"]) which_run(["python"])
assert ran_cmds == [["where", "python"]] assert ran_cmds == [["where", "python"]]
@@ -112,7 +122,7 @@ class TestWhichRun:
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n") mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
which_run(["python"]) which_run(["python"])
assert ran_cmds == [["which", "python"]] assert ran_cmds == [["which", "python"]]
@@ -120,7 +130,7 @@ class TestWhichRun:
"""找到命令时应打印 <cmd> -> <path>.""" """找到命令时应打印 <cmd> -> <path>."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n") mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
which_run(["python"]) which_run(["python"])
captured = capsys.readouterr() captured = capsys.readouterr()
assert "python" in captured.out assert "python" in captured.out
@@ -132,7 +142,7 @@ class TestWhichRun:
"""未找到命令时应打印未找到.""" """未找到命令时应打印未找到."""
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
mock_result = MagicMock(returncode=1, stdout="") mock_result = MagicMock(returncode=1, stdout="")
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
which_run(["nonexistent_cmd"]) which_run(["nonexistent_cmd"])
captured = capsys.readouterr() captured = capsys.readouterr()
assert "未找到" in captured.out assert "未找到" in captured.out
@@ -142,7 +152,7 @@ class TestWhichRun:
monkeypatch.setattr(Constants, "IS_WINDOWS", False) monkeypatch.setattr(Constants, "IS_WINDOWS", False)
ran_cmds: list[list[str]] = [] ran_cmds: list[list[str]] = []
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n") mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result) monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
which_run(["python", "pip"]) which_run(["python", "pip"])
assert ran_cmds == [["which", "python"], ["which", "pip"]] assert ran_cmds == [["which", "python"], ["which", "pip"]]
@@ -152,7 +162,7 @@ class TestWhichRun:
"""Windows where 返回多行时应取第一行.""" """Windows where 返回多行时应取第一行."""
monkeypatch.setattr(Constants, "IS_WINDOWS", True) monkeypatch.setattr(Constants, "IS_WINDOWS", True)
mock_result = MagicMock(returncode=0, stdout="C:\\first\\python.exe\nC:\\second\\python.exe\n") mock_result = MagicMock(returncode=0, stdout="C:\\first\\python.exe\nC:\\second\\python.exe\n")
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result) monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
which_run(["python"]) which_run(["python"])
captured = capsys.readouterr() captured = capsys.readouterr()
assert "C:\\first\\python.exe" in captured.out assert "C:\\first\\python.exe" in captured.out
+23 -29
View File
@@ -141,13 +141,11 @@ class TestMarkRemainingSkipped:
def test_marks_all_unexecuted_tasks(self) -> None: def test_marks_all_unexecuted_tasks(self) -> None:
"""未执行的任务应被标记为 SKIPPED.""" """未执行的任务应被标记为 SKIPPED."""
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", lambda: 1),
px.TaskSpec("a", lambda: 1), px.TaskSpec("b", lambda: 2, depends_on=("a",)),
px.TaskSpec("b", lambda: 2, depends_on=("a",)), px.TaskSpec("c", lambda: 3, depends_on=("b",)),
px.TaskSpec("c", lambda: 3, depends_on=("b",)), ])
]
)
report = RunReport() report = RunReport()
# 模拟 a 已完成 # 模拟 a 已完成
from datetime import datetime from datetime import datetime
@@ -171,12 +169,10 @@ class TestMarkRemainingSkipped:
def test_marks_all_when_none_executed(self) -> None: def test_marks_all_when_none_executed(self) -> None:
"""无已完成任务时,所有任务应被标记为 SKIPPED.""" """无已完成任务时,所有任务应被标记为 SKIPPED."""
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", lambda: 1),
px.TaskSpec("a", lambda: 1), px.TaskSpec("b", lambda: 2, depends_on=("a",)),
px.TaskSpec("b", lambda: 2, depends_on=("a",)), ])
]
)
report = RunReport() report = RunReport()
_mark_remaining_skipped(graph, report, None) _mark_remaining_skipped(graph, report, None)
assert len(report.results) == 2 assert len(report.results) == 2
@@ -223,14 +219,12 @@ def _make_cancellable_graph() -> px.Graph:
time.sleep(0.05) time.sleep(0.05)
return c + 1 return c + 1
return px.Graph.from_specs( return px.Graph.from_specs([
[ px.TaskSpec("a", a),
px.TaskSpec("a", a), px.TaskSpec("b", b, depends_on=("a",)),
px.TaskSpec("b", b, depends_on=("a",)), px.TaskSpec("c", c, depends_on=("b",)),
px.TaskSpec("c", c, depends_on=("b",)), px.TaskSpec("d", d, depends_on=("c",)),
px.TaskSpec("d", d, depends_on=("c",)), ])
]
)
class TestExternalCancel: class TestExternalCancel:
@@ -327,10 +321,12 @@ class TestKeyboardInterrupt:
graph = _make_cancellable_graph() graph = _make_cancellable_graph()
def _interrupt_after_first_layer( def _interrupt_after_first_layer(
graph: Any, layers: Any, context: Any, report: Any, backend: Any, on_event: Any, cancel_event: Any = None graph: Any,
layers: Any,
ctx: Any,
) -> None: ) -> None:
# 只运行第一层,然后抛 KeyboardInterrupt # 只运行第一层,然后抛 KeyboardInterrupt
executors.SequentialLayerRunner.execute(layers[0], graph, context, report, backend, 1, on_event) executors.SequentialLayerRunner.execute(layers[0], graph, ctx, 1)
raise KeyboardInterrupt raise KeyboardInterrupt
with patch.object(executors, "_drive_sequential", _interrupt_after_first_layer): with patch.object(executors, "_drive_sequential", _interrupt_after_first_layer):
@@ -350,12 +346,10 @@ class TestKeyboardInterrupt:
def fn() -> int: def fn() -> int:
return 1 return 1
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", fn),
px.TaskSpec("a", fn), px.TaskSpec("b", fn, depends_on=("a",)),
px.TaskSpec("b", fn, depends_on=("a",)), ])
]
)
async def _interrupt(*args: Any, **kwargs: Any) -> None: async def _interrupt(*args: Any, **kwargs: Any) -> None:
raise KeyboardInterrupt raise KeyboardInterrupt
+7 -9
View File
@@ -16,7 +16,7 @@ def _fn() -> None:
return None return None
def _make_result( def _make_result( # noqa: PLR0913
name: str, name: str,
status: TaskStatus = TaskStatus.SUCCESS, status: TaskStatus = TaskStatus.SUCCESS,
value: Any = None, value: Any = None,
@@ -27,10 +27,10 @@ def _make_result(
attempts: int = 1, attempts: int = 1,
) -> TaskResult[Any]: ) -> TaskResult[Any]:
"""构造测试用 TaskResult。""" """构造测试用 TaskResult。"""
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn, depends_on=depends_on, soft_depends_on=soft_depends_on) spec: TaskSpec[Any] = TaskSpec(name, _fn, depends_on=depends_on, soft_depends_on=soft_depends_on)
start = datetime(2024, 1, 1, 0, 0, 0) start = datetime(2024, 1, 1, 0, 0, 0)
end = datetime(2024, 1, 1, 0, 0, 1) if status != TaskStatus.PENDING else None end = datetime(2024, 1, 1, 0, 0, 1) if status != TaskStatus.PENDING else None
return TaskResult[Any]( return TaskResult(
spec=spec, spec=spec,
status=status, status=status,
value=value, value=value,
@@ -507,12 +507,10 @@ class TestRunReportIntegration:
return 1 return 1
# continue_on_error=True 使失败不抛异常而写入报告(success 保持 True # continue_on_error=True 使失败不抛异常而写入报告(success 保持 True
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("boom", boom, continue_on_error=True),
px.TaskSpec("boom", boom, continue_on_error=True), px.TaskSpec("downstream", downstream, depends_on=("boom",)),
px.TaskSpec("downstream", downstream, depends_on=("boom",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
# continue_on_error 不影响 success 标志,手动标记以触发诊断 # continue_on_error 不影响 success 标志,手动标记以触发诊断
report.success = False report.success = False
+21 -31
View File
@@ -565,12 +565,10 @@ class TestDagComposition:
def read_all(find_logs: list[Path]) -> list[str]: def read_all(find_logs: list[Path]) -> list[str]:
return [read_text(p) for p in find_logs] return [read_text(p) for p in find_logs]
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("find_logs", fn=find_logs, outputs={"paths": "$"}),
px.TaskSpec("find_logs", fn=find_logs, outputs={"paths": "$"}), px.TaskSpec("read_all", fn=read_all, depends_on=("find_logs",)),
px.TaskSpec("read_all", fn=read_all, depends_on=("find_logs",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
assert report.success assert report.success
@@ -592,12 +590,10 @@ class TestDagComposition:
def copy_all(find_sources: list[Path]) -> list[Path]: def copy_all(find_sources: list[Path]) -> list[Path]:
return [copy(p, tmp_path / "dst") for p in find_sources] return [copy(p, tmp_path / "dst") for p in find_sources]
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("find_sources", fn=find_sources, outputs={"files": "$"}),
px.TaskSpec("find_sources", fn=find_sources, outputs={"files": "$"}), px.TaskSpec("copy_all", fn=copy_all, depends_on=("find_sources",)),
px.TaskSpec("copy_all", fn=copy_all, depends_on=("find_sources",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
assert report.success assert report.success
@@ -619,12 +615,10 @@ class TestDagComposition:
def checksums(find_bins: list[Path]) -> dict[str, str]: def checksums(find_bins: list[Path]) -> dict[str, str]:
return {p.name: sha256(p) for p in find_bins} return {p.name: sha256(p) for p in find_bins}
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("find_bins", fn=find_bins, outputs={"paths": "$"}),
px.TaskSpec("find_bins", fn=find_bins, outputs={"paths": "$"}), px.TaskSpec("checksums", fn=checksums, depends_on=("find_bins",)),
px.TaskSpec("checksums", fn=checksums, depends_on=("find_bins",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
assert report.success assert report.success
@@ -643,12 +637,10 @@ class TestDagComposition:
assert write_data == len("dag-content") assert write_data == len("dag-content")
return read_text(target) return read_text(target)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("write_data", fn=write_data, outputs={"bytes": "$"}),
px.TaskSpec("write_data", fn=write_data, outputs={"bytes": "$"}), px.TaskSpec("read_back", fn=read_back, depends_on=("write_data",)),
px.TaskSpec("read_back", fn=read_back, depends_on=("write_data",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
assert report.success assert report.success
@@ -671,13 +663,11 @@ class TestDagComposition:
content = f"files: {summarize['count']}, size: {summarize['total_size']}" content = f"files: {summarize['count']}, size: {summarize['total_size']}"
return write_text(tmp_path / "summary.txt", content) return write_text(tmp_path / "summary.txt", content)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("find_files", fn=find_files, outputs={"paths": "$"}),
px.TaskSpec("find_files", fn=find_files, outputs={"paths": "$"}), px.TaskSpec("summarize", fn=summarize, depends_on=("find_files",)),
px.TaskSpec("summarize", fn=summarize, depends_on=("find_files",)), px.TaskSpec("write_summary", fn=write_summary, depends_on=("summarize",)),
px.TaskSpec("write_summary", fn=write_summary, depends_on=("summarize",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
assert report.success assert report.success
+48 -66
View File
@@ -21,7 +21,7 @@ def _fn() -> int:
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]: def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
return TaskSpec[Any](name, _fn, depends_on=deps) return TaskSpec(name, _fn, depends_on=deps)
def _result( def _result(
@@ -34,7 +34,7 @@ def _result(
) -> TaskResult[Any]: ) -> TaskResult[Any]:
"""构造带时间戳的 TaskResult.""" """构造带时间戳的 TaskResult."""
end = start + timedelta(seconds=duration) if duration > 0 else start end = start + timedelta(seconds=duration) if duration > 0 else start
return TaskResult[Any]( return TaskResult(
spec=_spec(name), spec=_spec(name),
status=status, status=status,
value=None, value=None,
@@ -46,7 +46,7 @@ def _result(
def _skipped_result(name: str, reason: str = "skip") -> TaskResult[Any]: def _skipped_result(name: str, reason: str = "skip") -> TaskResult[Any]:
"""构造 SKIPPED 结果(无时间戳).""" """构造 SKIPPED 结果(无时间戳)."""
return TaskResult[Any]( return TaskResult(
spec=_spec(name), spec=_spec(name),
status=TaskStatus.SKIPPED, status=TaskStatus.SKIPPED,
reason=reason, reason=reason,
@@ -95,13 +95,11 @@ class TestProfileReportConstruction:
report.results["a"] = _result("a", start, 1.0) report.results["a"] = _result("a", start, 1.0)
report.results["b"] = _result("b", start + timedelta(seconds=1), 2.0) report.results["b"] = _result("b", start + timedelta(seconds=1), 2.0)
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.5) report.results["c"] = _result("c", start + timedelta(seconds=3), 1.5)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), _spec("c", deps=("b",)),
_spec("c", deps=("b",)), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -138,13 +136,11 @@ class TestProfileReportConstruction:
report.results["a"] = _result("a", start, 1.0) report.results["a"] = _result("a", start, 1.0)
report.results["b"] = _result("b", start, 3.0) report.results["b"] = _result("b", start, 3.0)
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.0) report.results["c"] = _result("c", start + timedelta(seconds=3), 1.0)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b"),
_spec("b"), _spec("c", deps=("a", "b")),
_spec("c", deps=("a", "b")), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -192,12 +188,10 @@ class TestWaitTime:
report = px.RunReport() report = px.RunReport()
report.results["a"] = _result("a", start, 1.0) report.results["a"] = _result("a", start, 1.0)
report.results["b"] = _result("b", start + timedelta(seconds=1.5), 1.0) report.results["b"] = _result("b", start + timedelta(seconds=1.5), 1.0)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -210,12 +204,10 @@ class TestWaitTime:
report.results["a"] = _result("a", start, 2.0) report.results["a"] = _result("a", start, 2.0)
# b 在 a 还没完成时就开始(不应该但可能发生) # b 在 a 还没完成时就开始(不应该但可能发生)
report.results["b"] = _result("b", start + timedelta(seconds=1), 1.0) report.results["b"] = _result("b", start + timedelta(seconds=1), 1.0)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -228,12 +220,10 @@ class TestWaitTime:
report = px.RunReport() report = px.RunReport()
report.results["a"] = _result("a", start, 1.0) report.results["a"] = _result("a", start, 1.0)
report.results["b"] = _skipped_result("b") report.results["b"] = _skipped_result("b")
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -251,14 +241,12 @@ class TestCriticalPath:
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0) report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0) report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0) report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), _spec("c", deps=("a",)),
_spec("c", deps=("a",)), _spec("d", deps=("b", "c")),
_spec("d", deps=("b", "c")), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -291,7 +279,7 @@ class TestParallelism:
def test_no_timestamps_zero_parallelism(self) -> None: def test_no_timestamps_zero_parallelism(self) -> None:
"""所有任务无时间戳:并行度为 0.""" """所有任务无时间戳:并行度为 0."""
report = px.RunReport() report = px.RunReport()
report.results["a"] = TaskResult[Any](spec=_spec("a"), status=TaskStatus.SUCCESS) report.results["a"] = TaskResult(spec=_spec("a"), status=TaskStatus.SUCCESS)
graph = px.Graph.from_specs([_spec("a")]) graph = px.Graph.from_specs([_spec("a")])
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -405,14 +393,12 @@ class TestQueries:
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0) report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0) report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0) report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ _spec("a"),
_spec("a"), _spec("b", deps=("a",)),
_spec("b", deps=("a",)), _spec("c", deps=("a",)),
_spec("c", deps=("a",)), _spec("d", deps=("b", "c")),
_spec("d", deps=("b", "c")), ])
]
)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -551,13 +537,11 @@ class TestIntegrationWithRun:
time.sleep(0.01) # 确保任务有实际耗时,避免 duration 极小导致并行度计算为 0 time.sleep(0.01) # 确保任务有实际耗时,避免 duration 极小导致并行度计算为 0
return 1 return 1
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", slow),
px.TaskSpec("a", slow), px.TaskSpec("b", slow, depends_on=("a",)),
px.TaskSpec("b", slow, depends_on=("a",)), px.TaskSpec("c", slow, depends_on=("a",)),
px.TaskSpec("c", slow, depends_on=("a",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
@@ -576,13 +560,11 @@ class TestIntegrationWithRun:
time.sleep(0.05) time.sleep(0.05)
return 1 return 1
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", slow),
px.TaskSpec("a", slow), px.TaskSpec("b", slow),
px.TaskSpec("b", slow), px.TaskSpec("c", slow),
px.TaskSpec("c", slow), ])
]
)
report = px.run(graph, strategy="thread", max_workers=3) report = px.run(graph, strategy="thread", max_workers=3)
profile = ProfileReport.from_report(report, graph) profile = ProfileReport.from_report(report, graph)
+20 -24
View File
@@ -20,7 +20,7 @@ def _fn() -> int:
return 1 return 1
def _make_result( def _make_result( # noqa: PLR0913
name: str = "a", name: str = "a",
status: TaskStatus = TaskStatus.SUCCESS, status: TaskStatus = TaskStatus.SUCCESS,
value: Any = 42, value: Any = 42,
@@ -32,11 +32,11 @@ def _make_result(
reason: str | None = None, reason: str | None = None,
) -> TaskResult[Any]: ) -> TaskResult[Any]:
"""构造测试用 TaskResult 实例.""" """构造测试用 TaskResult 实例."""
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn, tags=tags, depends_on=depends_on) spec: TaskSpec[Any] = TaskSpec(name, _fn, tags=tags, depends_on=depends_on)
start = datetime(2024, 1, 1, 0, 0, 0) start = datetime(2024, 1, 1, 0, 0, 0)
# 用 timedelta 精确表达秒数,避免 int() 截断小数 # 用 timedelta 精确表达秒数,避免 int() 截断小数
end = start + timedelta(seconds=duration) if duration else None end = start + timedelta(seconds=duration) if duration else None
return TaskResult[Any]( return TaskResult(
spec=spec, spec=spec,
status=status, status=status,
value=value, value=value,
@@ -98,7 +98,7 @@ class TestRunReportSummary:
def test_summary_with_none_duration(self) -> None: def test_summary_with_none_duration(self) -> None:
"""未开始/未结束的任务 duration 为 None,不应计入总时长.""" """未开始/未结束的任务 duration 为 None,不应计入总时长."""
report = px.RunReport() report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type] spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.FAILED) report.results["a"] = TaskResult(spec=spec, status=TaskStatus.FAILED)
s = report.summary() s = report.summary()
assert s["total_duration_seconds"] == 0.0 assert s["total_duration_seconds"] == 0.0
@@ -135,8 +135,8 @@ class TestRunReportDescribe:
def test_describe_no_duration(self) -> None: def test_describe_no_duration(self) -> None:
"""无耗时的任务应显示为 '-'.""" """无耗时的任务应显示为 '-'."""
report = px.RunReport() report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type] spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING) report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
desc = report.describe() desc = report.describe()
assert "-" in desc # duration 显示为 "-" assert "-" in desc # duration 显示为 "-"
@@ -181,8 +181,8 @@ class TestRunReportQueries:
def test_durations_no_duration(self) -> None: def test_durations_no_duration(self) -> None:
"""无时长的任务应返回 0.0.""" """无时长的任务应返回 0.0."""
report = px.RunReport() report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type] spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING) report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
durs = report.durations() durs = report.durations()
assert durs["a"] == 0.0 assert durs["a"] == 0.0
@@ -257,8 +257,8 @@ class TestRunReportToDict:
def test_to_dict_no_timestamps(self) -> None: def test_to_dict_no_timestamps(self) -> None:
"""未开始/未结束的任务 started_at/finished_at/duration 应为 None。""" """未开始/未结束的任务 started_at/finished_at/duration 应为 None。"""
report = px.RunReport() report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type] spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING) report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
d = report.to_dict() d = report.to_dict()
entry = d["results"][0] entry = d["results"][0]
assert entry["started_at"] is None assert entry["started_at"] is None
@@ -439,7 +439,7 @@ class TestRunReportToHtml:
"""任务名含 HTML 特殊字符应被转义。""" """任务名含 HTML 特殊字符应被转义。"""
report = px.RunReport() report = px.RunReport()
malicious = "<script>evil()</script>" malicious = "<script>evil()</script>"
spec: TaskSpec[Any] = TaskSpec[Any](malicious, _fn) spec: TaskSpec[Any] = TaskSpec(malicious, _fn)
result: TaskResult[Any] = TaskResult( result: TaskResult[Any] = TaskResult(
spec=spec, spec=spec,
status=TaskStatus.SUCCESS, status=TaskStatus.SUCCESS,
@@ -538,7 +538,7 @@ class TestRunReportToHtml:
"""无时间戳时应渲染占位符。""" """无时间戳时应渲染占位符。"""
report = px.RunReport() report = px.RunReport()
# 构造无时间戳的结果 # 构造无时间戳的结果
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) spec: TaskSpec[Any] = TaskSpec("a", _fn)
result: TaskResult[Any] = TaskResult( result: TaskResult[Any] = TaskResult(
spec=spec, spec=spec,
status=TaskStatus.SUCCESS, status=TaskStatus.SUCCESS,
@@ -652,12 +652,10 @@ class TestRunReportSerializationIntegration:
def double(extract: list[int]) -> list[int]: def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract] return [x * 2 for x in extract]
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("extract", extract, tags=("ingest",)),
px.TaskSpec("extract", extract, tags=("ingest",)), px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
# 序列化 → 反序列化 # 序列化 → 反序列化
@@ -753,12 +751,10 @@ class TestRunId:
def task_b(a: int) -> int: def task_b(a: int) -> int:
return a * 2 return a * 2
graph = px.Graph.from_specs( graph = px.Graph.from_specs([
[ px.TaskSpec("a", task_a),
px.TaskSpec("a", task_a), px.TaskSpec("b", task_b, depends_on=("a",)),
px.TaskSpec("b", task_b, depends_on=("a",)), ])
]
)
report = px.run(graph, strategy="sequential") report = px.run(graph, strategy="sequential")
profile = report.profile(graph) profile = report.profile(graph)
assert profile.total_duration >= 0 assert profile.total_duration >= 0
+71
View File
@@ -133,6 +133,7 @@ class TestShMocked:
def test_sh_called_process_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None: def test_sh_called_process_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
"""CalledProcessError 被捕获,打印中文错误并返回 None。""" """CalledProcessError 被捕获,打印中文错误并返回 None。"""
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]: def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
raise subprocess.CalledProcessError(returncode=1, cmd=cmd) raise subprocess.CalledProcessError(returncode=1, cmd=cmd)
@@ -145,6 +146,7 @@ class TestShMocked:
def test_sh_file_not_found_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None: def test_sh_file_not_found_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
"""FileNotFoundError 被捕获,打印中文错误并返回 None。""" """FileNotFoundError 被捕获,打印中文错误并返回 None。"""
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]: def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
raise FileNotFoundError("[Errno 2] No such file") raise FileNotFoundError("[Errno 2] No such file")
@@ -156,6 +158,75 @@ class TestShMocked:
assert "命令未找到" in captured.out assert "命令未找到" in captured.out
class TestShCheckFalse:
"""check=False 模式:非零返回码返回 CompletedProcess(不返回 None)。"""
def test_check_false_nonzero_returns_completed_process(self) -> None:
"""check=False 时非零返回码返回 CompletedProcess 而非 None。"""
result = sh(["false"], check=False, label="容忍命令")
assert result is not None
assert result.returncode != 0
def test_check_false_success_returns_completed_process(self) -> None:
"""check=False 时成功也返回 CompletedProcess。"""
result = sh(["true"], check=False)
assert result is not None
assert result.returncode == 0
def test_check_false_file_not_found_returns_none(self, capsys: Any) -> None:
"""check=False 时命令未找到仍返回 NoneFileNotFoundError 非返回码)。"""
result = sh(["this_command_does_not_exist_xyz"], check=False, label="缺失")
assert result is None
captured = capsys.readouterr()
assert "缺失失败" in captured.out
assert "命令未找到" in captured.out
def test_check_false_capture_stdout(self) -> None:
"""check=False + capture=True 可读取 stdout(含非零返回码)。"""
result = sh("echo out; exit 3", capture=True, check=False, label="捕获")
assert result is not None
assert result.returncode == 3
assert result.stdout == "out\n"
class TestShTimeout:
"""timeout 参数:超时返回 None。"""
def test_timeout_expired_returns_none(self, capsys: Any) -> None:
"""超时时返回 None + 打印中文错误。"""
# sleep 5 但 timeout=0.1,必超时
result = sh(["sleep", "5"], timeout=0.1, label="睡眠")
assert result is None
captured = capsys.readouterr()
assert "睡眠超时" in captured.out
assert "timeout=0.1" in captured.out
def test_timeout_not_expired_returns_completed_process(self) -> None:
"""未超时正常返回 CompletedProcess。"""
result = sh(["true"], timeout=10, label="快速")
assert result is not None
assert result.returncode == 0
def test_timeout_passed_to_subprocess(self, monkeypatch: Any) -> None:
"""timeout 参数应传递给 subprocess.run。"""
captured: dict[str, Any] = {}
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
captured["timeout"] = kwargs.get("timeout")
return subprocess.CompletedProcess(cmd, 0)
monkeypatch.setattr(subprocess, "run", fake_run)
sh(["ls"], timeout=30)
assert captured["timeout"] == 30
def test_timeout_with_check_false(self, capsys: Any) -> None:
"""check=False + timeout 超时仍返回 None。"""
result = sh(["sleep", "5"], check=False, timeout=0.1, label="限时")
assert result is None
captured = capsys.readouterr()
assert "限时超时" in captured.out
class TestShExported: class TestShExported:
"""验证 px.sh 顶层导出。""" """验证 px.sh 顶层导出。"""
+3 -3
View File
@@ -354,7 +354,7 @@ class TestTaskSpecVerbose:
def test_verbose_default_is_false(self) -> None: def test_verbose_default_is_false(self) -> None:
"""verbose 默认应为 False.""" """verbose 默认应为 False."""
spec: px.TaskSpec[Any] = px.TaskSpec[Any]("a", cmd=[*ECHO_CMD, "hi"]) spec: px.TaskSpec[Any] = px.TaskSpec("a", cmd=[*ECHO_CMD, "hi"])
assert spec.verbose is False assert spec.verbose is False
def test_verbose_true_prints_command(self, capsys: pytest.CaptureFixture[str]) -> None: def test_verbose_true_prints_command(self, capsys: pytest.CaptureFixture[str]) -> None:
@@ -367,7 +367,7 @@ class TestTaskSpecVerbose:
def test_verbose_false_silent(self, capsys: pytest.CaptureFixture[str]) -> None: def test_verbose_false_silent(self, capsys: pytest.CaptureFixture[str]) -> None:
"""verbose=False 时不应打印命令信息.""" """verbose=False 时不应打印命令信息."""
graph = px.Graph.from_specs([px.TaskSpec[Any]("echo", cmd=[*ECHO_CMD, "silent"], verbose=False)]) graph = px.Graph.from_specs([px.TaskSpec("echo", cmd=[*ECHO_CMD, "silent"], verbose=False)])
_ = px.run(graph, strategy="sequential") _ = px.run(graph, strategy="sequential")
captured = capsys.readouterr() captured = capsys.readouterr()
assert "执行命令" not in captured.out assert "执行命令" not in captured.out
@@ -390,7 +390,7 @@ class TestTaskSpecVerbose:
import tempfile import tempfile
with tempfile.TemporaryDirectory() as tmpdir: with tempfile.TemporaryDirectory() as tmpdir:
graph = px.Graph.from_specs([px.TaskSpec[Any]("ls", cmd=ECHO_CMD, cwd=Path(tmpdir), verbose=True)]) graph = px.Graph.from_specs([px.TaskSpec("ls", cmd=ECHO_CMD, cwd=Path(tmpdir), verbose=True)])
_ = px.run(graph, strategy="sequential") _ = px.run(graph, strategy="sequential")
captured = capsys.readouterr() captured = capsys.readouterr()
assert "工作目录" in captured.out assert "工作目录" in captured.out
+1 -1
View File
@@ -8,7 +8,7 @@ skipsdist = true
[testenv] [testenv]
uv_sync = true uv_sync = true
deps = deps =
.[dev] .[dev,docs,fast,office]
commands = commands =
pytest -m "not slow" {posargs} pytest -m "not slow" {posargs}
passenv = passenv =
Generated
+1 -1
View File
@@ -1298,7 +1298,7 @@ wheels = [
[[package]] [[package]]
name = "pyflowx" name = "pyflowx"
version = "0.4.8" version = "0.4.11"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "pyyaml" }, { name = "pyyaml" },