Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3b52b047d2 | |||
| 529f94882c | |||
| 51b69a47bc | |||
| 9f87f86189 | |||
| 9ac68ddb2a | |||
| 2d7bbd1a1f | |||
| 18f4ee2957 | |||
| e0c9b0c821 | |||
| 8225f2fa3b | |||
| c5878e75ff | |||
| 5c7d27f682 | |||
| bbeba918cf |
@@ -30,5 +30,5 @@ jobs:
|
|||||||
- name: Ruff check
|
- name: Ruff check
|
||||||
run: ruff check src tests
|
run: ruff check src tests
|
||||||
|
|
||||||
- name: Tox test (py38, py313)
|
- name: Tox test (py310, py314)
|
||||||
run: uvx tox run -e py38,py313
|
run: uvx tox run -e py310,py314
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# 迭代 29: subprocess.run → px.sh 迁移
|
||||||
|
|
||||||
|
## 本轮目标
|
||||||
|
|
||||||
|
将 `ops/` 目录下所有 `subprocess.run(cmd, check=True)` 调用迁移到 `px.sh` 封装,统一错误处理与中文输出。
|
||||||
|
|
||||||
|
## 改动文件清单
|
||||||
|
|
||||||
|
### 源文件 (9 个)
|
||||||
|
|
||||||
|
| 文件 | 迁移调用数 | 保留调用数 | 保留原因 |
|
||||||
|
|------|-----------|-----------|---------|
|
||||||
|
| dev/gittool.py | 5 | 1 | `has_files()` 用 `check=False` 检查状态 |
|
||||||
|
| dev/piptool.py | 8 | 0 | 全部迁移,移除 `import subprocess` |
|
||||||
|
| dev/lscalc.py | 3 | 1 | Linux `pgrep` 用 `check=False` |
|
||||||
|
| dev/bumpversion.py | 3 | 0 | 全部迁移,移除 `import subprocess` |
|
||||||
|
| dev/packtool.py | 2 | 0 | 全部迁移,移除 `import subprocess` |
|
||||||
|
| dev/autofmt.py | 4 | 1 | `sync_pyproject_config` 用 `check=False` (失败可忽略) |
|
||||||
|
| infra/dockercmd.py | 1 | 0 | 全部迁移,移除 `import subprocess` |
|
||||||
|
| infra/msdownload.py | 1 | 0 | 全部迁移,移除 `import subprocess` |
|
||||||
|
| files/screenshot.py | 4 | 4 | Linux 回退模式用 `FileNotFoundError` 作为控制流 |
|
||||||
|
|
||||||
|
### 未迁移文件
|
||||||
|
|
||||||
|
- `infra/sshcopyid.py` — 使用 `timeout=` 参数(`px.sh` 不支持)
|
||||||
|
- `infra/envdev.py` — 全部 `check=False`
|
||||||
|
- `system/clr.py`, `system/reseticoncache.py`, `system/taskkill.py`, `system/which.py` — 全部 `check=False`
|
||||||
|
|
||||||
|
### 测试文件 (8 个)
|
||||||
|
|
||||||
|
- `test_piptool.py` — mock 目标从 `subprocess.run` 改为 `pyflowx.sh`,错误测试从 `side_effect` 改为 `return_value=None`
|
||||||
|
- `test_lscalc.py` — 迁移调用改 mock `pyflowx.sh`,Linux `check=False` 调用保留 `subprocess.run`
|
||||||
|
- `test_packtool.py` — mock 目标改为 `pyflowx.sh`
|
||||||
|
- `test_autofmt.py` — 迁移调用改 mock `pyflowx.sh`,`sync_pyproject_config` 保留 `subprocess.run`
|
||||||
|
- `test_screenshot.py` — Win/macOS 测试改 mock `pyflowx.sh`,Linux 回退测试保留 `subprocess.run`
|
||||||
|
- `test_bumpversion.py` — `_mock_subprocess` 改为 `_mock_sh`,mock `pyflowx.sh`
|
||||||
|
- `test_envdev.py` — 更新 dockercmd 错误消息断言(`px.sh` 消息格式与原自定义消息不同)
|
||||||
|
- `test_llm.py` — 更新 msdownload 错误消息断言
|
||||||
|
|
||||||
|
## 关键决策与依据
|
||||||
|
|
||||||
|
1. **`check=False` 调用不迁移**:`px.sh` 内部 `check=True`,失败时返回 `None` + 打印错误。`check=False` 调用需要检查 `returncode` / `stdout`,语义不同。
|
||||||
|
|
||||||
|
2. **screenshot Linux 回退不迁移**:`try: gnome-screenshot except FileNotFoundError: scrot` 模式中 `FileNotFoundError` 是控制流(回退到备用命令),不是错误。`px.sh` 会打印误导性错误信息。
|
||||||
|
|
||||||
|
3. **sshcopyid 不迁移**:使用 `timeout=` 参数控制 SSH 超时,`px.sh` 不支持此参数。
|
||||||
|
|
||||||
|
4. **测试 mock 策略**:迁移调用的测试从 mock `subprocess.run` 改为 mock `pyflowx.sh`,测试调用方的行为(处理 `None` 返回值)而非底层 `subprocess.run` 的异常。
|
||||||
|
|
||||||
|
5. **错误消息断言放宽**:`px.sh` 的错误消息格式(`"{label}失败 (returncode=N): ..."`)与原自定义消息不同,测试断言改为检查关键词(`"失败"`、`"命令未找到"`)而非精确消息。
|
||||||
|
|
||||||
|
## 验证结果
|
||||||
|
|
||||||
|
- 全部 1731 个测试通过
|
||||||
|
- 覆盖率 97.24%(门槛 95%)
|
||||||
|
- ruff check + format 全部通过
|
||||||
|
- 迁移后源文件覆盖率均为 100%
|
||||||
|
|
||||||
|
## 遗留事项
|
||||||
|
|
||||||
|
- `system/` 目录和 `infra/envdev.py` 的 `check=False` 调用未迁移(语义不同,不应迁移)
|
||||||
|
- `infra/sshcopyid.py` 未迁移(需要 `timeout=` 支持)
|
||||||
|
- 如需统一这些调用,可考虑扩展 `px.sh` 支持 `check=False` 和 `timeout` 参数
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# 迭代 30: px.sh 扩展 + 剩余 subprocess.run 迁移
|
||||||
|
|
||||||
|
## 本轮目标
|
||||||
|
|
||||||
|
扩展 `px.sh` 支持 `check=False` 和 `timeout` 参数,完成 `ops/` 目录下剩余 `subprocess.run` 调用的迁移,统一错误处理与中文输出。解决 iter-29 遗留的 `check=False` / `timeout=` / `FileNotFoundError` 控制流三类场景。
|
||||||
|
|
||||||
|
## 改动文件清单
|
||||||
|
|
||||||
|
### 核心扩展
|
||||||
|
|
||||||
|
| 文件 | 变更 |
|
||||||
|
|------|------|
|
||||||
|
| `shell.py` | 新增 `check: bool = True` 和 `timeout: float \| None = None` 参数;新增 `except subprocess.TimeoutExpired` 分支 |
|
||||||
|
|
||||||
|
扩展后的签名:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def sh(
|
||||||
|
cmd: list[str] | str,
|
||||||
|
*,
|
||||||
|
capture: bool = False,
|
||||||
|
label: str = "命令",
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float | None = None,
|
||||||
|
) -> subprocess.CompletedProcess[str] | None:
|
||||||
|
```
|
||||||
|
|
||||||
|
异常捕获范围:`CalledProcessError`、`FileNotFoundError`、`TimeoutExpired`(不捕获通用 `SubprocessError` 或 `OSError`)。
|
||||||
|
|
||||||
|
### 源文件迁移 (10 个)
|
||||||
|
|
||||||
|
| 文件 | 迁移调用数 | 迁移策略 |
|
||||||
|
|------|-----------|---------|
|
||||||
|
| `system/clr.py` | 1 | `check=False` → `px.sh(check=False)`,移除 `import subprocess` |
|
||||||
|
| `system/taskkill.py` | 1 | `check=False` + 三分支 returncode 检查 |
|
||||||
|
| `system/which.py` | 1 | `check=False` + `returncode == 0` 检查 |
|
||||||
|
| `system/reseticoncache.py` | 4 | 全部 `check=False`,移除 `import subprocess` |
|
||||||
|
| `dev/lscalc.py` | 1 | Linux `pgrep` 用 `check=False` + stdout 检查,移除 `import subprocess` |
|
||||||
|
| `dev/gittool.py` | 1 | `has_files()` 用 `check=False`,移除 `import subprocess` |
|
||||||
|
| `dev/autofmt.py` | 1 | `sync_pyproject_config` 用 `check=False`,移除 `import subprocess` |
|
||||||
|
| `files/screenshot.py` | 4 | Linux 回退:`try/except FileNotFoundError` → `if px.sh(...) is None: px.sh(...)`,移除 `import subprocess` |
|
||||||
|
| `infra/envdev.py` | 9 | 全部 `check=False`,移除 `import subprocess` |
|
||||||
|
| `infra/sshcopyid.py` | 1 | `timeout=` → `px.sh(timeout=...)`,三分支 returncode 检查,移除 `import subprocess` |
|
||||||
|
| `dev/piptool.py` | 0 | 仅 ruff format 格式化(frozenset 换行) |
|
||||||
|
|
||||||
|
### 测试文件 (7 个)
|
||||||
|
|
||||||
|
| 文件 | 变更 |
|
||||||
|
|------|------|
|
||||||
|
| `test_shell.py` | 新增 `TestShCheckFalse`(4 个测试)和 `TestShTimeout`(4 个测试) |
|
||||||
|
| `cli/test_system_run.py` | mock 目标从 `subprocess.run` 改为 `pyflowx.sh`,新增命令不可用测试 |
|
||||||
|
| `cli/test_reseticoncache.py` | mock 目标改为 `pyflowx.sh` |
|
||||||
|
| `cli/test_lscalc.py` | Linux 测试 mock 目标改为 `pyflowx.sh` |
|
||||||
|
| `cli/test_gittool.py` | `has_files` 测试 mock 目标改为 `pyflowx.sh` |
|
||||||
|
| `cli/test_autofmt.py` | `sync_pyproject_config` 测试 mock 目标改为 `pyflowx.sh` |
|
||||||
|
| `cli/test_screenshot.py` | Linux 测试 mock 目标改为 `pyflowx.sh`,scrot 回退测试从 `FileNotFoundError` side_effect 改为 `None` 返回值 |
|
||||||
|
| `cli/test_gittool_tool.py` | 合并 2 个失败测试为 `test_has_files_false_when_sh_fails` |
|
||||||
|
|
||||||
|
### 未迁移文件(确认不迁移)
|
||||||
|
|
||||||
|
- `command.py` — 核心命令执行器,`px.sh` 的同层抽象
|
||||||
|
- `conditions.py` — 条件检查,`subprocess.run` 检查进程/命令存在性
|
||||||
|
- `shell.py` — `px.sh` 自身实现
|
||||||
|
|
||||||
|
## 关键决策与依据
|
||||||
|
|
||||||
|
1. **`check=False` 语义**:`check=True`(默认)时非零返回码/命令未找到/超时返回 `None`;`check=False` 时任何返回码都返回 `CompletedProcess`(调用方检查 `returncode`),仅命令未找到/超时返回 `None`。
|
||||||
|
|
||||||
|
2. **`FileNotFoundError` 控制流迁移**:screenshot Linux 回退模式中 `try: gnome-screenshot except FileNotFoundError: scrot` 迁移为 `if px.sh(["gnome-screenshot", ...]) is None: px.sh(["scrot", ...])`。`px.sh` 捕获 `FileNotFoundError` 返回 `None`,语义等价且保留回退逻辑。
|
||||||
|
|
||||||
|
3. **异常捕获范围**:仅捕获 `CalledProcessError`、`FileNotFoundError`、`TimeoutExpired`。不捕获通用 `SubprocessError`(基类)或 `OSError`(非 `FileNotFoundError`),这些会向上传播,避免掩盖 bug。
|
||||||
|
|
||||||
|
4. **mock 兼容性**:`px.sh` 内部调用 `subprocess.run`,因此测试中 mock `subprocess.run` 仍能拦截 `px.sh` 的调用。`test_envdev.py` 和 `test_sshcopyid.py` 不需要更新 mock 目标。
|
||||||
|
|
||||||
|
5. **测试合并**:`test_gittool_tool.py` 中 `test_has_files_false_on_subprocess_error` 和 `test_has_files_false_on_oserror` 期望 `px.sh` 捕获通用 `SubprocessError`/`OSError`,但 `px.sh` 仅捕获特定异常。合并为 `test_has_files_false_when_sh_fails`,mock `pyflowx.sh` 返回 `None`(模拟 `px.sh` 捕获异常后的返回值)。
|
||||||
|
|
||||||
|
## 验证结果
|
||||||
|
|
||||||
|
- 全部 **1739** 个测试通过(修复 iter-29 后的 2 个失败测试)
|
||||||
|
- 覆盖率 **97.27%**(门槛 95%)
|
||||||
|
- ruff check 全部通过
|
||||||
|
- pyrefly check 0 errors
|
||||||
|
- P13 修改的源文件覆盖率均为 100%(shell.py / envdev.py / sshcopyid.py / screenshot.py / gittool.py 等)
|
||||||
|
|
||||||
|
## 遗留事项
|
||||||
|
|
||||||
|
- `command.py` 和 `conditions.py` 仍使用 `subprocess.run`(核心抽象,不应迁移)
|
||||||
|
- 6 个预存文件需要 ruff format(test_cancellation.py / test_diagnostics.py / test_fileops.py / test_profiling.py / test_report.py / piptool.py 中 piptool.py 已格式化,其余 5 个非本次范围)
|
||||||
|
- iter-21-25 应在 iter-26 前归档清理(dev-workflow.md 规则),属遗留清理事项
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
# 迭代 31:调度器性能深度优化(P14)
|
||||||
|
|
||||||
|
## 本轮目标
|
||||||
|
|
||||||
|
针对调度器执行热路径中的函数调用开销进行深度优化,提升四种执行策略
|
||||||
|
(sequential/thread/async/dependency)的 noop 吞吐。基于 cProfile
|
||||||
|
分析结果,消除高频调用路径中可避免的函数调用与对象创建。
|
||||||
|
|
||||||
|
## 改动文件清单
|
||||||
|
|
||||||
|
- `src/pyflowx/executors.py` — 4 项微优化 + 线程池复用
|
||||||
|
- `benchmarks/bench_runner.py` — 新增基准测试脚本(上一会话已建,本轮复用)
|
||||||
|
|
||||||
|
## 关键决策与依据
|
||||||
|
|
||||||
|
### 1. 线程池复用(dependency 策略主要优化)
|
||||||
|
|
||||||
|
**问题**:`asyncio.run()` 每次创建新事件循环,默认线程池也随之重建。
|
||||||
|
dependency 策略把同步任务卸载到线程池,每次 `run()` 都重复创建/销毁
|
||||||
|
线程池,500 任务的线程池提交占 36% 耗时。
|
||||||
|
|
||||||
|
**方案**:新增模块级 `_thread_pool`(`ThreadPoolExecutor`)+ 双重检查锁
|
||||||
|
惰性创建,`_submit_sync_task` 的 thread 分支改用 `_get_thread_pool()`。
|
||||||
|
`atexit` 注册 `_shutdown_thread_pool` 兜底关闭。
|
||||||
|
|
||||||
|
**依据**:与既有 `_process_pool` 复用模式一致。线程池跨 `run()` 调用
|
||||||
|
复用后,`submit` 的 `_adjust_thread_count` 不再每轮重建线程。
|
||||||
|
|
||||||
|
### 2. `_sort_by_priority` 单任务快速路径
|
||||||
|
|
||||||
|
**问题**:dependency 策略每轮就绪集常为 1 个任务,仍调用 `sorted()`
|
||||||
|
创建新列表 + 排序键函数。
|
||||||
|
|
||||||
|
**方案**:开头加 `if len(layer) <= 1: return layer`,跳过 `sorted()`。
|
||||||
|
|
||||||
|
**依据**:dependency 主循环 `while remaining or in_flight` 中
|
||||||
|
`_sort_by_priority(list(ready), all_specs)` 每轮调用,单任务就绪时
|
||||||
|
`sorted()` 的列表复制 + 键函数调用纯属开销。
|
||||||
|
|
||||||
|
### 3. `_build_context` 无依赖快速路径
|
||||||
|
|
||||||
|
**问题**:无依赖任务(最常见场景)仍创建空 dict + 遍历两个空元组。
|
||||||
|
|
||||||
|
**方案**:开头加 `if not spec.depends_on and not spec.soft_depends_on: return {}`。
|
||||||
|
|
||||||
|
**依据**:noop 基准全部无依赖,每个任务都走此路径。省去空元组遍历的
|
||||||
|
迭代器创建开销。
|
||||||
|
|
||||||
|
### 4. `_prepare_for_execution` 三无快速路径
|
||||||
|
|
||||||
|
**问题**:无依赖、无条件、无 `skip_if_missing` 的任务(最常见场景)
|
||||||
|
仍调用 `_upstream_skip_reason`(遍历空依赖返回 None)+ `should_execute`
|
||||||
|
(内部已有快速路径但有函数调用开销)。
|
||||||
|
|
||||||
|
**方案**:开头加
|
||||||
|
`if not spec.depends_on and not spec.conditions and not spec.skip_if_missing: return None`。
|
||||||
|
|
||||||
|
**依据**:无依赖时 `_upstream_skip_reason` 必返回 None(遍历空元组),
|
||||||
|
`should_execute` 的快速路径也已返回 `(True, None)`。直接返回 None 省去
|
||||||
|
两次函数调用。
|
||||||
|
|
||||||
|
### 5. `_run_hooks` 调用点内联
|
||||||
|
|
||||||
|
**问题**:`_run_hooks(spec.hooks, "pre_run", spec)` 内部用
|
||||||
|
`getattr(hooks, fn_name, None)` 查找钩子。大多数任务无钩子
|
||||||
|
(`TaskHooks` 默认全 None),仍付出函数调用 + `getattr` 开销。
|
||||||
|
4 个调用点:`SyncTaskRunner.run`(pre_run)、`AsyncTaskRunner._inner`
|
||||||
|
(pre_run)、`_mark_success`(post_run)、`_handle_failure`(on_failure)。
|
||||||
|
|
||||||
|
**方案**:每个调用点前加 `if spec.hooks.xxx is not None:` 守卫,钩子为
|
||||||
|
None 时跳过 `_run_hooks` 调用(省去函数调用 + getattr)。
|
||||||
|
|
||||||
|
**依据**:`TaskHooks` 是 `frozen=True, slots=True` dataclass,属性访问
|
||||||
|
为 O(1) slot 查找,比 `getattr` + 函数调用快。noop 基准无钩子,全部命中
|
||||||
|
快速路径。
|
||||||
|
|
||||||
|
## 验证结果
|
||||||
|
|
||||||
|
### 门禁
|
||||||
|
|
||||||
|
- ruff: All checks passed
|
||||||
|
- pyrefly: 0 errors
|
||||||
|
- pytest: 1739 passed
|
||||||
|
- coverage: 97.14%(≥95% 门槛)
|
||||||
|
|
||||||
|
### 基准对比(500 noop 任务)
|
||||||
|
|
||||||
|
| 策略 | 优化前 | 优化后 | 提升 |
|
||||||
|
|------|--------|--------|------|
|
||||||
|
| sequential | 5,032,712 ops/s | 5,664,117 ops/s | +12.6% |
|
||||||
|
| thread | 2,200,007 ops/s | 2,359,447 ops/s | +7.2% |
|
||||||
|
| async | 1,723,508 ops/s | 1,906,286 ops/s | +10.6% |
|
||||||
|
| dependency | 17,494 ops/s | 18,296 ops/s | +4.6% |
|
||||||
|
|
||||||
|
sequential 与 async 提升最显著(+10~12%),因为 noop 任务命中所有
|
||||||
|
快速路径,每任务节省的开销在总耗时中占比最大。thread 次之(+7%)。
|
||||||
|
dependency 受 asyncio 事件循环与线程池提交的固有开销主导,per-task
|
||||||
|
优化相对影响小(+4.6%)。
|
||||||
|
|
||||||
|
### cProfile 热路径变化
|
||||||
|
|
||||||
|
优化前 `build_call_args`(500 次, 0.007s)与 `_run_hooks` 占用可见。
|
||||||
|
优化后两者均跌出 top 30,剩余瓶颈为 asyncio + 线程池桥接的固有开销
|
||||||
|
(`run_in_executor` / `submit` / `_adjust_thread_count`),属架构性
|
||||||
|
开销,无法通过 per-task 微优化消除。
|
||||||
|
|
||||||
|
## 遗留事项
|
||||||
|
|
||||||
|
- dependency 策略对 noop 任务的固有并发开销仍较高(18K ops/s vs
|
||||||
|
sequential 5.6M ops/s),需真实 I/O 任务才能体现并行优势(设计预期)。
|
||||||
|
- `datetime.now()` 每任务调用 2 次(started_at + finished_at),约占
|
||||||
|
sequential 预算可观比例,但涉及 duration 正确性,未优化。
|
||||||
|
- ThreadedLayerRunner(thread 策略)每次 `run()` 创建新
|
||||||
|
`ThreadPoolExecutor`,但性能已可接受(2.3M ops/s),暂未复用。
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: "pyflowx-development"
|
name: "pyflowx-development"
|
||||||
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策,并同步迭代 21(P9)、迭代 22(P10 图片处理)、迭代 23(P11 文件操作底层 API)、迭代 24(P12 通用 DAG 构造器)与迭代 25(执行热路径性能优化)的行为变更。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化、任务编排、检查点恢复、监控导出、图片处理、文件操作底层 API、通用 DAG 构造器相关的功能时参考。"
|
description: "PyFlowX 项目(DAG 任务调度库)的开发知识库。归档迭代 06-20 中可复用的架构模式、踩坑总结与设计决策,并同步迭代 21(P9)、迭代 22(P10 图片处理)、迭代 23(P11 文件操作底层 API)、迭代 24(P12 通用 DAG 构造器)、迭代 25(执行热路径性能优化)与迭代 31(调度器深度性能优化)的行为变更。在设计与调度器、CLI、YAML 编排、取消机制、序列化、观测性、错误诊断、状态后端、性能优化、任务编排、检查点恢复、监控导出、图片处理、文件操作底层 API、通用 DAG 构造器相关的功能时参考。"
|
||||||
---
|
---
|
||||||
|
|
||||||
# PyFlowX 开发知识库
|
# PyFlowX 开发知识库
|
||||||
@@ -243,6 +243,35 @@ sequential(500) 从 764 → 930 ops/s(+22%),thread(500) 从 108 → 119 op
|
|||||||
- **适用边界**:async/dependency 策略对 noop 任务的固有并发开销(事件循环 +
|
- **适用边界**:async/dependency 策略对 noop 任务的固有并发开销(事件循环 +
|
||||||
线程池提交)主导,per-task 优化相对影响小;真实 I/O 任务才能体现并行优势。
|
线程池提交)主导,per-task 优化相对影响小;真实 I/O 任务才能体现并行优势。
|
||||||
|
|
||||||
|
### 调度器深度优化 5 项(iter-31)
|
||||||
|
|
||||||
|
sequential(500) 从 5.0M → 5.7M ops/s(+12.6%),async 从 1.7M → 1.9M
|
||||||
|
(+10.6%),thread 从 2.2M → 2.4M(+7.2%),dependency 从 17.5K → 18.3K
|
||||||
|
(+4.6%)。
|
||||||
|
|
||||||
|
- **线程池复用**:新增模块级 `_thread_pool`(`ThreadPoolExecutor`)+ 双重
|
||||||
|
检查锁惰性创建,`_submit_sync_task` 的 thread 分支改用 `_get_thread_pool()`。
|
||||||
|
`asyncio.run()` 每次创建新事件循环,默认线程池也随之重建;模块级缓存让
|
||||||
|
线程池跨 `run()` 调用复用。`atexit` 注册 `_shutdown_thread_pool` 兜底。
|
||||||
|
与既有 `_process_pool` 模式一致。
|
||||||
|
- **`_sort_by_priority` 单任务快速路径**:`if len(layer) <= 1: return layer`。
|
||||||
|
dependency 策略每轮就绪集常为 1 个任务,跳过 `sorted()` 的列表复制 + 键函数。
|
||||||
|
- **`_build_context` 无依赖快速路径**:
|
||||||
|
`if not spec.depends_on and not spec.soft_depends_on: return {}`,跳过两个
|
||||||
|
空元组遍历。
|
||||||
|
- **`_prepare_for_execution` 三无快速路径**:
|
||||||
|
`if not spec.depends_on and not spec.conditions and not spec.skip_if_missing: return None`,
|
||||||
|
省去 `_upstream_skip_reason`(无依赖必返回 None)+ `should_execute` 两次
|
||||||
|
函数调用。
|
||||||
|
- **`_run_hooks` 调用点内联**:4 个调用点(`SyncTaskRunner.run`/`AsyncTaskRunner._inner`
|
||||||
|
的 pre_run、`_mark_success` 的 post_run、`_handle_failure` 的 on_failure)
|
||||||
|
前加 `if spec.hooks.xxx is not None:` 守卫。`TaskHooks` 默认全 None,noop
|
||||||
|
基准全部命中快速路径,省去 `_run_hooks` 函数调用 + `getattr` 开销。
|
||||||
|
- **优化边界**:cProfile 确认优化后 `build_call_args` 与 `_run_hooks` 跌出
|
||||||
|
top 30。剩余瓶颈为 asyncio + 线程池桥接的固有开销
|
||||||
|
(`run_in_executor`/`submit`/`_adjust_thread_count`),属架构性开销,无法
|
||||||
|
通过 per-task 微优化消除。
|
||||||
|
|
||||||
### 事件驱动优先级调度(iter-27)
|
### 事件驱动优先级调度(iter-27)
|
||||||
|
|
||||||
DependencyRunner 从"一次性创建所有 asyncio Task"重写为"事件驱动按需创建":
|
DependencyRunner 从"一次性创建所有 asyncio Task"重写为"事件驱动按需创建":
|
||||||
@@ -625,38 +654,42 @@ DependencyRunner 主循环每轮调用 `_find_ready()` 扫描全部 `remaining`
|
|||||||
|
|
||||||
### 可复用模式
|
### 可复用模式
|
||||||
|
|
||||||
- **`subprocess.run` 的 `check` 参数分级策略**(iter-16):
|
- **`px.sh` 的 `check` 参数分级策略**(iter-16 → iter-30 演进):
|
||||||
- **失败需感知**(dockercmd/msdownload/sglang)→ `check=True` +
|
- **失败需感知**(dockercmd/msdownload/sglang)→ `px.sh(cmd, label="...")`(默认 `check=True`),失败返回 `None` + 中文打印
|
||||||
`try/except subprocess.CalledProcessError` 打印 rich 错误
|
- **失败可容忍但需检查**(taskkill/which/has_files)→ `px.sh(cmd, check=False, label="...")` + 检查 `returncode`/`stdout`
|
||||||
- **失败可容忍但需检查**(taskkill)→ 保持 `check=False`,但检查
|
- **失败无关紧要**(clr/reseticoncache/envdev)→ `px.sh(cmd, check=False, label="...")`
|
||||||
`returncode` 打印结果(pkill 返回 1 表示无匹配进程,非错误)
|
- **超时控制**(sshcopyid)→ `px.sh(cmd, timeout=N, label="...")`,超时返回 `None`
|
||||||
- **失败无关紧要**(clr/which)→ `check=False`,加注释说明原因
|
- **命令回退**(screenshot Linux)→ `if px.sh(cmd1, label="...") is None: px.sh(cmd2, label="...")`(替代 `try/except FileNotFoundError`)
|
||||||
- **错误路径统一处理模式**:
|
- **px.sh 统一错误处理**:
|
||||||
```python
|
```python
|
||||||
try:
|
result = px.sh(cmd, label="操作")
|
||||||
subprocess.run(cmd, check=True, ...)
|
if result is None:
|
||||||
except subprocess.CalledProcessError as e:
|
# 命令失败/未找到/超时,px.sh 已打印中文错误
|
||||||
_console.print(f"[red]命令失败 (returncode={e.returncode})[/]")
|
return
|
||||||
except FileNotFoundError:
|
|
||||||
_console.print(f"[red]命令未找到: {cmd[0]}[/]")
|
|
||||||
```
|
```
|
||||||
|
`px.sh` 内部捕获 `CalledProcessError`/`FileNotFoundError`/`TimeoutExpired`,不捕获通用 `SubprocessError`/`OSError`(向上传播)。
|
||||||
|
|
||||||
### 设计决策
|
### 设计决策
|
||||||
|
|
||||||
- **rich Console 而非 print**:CLI 工具错误用 `rich.console.Console`
|
- **px.sh 封装优于直接 subprocess.run**:统一中文错误输出 + `None` 返回值语义,
|
||||||
打印带颜色输出,与项目其他 CLI 工具一致。
|
调用方无需重复 try/except。`check=False` 时返回 `CompletedProcess`,调用方检查 `returncode`。
|
||||||
- **`check=True` + try/except 优于 `check=False` + 手动检查**:除非
|
- **`check=True` 默认优于 `check=False`**:除非命令返回非零是预期行为
|
||||||
命令返回非零是预期行为(如 taskkill pkill 无匹配、which 未找到命令),
|
(如 taskkill pkill 无匹配、which 未找到命令),否则用默认 `check=True`。
|
||||||
否则用 `check=True` 让 subprocess 抛异常,统一在 except 中处理。
|
- **异常捕获范围刻意收窄**:仅捕获 `CalledProcessError`/`FileNotFoundError`/`TimeoutExpired`,
|
||||||
|
通用 `SubprocessError`/`OSError` 向上传播,避免掩盖 bug。
|
||||||
|
|
||||||
### 踩坑总结
|
### 踩坑总结
|
||||||
|
|
||||||
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
|
- **`@patch` 装饰器禁用**:项目 `python-standards.md` 禁用 `@patch`
|
||||||
装饰器与 `mock.patch.object` 上下文。mock subprocess 异常用
|
装饰器与 `mock.patch.object` 上下文。mock `px.sh` 用
|
||||||
|
`monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)`;
|
||||||
|
mock `subprocess.run` 抛异常用
|
||||||
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
|
`monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))`
|
||||||
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
|
—— 用生成器抛异常的技巧让 lambda 能抛出指定异常。
|
||||||
- **mock 异常抛出技巧**:`lambda *_, **__: (_ for _ in ()).throw(err)`
|
- **mock 兼容性**:`px.sh` 内部调用 `subprocess.run`,因此 mock `subprocess.run`
|
||||||
利用生成器仅在迭代时抛异常的特性,让无参 lambda 能抛出指定异常。
|
仍能拦截 `px.sh` 的调用。但 mock `px.sh` 更直接(测试调用方行为而非底层实现)。
|
||||||
|
- **`SubprocessError`/`OSError` 不被 px.sh 捕获**:测试中不应期望 `px.sh` 捕获
|
||||||
|
通用异常基类,应 mock `px.sh` 返回 `None` 模拟捕获后的返回值。
|
||||||
|
|
||||||
## 十三、测试与质量保障
|
## 十三、测试与质量保障
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
"""PyFlowX 调度器性能基准测试.
|
||||||
|
|
||||||
|
测量 4 种执行策略(sequential/thread/async/dependency)的调度开销,
|
||||||
|
使用 noop 任务隔离调度器本身的开销。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
uv run python benchmarks/bench_runner.py # 运行所有基准(默认 500 任务)
|
||||||
|
uv run python benchmarks/bench_runner.py -s 1000 # 指定任务数
|
||||||
|
uv run python benchmarks/bench_runner.py --profile # cProfile 分析 sequential
|
||||||
|
uv run python benchmarks/bench_runner.py --profile --strategy thread
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import cProfile
|
||||||
|
import io
|
||||||
|
import pstats
|
||||||
|
import time
|
||||||
|
from typing import cast
|
||||||
|
|
||||||
|
import pyflowx as px
|
||||||
|
from pyflowx import TaskSpec
|
||||||
|
from pyflowx.executors import Strategy
|
||||||
|
|
||||||
|
_STRATEGIES: tuple[Strategy, ...] = ("sequential", "thread", "async", "dependency")
|
||||||
|
|
||||||
|
|
||||||
|
def _noop() -> None:
|
||||||
|
"""空操作任务,用于隔离调度器开销。"""
|
||||||
|
|
||||||
|
|
||||||
|
def build_graph(n: int) -> px.Graph:
|
||||||
|
"""构建 n 个无依赖 noop 任务的图。"""
|
||||||
|
specs = {f"task-{i}": TaskSpec(f"task-{i}", _noop) for i in range(n)}
|
||||||
|
return px.Graph(specs=specs)
|
||||||
|
|
||||||
|
|
||||||
|
def bench_strategy(n: int, strategy: Strategy) -> float:
|
||||||
|
"""测量指定策略的 ops/s。"""
|
||||||
|
graph = build_graph(n)
|
||||||
|
start = time.perf_counter()
|
||||||
|
px.run(graph, strategy=strategy)
|
||||||
|
elapsed = time.perf_counter() - start
|
||||||
|
return n / elapsed
|
||||||
|
|
||||||
|
|
||||||
|
def run_benchmarks(n: int) -> dict[str, float]:
|
||||||
|
"""运行所有基准测试,返回 {策略: ops/s}。"""
|
||||||
|
results: dict[str, float] = {}
|
||||||
|
for strategy in ("sequential", "thread", "async", "dependency"):
|
||||||
|
results[strategy] = bench_strategy(n, strategy)
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def run_profile(n: int, strategy: Strategy) -> None:
|
||||||
|
"""用 cProfile 分析指定策略的热路径。"""
|
||||||
|
profiler = cProfile.Profile()
|
||||||
|
profiler.enable()
|
||||||
|
graph = build_graph(n)
|
||||||
|
px.run(graph, strategy=strategy)
|
||||||
|
profiler.disable()
|
||||||
|
|
||||||
|
buf = io.StringIO()
|
||||||
|
stats = pstats.Stats(profiler, stream=buf).sort_stats("cumulative")
|
||||||
|
stats.print_stats(30)
|
||||||
|
print(buf.getvalue())
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description="PyFlowX 调度器性能基准测试")
|
||||||
|
parser.add_argument("-s", "--size", type=int, default=500, help="任务数量(默认 500)")
|
||||||
|
parser.add_argument("--profile", action="store_true", help="启用 cProfile 分析")
|
||||||
|
parser.add_argument("--strategy", default="sequential", help="分析的策略(默认 sequential)")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.profile:
|
||||||
|
strategy = cast("Strategy", args.strategy)
|
||||||
|
print(f"cProfile 分析: {strategy} 策略, {args.size} 任务")
|
||||||
|
print("=" * 70)
|
||||||
|
run_profile(args.size, strategy)
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"PyFlowX 调度器性能基准测试({args.size} noop 任务)")
|
||||||
|
print("=" * 70)
|
||||||
|
results = run_benchmarks(args.size)
|
||||||
|
for strategy, ops in results.items():
|
||||||
|
print(f" {strategy:12s}: {ops:8.1f} ops/s")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+5
-5
@@ -21,12 +21,13 @@ license = { text = "MIT" }
|
|||||||
name = "pyflowx"
|
name = "pyflowx"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
version = "0.4.9"
|
version = "0.4.11"
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
emlman = "pyflowx.cli.legacy.emlmanager:main"
|
emlman = "pyflowx.cli.legacy.emlmanager:main"
|
||||||
pf = "pyflowx.cli.pf:main"
|
pf = "pyflowx.cli.pf:main"
|
||||||
pxp = "pyflowx.cli.legacy.profiler:main"
|
pxp = "pyflowx.cli.legacy.profiler:main"
|
||||||
|
pyflowx = "pyflowx.cli.pf:main"
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
dev = [
|
dev = [
|
||||||
@@ -105,7 +106,6 @@ target-version = "py310"
|
|||||||
ignore = [
|
ignore = [
|
||||||
"E501", # line too long (handled by formatter)
|
"E501", # line too long (handled by formatter)
|
||||||
"PLC0415", # import should be at top-level (intentional for lazy imports)
|
"PLC0415", # import should be at top-level (intentional for lazy imports)
|
||||||
"PLR0913", # too many arguments
|
|
||||||
"PLR0915", # too many statements (intentional for complex methods)
|
"PLR0915", # too many statements (intentional for complex methods)
|
||||||
"PLR2004", # magic value comparison
|
"PLR2004", # magic value comparison
|
||||||
"PTH119", # os.path.basename (intentional for sys.argv)
|
"PTH119", # os.path.basename (intentional for sys.argv)
|
||||||
|
|||||||
@@ -112,7 +112,7 @@ from .task import (
|
|||||||
from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool
|
from .tools import ToolSpec, list_subcommands, list_tools, run_tool, tool
|
||||||
from .yaml_loader import load_yaml, parse_yaml_string
|
from .yaml_loader import load_yaml, parse_yaml_string
|
||||||
|
|
||||||
__version__ = "0.4.9"
|
__version__ = "0.4.11"
|
||||||
|
|
||||||
|
|
||||||
def graph(
|
def graph(
|
||||||
|
|||||||
+171
-152
@@ -52,6 +52,7 @@ import queue
|
|||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping
|
from collections.abc import Awaitable, Callable, Iterable, Iterator, Mapping
|
||||||
|
from dataclasses import dataclass, field
|
||||||
from dataclasses import replace as dc_replace
|
from dataclasses import replace as dc_replace
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -118,6 +119,34 @@ def _shutdown_process_pool() -> None:
|
|||||||
atexit.register(_shutdown_process_pool)
|
atexit.register(_shutdown_process_pool)
|
||||||
|
|
||||||
|
|
||||||
|
# 线程池复用:asyncio.run() 每次创建新事件循环,默认线程池也随之重建。
|
||||||
|
# 模块级缓存让线程池跨 run() 调用复用,避免重复创建/销毁线程的开销。
|
||||||
|
_thread_pool: concurrent.futures.ThreadPoolExecutor | None = None
|
||||||
|
_thread_pool_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def _get_thread_pool() -> concurrent.futures.ThreadPoolExecutor:
|
||||||
|
"""获取复用的线程池(惰性创建)。"""
|
||||||
|
global _thread_pool # noqa: PLW0603
|
||||||
|
if _thread_pool is None:
|
||||||
|
with _thread_pool_lock:
|
||||||
|
if _thread_pool is None:
|
||||||
|
_thread_pool = concurrent.futures.ThreadPoolExecutor()
|
||||||
|
return _thread_pool
|
||||||
|
|
||||||
|
|
||||||
|
def _shutdown_thread_pool() -> None:
|
||||||
|
"""关闭复用的线程池。"""
|
||||||
|
global _thread_pool # noqa: PLW0603
|
||||||
|
if _thread_pool is not None:
|
||||||
|
pool = _thread_pool
|
||||||
|
_thread_pool = None
|
||||||
|
pool.shutdown(wait=False)
|
||||||
|
|
||||||
|
|
||||||
|
atexit.register(_shutdown_thread_pool)
|
||||||
|
|
||||||
|
|
||||||
def _run_in_process(fn: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
|
def _run_in_process(fn: Any, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
|
||||||
"""模块级函数:在进程池中执行任务(须可 pickle)。
|
"""模块级函数:在进程池中执行任务(须可 pickle)。
|
||||||
|
|
||||||
@@ -132,6 +161,24 @@ EventCallback = Callable[[TaskEvent], None]
|
|||||||
Strategy = Literal["sequential", "thread", "async", "dependency"]
|
Strategy = Literal["sequential", "thread", "async", "dependency"]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _ExecContext:
|
||||||
|
"""执行上下文:捆绑 run() 调用链中共享的状态,减少参数传递。
|
||||||
|
|
||||||
|
将 context/report/backend/on_event/concurrency_limits/cancel_event
|
||||||
|
打包为单一参数,使调用链中每个函数的参数数 ≤5(PLR0913)。
|
||||||
|
frozen=True 保证调用链中不可意外替换整体引用,但不阻止对
|
||||||
|
context/report 等可变属性的内部修改(如 ``ctx.context[name] = value``)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
context: dict[str, Any]
|
||||||
|
report: RunReport
|
||||||
|
backend: StateBackend
|
||||||
|
on_event: EventCallback | None
|
||||||
|
concurrency_limits: Mapping[str, int] = field(default_factory=dict)
|
||||||
|
cancel_event: threading.Event | CancelToken | None = None
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
# 无状态公共辅助
|
# 无状态公共辅助
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
@@ -193,6 +240,9 @@ def _build_context(
|
|||||||
硬依赖:若上游 SKIPPED/FAILED 则不注入(本任务通常也会被跳过)。
|
硬依赖:若上游 SKIPPED/FAILED 则不注入(本任务通常也会被跳过)。
|
||||||
软依赖:上游成功则注入其值;否则注入 ``spec.defaults`` 中的默认值(或 ``None``)。
|
软依赖:上游成功则注入其值;否则注入 ``spec.defaults`` 中的默认值(或 ``None``)。
|
||||||
"""
|
"""
|
||||||
|
# 快速路径:无依赖任务直接返回空 dict,跳过两个空元组遍历。
|
||||||
|
if not spec.depends_on and not spec.soft_depends_on:
|
||||||
|
return {}
|
||||||
ctx: dict[str, Any] = {}
|
ctx: dict[str, Any] = {}
|
||||||
for dep in spec.depends_on:
|
for dep in spec.depends_on:
|
||||||
if dep in global_context:
|
if dep in global_context:
|
||||||
@@ -208,32 +258,28 @@ def _build_context(
|
|||||||
|
|
||||||
|
|
||||||
def _apply_cached(
|
def _apply_cached(
|
||||||
name: str,
|
|
||||||
spec: TaskSpec[Any],
|
spec: TaskSpec[Any],
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
) -> bool:
|
) -> bool:
|
||||||
"""若 ``name`` 命中缓存,写入 context/report 并返回 True。
|
"""若 ``spec.name`` 命中缓存,写入 context/report 并返回 True。
|
||||||
|
|
||||||
单次 ``backend.get`` + ``KeyError`` 回退,避免 ``has`` + ``get`` 双重
|
单次 ``backend.get`` + ``KeyError`` 回退,避免 ``has`` + ``get`` 双重
|
||||||
哈希查找与双重 TTL 判断。
|
哈希查找与双重 TTL 判断。
|
||||||
"""
|
"""
|
||||||
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
|
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
|
||||||
storage_key = spec.name if spec.cache_key is None else spec.storage_key(context)
|
storage_key = spec.name if spec.cache_key is None else spec.storage_key(ctx.context)
|
||||||
try:
|
try:
|
||||||
cached = backend.get(storage_key)
|
cached = ctx.backend.get(storage_key)
|
||||||
except KeyError:
|
except KeyError:
|
||||||
return False
|
return False
|
||||||
context[name] = cached
|
ctx.context[spec.name] = cached
|
||||||
result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中")
|
result = TaskResult(spec=spec, status=TaskStatus.SKIPPED, value=cached, reason="缓存命中")
|
||||||
report.results[name] = result
|
ctx.report.results[spec.name] = result
|
||||||
_emit(on_event, result)
|
_emit(ctx.on_event, result)
|
||||||
logger.info(
|
logger.info(
|
||||||
"task %r skipped (cached)",
|
"task %r skipped (cached)",
|
||||||
name,
|
spec.name,
|
||||||
extra={"run_id": report.run_id, "task_name": name, "status": TaskStatus.SKIPPED.value},
|
extra={"run_id": ctx.report.run_id, "task_name": spec.name, "status": TaskStatus.SKIPPED.value},
|
||||||
)
|
)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
@@ -244,6 +290,10 @@ def _sort_by_priority(layer: list[str], specs: Mapping[str, TaskSpec[Any]]) -> l
|
|||||||
接受预构建的 ``{name: spec}`` 映射,避免在排序键函数中重复调用
|
接受预构建的 ``{name: spec}`` 映射,避免在排序键函数中重复调用
|
||||||
``graph.resolved_spec``(即便有缓存也省去 N 次字典查询)。
|
``graph.resolved_spec``(即便有缓存也省去 N 次字典查询)。
|
||||||
"""
|
"""
|
||||||
|
# 快速路径:单任务或空层无需排序,跳过 sorted() 调用开销。
|
||||||
|
# dependency 策略每轮就绪集常为 1 个任务,此路径命中率高。
|
||||||
|
if len(layer) <= 1:
|
||||||
|
return layer
|
||||||
return sorted(layer, key=lambda n: -specs[n].priority)
|
return sorted(layer, key=lambda n: -specs[n].priority)
|
||||||
|
|
||||||
|
|
||||||
@@ -304,6 +354,11 @@ def _prepare_for_execution(
|
|||||||
返回 SKIPPED TaskResult 或 ``None``(继续执行)。
|
返回 SKIPPED TaskResult 或 ``None``(继续执行)。
|
||||||
条件判断委托给 :meth:`TaskSpec.should_execute`,避免重复实现。
|
条件判断委托给 :meth:`TaskSpec.should_execute`,避免重复实现。
|
||||||
"""
|
"""
|
||||||
|
# 快速路径:无依赖、无条件、无 skip_if_missing 时直接放行(最常见场景),
|
||||||
|
# 省去 _upstream_skip_reason 与 should_execute 两次函数调用开销。
|
||||||
|
# 无依赖时 _upstream_skip_reason 必返回 None(遍历空元组),故可安全跳过。
|
||||||
|
if not spec.depends_on and not spec.conditions and not spec.skip_if_missing:
|
||||||
|
return None
|
||||||
# 1. 上游被跳过/失败
|
# 1. 上游被跳过/失败
|
||||||
skip_reason = _upstream_skip_reason(spec, report)
|
skip_reason = _upstream_skip_reason(spec, report)
|
||||||
# 2. 条件 / skip_if_missing(单一来源:TaskSpec.should_execute)
|
# 2. 条件 / skip_if_missing(单一来源:TaskSpec.should_execute)
|
||||||
@@ -345,33 +400,31 @@ def _mark_success(spec: TaskSpec[Any], result: TaskResult[Any], value: Any) -> N
|
|||||||
result.value = value
|
result.value = value
|
||||||
result.status = TaskStatus.SUCCESS
|
result.status = TaskStatus.SUCCESS
|
||||||
result.finished_at = datetime.now()
|
result.finished_at = datetime.now()
|
||||||
_run_hooks(spec.hooks, "post_run", spec, value)
|
if spec.hooks.post_run is not None:
|
||||||
|
_run_hooks(spec.hooks, "post_run", spec, value)
|
||||||
|
|
||||||
|
|
||||||
def _finalize_failure(
|
def _finalize_failure(
|
||||||
result: TaskResult[Any],
|
result: TaskResult[Any],
|
||||||
layer_idx: int | None,
|
layer_idx: int | None,
|
||||||
on_event: EventCallback | None,
|
ctx: _ExecContext,
|
||||||
continue_on_error: bool,
|
continue_on_error: bool,
|
||||||
name: str,
|
|
||||||
report: RunReport | None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。
|
"""标记任务为 FAILED。若 ``continue_on_error`` 为真则不抛出异常。
|
||||||
|
|
||||||
失败结果在抛出前写入 ``report.results[name]``,使流式 API
|
失败结果在抛出前写入 ``ctx.report.results``,使流式 API
|
||||||
(:func:`run_iter`) 能在 re-raise 前yield 该结果。
|
(:func:`run_iter`) 能在 re-raise 前yield 该结果。
|
||||||
"""
|
"""
|
||||||
result.status = TaskStatus.FAILED
|
result.status = TaskStatus.FAILED
|
||||||
result.finished_at = datetime.now()
|
result.finished_at = datetime.now()
|
||||||
if report is not None:
|
ctx.report.results[result.spec.name] = result
|
||||||
report.results[name] = result
|
_emit(ctx.on_event, result)
|
||||||
_emit(on_event, result)
|
|
||||||
if continue_on_error:
|
if continue_on_error:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"task %r failed but continue_on_error=True; continuing.",
|
"task %r failed but continue_on_error=True; continuing.",
|
||||||
result.spec.name,
|
result.spec.name,
|
||||||
extra={
|
extra={
|
||||||
"run_id": report.run_id if report is not None else "-",
|
"run_id": ctx.report.run_id,
|
||||||
"task_name": result.spec.name,
|
"task_name": result.spec.name,
|
||||||
"status": TaskStatus.FAILED.value,
|
"status": TaskStatus.FAILED.value,
|
||||||
"attempts": result.attempts,
|
"attempts": result.attempts,
|
||||||
@@ -384,7 +437,7 @@ def _finalize_failure(
|
|||||||
cause=result.error if result.error is not None else RuntimeError("unknown"),
|
cause=result.error if result.error is not None else RuntimeError("unknown"),
|
||||||
attempts=result.attempts,
|
attempts=result.attempts,
|
||||||
layer=layer_idx,
|
layer=layer_idx,
|
||||||
report=report,
|
report=ctx.report,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -393,9 +446,7 @@ def _handle_failure(
|
|||||||
result: TaskResult[Any],
|
result: TaskResult[Any],
|
||||||
exc: BaseException,
|
exc: BaseException,
|
||||||
layer_idx: int | None,
|
layer_idx: int | None,
|
||||||
on_event: EventCallback | None,
|
ctx: _ExecContext,
|
||||||
name: str,
|
|
||||||
report: RunReport | None,
|
|
||||||
) -> bool:
|
) -> bool:
|
||||||
"""统一处理失败:超时转换、重试决策、finalize。
|
"""统一处理失败:超时转换、重试决策、finalize。
|
||||||
|
|
||||||
@@ -405,7 +456,7 @@ def _handle_failure(
|
|||||||
``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。
|
``True`` 表示已 finalize(不再重试);``False`` 表示应继续重试。
|
||||||
"""
|
"""
|
||||||
# asyncio.TimeoutError → TaskTimeoutError(统一异常类型)
|
# asyncio.TimeoutError → TaskTimeoutError(统一异常类型)
|
||||||
run_id = report.run_id if report is not None else "-"
|
run_id = ctx.report.run_id
|
||||||
if isinstance(exc, asyncio.TimeoutError):
|
if isinstance(exc, asyncio.TimeoutError):
|
||||||
exc = TaskTimeoutError(spec.name, spec.timeout or 0.0)
|
exc = TaskTimeoutError(spec.name, spec.timeout or 0.0)
|
||||||
logger.warning(
|
logger.warning(
|
||||||
@@ -439,8 +490,9 @@ def _handle_failure(
|
|||||||
result.error = exc
|
result.error = exc
|
||||||
if _should_retry(spec, result.attempts, exc):
|
if _should_retry(spec, result.attempts, exc):
|
||||||
return False
|
return False
|
||||||
_run_hooks(spec.hooks, "on_failure", spec, exc)
|
if spec.hooks.on_failure is not None:
|
||||||
_finalize_failure(result, layer_idx, on_event, spec.continue_on_error, name, report)
|
_run_hooks(spec.hooks, "on_failure", spec, exc)
|
||||||
|
_finalize_failure(result, layer_idx, ctx, spec.continue_on_error)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
@@ -453,21 +505,21 @@ class SyncTaskRunner:
|
|||||||
@staticmethod
|
@staticmethod
|
||||||
def run(
|
def run(
|
||||||
spec: TaskSpec[Any],
|
spec: TaskSpec[Any],
|
||||||
context: Mapping[str, Any],
|
task_ctx: Mapping[str, Any],
|
||||||
layer_idx: int | None,
|
layer_idx: int | None,
|
||||||
on_event: EventCallback | None = None,
|
ctx: _ExecContext,
|
||||||
report: RunReport | None = None,
|
|
||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
skipped = _prepare_for_execution(spec, context, report, on_event)
|
skipped = _prepare_for_execution(spec, task_ctx, ctx.report, ctx.on_event)
|
||||||
if skipped is not None:
|
if skipped is not None:
|
||||||
return skipped
|
return skipped
|
||||||
|
|
||||||
result: TaskResult[Any] = TaskResult(spec=spec)
|
result: TaskResult[Any] = TaskResult(spec=spec)
|
||||||
result.started_at = datetime.now()
|
result.started_at = datetime.now()
|
||||||
args, kwargs = build_call_args(spec, context)
|
args, kwargs = build_call_args(spec, task_ctx)
|
||||||
|
|
||||||
_run_hooks(spec.hooks, "pre_run", spec)
|
if spec.hooks.pre_run is not None:
|
||||||
_emit_running(on_event, spec)
|
_run_hooks(spec.hooks, "pre_run", spec)
|
||||||
|
_emit_running(ctx.on_event, spec)
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
result.attempts += 1
|
result.attempts += 1
|
||||||
@@ -481,7 +533,7 @@ class SyncTaskRunner:
|
|||||||
_mark_success(spec, result, value)
|
_mark_success(spec, result, value)
|
||||||
return result
|
return result
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
|
if _handle_failure(spec, result, exc, layer_idx, ctx):
|
||||||
return result
|
return result
|
||||||
wait = spec.retry.wait_seconds(result.attempts)
|
wait = spec.retry.wait_seconds(result.attempts)
|
||||||
if wait > 0:
|
if wait > 0:
|
||||||
@@ -494,24 +546,24 @@ class AsyncTaskRunner:
|
|||||||
@staticmethod
|
@staticmethod
|
||||||
async def run(
|
async def run(
|
||||||
spec: TaskSpec[Any],
|
spec: TaskSpec[Any],
|
||||||
context: Mapping[str, Any],
|
task_ctx: Mapping[str, Any],
|
||||||
layer_idx: int | None,
|
layer_idx: int | None,
|
||||||
on_event: EventCallback | None = None,
|
ctx: _ExecContext,
|
||||||
report: RunReport | None = None,
|
|
||||||
semaphore: asyncio.Semaphore | None = None,
|
semaphore: asyncio.Semaphore | None = None,
|
||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
skipped = _prepare_for_execution(spec, context, report, on_event)
|
skipped = _prepare_for_execution(spec, task_ctx, ctx.report, ctx.on_event)
|
||||||
if skipped is not None:
|
if skipped is not None:
|
||||||
return skipped
|
return skipped
|
||||||
|
|
||||||
async def _inner() -> TaskResult[Any]:
|
async def _inner() -> TaskResult[Any]:
|
||||||
result: TaskResult[Any] = TaskResult(spec=spec)
|
result: TaskResult[Any] = TaskResult(spec=spec)
|
||||||
result.started_at = datetime.now()
|
result.started_at = datetime.now()
|
||||||
args, kwargs = build_call_args(spec, context)
|
args, kwargs = build_call_args(spec, task_ctx)
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
|
|
||||||
_run_hooks(spec.hooks, "pre_run", spec)
|
if spec.hooks.pre_run is not None:
|
||||||
_emit_running(on_event, spec)
|
_run_hooks(spec.hooks, "pre_run", spec)
|
||||||
|
_emit_running(ctx.on_event, spec)
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
result.attempts += 1
|
result.attempts += 1
|
||||||
@@ -520,7 +572,7 @@ class AsyncTaskRunner:
|
|||||||
_mark_success(spec, result, value)
|
_mark_success(spec, result, value)
|
||||||
return result
|
return result
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
if _handle_failure(spec, result, exc, layer_idx, on_event, spec.name, report):
|
if _handle_failure(spec, result, exc, layer_idx, ctx):
|
||||||
return result
|
return result
|
||||||
wait = spec.retry.wait_seconds(result.attempts)
|
wait = spec.retry.wait_seconds(result.attempts)
|
||||||
if wait > 0:
|
if wait > 0:
|
||||||
@@ -584,8 +636,8 @@ def _submit_sync_task(
|
|||||||
proc_fn = partial(_run_in_process, spec.effective_fn, args, kwargs)
|
proc_fn = partial(_run_in_process, spec.effective_fn, args, kwargs)
|
||||||
return loop.run_in_executor(pool, proc_fn)
|
return loop.run_in_executor(pool, proc_fn)
|
||||||
|
|
||||||
# thread(默认):线程池执行。
|
# thread(默认):复用模块级线程池,避免每次 asyncio.run() 创建新线程池的开销。
|
||||||
return loop.run_in_executor(None, fn_call)
|
return loop.run_in_executor(_get_thread_pool(), fn_call)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
@@ -594,10 +646,7 @@ def _submit_sync_task(
|
|||||||
def _filter_and_sort(
|
def _filter_and_sort(
|
||||||
layer: list[str],
|
layer: list[str],
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
) -> tuple[list[str], dict[str, TaskSpec[Any]]]:
|
) -> tuple[list[str], dict[str, TaskSpec[Any]]]:
|
||||||
"""过滤掉已命中缓存的任务,按优先级排序返回待运行列表与 specs 映射。
|
"""过滤掉已命中缓存的任务,按优先级排序返回待运行列表与 specs 映射。
|
||||||
|
|
||||||
@@ -611,36 +660,32 @@ def _filter_and_sort(
|
|||||||
spec = graph.resolved_spec(name)
|
spec = graph.resolved_spec(name)
|
||||||
specs[name] = spec
|
specs[name] = spec
|
||||||
# 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过
|
# 检查点恢复:已在 report 中的 SUCCESS 任务直接跳过
|
||||||
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
|
if name in ctx.report.results and ctx.report.results[name].status == TaskStatus.SUCCESS:
|
||||||
context[name] = report.results[name].value
|
ctx.context[name] = ctx.report.results[name].value
|
||||||
continue
|
continue
|
||||||
if not _apply_cached(name, spec, context, report, backend, on_event):
|
if not _apply_cached(spec, ctx):
|
||||||
to_run.append(name)
|
to_run.append(name)
|
||||||
return _sort_by_priority(to_run, specs), specs
|
return _sort_by_priority(to_run, specs), specs
|
||||||
|
|
||||||
|
|
||||||
def _store_result(
|
def _store_result(
|
||||||
name: str,
|
|
||||||
result: TaskResult[Any],
|
result: TaskResult[Any],
|
||||||
spec: TaskSpec[Any],
|
spec: TaskSpec[Any],
|
||||||
task_ctx: dict[str, Any],
|
task_ctx: dict[str, Any],
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
"""存储任务结果到 context/report/backend 并触发事件。
|
"""存储任务结果到 context/report/backend 并触发事件。
|
||||||
|
|
||||||
``spec`` 与 ``task_ctx`` 由调用方在执行前已构建,直接复用避免重复
|
``spec`` 与 ``task_ctx`` 由调用方在执行前已构建,直接复用避免重复
|
||||||
``resolved_spec`` / ``_build_context`` 调用。
|
``resolved_spec`` / ``_build_context`` 调用。
|
||||||
"""
|
"""
|
||||||
context[name] = result.value
|
ctx.context[spec.name] = result.value
|
||||||
if result.status == TaskStatus.SUCCESS:
|
if result.status == TaskStatus.SUCCESS:
|
||||||
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
|
# 快速路径:无 cache_key 时存储键即为任务名,避免函数调用开销。
|
||||||
key = spec.name if spec.cache_key is None else spec.storage_key(task_ctx)
|
key = spec.name if spec.cache_key is None else spec.storage_key(task_ctx)
|
||||||
backend.save(key, result.value)
|
ctx.backend.save(key, result.value)
|
||||||
report.results[name] = result
|
ctx.report.results[spec.name] = result
|
||||||
_emit(on_event, result)
|
_emit(ctx.on_event, result)
|
||||||
|
|
||||||
|
|
||||||
def _build_semaphores(
|
def _build_semaphores(
|
||||||
@@ -676,18 +721,15 @@ class SequentialLayerRunner:
|
|||||||
def execute(
|
def execute(
|
||||||
layer: list[str],
|
layer: list[str],
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
layer_idx: int,
|
layer_idx: int,
|
||||||
on_event: EventCallback | None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event)
|
to_run, specs = _filter_and_sort(layer, graph, ctx)
|
||||||
for name in to_run:
|
for name in to_run:
|
||||||
spec = specs[name]
|
spec = specs[name]
|
||||||
task_ctx = _build_context(spec, context, report)
|
task_ctx = _build_context(spec, ctx.context, ctx.report)
|
||||||
result = SyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report)
|
result = SyncTaskRunner.run(spec, task_ctx, layer_idx, ctx)
|
||||||
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
|
_store_result(result, spec, task_ctx, ctx)
|
||||||
|
|
||||||
|
|
||||||
class ThreadedLayerRunner:
|
class ThreadedLayerRunner:
|
||||||
@@ -697,29 +739,25 @@ class ThreadedLayerRunner:
|
|||||||
def execute(
|
def execute(
|
||||||
layer: list[str],
|
layer: list[str],
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
layer_idx: int,
|
layer_idx: int,
|
||||||
on_event: EventCallback | None,
|
|
||||||
pool: concurrent.futures.ThreadPoolExecutor,
|
pool: concurrent.futures.ThreadPoolExecutor,
|
||||||
concurrency_limits: Mapping[str, int],
|
|
||||||
) -> None:
|
) -> None:
|
||||||
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event)
|
to_run, specs = _filter_and_sort(layer, graph, ctx)
|
||||||
if not to_run:
|
if not to_run:
|
||||||
return
|
return
|
||||||
semaphores = _build_semaphores(to_run, specs, threading.Semaphore, concurrency_limits)
|
semaphores = _build_semaphores(to_run, specs, threading.Semaphore, ctx.concurrency_limits)
|
||||||
context_snapshot = dict(context)
|
context_snapshot = dict(ctx.context)
|
||||||
lock = threading.Lock()
|
lock = threading.Lock()
|
||||||
|
|
||||||
def _run_threaded_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
|
def _run_threaded_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
|
||||||
spec = specs[name]
|
spec = specs[name]
|
||||||
task_ctx = _build_context(spec, context_snapshot, report)
|
task_ctx = _build_context(spec, context_snapshot, ctx.report)
|
||||||
sem = _get_sem(semaphores, spec)
|
sem = _get_sem(semaphores, spec)
|
||||||
if sem is not None:
|
if sem is not None:
|
||||||
sem.acquire()
|
sem.acquire()
|
||||||
try:
|
try:
|
||||||
return task_ctx, SyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report)
|
return task_ctx, SyncTaskRunner.run(spec, task_ctx, layer_idx, ctx)
|
||||||
finally:
|
finally:
|
||||||
if sem is not None:
|
if sem is not None:
|
||||||
sem.release()
|
sem.release()
|
||||||
@@ -735,7 +773,7 @@ class ThreadedLayerRunner:
|
|||||||
finally:
|
finally:
|
||||||
with lock:
|
with lock:
|
||||||
for name, (task_ctx, result) in completed.items():
|
for name, (task_ctx, result) in completed.items():
|
||||||
_store_result(name, result, specs[name], task_ctx, context, report, backend, on_event)
|
_store_result(result, specs[name], task_ctx, ctx)
|
||||||
|
|
||||||
|
|
||||||
class AsyncLayerRunner:
|
class AsyncLayerRunner:
|
||||||
@@ -745,29 +783,25 @@ class AsyncLayerRunner:
|
|||||||
async def execute(
|
async def execute(
|
||||||
layer: list[str],
|
layer: list[str],
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
layer_idx: int,
|
layer_idx: int,
|
||||||
on_event: EventCallback | None,
|
|
||||||
concurrency_limits: Mapping[str, int],
|
|
||||||
) -> None:
|
) -> None:
|
||||||
to_run, specs = _filter_and_sort(layer, graph, context, report, backend, on_event)
|
to_run, specs = _filter_and_sort(layer, graph, ctx)
|
||||||
if not to_run:
|
if not to_run:
|
||||||
return
|
return
|
||||||
semaphores = _build_semaphores(to_run, specs, asyncio.Semaphore, concurrency_limits)
|
semaphores = _build_semaphores(to_run, specs, asyncio.Semaphore, ctx.concurrency_limits)
|
||||||
context_snapshot = dict(context)
|
context_snapshot = dict(ctx.context)
|
||||||
|
|
||||||
async def _run_async_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
|
async def _run_async_task(name: str) -> tuple[dict[str, Any], TaskResult[Any]]:
|
||||||
spec = specs[name]
|
spec = specs[name]
|
||||||
task_ctx = _build_context(spec, context_snapshot, report)
|
task_ctx = _build_context(spec, context_snapshot, ctx.report)
|
||||||
sem = _get_sem(semaphores, spec)
|
sem = _get_sem(semaphores, spec)
|
||||||
result = await AsyncTaskRunner.run(spec, task_ctx, layer_idx, on_event, report, sem)
|
result = await AsyncTaskRunner.run(spec, task_ctx, layer_idx, ctx, sem)
|
||||||
return task_ctx, result
|
return task_ctx, result
|
||||||
|
|
||||||
results = await asyncio.gather(*[_run_async_task(name) for name in to_run])
|
results = await asyncio.gather(*[_run_async_task(name) for name in to_run])
|
||||||
for name, (task_ctx, result) in zip(to_run, results, strict=True):
|
for name, (task_ctx, result) in zip(to_run, results, strict=True):
|
||||||
_store_result(name, result, specs[name], task_ctx, context, report, backend, on_event)
|
_store_result(result, specs[name], task_ctx, ctx)
|
||||||
|
|
||||||
|
|
||||||
def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
|
def _extract_dynamic_specs(result: TaskResult[Any], spec: TaskSpec[Any]) -> list[TaskSpec[Any]]:
|
||||||
@@ -806,16 +840,11 @@ class DependencyRunner:
|
|||||||
@staticmethod
|
@staticmethod
|
||||||
async def execute(
|
async def execute(
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
concurrency_limits: Mapping[str, int],
|
|
||||||
cancel_event: threading.Event | CancelToken | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
all_names = list(graph.all_specs().keys())
|
all_names = list(graph.all_specs().keys())
|
||||||
all_specs: dict[str, TaskSpec[Any]] = {name: graph.resolved_spec(name) for name in all_names}
|
all_specs: dict[str, TaskSpec[Any]] = {name: graph.resolved_spec(name) for name in all_names}
|
||||||
semaphores = _build_semaphores(all_names, all_specs, asyncio.Semaphore, concurrency_limits)
|
semaphores = _build_semaphores(all_names, all_specs, asyncio.Semaphore, ctx.concurrency_limits)
|
||||||
|
|
||||||
# 事件驱动调度:跟踪 completed / in_flight / remaining。
|
# 事件驱动调度:跟踪 completed / in_flight / remaining。
|
||||||
# 仅当任务依赖全部完成时才创建 asyncio Task,按优先级降序调度。
|
# 仅当任务依赖全部完成时才创建 asyncio Task,按优先级降序调度。
|
||||||
@@ -825,10 +854,10 @@ class DependencyRunner:
|
|||||||
|
|
||||||
# 检查点恢复:已在 report 中的 SUCCESS 任务直接标记完成,跳过调度。
|
# 检查点恢复:已在 report 中的 SUCCESS 任务直接标记完成,跳过调度。
|
||||||
for name in list(remaining):
|
for name in list(remaining):
|
||||||
if name in report.results and report.results[name].status == TaskStatus.SUCCESS:
|
if name in ctx.report.results and ctx.report.results[name].status == TaskStatus.SUCCESS:
|
||||||
completed.add(name)
|
completed.add(name)
|
||||||
remaining.discard(name)
|
remaining.discard(name)
|
||||||
context[name] = report.results[name].value
|
ctx.context[name] = ctx.report.results[name].value
|
||||||
|
|
||||||
# 增量就绪集:用 in_degree 计数器 + dependents 反向邻接表替代每轮 O(N) 扫描。
|
# 增量就绪集:用 in_degree 计数器 + dependents 反向邻接表替代每轮 O(N) 扫描。
|
||||||
# 每轮调度开销从 O(N*D) 降至 O(D_out),大图(10k+ 任务)显著加速。
|
# 每轮调度开销从 O(N*D) 降至 O(D_out),大图(10k+ 任务)显著加速。
|
||||||
@@ -850,7 +879,7 @@ class DependencyRunner:
|
|||||||
all_specs[final_spec.name] = graph.resolved_spec(final_spec.name)
|
all_specs[final_spec.name] = graph.resolved_spec(final_spec.name)
|
||||||
remaining.add(final_spec.name)
|
remaining.add(final_spec.name)
|
||||||
if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores:
|
if final_spec.concurrency_key and final_spec.concurrency_key not in semaphores:
|
||||||
limit = concurrency_limits.get(final_spec.concurrency_key, 1)
|
limit = ctx.concurrency_limits.get(final_spec.concurrency_key, 1)
|
||||||
semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit)
|
semaphores[final_spec.concurrency_key] = asyncio.Semaphore(limit)
|
||||||
# 计算新任务的 in_degree(spawner 可能尚未标记 completed,计入 unsatisfied)
|
# 计算新任务的 in_degree(spawner 可能尚未标记 completed,计入 unsatisfied)
|
||||||
deps = (*final_spec.depends_on, *final_spec.soft_depends_on)
|
deps = (*final_spec.depends_on, *final_spec.soft_depends_on)
|
||||||
@@ -867,34 +896,34 @@ class DependencyRunner:
|
|||||||
spec = all_specs[name]
|
spec = all_specs[name]
|
||||||
|
|
||||||
# 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED
|
# 取消检查:依赖完成后、执行前检查;已取消则标记 SKIPPED
|
||||||
if _is_cancelled(cancel_event) and name not in report.results:
|
if _is_cancelled(ctx.cancel_event) and name not in ctx.report.results:
|
||||||
result: TaskResult[Any] = TaskResult(
|
result: TaskResult[Any] = TaskResult(
|
||||||
spec=spec,
|
spec=spec,
|
||||||
status=TaskStatus.SKIPPED,
|
status=TaskStatus.SKIPPED,
|
||||||
finished_at=datetime.now(),
|
finished_at=datetime.now(),
|
||||||
reason="执行被取消",
|
reason="执行被取消",
|
||||||
)
|
)
|
||||||
report.results[name] = result
|
ctx.report.results[name] = result
|
||||||
_emit(on_event, result)
|
_emit(ctx.on_event, result)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
task_ctx = _build_context(spec, context, report)
|
task_ctx = _build_context(spec, ctx.context, ctx.report)
|
||||||
if _apply_cached(name, spec, context, report, backend, on_event):
|
if _apply_cached(spec, ctx):
|
||||||
return report.results[name]
|
return ctx.report.results[spec.name]
|
||||||
|
|
||||||
sem = _get_sem(semaphores, spec)
|
sem = _get_sem(semaphores, spec)
|
||||||
result = await AsyncTaskRunner.run(spec, task_ctx, None, on_event, report, sem)
|
result = await AsyncTaskRunner.run(spec, task_ctx, None, ctx, sem)
|
||||||
|
|
||||||
# 动态任务生成:提取 specs,替换 result.value 为生成任务名列表
|
# 动态任务生成:提取 specs,替换 result.value 为生成任务名列表
|
||||||
spawned = _extract_dynamic_specs(result, spec)
|
spawned = _extract_dynamic_specs(result, spec)
|
||||||
if spawned:
|
if spawned:
|
||||||
spawned_names = [s.name for s in spawned]
|
spawned_names = [s.name for s in spawned]
|
||||||
result = dc_replace(result, value=spawned_names)
|
result = dc_replace(result, value=spawned_names)
|
||||||
context[name] = result.value
|
ctx.context[name] = result.value
|
||||||
report.results[name] = result
|
ctx.report.results[name] = result
|
||||||
_emit(on_event, result)
|
_emit(ctx.on_event, result)
|
||||||
else:
|
else:
|
||||||
_store_result(name, result, spec, task_ctx, context, report, backend, on_event)
|
_store_result(result, spec, task_ctx, ctx)
|
||||||
|
|
||||||
# 注册动态生成的 specs,接入增量调度结构
|
# 注册动态生成的 specs,接入增量调度结构
|
||||||
for raw_spec in spawned:
|
for raw_spec in spawned:
|
||||||
@@ -1018,13 +1047,8 @@ def _apply_subgraph_filter(
|
|||||||
def _dispatch_strategy(
|
def _dispatch_strategy(
|
||||||
strategy: Strategy,
|
strategy: Strategy,
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
effective_callback: EventCallback | None,
|
|
||||||
max_workers: int | None,
|
max_workers: int | None,
|
||||||
limits: Mapping[str, int],
|
|
||||||
cancel_event: threading.Event | CancelToken | None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
"""按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。
|
"""按策略派发执行,将策略分支从 :func:`run` 中提取出来控制 PLR0912。
|
||||||
|
|
||||||
@@ -1038,15 +1062,15 @@ def _dispatch_strategy(
|
|||||||
)
|
)
|
||||||
if strategy == "sequential":
|
if strategy == "sequential":
|
||||||
layers = graph.layers()
|
layers = graph.layers()
|
||||||
_drive_sequential(graph, layers, context, report, backend, effective_callback, cancel_event)
|
_drive_sequential(graph, layers, ctx)
|
||||||
elif strategy == "thread":
|
elif strategy == "thread":
|
||||||
layers = graph.layers()
|
layers = graph.layers()
|
||||||
_drive_threaded(graph, layers, context, report, backend, effective_callback, max_workers, limits, cancel_event)
|
_drive_threaded(graph, layers, ctx, max_workers)
|
||||||
elif strategy == "async":
|
elif strategy == "async":
|
||||||
layers = graph.layers()
|
layers = graph.layers()
|
||||||
asyncio.run(_async_drive(graph, layers, context, report, backend, effective_callback, limits, cancel_event))
|
asyncio.run(_async_drive(graph, layers, ctx))
|
||||||
elif strategy == "dependency":
|
elif strategy == "dependency":
|
||||||
asyncio.run(DependencyRunner.execute(graph, context, report, backend, effective_callback, limits, cancel_event))
|
asyncio.run(DependencyRunner.execute(graph, ctx))
|
||||||
else: # pragma: no cover - Strategy Literal 已穷尽所有取值
|
else: # pragma: no cover - Strategy Literal 已穷尽所有取值
|
||||||
raise ValueError(f"Unknown strategy: {strategy!r}")
|
raise ValueError(f"Unknown strategy: {strategy!r}")
|
||||||
|
|
||||||
@@ -1105,7 +1129,7 @@ def _apply_resume(
|
|||||||
logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored})
|
logger.info("从检查点恢复 %d 个任务", restored, extra={"restored": restored})
|
||||||
|
|
||||||
|
|
||||||
def run(
|
def run( # noqa: PLR0913
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
strategy: Strategy = "dependency",
|
strategy: Strategy = "dependency",
|
||||||
*,
|
*,
|
||||||
@@ -1221,15 +1245,24 @@ def run(
|
|||||||
internal_cancel = CancelToken()
|
internal_cancel = CancelToken()
|
||||||
effective_cancel: CancelToken | threading.Event | None = cancel_event
|
effective_cancel: CancelToken | threading.Event | None = cancel_event
|
||||||
|
|
||||||
|
# 打包执行上下文:将 context/report/backend/on_event/concurrency_limits/cancel_event
|
||||||
|
# 捆绑为单一参数传递给调用链,使每个内部函数参数数 ≤5(PLR0913)。
|
||||||
|
ctx = _ExecContext(
|
||||||
|
context=context,
|
||||||
|
report=report,
|
||||||
|
backend=backend,
|
||||||
|
on_event=effective_callback,
|
||||||
|
concurrency_limits=limits,
|
||||||
|
cancel_event=effective_cancel,
|
||||||
|
)
|
||||||
|
|
||||||
# backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend);
|
# backend.batch():将每任务一次落盘降为整次运行一次(JSONBackend);
|
||||||
# MemoryBackend 为 no-op。即使中途抛出 TaskFailedError,batch 退出时
|
# MemoryBackend 为 no-op。即使中途抛出 TaskFailedError,batch 退出时
|
||||||
# 仍会 flush 一次,保留已成功任务的结果以便断点续跑。
|
# 仍会 flush 一次,保留已成功任务的结果以便断点续跑。
|
||||||
try:
|
try:
|
||||||
with backend.batch():
|
with backend.batch():
|
||||||
try:
|
try:
|
||||||
_dispatch_strategy(
|
_dispatch_strategy(strategy, graph, ctx, max_workers)
|
||||||
strategy, graph, context, report, backend, effective_callback, max_workers, limits, effective_cancel
|
|
||||||
)
|
|
||||||
except TaskFailedError:
|
except TaskFailedError:
|
||||||
report.success = False
|
report.success = False
|
||||||
raise
|
raise
|
||||||
@@ -1267,7 +1300,7 @@ def run(
|
|||||||
return report
|
return report
|
||||||
|
|
||||||
|
|
||||||
def run_iter(
|
def run_iter( # noqa: PLR0913
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
strategy: Strategy = "dependency",
|
strategy: Strategy = "dependency",
|
||||||
*,
|
*,
|
||||||
@@ -1369,53 +1402,39 @@ def _print_dry_run(graph: Graph, layers: list[list[str]]) -> None:
|
|||||||
def _drive_sequential(
|
def _drive_sequential(
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
layers: list[list[str]],
|
layers: list[list[str]],
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
cancel_event: threading.Event | CancelToken | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
for idx, layer in enumerate(layers, 1):
|
for idx, layer in enumerate(layers, 1):
|
||||||
if _is_cancelled(cancel_event):
|
if _is_cancelled(ctx.cancel_event):
|
||||||
return
|
return
|
||||||
SequentialLayerRunner.execute(layer, graph, context, report, backend, idx, on_event)
|
SequentialLayerRunner.execute(layer, graph, ctx, idx)
|
||||||
|
|
||||||
|
|
||||||
def _drive_threaded(
|
def _drive_threaded(
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
layers: list[list[str]],
|
layers: list[list[str]],
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
max_workers: int | None,
|
max_workers: int | None,
|
||||||
concurrency_limits: Mapping[str, int],
|
|
||||||
cancel_event: threading.Event | CancelToken | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
|
# 线程池在整个 run() 内复用,避免逐层创建/销毁线程的开销。
|
||||||
max_layer_size = max((len(layer) for layer in layers), default=1)
|
max_layer_size = max((len(layer) for layer in layers), default=1)
|
||||||
pool_workers = max_workers or max(1, min(32, max_layer_size))
|
pool_workers = max_workers or max(1, min(32, max_layer_size))
|
||||||
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
|
with concurrent.futures.ThreadPoolExecutor(max_workers=pool_workers) as pool:
|
||||||
for idx, layer in enumerate(layers, 1):
|
for idx, layer in enumerate(layers, 1):
|
||||||
if _is_cancelled(cancel_event):
|
if _is_cancelled(ctx.cancel_event):
|
||||||
return
|
return
|
||||||
ThreadedLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, pool, concurrency_limits)
|
ThreadedLayerRunner.execute(layer, graph, ctx, idx, pool)
|
||||||
|
|
||||||
|
|
||||||
async def _async_drive(
|
async def _async_drive(
|
||||||
graph: Graph,
|
graph: Graph,
|
||||||
layers: list[list[str]],
|
layers: list[list[str]],
|
||||||
context: dict[str, Any],
|
ctx: _ExecContext,
|
||||||
report: RunReport,
|
|
||||||
backend: StateBackend,
|
|
||||||
on_event: EventCallback | None,
|
|
||||||
concurrency_limits: Mapping[str, int],
|
|
||||||
cancel_event: threading.Event | CancelToken | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
for idx, layer in enumerate(layers, 1):
|
for idx, layer in enumerate(layers, 1):
|
||||||
if _is_cancelled(cancel_event):
|
if _is_cancelled(ctx.cancel_event):
|
||||||
return
|
return
|
||||||
await AsyncLayerRunner.execute(layer, graph, context, report, backend, idx, on_event, concurrency_limits)
|
await AsyncLayerRunner.execute(layer, graph, ctx, idx)
|
||||||
|
|
||||||
|
|
||||||
def _mark_remaining_skipped(
|
def _mark_remaining_skipped(
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ __all__ = [
|
|||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
# 查找与遍历
|
# 查找与遍历
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
def find(
|
def find( # noqa: PLR0913
|
||||||
root: str | Path,
|
root: str | Path,
|
||||||
pattern: str = "*",
|
pattern: str = "*",
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -6,7 +6,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import ast
|
import ast
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -69,7 +68,7 @@ def format_with_ruff(target: Path, fix: bool = True) -> None:
|
|||||||
if fix:
|
if fix:
|
||||||
cmd.append("--fix")
|
cmd.append("--fix")
|
||||||
|
|
||||||
subprocess.run(cmd, check=True)
|
px.sh(cmd, label="ruff format")
|
||||||
print(f"ruff format 完成: {target}")
|
print(f"ruff format 完成: {target}")
|
||||||
|
|
||||||
|
|
||||||
@@ -87,7 +86,7 @@ def lint_with_ruff(target: Path, fix: bool = True) -> None:
|
|||||||
if fix:
|
if fix:
|
||||||
cmd.extend(["--fix", "--unsafe-fixes"])
|
cmd.extend(["--fix", "--unsafe-fixes"])
|
||||||
|
|
||||||
subprocess.run(cmd, check=True)
|
px.sh(cmd, label="ruff check")
|
||||||
print(f"ruff check 完成: {target}")
|
print(f"ruff check 完成: {target}")
|
||||||
|
|
||||||
|
|
||||||
@@ -210,7 +209,7 @@ def sync_pyproject_config(root_dir: Path = Path()) -> None:
|
|||||||
print(f"找到 {len(sub_tomls)} 个子项目配置文件")
|
print(f"找到 {len(sub_tomls)} 个子项目配置文件")
|
||||||
|
|
||||||
for sub_toml in sub_tomls:
|
for sub_toml in sub_tomls:
|
||||||
subprocess.run(["ruff", "format", str(sub_toml)], check=False)
|
px.sh(["ruff", "format", str(sub_toml)], check=False, label="ruff format")
|
||||||
|
|
||||||
print("配置同步完成")
|
print("配置同步完成")
|
||||||
|
|
||||||
@@ -223,6 +222,6 @@ def format_all(root_dir: Path) -> None:
|
|||||||
root_dir : Path
|
root_dir : Path
|
||||||
根目录
|
根目录
|
||||||
"""
|
"""
|
||||||
subprocess.run(["ruff", "format", str(root_dir)], check=True)
|
px.sh(["ruff", "format", str(root_dir)], label="ruff format")
|
||||||
subprocess.run(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], check=True)
|
px.sh(["ruff", "check", "--fix", "--unsafe-fixes", str(root_dir)], label="ruff check")
|
||||||
print(f"格式化完成: {root_dir}")
|
print(f"格式化完成: {root_dir}")
|
||||||
|
|||||||
@@ -15,7 +15,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
import re
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
@@ -221,12 +220,12 @@ def bump_project_version(part: BumpVersionType = "patch", no_tag: bool = False)
|
|||||||
|
|
||||||
# 阶段 3: git add (按文件名) + commit + tag
|
# 阶段 3: git add (按文件名) + commit + tag
|
||||||
relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)]
|
relative_files = [str(file.relative_to(cwd)) for file in sorted(all_files)]
|
||||||
subprocess.run(["git", "add", *relative_files], check=True)
|
px.sh(["git", "add", *relative_files], label="git add")
|
||||||
subprocess.run(["git", "commit", "-m", f"bump version to {new_version}"], check=True)
|
px.sh(["git", "commit", "-m", f"bump version to {new_version}"], label="git commit")
|
||||||
|
|
||||||
if not no_tag:
|
if not no_tag:
|
||||||
tag_name = f"v{new_version}"
|
tag_name = f"v{new_version}"
|
||||||
subprocess.run(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], check=True)
|
px.sh(["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"], label="git tag")
|
||||||
print(f"已创建标签: {tag_name}")
|
print(f"已创建标签: {tag_name}")
|
||||||
|
|
||||||
return new_version
|
return new_version
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -40,16 +39,10 @@ def not_has_git_repo() -> bool:
|
|||||||
|
|
||||||
def has_files() -> bool:
|
def has_files() -> bool:
|
||||||
"""检查当前 Git 仓库是否有未提交的更改."""
|
"""检查当前 Git 仓库是否有未提交的更改."""
|
||||||
try:
|
result = px.sh(["git", "status", "--porcelain"], capture=True, check=False, label="git status")
|
||||||
result = subprocess.run(
|
if result is None:
|
||||||
["git", "status", "--porcelain"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
return bool(result.stdout.strip())
|
|
||||||
except (subprocess.SubprocessError, OSError):
|
|
||||||
return False
|
return False
|
||||||
|
return bool(result.stdout.strip())
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
@@ -69,8 +62,8 @@ def git_add_commit(message: str = "chore: update") -> None:
|
|||||||
if not has_files():
|
if not has_files():
|
||||||
print("没有文件需要提交")
|
print("没有文件需要提交")
|
||||||
return
|
return
|
||||||
subprocess.run(["git", "add", "."], check=True)
|
px.sh(["git", "add", "."], label="git add")
|
||||||
subprocess.run(["git", "commit", "-m", message], check=True)
|
px.sh(["git", "commit", "-m", message], label="git commit")
|
||||||
|
|
||||||
|
|
||||||
@px.tool("gittool", subcommand="i", help="初始化并提交")
|
@px.tool("gittool", subcommand="i", help="初始化并提交")
|
||||||
@@ -83,10 +76,10 @@ def git_init_add_commit(message: str = "init commit") -> None:
|
|||||||
提交信息 (默认: ``init commit``)
|
提交信息 (默认: ``init commit``)
|
||||||
"""
|
"""
|
||||||
if not_has_git_repo():
|
if not_has_git_repo():
|
||||||
subprocess.run(["git", "init"], check=True)
|
px.sh(["git", "init"], label="git init")
|
||||||
if has_files():
|
if has_files():
|
||||||
subprocess.run(["git", "add", "."], check=True)
|
px.sh(["git", "add", "."], label="git add")
|
||||||
subprocess.run(["git", "commit", "-m", message], check=True)
|
px.sh(["git", "commit", "-m", message], label="git commit")
|
||||||
else:
|
else:
|
||||||
print("没有文件需要提交")
|
print("没有文件需要提交")
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -49,13 +48,8 @@ def run_ls_dyna(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
|
|||||||
return
|
return
|
||||||
|
|
||||||
cmd = get_ls_dyna_command(input_file, ncpu)
|
cmd = get_ls_dyna_command(input_file, ncpu)
|
||||||
try:
|
if px.sh(cmd, label="LS-DYNA 计算") is not None:
|
||||||
subprocess.run(cmd, check=True)
|
|
||||||
print(f"LS-DYNA 计算完成: {input_file}")
|
print(f"LS-DYNA 计算完成: {input_file}")
|
||||||
except FileNotFoundError:
|
|
||||||
print("未找到 ls-dyna_mpp 命令")
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"LS-DYNA 计算失败: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
@px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算")
|
@px.tool("lscalc", subcommand="mpi", help="运行 LS-DYNA MPI 计算")
|
||||||
@@ -75,37 +69,24 @@ def run_ls_dyna_mpi(input_file: str, ncpu: int = _DEFAULT_NCPU) -> None:
|
|||||||
return
|
return
|
||||||
|
|
||||||
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
|
cmd = ["mpirun", "-np", str(ncpu), "ls-dyna_mpp", f"i={input_file}"]
|
||||||
try:
|
if px.sh(cmd, label="LS-DYNA MPI 计算") is not None:
|
||||||
subprocess.run(cmd, check=True)
|
|
||||||
print(f"LS-DYNA MPI 计算完成: {input_file}")
|
print(f"LS-DYNA MPI 计算完成: {input_file}")
|
||||||
except FileNotFoundError:
|
|
||||||
print("未找到 mpirun 或 ls-dyna_mpp 命令")
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"LS-DYNA MPI 计算失败: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
@px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态")
|
@px.tool("lscalc", subcommand="status", help="检查 LS-DYNA 进程状态")
|
||||||
def check_ls_dyna_status() -> None:
|
def check_ls_dyna_status() -> None:
|
||||||
"""检查 LS-DYNA 进程状态."""
|
"""检查 LS-DYNA 进程状态."""
|
||||||
try:
|
if Constants.IS_WINDOWS:
|
||||||
if Constants.IS_WINDOWS:
|
result = px.sh(
|
||||||
result = subprocess.run(
|
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
|
||||||
["tasklist", "/fi", "imagename eq ls-dyna_mpp.exe"],
|
capture=True,
|
||||||
capture_output=True,
|
label="tasklist",
|
||||||
text=True,
|
)
|
||||||
check=True,
|
if result is not None:
|
||||||
)
|
|
||||||
print(result.stdout)
|
print(result.stdout)
|
||||||
|
else:
|
||||||
|
result = px.sh(["pgrep", "-f", "ls-dyna"], capture=True, check=False, label="pgrep")
|
||||||
|
if result is not None and result.stdout.strip():
|
||||||
|
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
|
||||||
else:
|
else:
|
||||||
result = subprocess.run(
|
print("没有运行中的 LS-DYNA 进程")
|
||||||
["pgrep", "-f", "ls-dyna"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
if result.stdout.strip():
|
|
||||||
print(f"运行中的 LS-DYNA 进程 PID: {result.stdout.strip()}")
|
|
||||||
else:
|
|
||||||
print("没有运行中的 LS-DYNA 进程")
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"检查进程状态失败: {e}")
|
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import platform
|
import platform
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
|
||||||
import urllib.request
|
import urllib.request
|
||||||
import zipfile
|
import zipfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -124,7 +123,7 @@ def pack_dependencies(lib_dir: Path = Path("libs"), dependencies: list[str] | No
|
|||||||
]
|
]
|
||||||
cmd.extend(dependencies)
|
cmd.extend(dependencies)
|
||||||
|
|
||||||
subprocess.run(cmd, check=True)
|
px.sh(cmd, label="pip install")
|
||||||
print(f"依赖打包完成: {lib_dir}")
|
print(f"依赖打包完成: {lib_dir}")
|
||||||
|
|
||||||
|
|
||||||
@@ -150,7 +149,7 @@ def pack_wheel(project_dir: Path = Path(), output_dir: Path = Path("dist")) -> N
|
|||||||
str(project_dir),
|
str(project_dir),
|
||||||
]
|
]
|
||||||
|
|
||||||
subprocess.run(cmd, check=True)
|
px.sh(cmd, label="pip wheel")
|
||||||
print(f"Wheel 打包完成: {output_dir}")
|
print(f"Wheel 打包完成: {output_dir}")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import fnmatch
|
import fnmatch
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -27,12 +26,10 @@ __all__ = [
|
|||||||
PACKAGE_DIR = "packages"
|
PACKAGE_DIR = "packages"
|
||||||
REQUIREMENTS_FILE = "requirements.txt"
|
REQUIREMENTS_FILE = "requirements.txt"
|
||||||
|
|
||||||
_PROTECTED_PACKAGES: frozenset[str] = frozenset(
|
_PROTECTED_PACKAGES: frozenset[str] = frozenset({
|
||||||
{
|
"pyflowx",
|
||||||
"pyflowx",
|
"bitool",
|
||||||
"bitool",
|
})
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
@@ -42,20 +39,14 @@ _PROTECTED_PACKAGES: frozenset[str] = frozenset(
|
|||||||
|
|
||||||
def _get_installed_packages() -> list[str]:
|
def _get_installed_packages() -> list[str]:
|
||||||
"""获取当前环境中所有已安装的包名."""
|
"""获取当前环境中所有已安装的包名."""
|
||||||
try:
|
result = px.sh(["pip", "list", "--format=freeze"], capture=True, label="pip list")
|
||||||
result = subprocess.run(
|
if result is None:
|
||||||
["pip", "list", "--format=freeze"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
packages: list[str] = []
|
|
||||||
for line in result.stdout.strip().split("\n"):
|
|
||||||
if line and "==" in line:
|
|
||||||
pkg_name = line.split("==")[0].strip()
|
|
||||||
packages.append(pkg_name)
|
|
||||||
except (subprocess.SubprocessError, OSError):
|
|
||||||
return []
|
return []
|
||||||
|
packages: list[str] = []
|
||||||
|
for line in result.stdout.strip().split("\n"):
|
||||||
|
if line and "==" in line:
|
||||||
|
pkg_name = line.split("==")[0].strip()
|
||||||
|
packages.append(pkg_name)
|
||||||
return packages
|
return packages
|
||||||
|
|
||||||
|
|
||||||
@@ -92,7 +83,7 @@ def pip_install(packages: list[str]) -> None:
|
|||||||
packages : list[str]
|
packages : list[str]
|
||||||
包名列表
|
包名列表
|
||||||
"""
|
"""
|
||||||
subprocess.run(["pip", "install", *packages], check=True)
|
px.sh(["pip", "install", *packages], label="pip install")
|
||||||
print(f"安装完成: {', '.join(packages)}")
|
print(f"安装完成: {', '.join(packages)}")
|
||||||
|
|
||||||
|
|
||||||
@@ -114,7 +105,7 @@ def pip_uninstall(packages: list[str]) -> None:
|
|||||||
if not packages_to_uninstall:
|
if not packages_to_uninstall:
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["pip", "uninstall", "-y", *packages_to_uninstall], check=True)
|
px.sh(["pip", "uninstall", "-y", *packages_to_uninstall], label="pip uninstall")
|
||||||
|
|
||||||
|
|
||||||
@px.tool("piptool", subcommand="r", help="重装包")
|
@px.tool("piptool", subcommand="r", help="重装包")
|
||||||
@@ -133,10 +124,10 @@ def pip_reinstall(packages: list[str], offline: bool = False) -> None:
|
|||||||
print("所有指定的包均为受保护包, 跳过重装")
|
print("所有指定的包均为受保护包, 跳过重装")
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["pip", "uninstall", "-y", *safe_ps], check=True)
|
px.sh(["pip", "uninstall", "-y", *safe_ps], label="pip uninstall")
|
||||||
|
|
||||||
options = ["--no-index", "--find-links", "."] if offline else []
|
options = ["--no-index", "--find-links", "."] if offline else []
|
||||||
subprocess.run(["pip", "install", *options, *safe_ps], check=True)
|
px.sh(["pip", "install", *options, *safe_ps], label="pip install")
|
||||||
|
|
||||||
|
|
||||||
@px.tool("piptool", subcommand="d", help="下载包")
|
@px.tool("piptool", subcommand="d", help="下载包")
|
||||||
@@ -151,27 +142,20 @@ def pip_download(packages: list[str], offline: bool = False) -> None:
|
|||||||
离线模式
|
离线模式
|
||||||
"""
|
"""
|
||||||
options = ["--no-index", "--find-links", "."] if offline else []
|
options = ["--no-index", "--find-links", "."] if offline else []
|
||||||
subprocess.run(
|
px.sh(["pip", "download", *packages, *options, "-d", PACKAGE_DIR], label="pip download")
|
||||||
["pip", "download", *packages, *options, "-d", PACKAGE_DIR],
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@px.tool("piptool", subcommand="up", help="升级 pip")
|
@px.tool("piptool", subcommand="up", help="升级 pip")
|
||||||
def pip_upgrade() -> None:
|
def pip_upgrade() -> None:
|
||||||
"""升级 pip 到最新版本."""
|
"""升级 pip 到最新版本."""
|
||||||
subprocess.run(["python", "-m", "pip", "install", "--upgrade", "pip"], check=True)
|
px.sh(["python", "-m", "pip", "install", "--upgrade", "pip"], label="pip 升级")
|
||||||
print("pip 升级完成")
|
print("pip 升级完成")
|
||||||
|
|
||||||
|
|
||||||
@px.tool("piptool", subcommand="f", help="导出依赖")
|
@px.tool("piptool", subcommand="f", help="导出依赖")
|
||||||
def pip_freeze() -> None:
|
def pip_freeze() -> None:
|
||||||
"""冻结依赖到 requirements.txt."""
|
"""冻结依赖到 requirements.txt."""
|
||||||
result = subprocess.run(
|
result = px.sh(["pip", "freeze", "--exclude-editable"], capture=True, label="pip freeze")
|
||||||
["pip", "freeze", "--exclude-editable"],
|
if result is not None:
|
||||||
capture_output=True,
|
Path(REQUIREMENTS_FILE).write_text(result.stdout)
|
||||||
text=True,
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
Path(REQUIREMENTS_FILE).write_text(result.stdout)
|
|
||||||
print(f"依赖已导出到 {REQUIREMENTS_FILE}")
|
print(f"依赖已导出到 {REQUIREMENTS_FILE}")
|
||||||
|
|||||||
@@ -99,7 +99,7 @@ def image_resize(
|
|||||||
|
|
||||||
|
|
||||||
@px.tool("imagetool", subcommand="c", help="裁剪图片")
|
@px.tool("imagetool", subcommand="c", help="裁剪图片")
|
||||||
def image_crop(
|
def image_crop( # noqa: PLR0913
|
||||||
input_path: Path,
|
input_path: Path,
|
||||||
output_path: Path,
|
output_path: Path,
|
||||||
left: int,
|
left: int,
|
||||||
@@ -213,7 +213,7 @@ def image_convert(
|
|||||||
|
|
||||||
|
|
||||||
@px.tool("imagetool", subcommand="wm", help="添加文字水印")
|
@px.tool("imagetool", subcommand="wm", help="添加文字水印")
|
||||||
def image_watermark(
|
def image_watermark( # noqa: PLR0913
|
||||||
input_path: Path,
|
input_path: Path,
|
||||||
output_path: Path,
|
output_path: Path,
|
||||||
text: str,
|
text: str,
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -59,14 +58,11 @@ $bitmap.Save('{output_path.as_posix()}')
|
|||||||
$graphics.Dispose()
|
$graphics.Dispose()
|
||||||
$bitmap.Dispose()
|
$bitmap.Dispose()
|
||||||
"""
|
"""
|
||||||
subprocess.run(["powershell", "-Command", ps_script], check=True)
|
px.sh(["powershell", "-Command", ps_script], label="截图")
|
||||||
elif Constants.IS_MACOS:
|
elif Constants.IS_MACOS:
|
||||||
subprocess.run(["screencapture", "-x", str(output_path)], check=True)
|
px.sh(["screencapture", "-x", str(output_path)], label="截图")
|
||||||
else:
|
elif px.sh(["gnome-screenshot", "-f", str(output_path)], label="截图") is None:
|
||||||
try:
|
px.sh(["scrot", str(output_path)], label="截图")
|
||||||
subprocess.run(["gnome-screenshot", "-f", str(output_path)], check=True)
|
|
||||||
except FileNotFoundError:
|
|
||||||
subprocess.run(["scrot", str(output_path)], check=True)
|
|
||||||
|
|
||||||
print(f"截图已保存: {output_path}")
|
print(f"截图已保存: {output_path}")
|
||||||
|
|
||||||
@@ -104,13 +100,10 @@ $bitmap.Save('{output_path.as_posix()}')
|
|||||||
$graphics.Dispose()
|
$graphics.Dispose()
|
||||||
$bitmap.Dispose()
|
$bitmap.Dispose()
|
||||||
"""
|
"""
|
||||||
subprocess.run(["powershell", "-Command", ps_script], check=True)
|
px.sh(["powershell", "-Command", ps_script], label="截图")
|
||||||
elif Constants.IS_MACOS:
|
elif Constants.IS_MACOS:
|
||||||
subprocess.run(["screencapture", "-i", str(output_path)], check=True)
|
px.sh(["screencapture", "-i", str(output_path)], label="截图")
|
||||||
else:
|
elif px.sh(["gnome-screenshot", "-a", "-f", str(output_path)], label="截图") is None:
|
||||||
try:
|
px.sh(["scrot", "-s", str(output_path)], label="截图")
|
||||||
subprocess.run(["gnome-screenshot", "-a", "-f", str(output_path)], check=True)
|
|
||||||
except FileNotFoundError:
|
|
||||||
subprocess.run(["scrot", "-s", str(output_path)], check=True)
|
|
||||||
|
|
||||||
print(f"截图已保存: {output_path}")
|
print(f"截图已保存: {output_path}")
|
||||||
|
|||||||
@@ -6,7 +6,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import getpass
|
import getpass
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
|
|
||||||
@@ -23,15 +22,12 @@ def docker_login_tencent(username: str = "") -> None:
|
|||||||
Docker 用户名 (为空时由 docker 交互式提示输入)
|
Docker 用户名 (为空时由 docker 交互式提示输入)
|
||||||
"""
|
"""
|
||||||
user = username or getpass.getuser()
|
user = username or getpass.getuser()
|
||||||
try:
|
if (
|
||||||
subprocess.run(
|
px.sh(
|
||||||
["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT],
|
["docker", "login", "--username", user, _DOCKER_MIRROR_TENCENT],
|
||||||
check=True,
|
label="docker login",
|
||||||
)
|
)
|
||||||
except subprocess.CalledProcessError as e:
|
is None
|
||||||
print(f"docker login 失败 (returncode={e.returncode}): {e}")
|
):
|
||||||
return
|
|
||||||
except FileNotFoundError:
|
|
||||||
print("docker 命令未找到,请确认 Docker 已安装并在 PATH 中")
|
|
||||||
return
|
return
|
||||||
print(f"已登录腾讯云镜像仓库 (用户: {user})")
|
print(f"已登录腾讯云镜像仓库 (用户: {user})")
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ from __future__ import annotations
|
|||||||
import getpass
|
import getpass
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
@@ -289,7 +288,7 @@ def download_rustup_script() -> None:
|
|||||||
|
|
||||||
if Constants.IS_WINDOWS:
|
if Constants.IS_WINDOWS:
|
||||||
print("下载 rustup-init.exe...")
|
print("下载 rustup-init.exe...")
|
||||||
subprocess.run(
|
px.sh(
|
||||||
[
|
[
|
||||||
"powershell",
|
"powershell",
|
||||||
"-Command",
|
"-Command",
|
||||||
@@ -300,12 +299,14 @@ def download_rustup_script() -> None:
|
|||||||
"rustup-init.exe",
|
"rustup-init.exe",
|
||||||
],
|
],
|
||||||
check=False,
|
check=False,
|
||||||
|
label="下载 rustup",
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
print("下载 rustup-init.sh...")
|
print("下载 rustup-init.sh...")
|
||||||
subprocess.run(
|
px.sh(
|
||||||
["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
|
["curl", "-fsSL", _RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
|
||||||
check=False,
|
check=False,
|
||||||
|
label="下载 rustup",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -328,7 +329,7 @@ def install_rust_toolchain(version: str = "stable") -> None:
|
|||||||
print("rustup 未安装, 跳过工具链安装")
|
print("rustup 未安装, 跳过工具链安装")
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["rustup", "toolchain", "install", version], check=False)
|
px.sh(["rustup", "toolchain", "install", version], check=False, label="安装 Rust 工具链")
|
||||||
print(f"Rust 工具链 {version} 安装完成")
|
print(f"Rust 工具链 {version} 安装完成")
|
||||||
|
|
||||||
|
|
||||||
@@ -364,9 +365,9 @@ def setup_linux_system_mirror() -> None:
|
|||||||
return
|
return
|
||||||
|
|
||||||
print("下载 linuxmirrors 脚本...")
|
print("下载 linuxmirrors 脚本...")
|
||||||
subprocess.run(_DOWNLOAD_MIRROR_SCRIPT, shell=True, check=False)
|
px.sh(_DOWNLOAD_MIRROR_SCRIPT, check=False, label="下载镜像脚本")
|
||||||
print("安装 linuxmirrors...")
|
print("安装 linuxmirrors...")
|
||||||
subprocess.run(_INSTALL_MIRROR_SCRIPT, shell=True, check=False)
|
px.sh(_INSTALL_MIRROR_SCRIPT, check=False, label="安装镜像")
|
||||||
|
|
||||||
|
|
||||||
@px.tool(
|
@px.tool(
|
||||||
@@ -381,7 +382,7 @@ def install_linux_qt_libs() -> None:
|
|||||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"):
|
if not ensure_platform(Constants.IS_LINUX, "install_linux_qt_libs", "Linux"):
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False)
|
px.sh(["sudo", "apt", "install", "-y", *_QT_LIBS], check=False, label="安装 Qt 依赖")
|
||||||
print("Qt 依赖库安装完成")
|
print("Qt 依赖库安装完成")
|
||||||
|
|
||||||
|
|
||||||
@@ -397,7 +398,7 @@ def install_linux_fonts() -> None:
|
|||||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"):
|
if not ensure_platform(Constants.IS_LINUX, "install_linux_fonts", "Linux"):
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False)
|
px.sh(["sudo", "apt", "install", "-y", *_CHINESE_FONTS], check=False, label="安装中文字体")
|
||||||
print("中文字体安装完成")
|
print("中文字体安装完成")
|
||||||
|
|
||||||
|
|
||||||
@@ -413,6 +414,6 @@ def install_linux_docker() -> None:
|
|||||||
if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"):
|
if not ensure_platform(Constants.IS_LINUX, "install_linux_docker", "Linux"):
|
||||||
return
|
return
|
||||||
|
|
||||||
subprocess.run(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False)
|
px.sh(["sudo", "apt", "install", "-y", "docker-compose-v2"], check=False, label="安装 Docker")
|
||||||
subprocess.run(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False)
|
px.sh(["sudo", "usermod", "-aG", "docker", getpass.getuser()], check=False, label="加入 docker 组")
|
||||||
print("Docker 安装完成 (需重新登录以生效 docker 用户组)")
|
print("Docker 安装完成 (需重新登录以生效 docker 用户组)")
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -38,9 +37,4 @@ def msdownload_run(name: str, target_type: str = "model", download_dir: str | No
|
|||||||
|
|
||||||
cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)]
|
cmd = ["uvx", "modelscope", "download", f"--{target_type}", name, "--local_dir", str(out_dir)]
|
||||||
print(f"下载 {target_type}: {name} -> {out_dir}")
|
print(f"下载 {target_type}: {name} -> {out_dir}")
|
||||||
try:
|
px.sh(cmd, label="下载")
|
||||||
subprocess.run(cmd, check=True)
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"下载失败 (returncode={e.returncode}): {e}")
|
|
||||||
except FileNotFoundError:
|
|
||||||
print("uvx 命令未找到,请确认 uv 已安装并在 PATH 中")
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ def install_sglang() -> None:
|
|||||||
|
|
||||||
|
|
||||||
@px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"])
|
@px.tool("sglang", subcommand="run", help="启动 SGLang 服务", needs=["install"])
|
||||||
def run_sglang(
|
def run_sglang( # noqa: PLR0913
|
||||||
model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ",
|
model: str = "~/.models/Qwen2.5-Coder-32B-Instruct-AWQ",
|
||||||
port: int = 8000,
|
port: int = 8000,
|
||||||
ctx_len: int = 32768,
|
ctx_len: int = 32768,
|
||||||
|
|||||||
@@ -5,7 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -15,7 +14,7 @@ __all__ = ["ssh_copy_id"]
|
|||||||
|
|
||||||
|
|
||||||
@px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器")
|
@px.tool("sshcopyid", help="部署 SSH 公钥到远程服务器")
|
||||||
def ssh_copy_id(
|
def ssh_copy_id( # noqa: PLR0913
|
||||||
hostname: str,
|
hostname: str,
|
||||||
username: str,
|
username: str,
|
||||||
password: str,
|
password: str,
|
||||||
@@ -51,34 +50,27 @@ def ssh_copy_id(
|
|||||||
cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys
|
cd ~/.ssh && touch authorized_keys && chmod 600 authorized_keys
|
||||||
grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys"""
|
grep -qF '{pub_key.split()[1]}' authorized_keys 2>/dev/null || echo '{pub_key}' >> authorized_keys"""
|
||||||
|
|
||||||
try:
|
result = px.sh(
|
||||||
subprocess.run(
|
[
|
||||||
[
|
"sshpass",
|
||||||
"sshpass",
|
"-p",
|
||||||
"-p",
|
password,
|
||||||
password,
|
"ssh",
|
||||||
"ssh",
|
"-p",
|
||||||
"-p",
|
str(port),
|
||||||
str(port),
|
"-o",
|
||||||
"-o",
|
"StrictHostKeyChecking=no",
|
||||||
"StrictHostKeyChecking=no",
|
"-o",
|
||||||
"-o",
|
"UserKnownHostsFile=/dev/null",
|
||||||
"UserKnownHostsFile=/dev/null",
|
"-o",
|
||||||
"-o",
|
f"ConnectTimeout={timeout}",
|
||||||
f"ConnectTimeout={timeout}",
|
f"{username}@{hostname}",
|
||||||
f"{username}@{hostname}",
|
script,
|
||||||
script,
|
],
|
||||||
],
|
timeout=timeout,
|
||||||
check=True,
|
label="SSH 密钥部署",
|
||||||
timeout=timeout,
|
)
|
||||||
)
|
if result is None:
|
||||||
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}")
|
print(f"可手动执行: ssh-copy-id -p {port} {username}@{hostname}")
|
||||||
except FileNotFoundError:
|
|
||||||
print(f"未找到 sshpass 工具, 请手动执行: ssh-copy-id -p {port} {username}@{hostname}")
|
|
||||||
sys.exit(1)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
print("SSH 连接超时")
|
|
||||||
sys.exit(1)
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"SSH 执行失败: {e}")
|
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
print(f"SSH 密钥已部署到 {username}@{hostname}:{port}")
|
||||||
|
|||||||
@@ -5,8 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
from pyflowx.ops._common import platform_command
|
from pyflowx.ops._common import platform_command
|
||||||
|
|
||||||
@@ -18,4 +16,4 @@ def clear_screen_run() -> None:
|
|||||||
Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
|
Windows 调用 ``cls``, Linux/macOS 调用 ``clear``.
|
||||||
"""
|
"""
|
||||||
# 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍
|
# 清屏失败(如非 TTY 环境)不影响后续工作,故 check=False 容忍
|
||||||
subprocess.run(platform_command(["cls"], ["clear"]), check=False)
|
px.sh(platform_command(["cls"], ["clear"]), check=False, label="清屏")
|
||||||
|
|||||||
@@ -7,7 +7,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
@@ -34,19 +33,20 @@ def reset_icon_cache_run() -> None:
|
|||||||
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
|
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
|
||||||
|
|
||||||
print("正在终止 explorer 进程...")
|
print("正在终止 explorer 进程...")
|
||||||
subprocess.run(["taskkill", "/f", "/im", "explorer.exe"], check=False)
|
px.sh(["taskkill", "/f", "/im", "explorer.exe"], check=False, label="终止 explorer")
|
||||||
|
|
||||||
if icon_cache_db.exists():
|
if icon_cache_db.exists():
|
||||||
print(f"删除图标缓存: {icon_cache_db}")
|
print(f"删除图标缓存: {icon_cache_db}")
|
||||||
subprocess.run(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False)
|
px.sh(["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)], check=False, label="删除图标缓存")
|
||||||
|
|
||||||
if explorer_cache_dir.exists():
|
if explorer_cache_dir.exists():
|
||||||
print(f"清理 Explorer 缓存: {explorer_cache_dir}")
|
print(f"清理 Explorer 缓存: {explorer_cache_dir}")
|
||||||
subprocess.run(
|
px.sh(
|
||||||
["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
|
["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
|
||||||
check=False,
|
check=False,
|
||||||
|
label="清理 Explorer 缓存",
|
||||||
)
|
)
|
||||||
|
|
||||||
print("重启 explorer...")
|
print("重启 explorer...")
|
||||||
subprocess.run(["cmd", "/c", "start", "explorer.exe"], check=False)
|
px.sh(["cmd", "/c", "start", "explorer.exe"], check=False, label="重启 explorer")
|
||||||
print("图标缓存已重置")
|
print("图标缓存已重置")
|
||||||
|
|||||||
@@ -5,8 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
from pyflowx.ops._common import platform_command
|
from pyflowx.ops._common import platform_command
|
||||||
|
|
||||||
@@ -28,8 +26,10 @@ def taskkill_run(process_names: list[str]) -> None:
|
|||||||
for name in process_names:
|
for name in process_names:
|
||||||
print(f"终止进程: {name}")
|
print(f"终止进程: {name}")
|
||||||
# pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode
|
# pkill 返回 1 表示无匹配进程(非错误),故 check=False + 手动检查 returncode
|
||||||
result = subprocess.run([*cmd_prefix, f"{name}*"], check=False)
|
result = px.sh([*cmd_prefix, f"{name}*"], check=False, label="终止进程")
|
||||||
if result.returncode == 0:
|
if result is None:
|
||||||
|
print(f" 终止命令不可用: {name}")
|
||||||
|
elif result.returncode == 0:
|
||||||
print(f" 已发送终止信号: {name}")
|
print(f" 已发送终止信号: {name}")
|
||||||
else:
|
else:
|
||||||
print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}")
|
print(f" 未找到匹配进程或终止失败 (returncode={result.returncode}): {name}")
|
||||||
|
|||||||
@@ -5,8 +5,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
from pyflowx.ops._common import platform_command
|
from pyflowx.ops._common import platform_command
|
||||||
|
|
||||||
@@ -27,8 +25,8 @@ def which_run(commands: list[str]) -> None:
|
|||||||
|
|
||||||
for cmd in commands:
|
for cmd in commands:
|
||||||
# where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode
|
# where/which 返回非零表示未找到命令(正常情况),故 check=False + 手动检查 returncode
|
||||||
result = subprocess.run([which_cmd, cmd], capture_output=True, text=True, check=False)
|
result = px.sh([which_cmd, cmd], capture=True, check=False, label="which")
|
||||||
if result.returncode == 0:
|
if result is not None and result.returncode == 0:
|
||||||
# Windows 的 where 可能返回多行, 取第一个
|
# Windows 的 where 可能返回多行, 取第一个
|
||||||
path = result.stdout.strip().split("\n")[0].strip()
|
path = result.stdout.strip().split("\n")[0].strip()
|
||||||
print(f"{cmd} -> {path}")
|
print(f"{cmd} -> {path}")
|
||||||
|
|||||||
+30
-6
@@ -12,9 +12,14 @@
|
|||||||
result = px.sh("echo hello", capture=True)
|
result = px.sh("echo hello", capture=True)
|
||||||
if result is not None:
|
if result is not None:
|
||||||
print(result.stdout.strip())
|
print(result.stdout.strip())
|
||||||
|
# check=False 时返回 CompletedProcess(含非零返回码),调用方检查 returncode
|
||||||
|
res = px.sh(["git", "status", "--porcelain"], capture=True, check=False)
|
||||||
|
if res is not None and res.returncode == 0:
|
||||||
|
print(res.stdout)
|
||||||
|
|
||||||
设计定位:简单"执行 + 报错"场景的快捷方式。需要 ``capture_output`` +
|
设计定位:简单"执行 + 报错"场景的快捷方式。``check=False`` 模式覆盖
|
||||||
手动检查返回码等精细控制时,直接使用 :func:`subprocess.run`。
|
"容忍非零返回码 + 手动检查"场景;仍需更精细控制时直接使用
|
||||||
|
:func:`subprocess.run`。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -29,12 +34,17 @@ def sh(
|
|||||||
*,
|
*,
|
||||||
capture: bool = False,
|
capture: bool = False,
|
||||||
label: str = "命令",
|
label: str = "命令",
|
||||||
|
check: bool = True,
|
||||||
|
timeout: float | None = None,
|
||||||
) -> subprocess.CompletedProcess[str] | None:
|
) -> subprocess.CompletedProcess[str] | None:
|
||||||
"""执行 shell 命令,统一错误处理与中文输出。
|
"""执行 shell 命令,统一错误处理与中文输出。
|
||||||
|
|
||||||
``list[str]`` 直接执行;``str`` 走 ``shell=True``(支持管道/重定向等语法)。
|
``list[str]`` 直接执行;``str`` 走 ``shell=True``(支持管道/重定向等语法)。
|
||||||
成功返回 :class:`subprocess.CompletedProcess`;失败时打印中文错误信息
|
|
||||||
并返回 ``None``(不抛异常),调用方用 ``result is None`` 判断是否失败。
|
* ``check=True``(默认):成功返回 :class:`subprocess.CompletedProcess`;
|
||||||
|
非零返回码 / 命令未找到 / 超时均打印中文错误并返回 ``None``。
|
||||||
|
* ``check=False``:任何返回码都返回 :class:`subprocess.CompletedProcess`
|
||||||
|
(调用方检查 ``returncode``);仅命令未找到 / 超时返回 ``None``。
|
||||||
|
|
||||||
Parameters
|
Parameters
|
||||||
----------
|
----------
|
||||||
@@ -45,11 +55,18 @@ def sh(
|
|||||||
可从返回值的 ``.stdout`` / ``.stderr`` 读取(默认 ``False``)
|
可从返回值的 ``.stdout`` / ``.stderr`` 读取(默认 ``False``)
|
||||||
label : str
|
label : str
|
||||||
命令标签,用于错误信息中标识命令用途(默认 ``"命令"``)
|
命令标签,用于错误信息中标识命令用途(默认 ``"命令"``)
|
||||||
|
check : bool
|
||||||
|
为 ``True`` 时非零返回码视为失败(打印错误并返回 ``None``);
|
||||||
|
为 ``False`` 时总是返回 :class:`subprocess.CompletedProcess`,
|
||||||
|
由调用方检查 ``returncode``(默认 ``True``)
|
||||||
|
timeout : float | None
|
||||||
|
命令超时秒数;超时打印错误并返回 ``None``(默认 ``None`` 不限时)
|
||||||
|
|
||||||
Returns
|
Returns
|
||||||
-------
|
-------
|
||||||
subprocess.CompletedProcess[str] | None
|
subprocess.CompletedProcess[str] | None
|
||||||
成功时返回执行结果;失败时返回 ``None``
|
``check=True`` 时失败返回 ``None``;``check=False`` 时仅命令未找到 /
|
||||||
|
超时返回 ``None``,其余返回 :class:`subprocess.CompletedProcess`
|
||||||
|
|
||||||
Examples
|
Examples
|
||||||
--------
|
--------
|
||||||
@@ -58,6 +75,9 @@ def sh(
|
|||||||
>>> result = px.sh("pip --version", capture=True, label="pip")
|
>>> result = px.sh("pip --version", capture=True, label="pip")
|
||||||
>>> if result is not None:
|
>>> if result is not None:
|
||||||
... print(result.stdout)
|
... print(result.stdout)
|
||||||
|
>>> res = px.sh(["git", "status"], check=False, label="git") # 容忍非零返回码
|
||||||
|
>>> if res is not None and res.returncode == 0:
|
||||||
|
... print("git 可用")
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
return subprocess.run(
|
return subprocess.run(
|
||||||
@@ -65,7 +85,8 @@ def sh(
|
|||||||
shell=isinstance(cmd, str),
|
shell=isinstance(cmd, str),
|
||||||
capture_output=capture,
|
capture_output=capture,
|
||||||
text=True,
|
text=True,
|
||||||
check=True,
|
check=check,
|
||||||
|
timeout=timeout,
|
||||||
)
|
)
|
||||||
except subprocess.CalledProcessError as e:
|
except subprocess.CalledProcessError as e:
|
||||||
print(f"{label}失败 (returncode={e.returncode}): {e}")
|
print(f"{label}失败 (returncode={e.returncode}): {e}")
|
||||||
@@ -73,3 +94,6 @@ def sh(
|
|||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
print(f"{label}失败: 命令未找到,请确认已安装并在 PATH 中")
|
print(f"{label}失败: 命令未找到,请确认已安装并在 PATH 中")
|
||||||
return None
|
return None
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
print(f"{label}超时 (timeout={timeout}s)")
|
||||||
|
return None
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ import threading
|
|||||||
import time
|
import time
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from collections import OrderedDict
|
from collections import OrderedDict
|
||||||
from collections.abc import Iterator, Mapping
|
from collections.abc import Generator, Iterator, Mapping
|
||||||
from contextlib import AbstractContextManager, contextmanager, nullcontext
|
from contextlib import AbstractContextManager, contextmanager, nullcontext
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -328,7 +328,7 @@ class JSONBackend(_TTLStateBackendMixin):
|
|||||||
|
|
||||||
@override
|
@override
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def batch(self) -> Iterator[None]:
|
def batch(self) -> Generator[None, None, None]:
|
||||||
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
|
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
|
||||||
|
|
||||||
将整次运行 N 个任务的 N 次全量落盘降为 1 次。
|
将整次运行 N 个任务的 N 次全量落盘降为 1 次。
|
||||||
@@ -462,7 +462,7 @@ class SQLiteBackend(_TTLStateBackendMixin):
|
|||||||
|
|
||||||
@override
|
@override
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def batch(self) -> Iterator[None]:
|
def batch(self) -> Generator[None, None, None]:
|
||||||
"""进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。
|
"""进入批量模式:``save`` 暂不 commit,退出时统一 commit 一次。
|
||||||
|
|
||||||
将整次运行 N 个任务的 N 次 commit 降为 1 次。
|
将整次运行 N 个任务的 N 次 commit 降为 1 次。
|
||||||
|
|||||||
+1
-1
@@ -513,7 +513,7 @@ def _task_noop() -> None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def task(
|
def task( # noqa: PLR0913
|
||||||
fn: TaskFn[Any] | None = None,
|
fn: TaskFn[Any] | None = None,
|
||||||
*,
|
*,
|
||||||
cmd: TaskCmd | None = None,
|
cmd: TaskCmd | None = None,
|
||||||
|
|||||||
@@ -114,7 +114,7 @@ _TOOL_REGISTRY: dict[str, dict[str | None, ToolSpec]] = {}
|
|||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
# @tool 装饰器 + 注册表
|
# @tool 装饰器 + 注册表
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
def tool(
|
def tool( # noqa: PLR0913
|
||||||
name: str,
|
name: str,
|
||||||
*,
|
*,
|
||||||
subcommand: str | None = None,
|
subcommand: str | None = None,
|
||||||
|
|||||||
@@ -16,14 +16,14 @@ class TestFormatWithRuff:
|
|||||||
|
|
||||||
def test_format_with_ruff(self, tmp_path: Path) -> None:
|
def test_format_with_ruff(self, tmp_path: Path) -> None:
|
||||||
"""Should format with ruff."""
|
"""Should format with ruff."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.format_with_ruff(tmp_path, fix=True)
|
dev.format_with_ruff(tmp_path, fix=True)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_format_with_ruff_no_fix(self, tmp_path: Path) -> None:
|
def test_format_with_ruff_no_fix(self, tmp_path: Path) -> None:
|
||||||
"""Should format with ruff without fix."""
|
"""Should format with ruff without fix."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.format_with_ruff(tmp_path, fix=False)
|
dev.format_with_ruff(tmp_path, fix=False)
|
||||||
# Should not include --fix flag
|
# Should not include --fix flag
|
||||||
@@ -39,14 +39,14 @@ class TestLintWithRuff:
|
|||||||
|
|
||||||
def test_lint_with_ruff(self, tmp_path: Path) -> None:
|
def test_lint_with_ruff(self, tmp_path: Path) -> None:
|
||||||
"""Should lint with ruff."""
|
"""Should lint with ruff."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.lint_with_ruff(tmp_path, fix=True)
|
dev.lint_with_ruff(tmp_path, fix=True)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_lint_with_ruff_no_fix(self, tmp_path: Path) -> None:
|
def test_lint_with_ruff_no_fix(self, tmp_path: Path) -> None:
|
||||||
"""Should lint with ruff without fix."""
|
"""Should lint with ruff without fix."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.lint_with_ruff(tmp_path, fix=False)
|
dev.lint_with_ruff(tmp_path, fix=False)
|
||||||
# Should not include --fix flag
|
# Should not include --fix flag
|
||||||
@@ -176,7 +176,7 @@ class TestSyncPyprojectConfig:
|
|||||||
sub_toml = sub_dir / "pyproject.toml"
|
sub_toml = sub_dir / "pyproject.toml"
|
||||||
sub_toml.write_text("[tool.ruff]\n")
|
sub_toml.write_text("[tool.ruff]\n")
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.sync_pyproject_config(tmp_path)
|
dev.sync_pyproject_config(tmp_path)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
@@ -190,7 +190,7 @@ class TestSyncPyprojectConfig:
|
|||||||
sub_toml = sub_dir / "pyproject.toml"
|
sub_toml = sub_dir / "pyproject.toml"
|
||||||
sub_toml.write_text("[tool.ruff]\n")
|
sub_toml.write_text("[tool.ruff]\n")
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.sync_pyproject_config(tmp_path)
|
dev.sync_pyproject_config(tmp_path)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
@@ -204,14 +204,14 @@ class TestFormatAll:
|
|||||||
|
|
||||||
def test_format_all_runs_ruff_format(self, tmp_path: Path) -> None:
|
def test_format_all_runs_ruff_format(self, tmp_path: Path) -> None:
|
||||||
"""Should run ruff format."""
|
"""Should run ruff format."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.format_all(tmp_path)
|
dev.format_all(tmp_path)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_format_all_runs_ruff_check(self, tmp_path: Path) -> None:
|
def test_format_all_runs_ruff_check(self, tmp_path: Path) -> None:
|
||||||
"""Should run ruff check."""
|
"""Should run ruff check."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.format_all(tmp_path)
|
dev.format_all(tmp_path)
|
||||||
# Should call ruff format and ruff check
|
# Should call ruff format and ruff check
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
import subprocess
|
import subprocess
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -9,6 +10,13 @@ import pytest
|
|||||||
|
|
||||||
from pyflowx.ops.dev import bumpversion
|
from pyflowx.ops.dev import bumpversion
|
||||||
|
|
||||||
|
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[a-zA-Z]")
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_ansi(text: str) -> str:
|
||||||
|
"""移除 ANSI 转义码,使断言在 CI(富终端) 与本地环境一致."""
|
||||||
|
return _ANSI_RE.sub("", text)
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
@pytest.fixture(autouse=True)
|
||||||
def auto_use_tmp_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
def auto_use_tmp_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
@@ -104,15 +112,15 @@ class TestBumpProjectVersion:
|
|||||||
"""Test bump_project_version function."""
|
"""Test bump_project_version function."""
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _mock_subprocess(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
|
def _mock_sh(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
|
||||||
"""Mock subprocess.run, 返回调用记录列表."""
|
"""Mock px.sh, 返回调用记录列表."""
|
||||||
calls: list[list[str]] = []
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
def fake_run(cmd: list[str], **_kwargs: object) -> subprocess.CompletedProcess[bytes]:
|
def fake_sh(cmd: list[str], **_kwargs: object) -> subprocess.CompletedProcess[bytes]:
|
||||||
calls.append(cmd)
|
calls.append(cmd)
|
||||||
return subprocess.CompletedProcess(cmd, 0, b"", b"")
|
return subprocess.CompletedProcess(cmd, 0, b"", b"")
|
||||||
|
|
||||||
monkeypatch.setattr(subprocess, "run", fake_run)
|
monkeypatch.setattr("pyflowx.sh", fake_sh)
|
||||||
return calls
|
return calls
|
||||||
|
|
||||||
def test_unsynced_files_synchronized(
|
def test_unsynced_files_synchronized(
|
||||||
@@ -129,7 +137,7 @@ class TestBumpProjectVersion:
|
|||||||
pyproj = tmp_path / "pyproject.toml"
|
pyproj = tmp_path / "pyproject.toml"
|
||||||
pyproj.write_text('version = "0.3.5"', encoding="utf-8")
|
pyproj.write_text('version = "0.3.5"', encoding="utf-8")
|
||||||
|
|
||||||
calls = self._mock_subprocess(monkeypatch)
|
calls = self._mock_sh(monkeypatch)
|
||||||
|
|
||||||
result = bumpversion.bump_project_version("patch")
|
result = bumpversion.bump_project_version("patch")
|
||||||
|
|
||||||
@@ -163,7 +171,7 @@ class TestBumpProjectVersion:
|
|||||||
"""有文件但所有文件都无版本号返回 None."""
|
"""有文件但所有文件都无版本号返回 None."""
|
||||||
f = tmp_path / "__init__.py"
|
f = tmp_path / "__init__.py"
|
||||||
f.write_text("# no version here", encoding="utf-8")
|
f.write_text("# no version here", encoding="utf-8")
|
||||||
self._mock_subprocess(monkeypatch)
|
self._mock_sh(monkeypatch)
|
||||||
|
|
||||||
assert bumpversion.bump_project_version("patch") is None
|
assert bumpversion.bump_project_version("patch") is None
|
||||||
assert "未能从任何文件读取版本号" in capsys.readouterr().out
|
assert "未能从任何文件读取版本号" in capsys.readouterr().out
|
||||||
@@ -173,7 +181,7 @@ class TestBumpProjectVersion:
|
|||||||
pyproj = tmp_path / "pyproject.toml"
|
pyproj = tmp_path / "pyproject.toml"
|
||||||
pyproj.write_text('version = "1.0.0"', encoding="utf-8")
|
pyproj.write_text('version = "1.0.0"', encoding="utf-8")
|
||||||
|
|
||||||
calls = self._mock_subprocess(monkeypatch)
|
calls = self._mock_sh(monkeypatch)
|
||||||
|
|
||||||
assert bumpversion.bump_project_version("patch", no_tag=True) == "1.0.1"
|
assert bumpversion.bump_project_version("patch", no_tag=True) == "1.0.1"
|
||||||
assert not any(c[:2] == ["git", "tag"] for c in calls)
|
assert not any(c[:2] == ["git", "tag"] for c in calls)
|
||||||
@@ -186,7 +194,57 @@ class TestBumpProjectVersion:
|
|||||||
pyproj = tmp_path / "pyproject.toml"
|
pyproj = tmp_path / "pyproject.toml"
|
||||||
pyproj.write_text('version = "1.0.0"', encoding="utf-8")
|
pyproj.write_text('version = "1.0.0"', encoding="utf-8")
|
||||||
|
|
||||||
self._mock_subprocess(monkeypatch)
|
self._mock_sh(monkeypatch)
|
||||||
|
|
||||||
assert bumpversion.bump_project_version("patch") == "1.0.1"
|
assert bumpversion.bump_project_version("patch") == "1.0.1"
|
||||||
assert venv_init.read_text(encoding="utf-8") == '__version__ = "0.1.0"'
|
assert venv_init.read_text(encoding="utf-8") == '__version__ = "0.1.0"'
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
# CLI 调度测试 (pf bumpversion -h / pf bumpversion --part X --dry-run)
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
class TestBumpversionCliDispatch:
|
||||||
|
"""``pf bumpversion`` CLI 调度测试 (不执行真实 bump)."""
|
||||||
|
|
||||||
|
def test_bumpversion_h_shows_help(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf bumpversion -h 应返回 0 并显示 --part 选项."""
|
||||||
|
from pyflowx.cli.pf import PfApp
|
||||||
|
|
||||||
|
app = PfApp(["bumpversion", "-h"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = _strip_ansi(capsys.readouterr().out)
|
||||||
|
assert "--part" in out
|
||||||
|
assert "patch" in out
|
||||||
|
|
||||||
|
def test_bumpversion_part_patch_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf bumpversion --part patch --dry-run 应打印执行计划, 不执行 bump."""
|
||||||
|
from pyflowx.cli.pf import PfApp
|
||||||
|
|
||||||
|
app = PfApp(["bumpversion", "--part", "patch", "--dry-run"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Dry run" in out
|
||||||
|
assert "bumpversion" in out
|
||||||
|
|
||||||
|
def test_bumpversion_part_minor_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf bumpversion --part minor --dry-run 应打印执行计划."""
|
||||||
|
from pyflowx.cli.pf import PfApp
|
||||||
|
|
||||||
|
app = PfApp(["bumpversion", "--part", "minor", "--dry-run"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Dry run" in out
|
||||||
|
|
||||||
|
def test_bumpversion_default_part_is_patch(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf bumpversion --dry-run (无 --part) 默认使用 patch."""
|
||||||
|
from pyflowx.cli.pf import PfApp
|
||||||
|
|
||||||
|
app = PfApp(["bumpversion", "--dry-run"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Dry run" in out
|
||||||
|
assert "patch" in out
|
||||||
|
|||||||
@@ -178,7 +178,8 @@ class TestDockerLoginTencent:
|
|||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))
|
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(err))
|
||||||
docker_login_tencent("myuser")
|
docker_login_tencent("myuser")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "docker login 失败" in captured.out
|
assert "docker login" in captured.out
|
||||||
|
assert "失败" in captured.out
|
||||||
assert "returncode=1" in captured.out
|
assert "returncode=1" in captured.out
|
||||||
assert "已登录" not in captured.out
|
assert "已登录" not in captured.out
|
||||||
|
|
||||||
@@ -189,7 +190,7 @@ class TestDockerLoginTencent:
|
|||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
|
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
|
||||||
docker_login_tencent("myuser")
|
docker_login_tencent("myuser")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "docker 命令未找到" in captured.out
|
assert "命令未找到" in captured.out
|
||||||
assert "已登录" not in captured.out
|
assert "已登录" not in captured.out
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ class TestHasFiles:
|
|||||||
class _FakeResult:
|
class _FakeResult:
|
||||||
stdout = " M test.txt\n"
|
stdout = " M test.txt\n"
|
||||||
|
|
||||||
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult())
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: _FakeResult())
|
||||||
result = dev.has_files()
|
result = dev.has_files()
|
||||||
assert result is True
|
assert result is True
|
||||||
|
|
||||||
@@ -63,7 +63,7 @@ class TestHasFiles:
|
|||||||
class _FakeResult:
|
class _FakeResult:
|
||||||
stdout = ""
|
stdout = ""
|
||||||
|
|
||||||
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult())
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: _FakeResult())
|
||||||
result = dev.has_files()
|
result = dev.has_files()
|
||||||
assert result is False
|
assert result is False
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
|
||||||
@@ -115,22 +114,9 @@ class TestGittoolHelpers:
|
|||||||
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult())
|
monkeypatch.setattr("subprocess.run", lambda *_, **__: _FakeResult())
|
||||||
assert gittool.has_files() is True
|
assert gittool.has_files() is True
|
||||||
|
|
||||||
def test_has_files_false_on_subprocess_error(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_has_files_false_when_sh_fails(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
"""subprocess.SubprocessError 时 has_files 应返回 False."""
|
"""px.sh 返回 None(命令失败/未找到)时 has_files 应返回 False."""
|
||||||
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)
|
||||||
def _raise(*args: object, **kwargs: object) -> None:
|
|
||||||
raise subprocess.SubprocessError("fail")
|
|
||||||
|
|
||||||
monkeypatch.setattr("subprocess.run", _raise)
|
|
||||||
assert gittool.has_files() is False
|
|
||||||
|
|
||||||
def test_has_files_false_on_oserror(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
||||||
"""OSError 时 has_files 应返回 False."""
|
|
||||||
|
|
||||||
def _raise(*args: object, **kwargs: object) -> None:
|
|
||||||
raise OSError("fail")
|
|
||||||
|
|
||||||
monkeypatch.setattr("subprocess.run", _raise)
|
|
||||||
assert gittool.has_files() is False
|
assert gittool.has_files() is False
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -90,7 +90,7 @@ class TestMsdownloadRun:
|
|||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
|
monkeypatch.setattr(subprocess, "run", lambda *_, **__: (_ for _ in ()).throw(FileNotFoundError()))
|
||||||
msdownload_run("foo/bar")
|
msdownload_run("foo/bar")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "uvx 命令未找到" in captured.out
|
assert "命令未找到" in captured.out
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ class TestRunLsDyna:
|
|||||||
input_file = tmp_path / "input.k"
|
input_file = tmp_path / "input.k"
|
||||||
input_file.write_text("LS-DYNA input")
|
input_file.write_text("LS-DYNA input")
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
system.run_ls_dyna(str(input_file), ncpu=4)
|
system.run_ls_dyna(str(input_file), ncpu=4)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
@@ -60,7 +60,7 @@ class TestRunLsDyna:
|
|||||||
input_file = tmp_path / "input.k"
|
input_file = tmp_path / "input.k"
|
||||||
input_file.write_text("LS-DYNA input")
|
input_file.write_text("LS-DYNA input")
|
||||||
|
|
||||||
with patch("subprocess.run", side_effect=FileNotFoundError):
|
with patch("pyflowx.sh", return_value=None):
|
||||||
system.run_ls_dyna(str(input_file), ncpu=4)
|
system.run_ls_dyna(str(input_file), ncpu=4)
|
||||||
# Should print error message
|
# Should print error message
|
||||||
|
|
||||||
@@ -76,7 +76,7 @@ class TestRunLsDynaMpi:
|
|||||||
input_file = tmp_path / "input.k"
|
input_file = tmp_path / "input.k"
|
||||||
input_file.write_text("LS-DYNA input")
|
input_file.write_text("LS-DYNA input")
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
system.run_ls_dyna_mpi(str(input_file), ncpu=8)
|
system.run_ls_dyna_mpi(str(input_file), ncpu=8)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
@@ -97,14 +97,14 @@ class TestCheckLsDynaStatus:
|
|||||||
|
|
||||||
def test_check_ls_dyna_status_windows(self) -> None:
|
def test_check_ls_dyna_status_windows(self) -> None:
|
||||||
"""Should check LS-DYNA status on Windows."""
|
"""Should check LS-DYNA status on Windows."""
|
||||||
with patch.object(Constants, "IS_WINDOWS", True), patch("subprocess.run") as mock_run:
|
with patch.object(Constants, "IS_WINDOWS", True), patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(stdout="ls-dyna_mpp.exe", returncode=0)
|
mock_run.return_value = MagicMock(stdout="ls-dyna_mpp.exe", returncode=0)
|
||||||
system.check_ls_dyna_status()
|
system.check_ls_dyna_status()
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_check_ls_dyna_status_linux(self) -> None:
|
def test_check_ls_dyna_status_linux(self) -> None:
|
||||||
"""Should check LS-DYNA status on Linux."""
|
"""Should check LS-DYNA status on Linux."""
|
||||||
with patch.object(Constants, "IS_WINDOWS", False), patch("subprocess.run") as mock_run:
|
with patch.object(Constants, "IS_WINDOWS", False), patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(stdout="1234", returncode=0)
|
mock_run.return_value = MagicMock(stdout="1234", returncode=0)
|
||||||
system.check_ls_dyna_status()
|
system.check_ls_dyna_status()
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ class TestPackDependencies:
|
|||||||
"""Should pack dependencies."""
|
"""Should pack dependencies."""
|
||||||
lib_dir = tmp_path / "libs"
|
lib_dir = tmp_path / "libs"
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
system.pack_dependencies(lib_dir, ["numpy", "pandas"])
|
system.pack_dependencies(lib_dir, ["numpy", "pandas"])
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
@@ -86,7 +86,7 @@ class TestPackWheel:
|
|||||||
(project_dir / "pyproject.toml").write_text("[project]\nname = 'test'")
|
(project_dir / "pyproject.toml").write_text("[project]\nname = 'test'")
|
||||||
output_dir = tmp_path / "dist"
|
output_dir = tmp_path / "dist"
|
||||||
|
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
system.pack_wheel(project_dir, output_dir)
|
system.pack_wheel(project_dir, output_dir)
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|||||||
+12
-13
@@ -2,7 +2,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from unittest.mock import MagicMock, patch
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
@@ -17,7 +16,7 @@ class TestGetInstalledPackages:
|
|||||||
|
|
||||||
def test_get_installed_packages_success(self) -> None:
|
def test_get_installed_packages_success(self) -> None:
|
||||||
"""Should get installed packages."""
|
"""Should get installed packages."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0\n", returncode=0)
|
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0\n", returncode=0)
|
||||||
result = dev._get_installed_packages()
|
result = dev._get_installed_packages()
|
||||||
assert "numpy" in result
|
assert "numpy" in result
|
||||||
@@ -25,20 +24,20 @@ class TestGetInstalledPackages:
|
|||||||
|
|
||||||
def test_get_installed_packages_empty(self) -> None:
|
def test_get_installed_packages_empty(self) -> None:
|
||||||
"""Should handle empty output."""
|
"""Should handle empty output."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(stdout="", returncode=0)
|
mock_run.return_value = MagicMock(stdout="", returncode=0)
|
||||||
result = dev._get_installed_packages()
|
result = dev._get_installed_packages()
|
||||||
assert result == []
|
assert result == []
|
||||||
|
|
||||||
def test_get_installed_packages_error(self) -> None:
|
def test_get_installed_packages_error(self) -> None:
|
||||||
"""Should handle subprocess error."""
|
"""Should handle subprocess error."""
|
||||||
with patch("subprocess.run", side_effect=subprocess.SubprocessError):
|
with patch("pyflowx.sh", return_value=None):
|
||||||
result = dev._get_installed_packages()
|
result = dev._get_installed_packages()
|
||||||
assert result == []
|
assert result == []
|
||||||
|
|
||||||
def test_get_installed_packages_oserror(self) -> None:
|
def test_get_installed_packages_oserror(self) -> None:
|
||||||
"""Should handle OSError."""
|
"""Should handle OSError."""
|
||||||
with patch("subprocess.run", side_effect=OSError):
|
with patch("pyflowx.sh", return_value=None):
|
||||||
result = dev._get_installed_packages()
|
result = dev._get_installed_packages()
|
||||||
assert result == []
|
assert result == []
|
||||||
|
|
||||||
@@ -106,14 +105,14 @@ class TestPipUninstall:
|
|||||||
|
|
||||||
def test_pip_uninstall_single_package(self) -> None:
|
def test_pip_uninstall_single_package(self) -> None:
|
||||||
"""Should uninstall single package."""
|
"""Should uninstall single package."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_uninstall(["numpy"])
|
dev.pip_uninstall(["numpy"])
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_pip_uninstall_multiple_packages(self) -> None:
|
def test_pip_uninstall_multiple_packages(self) -> None:
|
||||||
"""Should uninstall multiple packages."""
|
"""Should uninstall multiple packages."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_uninstall(["numpy", "pandas", "scipy"])
|
dev.pip_uninstall(["numpy", "pandas", "scipy"])
|
||||||
# Should call pip uninstall
|
# Should call pip uninstall
|
||||||
@@ -123,7 +122,7 @@ class TestPipUninstall:
|
|||||||
"""Should handle wildcard in package name."""
|
"""Should handle wildcard in package name."""
|
||||||
with (
|
with (
|
||||||
patch.object(dev, "_expand_wildcard_packages", return_value=["numpy", "numpy-core"]),
|
patch.object(dev, "_expand_wildcard_packages", return_value=["numpy", "numpy-core"]),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_uninstall(["numpy*"])
|
dev.pip_uninstall(["numpy*"])
|
||||||
@@ -149,7 +148,7 @@ class TestPipReinstall:
|
|||||||
|
|
||||||
def test_pip_reinstall_single_package(self) -> None:
|
def test_pip_reinstall_single_package(self) -> None:
|
||||||
"""Should reinstall single package."""
|
"""Should reinstall single package."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_reinstall(["numpy"])
|
dev.pip_reinstall(["numpy"])
|
||||||
# Should call pip uninstall and pip install
|
# Should call pip uninstall and pip install
|
||||||
@@ -157,7 +156,7 @@ class TestPipReinstall:
|
|||||||
|
|
||||||
def test_pip_reinstall_offline(self) -> None:
|
def test_pip_reinstall_offline(self) -> None:
|
||||||
"""Should reinstall packages offline."""
|
"""Should reinstall packages offline."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_reinstall(["numpy"], offline=True)
|
dev.pip_reinstall(["numpy"], offline=True)
|
||||||
# Should call pip install with offline flags
|
# Should call pip install with offline flags
|
||||||
@@ -177,14 +176,14 @@ class TestPipDownload:
|
|||||||
|
|
||||||
def test_pip_download_single_package(self) -> None:
|
def test_pip_download_single_package(self) -> None:
|
||||||
"""Should download single package."""
|
"""Should download single package."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_download(["numpy"])
|
dev.pip_download(["numpy"])
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|
||||||
def test_pip_download_offline(self) -> None:
|
def test_pip_download_offline(self) -> None:
|
||||||
"""Should download packages offline."""
|
"""Should download packages offline."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
dev.pip_download(["numpy"], offline=True)
|
dev.pip_download(["numpy"], offline=True)
|
||||||
# Should call pip download with offline flags
|
# Should call pip download with offline flags
|
||||||
@@ -199,7 +198,7 @@ class TestPipFreeze:
|
|||||||
|
|
||||||
def test_pip_freeze(self, tmp_path: Path) -> None:
|
def test_pip_freeze(self, tmp_path: Path) -> None:
|
||||||
"""Should freeze dependencies."""
|
"""Should freeze dependencies."""
|
||||||
with patch("subprocess.run") as mock_run:
|
with patch("pyflowx.sh") as mock_run:
|
||||||
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0", returncode=0)
|
mock_run.return_value = MagicMock(stdout="numpy==1.0.0\npandas==2.0.0", returncode=0)
|
||||||
dev.pip_freeze()
|
dev.pip_freeze()
|
||||||
assert mock_run.called
|
assert mock_run.called
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ def _fn() -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
|
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
|
||||||
return TaskSpec[Any](name, _fn, depends_on=deps)
|
return TaskSpec(name, _fn, depends_on=deps)
|
||||||
|
|
||||||
|
|
||||||
def _result(
|
def _result(
|
||||||
@@ -40,7 +40,7 @@ def _result(
|
|||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
"""构造带时间戳的 TaskResult."""
|
"""构造带时间戳的 TaskResult."""
|
||||||
end = start + timedelta(seconds=duration) if duration > 0 else start
|
end = start + timedelta(seconds=duration) if duration > 0 else start
|
||||||
return TaskResult[Any](
|
return TaskResult(
|
||||||
spec=_spec(name),
|
spec=_spec(name),
|
||||||
status=status,
|
status=status,
|
||||||
value=None,
|
value=None,
|
||||||
@@ -139,7 +139,7 @@ class TestToHtml:
|
|||||||
start = datetime(2024, 1, 1, 0, 0, 0)
|
start = datetime(2024, 1, 1, 0, 0, 0)
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
report.results["a"] = _result("a", start, 1.0)
|
report.results["a"] = _result("a", start, 1.0)
|
||||||
report.results["b"] = TaskResult[Any](
|
report.results["b"] = TaskResult(
|
||||||
spec=_spec("b"),
|
spec=_spec("b"),
|
||||||
status=TaskStatus.SKIPPED,
|
status=TaskStatus.SKIPPED,
|
||||||
reason="skip",
|
reason="skip",
|
||||||
|
|||||||
@@ -1,9 +1,12 @@
|
|||||||
"""Tests for ops.pymake 模块 (@px.tool 注册验证)."""
|
"""Tests for ops.pymake 模块 (@px.tool 注册验证 + CLI 调度)."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
import pyflowx as px
|
import pyflowx as px
|
||||||
import pyflowx.ops.dev.pymake
|
import pyflowx.ops.dev.pymake
|
||||||
|
from pyflowx.cli.pf import PfApp
|
||||||
from pyflowx.tools import _TOOL_REGISTRY, get_tool
|
from pyflowx.tools import _TOOL_REGISTRY, get_tool
|
||||||
|
|
||||||
|
|
||||||
@@ -106,3 +109,58 @@ class TestPymakeRegistration:
|
|||||||
for name in ("ba", "bump", "cov", "tc", "p"):
|
for name in ("ba", "bump", "cov", "tc", "p"):
|
||||||
spec = get_tool("pymake", name)
|
spec = get_tool("pymake", name)
|
||||||
assert spec.cmd is None, f"{name} 应为聚合任务 (无 cmd)"
|
assert spec.cmd is None, f"{name} 应为聚合任务 (无 cmd)"
|
||||||
|
|
||||||
|
def test_bumpversion_cmd_uses_part_option(self) -> None:
|
||||||
|
"""bumpversion 隐藏任务 cmd 应使用 --part patch (非位置参数)."""
|
||||||
|
spec = get_tool("pymake", "bumpversion")
|
||||||
|
assert spec.cmd is not None
|
||||||
|
assert "--part" in spec.cmd
|
||||||
|
assert "patch" in spec.cmd
|
||||||
|
|
||||||
|
def test_bumpmi_cmd_uses_part_option(self) -> None:
|
||||||
|
"""bumpmi 子命令 cmd 应使用 --part minor (非位置参数)."""
|
||||||
|
spec = get_tool("pymake", "bumpmi")
|
||||||
|
assert spec.cmd is not None
|
||||||
|
assert "--part" in spec.cmd
|
||||||
|
assert "minor" in spec.cmd
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
# CLI 调度测试 (pf pymake -h / pf pymake bump --dry-run)
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
class TestPymakeCliDispatch:
|
||||||
|
"""``pf pymake`` CLI 调度测试 (不执行真实命令)."""
|
||||||
|
|
||||||
|
def test_pymake_h_shows_help(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf pymake -h 应返回 0 并打印子命令列表."""
|
||||||
|
app = PfApp(["pymake", "-h"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Commands" in out
|
||||||
|
assert "bump" in out
|
||||||
|
|
||||||
|
def test_pymake_help_long_form(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf pymake --help 应与 -h 等价."""
|
||||||
|
app = PfApp(["pymake", "--help"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Commands" in out
|
||||||
|
|
||||||
|
def test_pymake_bump_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf pymake bump --dry-run 应打印执行计划, 不执行真实命令."""
|
||||||
|
app = PfApp(["pymake", "bump", "--dry-run"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Dry run" in out
|
||||||
|
assert "bumpversion" in out
|
||||||
|
|
||||||
|
def test_pymake_bumpmi_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
"""pf pymake bumpmi --dry-run 应打印执行计划."""
|
||||||
|
app = PfApp(["pymake", "bumpmi", "--dry-run"])
|
||||||
|
rc = app.run()
|
||||||
|
assert rc == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Dry run" in out
|
||||||
|
|||||||
@@ -2,8 +2,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from pyflowx.conditions import Constants
|
from pyflowx.conditions import Constants
|
||||||
@@ -46,7 +44,7 @@ class TestResetIconCacheRun:
|
|||||||
monkeypatch.setattr(Path, "exists", lambda _self: False)
|
monkeypatch.setattr(Path, "exists", lambda _self: False)
|
||||||
|
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd))
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd))
|
||||||
|
|
||||||
reset_icon_cache_run()
|
reset_icon_cache_run()
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
@@ -69,7 +67,7 @@ class TestResetIconCacheRun:
|
|||||||
monkeypatch.setattr(Path, "exists", lambda _self: True)
|
monkeypatch.setattr(Path, "exists", lambda _self: True)
|
||||||
|
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd))
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd))
|
||||||
|
|
||||||
reset_icon_cache_run()
|
reset_icon_cache_run()
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ class TestTakeScreenshotFull:
|
|||||||
patch.object(Constants, "IS_WINDOWS", True),
|
patch.object(Constants, "IS_WINDOWS", True),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_full()
|
media.take_screenshot_full()
|
||||||
@@ -53,7 +53,7 @@ class TestTakeScreenshotFull:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", True),
|
patch.object(Constants, "IS_MACOS", True),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_full()
|
media.take_screenshot_full()
|
||||||
@@ -65,7 +65,7 @@ class TestTakeScreenshotFull:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_full()
|
media.take_screenshot_full()
|
||||||
@@ -77,9 +77,9 @@ class TestTakeScreenshotFull:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.side_effect = [FileNotFoundError(), MagicMock(returncode=0)]
|
mock_run.side_effect = [None, MagicMock(returncode=0)]
|
||||||
media.take_screenshot_full()
|
media.take_screenshot_full()
|
||||||
assert mock_run.call_count == 2
|
assert mock_run.call_count == 2
|
||||||
|
|
||||||
@@ -96,7 +96,7 @@ class TestTakeScreenshotArea:
|
|||||||
patch.object(Constants, "IS_WINDOWS", True),
|
patch.object(Constants, "IS_WINDOWS", True),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_area()
|
media.take_screenshot_area()
|
||||||
@@ -108,7 +108,7 @@ class TestTakeScreenshotArea:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", True),
|
patch.object(Constants, "IS_MACOS", True),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_area()
|
media.take_screenshot_area()
|
||||||
@@ -120,7 +120,7 @@ class TestTakeScreenshotArea:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.return_value = MagicMock(returncode=0)
|
mock_run.return_value = MagicMock(returncode=0)
|
||||||
media.take_screenshot_area()
|
media.take_screenshot_area()
|
||||||
@@ -132,8 +132,8 @@ class TestTakeScreenshotArea:
|
|||||||
patch.object(Constants, "IS_WINDOWS", False),
|
patch.object(Constants, "IS_WINDOWS", False),
|
||||||
patch.object(Constants, "IS_MACOS", False),
|
patch.object(Constants, "IS_MACOS", False),
|
||||||
patch.object(Path, "home", return_value=tmp_path),
|
patch.object(Path, "home", return_value=tmp_path),
|
||||||
patch("subprocess.run") as mock_run,
|
patch("pyflowx.sh") as mock_run,
|
||||||
):
|
):
|
||||||
mock_run.side_effect = [FileNotFoundError(), MagicMock(returncode=0)]
|
mock_run.side_effect = [None, MagicMock(returncode=0)]
|
||||||
media.take_screenshot_area()
|
media.take_screenshot_area()
|
||||||
assert mock_run.call_count == 2
|
assert mock_run.call_count == 2
|
||||||
|
|||||||
@@ -2,7 +2,6 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from unittest.mock import MagicMock
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -23,7 +22,7 @@ class TestClearScreenRun:
|
|||||||
"""Windows 上应调用 cls."""
|
"""Windows 上应调用 cls."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
||||||
clear_screen_run()
|
clear_screen_run()
|
||||||
assert ran_cmds == [["cls"]]
|
assert ran_cmds == [["cls"]]
|
||||||
|
|
||||||
@@ -31,7 +30,7 @@ class TestClearScreenRun:
|
|||||||
"""非 Windows 上应调用 clear."""
|
"""非 Windows 上应调用 clear."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
||||||
clear_screen_run()
|
clear_screen_run()
|
||||||
assert ran_cmds == [["clear"]]
|
assert ran_cmds == [["clear"]]
|
||||||
|
|
||||||
@@ -46,7 +45,7 @@ class TestTaskkillRun:
|
|||||||
"""Windows 上应使用 taskkill /f /im 并加通配符."""
|
"""Windows 上应使用 taskkill /f /im 并加通配符."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
||||||
taskkill_run(["chrome.exe", "python.exe"])
|
taskkill_run(["chrome.exe", "python.exe"])
|
||||||
assert ran_cmds == [
|
assert ran_cmds == [
|
||||||
["taskkill", "/f", "/im", "chrome.exe*"],
|
["taskkill", "/f", "/im", "chrome.exe*"],
|
||||||
@@ -57,22 +56,22 @@ class TestTaskkillRun:
|
|||||||
"""非 Windows 上应使用 pkill -f 并加通配符."""
|
"""非 Windows 上应使用 pkill -f 并加通配符."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or MagicMock())
|
||||||
taskkill_run(["chrome"])
|
taskkill_run(["chrome"])
|
||||||
assert ran_cmds == [["pkill", "-f", "chrome*"]]
|
assert ran_cmds == [["pkill", "-f", "chrome*"]]
|
||||||
|
|
||||||
def test_empty_list_does_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_empty_list_does_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
"""空列表不应调用 subprocess."""
|
"""空列表不应调用 px.sh."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
called: list[list[str]] = []
|
called: list[list[str]] = []
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: called.append(cmd) or MagicMock())
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: called.append(cmd) or MagicMock())
|
||||||
taskkill_run([])
|
taskkill_run([])
|
||||||
assert called == []
|
assert called == []
|
||||||
|
|
||||||
def test_prints_progress(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
|
def test_prints_progress(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
"""应打印每个进程的终止信息."""
|
"""应打印每个进程的终止信息."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: MagicMock(returncode=0))
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: MagicMock(returncode=0))
|
||||||
taskkill_run(["chrome.exe"])
|
taskkill_run(["chrome.exe"])
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "chrome.exe" in captured.out
|
assert "chrome.exe" in captured.out
|
||||||
@@ -84,13 +83,24 @@ class TestTaskkillRun:
|
|||||||
"""returncode 非零时应打印未找到/失败信息."""
|
"""returncode 非零时应打印未找到/失败信息."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
mock_result = MagicMock(returncode=1)
|
mock_result = MagicMock(returncode=1)
|
||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
|
||||||
taskkill_run(["nonexistent"])
|
taskkill_run(["nonexistent"])
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "未找到匹配进程或终止失败" in captured.out
|
assert "未找到匹配进程或终止失败" in captured.out
|
||||||
assert "returncode=1" in captured.out
|
assert "returncode=1" in captured.out
|
||||||
assert "已发送终止信号" not in captured.out
|
assert "已发送终止信号" not in captured.out
|
||||||
|
|
||||||
|
def test_command_unavailable_prints_message(
|
||||||
|
self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
"""px.sh 返回 None(命令未找到)时应打印不可用信息."""
|
||||||
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: None)
|
||||||
|
taskkill_run(["chrome"])
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "终止命令不可用" in captured.out
|
||||||
|
assert "已发送终止信号" not in captured.out
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------- #
|
# ---------------------------------------------------------------------- #
|
||||||
# which_run
|
# which_run
|
||||||
@@ -103,7 +113,7 @@ class TestWhichRun:
|
|||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
mock_result = MagicMock(returncode=0, stdout="C:\\Windows\\python.exe\n")
|
mock_result = MagicMock(returncode=0, stdout="C:\\Windows\\python.exe\n")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
||||||
which_run(["python"])
|
which_run(["python"])
|
||||||
assert ran_cmds == [["where", "python"]]
|
assert ran_cmds == [["where", "python"]]
|
||||||
|
|
||||||
@@ -112,7 +122,7 @@ class TestWhichRun:
|
|||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
||||||
which_run(["python"])
|
which_run(["python"])
|
||||||
assert ran_cmds == [["which", "python"]]
|
assert ran_cmds == [["which", "python"]]
|
||||||
|
|
||||||
@@ -120,7 +130,7 @@ class TestWhichRun:
|
|||||||
"""找到命令时应打印 <cmd> -> <path>."""
|
"""找到命令时应打印 <cmd> -> <path>."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
|
||||||
which_run(["python"])
|
which_run(["python"])
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "python" in captured.out
|
assert "python" in captured.out
|
||||||
@@ -132,7 +142,7 @@ class TestWhichRun:
|
|||||||
"""未找到命令时应打印未找到."""
|
"""未找到命令时应打印未找到."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
mock_result = MagicMock(returncode=1, stdout="")
|
mock_result = MagicMock(returncode=1, stdout="")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
|
||||||
which_run(["nonexistent_cmd"])
|
which_run(["nonexistent_cmd"])
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "未找到" in captured.out
|
assert "未找到" in captured.out
|
||||||
@@ -142,7 +152,7 @@ class TestWhichRun:
|
|||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
|
||||||
ran_cmds: list[list[str]] = []
|
ran_cmds: list[list[str]] = []
|
||||||
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
mock_result = MagicMock(returncode=0, stdout="/usr/bin/python\n")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda cmd, **_: ran_cmds.append(cmd) or mock_result)
|
||||||
which_run(["python", "pip"])
|
which_run(["python", "pip"])
|
||||||
assert ran_cmds == [["which", "python"], ["which", "pip"]]
|
assert ran_cmds == [["which", "python"], ["which", "pip"]]
|
||||||
|
|
||||||
@@ -152,7 +162,7 @@ class TestWhichRun:
|
|||||||
"""Windows where 返回多行时应取第一行."""
|
"""Windows where 返回多行时应取第一行."""
|
||||||
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
|
||||||
mock_result = MagicMock(returncode=0, stdout="C:\\first\\python.exe\nC:\\second\\python.exe\n")
|
mock_result = MagicMock(returncode=0, stdout="C:\\first\\python.exe\nC:\\second\\python.exe\n")
|
||||||
monkeypatch.setattr(subprocess, "run", lambda *_, **__: mock_result)
|
monkeypatch.setattr("pyflowx.sh", lambda *_, **__: mock_result)
|
||||||
which_run(["python"])
|
which_run(["python"])
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "C:\\first\\python.exe" in captured.out
|
assert "C:\\first\\python.exe" in captured.out
|
||||||
|
|||||||
+23
-29
@@ -141,13 +141,11 @@ class TestMarkRemainingSkipped:
|
|||||||
|
|
||||||
def test_marks_all_unexecuted_tasks(self) -> None:
|
def test_marks_all_unexecuted_tasks(self) -> None:
|
||||||
"""未执行的任务应被标记为 SKIPPED."""
|
"""未执行的任务应被标记为 SKIPPED."""
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", lambda: 1),
|
||||||
px.TaskSpec("a", lambda: 1),
|
px.TaskSpec("b", lambda: 2, depends_on=("a",)),
|
||||||
px.TaskSpec("b", lambda: 2, depends_on=("a",)),
|
px.TaskSpec("c", lambda: 3, depends_on=("b",)),
|
||||||
px.TaskSpec("c", lambda: 3, depends_on=("b",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = RunReport()
|
report = RunReport()
|
||||||
# 模拟 a 已完成
|
# 模拟 a 已完成
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -171,12 +169,10 @@ class TestMarkRemainingSkipped:
|
|||||||
|
|
||||||
def test_marks_all_when_none_executed(self) -> None:
|
def test_marks_all_when_none_executed(self) -> None:
|
||||||
"""无已完成任务时,所有任务应被标记为 SKIPPED."""
|
"""无已完成任务时,所有任务应被标记为 SKIPPED."""
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", lambda: 1),
|
||||||
px.TaskSpec("a", lambda: 1),
|
px.TaskSpec("b", lambda: 2, depends_on=("a",)),
|
||||||
px.TaskSpec("b", lambda: 2, depends_on=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = RunReport()
|
report = RunReport()
|
||||||
_mark_remaining_skipped(graph, report, None)
|
_mark_remaining_skipped(graph, report, None)
|
||||||
assert len(report.results) == 2
|
assert len(report.results) == 2
|
||||||
@@ -223,14 +219,12 @@ def _make_cancellable_graph() -> px.Graph:
|
|||||||
time.sleep(0.05)
|
time.sleep(0.05)
|
||||||
return c + 1
|
return c + 1
|
||||||
|
|
||||||
return px.Graph.from_specs(
|
return px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", a),
|
||||||
px.TaskSpec("a", a),
|
px.TaskSpec("b", b, depends_on=("a",)),
|
||||||
px.TaskSpec("b", b, depends_on=("a",)),
|
px.TaskSpec("c", c, depends_on=("b",)),
|
||||||
px.TaskSpec("c", c, depends_on=("b",)),
|
px.TaskSpec("d", d, depends_on=("c",)),
|
||||||
px.TaskSpec("d", d, depends_on=("c",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TestExternalCancel:
|
class TestExternalCancel:
|
||||||
@@ -327,10 +321,12 @@ class TestKeyboardInterrupt:
|
|||||||
graph = _make_cancellable_graph()
|
graph = _make_cancellable_graph()
|
||||||
|
|
||||||
def _interrupt_after_first_layer(
|
def _interrupt_after_first_layer(
|
||||||
graph: Any, layers: Any, context: Any, report: Any, backend: Any, on_event: Any, cancel_event: Any = None
|
graph: Any,
|
||||||
|
layers: Any,
|
||||||
|
ctx: Any,
|
||||||
) -> None:
|
) -> None:
|
||||||
# 只运行第一层,然后抛 KeyboardInterrupt
|
# 只运行第一层,然后抛 KeyboardInterrupt
|
||||||
executors.SequentialLayerRunner.execute(layers[0], graph, context, report, backend, 1, on_event)
|
executors.SequentialLayerRunner.execute(layers[0], graph, ctx, 1)
|
||||||
raise KeyboardInterrupt
|
raise KeyboardInterrupt
|
||||||
|
|
||||||
with patch.object(executors, "_drive_sequential", _interrupt_after_first_layer):
|
with patch.object(executors, "_drive_sequential", _interrupt_after_first_layer):
|
||||||
@@ -350,12 +346,10 @@ class TestKeyboardInterrupt:
|
|||||||
def fn() -> int:
|
def fn() -> int:
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", fn),
|
||||||
px.TaskSpec("a", fn),
|
px.TaskSpec("b", fn, depends_on=("a",)),
|
||||||
px.TaskSpec("b", fn, depends_on=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _interrupt(*args: Any, **kwargs: Any) -> None:
|
async def _interrupt(*args: Any, **kwargs: Any) -> None:
|
||||||
raise KeyboardInterrupt
|
raise KeyboardInterrupt
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ def _fn() -> None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _make_result(
|
def _make_result( # noqa: PLR0913
|
||||||
name: str,
|
name: str,
|
||||||
status: TaskStatus = TaskStatus.SUCCESS,
|
status: TaskStatus = TaskStatus.SUCCESS,
|
||||||
value: Any = None,
|
value: Any = None,
|
||||||
@@ -27,10 +27,10 @@ def _make_result(
|
|||||||
attempts: int = 1,
|
attempts: int = 1,
|
||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
"""构造测试用 TaskResult。"""
|
"""构造测试用 TaskResult。"""
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn, depends_on=depends_on, soft_depends_on=soft_depends_on)
|
spec: TaskSpec[Any] = TaskSpec(name, _fn, depends_on=depends_on, soft_depends_on=soft_depends_on)
|
||||||
start = datetime(2024, 1, 1, 0, 0, 0)
|
start = datetime(2024, 1, 1, 0, 0, 0)
|
||||||
end = datetime(2024, 1, 1, 0, 0, 1) if status != TaskStatus.PENDING else None
|
end = datetime(2024, 1, 1, 0, 0, 1) if status != TaskStatus.PENDING else None
|
||||||
return TaskResult[Any](
|
return TaskResult(
|
||||||
spec=spec,
|
spec=spec,
|
||||||
status=status,
|
status=status,
|
||||||
value=value,
|
value=value,
|
||||||
@@ -507,12 +507,10 @@ class TestRunReportIntegration:
|
|||||||
return 1
|
return 1
|
||||||
|
|
||||||
# continue_on_error=True 使失败不抛异常而写入报告(success 保持 True)
|
# continue_on_error=True 使失败不抛异常而写入报告(success 保持 True)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("boom", boom, continue_on_error=True),
|
||||||
px.TaskSpec("boom", boom, continue_on_error=True),
|
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
|
||||||
px.TaskSpec("downstream", downstream, depends_on=("boom",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
# continue_on_error 不影响 success 标志,手动标记以触发诊断
|
# continue_on_error 不影响 success 标志,手动标记以触发诊断
|
||||||
report.success = False
|
report.success = False
|
||||||
|
|||||||
+21
-31
@@ -565,12 +565,10 @@ class TestDagComposition:
|
|||||||
def read_all(find_logs: list[Path]) -> list[str]:
|
def read_all(find_logs: list[Path]) -> list[str]:
|
||||||
return [read_text(p) for p in find_logs]
|
return [read_text(p) for p in find_logs]
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("find_logs", fn=find_logs, outputs={"paths": "$"}),
|
||||||
px.TaskSpec("find_logs", fn=find_logs, outputs={"paths": "$"}),
|
px.TaskSpec("read_all", fn=read_all, depends_on=("find_logs",)),
|
||||||
px.TaskSpec("read_all", fn=read_all, depends_on=("find_logs",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
assert report.success
|
assert report.success
|
||||||
@@ -592,12 +590,10 @@ class TestDagComposition:
|
|||||||
def copy_all(find_sources: list[Path]) -> list[Path]:
|
def copy_all(find_sources: list[Path]) -> list[Path]:
|
||||||
return [copy(p, tmp_path / "dst") for p in find_sources]
|
return [copy(p, tmp_path / "dst") for p in find_sources]
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("find_sources", fn=find_sources, outputs={"files": "$"}),
|
||||||
px.TaskSpec("find_sources", fn=find_sources, outputs={"files": "$"}),
|
px.TaskSpec("copy_all", fn=copy_all, depends_on=("find_sources",)),
|
||||||
px.TaskSpec("copy_all", fn=copy_all, depends_on=("find_sources",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
assert report.success
|
assert report.success
|
||||||
@@ -619,12 +615,10 @@ class TestDagComposition:
|
|||||||
def checksums(find_bins: list[Path]) -> dict[str, str]:
|
def checksums(find_bins: list[Path]) -> dict[str, str]:
|
||||||
return {p.name: sha256(p) for p in find_bins}
|
return {p.name: sha256(p) for p in find_bins}
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("find_bins", fn=find_bins, outputs={"paths": "$"}),
|
||||||
px.TaskSpec("find_bins", fn=find_bins, outputs={"paths": "$"}),
|
px.TaskSpec("checksums", fn=checksums, depends_on=("find_bins",)),
|
||||||
px.TaskSpec("checksums", fn=checksums, depends_on=("find_bins",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
assert report.success
|
assert report.success
|
||||||
@@ -643,12 +637,10 @@ class TestDagComposition:
|
|||||||
assert write_data == len("dag-content")
|
assert write_data == len("dag-content")
|
||||||
return read_text(target)
|
return read_text(target)
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("write_data", fn=write_data, outputs={"bytes": "$"}),
|
||||||
px.TaskSpec("write_data", fn=write_data, outputs={"bytes": "$"}),
|
px.TaskSpec("read_back", fn=read_back, depends_on=("write_data",)),
|
||||||
px.TaskSpec("read_back", fn=read_back, depends_on=("write_data",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
assert report.success
|
assert report.success
|
||||||
@@ -671,13 +663,11 @@ class TestDagComposition:
|
|||||||
content = f"files: {summarize['count']}, size: {summarize['total_size']}"
|
content = f"files: {summarize['count']}, size: {summarize['total_size']}"
|
||||||
return write_text(tmp_path / "summary.txt", content)
|
return write_text(tmp_path / "summary.txt", content)
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("find_files", fn=find_files, outputs={"paths": "$"}),
|
||||||
px.TaskSpec("find_files", fn=find_files, outputs={"paths": "$"}),
|
px.TaskSpec("summarize", fn=summarize, depends_on=("find_files",)),
|
||||||
px.TaskSpec("summarize", fn=summarize, depends_on=("find_files",)),
|
px.TaskSpec("write_summary", fn=write_summary, depends_on=("summarize",)),
|
||||||
px.TaskSpec("write_summary", fn=write_summary, depends_on=("summarize",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
assert report.success
|
assert report.success
|
||||||
|
|||||||
+48
-66
@@ -21,7 +21,7 @@ def _fn() -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
|
def _spec(name: str, deps: tuple[str, ...] = ()) -> TaskSpec[Any]:
|
||||||
return TaskSpec[Any](name, _fn, depends_on=deps)
|
return TaskSpec(name, _fn, depends_on=deps)
|
||||||
|
|
||||||
|
|
||||||
def _result(
|
def _result(
|
||||||
@@ -34,7 +34,7 @@ def _result(
|
|||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
"""构造带时间戳的 TaskResult."""
|
"""构造带时间戳的 TaskResult."""
|
||||||
end = start + timedelta(seconds=duration) if duration > 0 else start
|
end = start + timedelta(seconds=duration) if duration > 0 else start
|
||||||
return TaskResult[Any](
|
return TaskResult(
|
||||||
spec=_spec(name),
|
spec=_spec(name),
|
||||||
status=status,
|
status=status,
|
||||||
value=None,
|
value=None,
|
||||||
@@ -46,7 +46,7 @@ def _result(
|
|||||||
|
|
||||||
def _skipped_result(name: str, reason: str = "skip") -> TaskResult[Any]:
|
def _skipped_result(name: str, reason: str = "skip") -> TaskResult[Any]:
|
||||||
"""构造 SKIPPED 结果(无时间戳)."""
|
"""构造 SKIPPED 结果(无时间戳)."""
|
||||||
return TaskResult[Any](
|
return TaskResult(
|
||||||
spec=_spec(name),
|
spec=_spec(name),
|
||||||
status=TaskStatus.SKIPPED,
|
status=TaskStatus.SKIPPED,
|
||||||
reason=reason,
|
reason=reason,
|
||||||
@@ -95,13 +95,11 @@ class TestProfileReportConstruction:
|
|||||||
report.results["a"] = _result("a", start, 1.0)
|
report.results["a"] = _result("a", start, 1.0)
|
||||||
report.results["b"] = _result("b", start + timedelta(seconds=1), 2.0)
|
report.results["b"] = _result("b", start + timedelta(seconds=1), 2.0)
|
||||||
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.5)
|
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.5)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
_spec("c", deps=("b",)),
|
||||||
_spec("c", deps=("b",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -138,13 +136,11 @@ class TestProfileReportConstruction:
|
|||||||
report.results["a"] = _result("a", start, 1.0)
|
report.results["a"] = _result("a", start, 1.0)
|
||||||
report.results["b"] = _result("b", start, 3.0)
|
report.results["b"] = _result("b", start, 3.0)
|
||||||
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.0)
|
report.results["c"] = _result("c", start + timedelta(seconds=3), 1.0)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b"),
|
||||||
_spec("b"),
|
_spec("c", deps=("a", "b")),
|
||||||
_spec("c", deps=("a", "b")),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -192,12 +188,10 @@ class TestWaitTime:
|
|||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
report.results["a"] = _result("a", start, 1.0)
|
report.results["a"] = _result("a", start, 1.0)
|
||||||
report.results["b"] = _result("b", start + timedelta(seconds=1.5), 1.0)
|
report.results["b"] = _result("b", start + timedelta(seconds=1.5), 1.0)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -210,12 +204,10 @@ class TestWaitTime:
|
|||||||
report.results["a"] = _result("a", start, 2.0)
|
report.results["a"] = _result("a", start, 2.0)
|
||||||
# b 在 a 还没完成时就开始(不应该但可能发生)
|
# b 在 a 还没完成时就开始(不应该但可能发生)
|
||||||
report.results["b"] = _result("b", start + timedelta(seconds=1), 1.0)
|
report.results["b"] = _result("b", start + timedelta(seconds=1), 1.0)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -228,12 +220,10 @@ class TestWaitTime:
|
|||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
report.results["a"] = _result("a", start, 1.0)
|
report.results["a"] = _result("a", start, 1.0)
|
||||||
report.results["b"] = _skipped_result("b")
|
report.results["b"] = _skipped_result("b")
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -251,14 +241,12 @@ class TestCriticalPath:
|
|||||||
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
|
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
|
||||||
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
|
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
|
||||||
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
|
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
_spec("c", deps=("a",)),
|
||||||
_spec("c", deps=("a",)),
|
_spec("d", deps=("b", "c")),
|
||||||
_spec("d", deps=("b", "c")),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -291,7 +279,7 @@ class TestParallelism:
|
|||||||
def test_no_timestamps_zero_parallelism(self) -> None:
|
def test_no_timestamps_zero_parallelism(self) -> None:
|
||||||
"""所有任务无时间戳:并行度为 0."""
|
"""所有任务无时间戳:并行度为 0."""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
report.results["a"] = TaskResult[Any](spec=_spec("a"), status=TaskStatus.SUCCESS)
|
report.results["a"] = TaskResult(spec=_spec("a"), status=TaskStatus.SUCCESS)
|
||||||
graph = px.Graph.from_specs([_spec("a")])
|
graph = px.Graph.from_specs([_spec("a")])
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
@@ -405,14 +393,12 @@ class TestQueries:
|
|||||||
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
|
report.results["b"] = _result("b", start + timedelta(seconds=1), 3.0)
|
||||||
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
|
report.results["c"] = _result("c", start + timedelta(seconds=1), 1.0)
|
||||||
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
|
report.results["d"] = _result("d", start + timedelta(seconds=4), 1.0)
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
_spec("a"),
|
||||||
_spec("a"),
|
_spec("b", deps=("a",)),
|
||||||
_spec("b", deps=("a",)),
|
_spec("c", deps=("a",)),
|
||||||
_spec("c", deps=("a",)),
|
_spec("d", deps=("b", "c")),
|
||||||
_spec("d", deps=("b", "c")),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|
||||||
@@ -551,13 +537,11 @@ class TestIntegrationWithRun:
|
|||||||
time.sleep(0.01) # 确保任务有实际耗时,避免 duration 极小导致并行度计算为 0
|
time.sleep(0.01) # 确保任务有实际耗时,避免 duration 极小导致并行度计算为 0
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", slow),
|
||||||
px.TaskSpec("a", slow),
|
px.TaskSpec("b", slow, depends_on=("a",)),
|
||||||
px.TaskSpec("b", slow, depends_on=("a",)),
|
px.TaskSpec("c", slow, depends_on=("a",)),
|
||||||
px.TaskSpec("c", slow, depends_on=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
@@ -576,13 +560,11 @@ class TestIntegrationWithRun:
|
|||||||
time.sleep(0.05)
|
time.sleep(0.05)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", slow),
|
||||||
px.TaskSpec("a", slow),
|
px.TaskSpec("b", slow),
|
||||||
px.TaskSpec("b", slow),
|
px.TaskSpec("c", slow),
|
||||||
px.TaskSpec("c", slow),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="thread", max_workers=3)
|
report = px.run(graph, strategy="thread", max_workers=3)
|
||||||
|
|
||||||
profile = ProfileReport.from_report(report, graph)
|
profile = ProfileReport.from_report(report, graph)
|
||||||
|
|||||||
+20
-24
@@ -20,7 +20,7 @@ def _fn() -> int:
|
|||||||
return 1
|
return 1
|
||||||
|
|
||||||
|
|
||||||
def _make_result(
|
def _make_result( # noqa: PLR0913
|
||||||
name: str = "a",
|
name: str = "a",
|
||||||
status: TaskStatus = TaskStatus.SUCCESS,
|
status: TaskStatus = TaskStatus.SUCCESS,
|
||||||
value: Any = 42,
|
value: Any = 42,
|
||||||
@@ -32,11 +32,11 @@ def _make_result(
|
|||||||
reason: str | None = None,
|
reason: str | None = None,
|
||||||
) -> TaskResult[Any]:
|
) -> TaskResult[Any]:
|
||||||
"""构造测试用 TaskResult 实例."""
|
"""构造测试用 TaskResult 实例."""
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any](name, _fn, tags=tags, depends_on=depends_on)
|
spec: TaskSpec[Any] = TaskSpec(name, _fn, tags=tags, depends_on=depends_on)
|
||||||
start = datetime(2024, 1, 1, 0, 0, 0)
|
start = datetime(2024, 1, 1, 0, 0, 0)
|
||||||
# 用 timedelta 精确表达秒数,避免 int() 截断小数
|
# 用 timedelta 精确表达秒数,避免 int() 截断小数
|
||||||
end = start + timedelta(seconds=duration) if duration else None
|
end = start + timedelta(seconds=duration) if duration else None
|
||||||
return TaskResult[Any](
|
return TaskResult(
|
||||||
spec=spec,
|
spec=spec,
|
||||||
status=status,
|
status=status,
|
||||||
value=value,
|
value=value,
|
||||||
@@ -98,7 +98,7 @@ class TestRunReportSummary:
|
|||||||
def test_summary_with_none_duration(self) -> None:
|
def test_summary_with_none_duration(self) -> None:
|
||||||
"""未开始/未结束的任务 duration 为 None,不应计入总时长."""
|
"""未开始/未结束的任务 duration 为 None,不应计入总时长."""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
|
spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
|
||||||
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.FAILED)
|
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.FAILED)
|
||||||
s = report.summary()
|
s = report.summary()
|
||||||
assert s["total_duration_seconds"] == 0.0
|
assert s["total_duration_seconds"] == 0.0
|
||||||
@@ -135,8 +135,8 @@ class TestRunReportDescribe:
|
|||||||
def test_describe_no_duration(self) -> None:
|
def test_describe_no_duration(self) -> None:
|
||||||
"""无耗时的任务应显示为 '-'."""
|
"""无耗时的任务应显示为 '-'."""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
|
spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
|
||||||
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
|
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
|
||||||
desc = report.describe()
|
desc = report.describe()
|
||||||
assert "-" in desc # duration 显示为 "-"
|
assert "-" in desc # duration 显示为 "-"
|
||||||
|
|
||||||
@@ -181,8 +181,8 @@ class TestRunReportQueries:
|
|||||||
def test_durations_no_duration(self) -> None:
|
def test_durations_no_duration(self) -> None:
|
||||||
"""无时长的任务应返回 0.0."""
|
"""无时长的任务应返回 0.0."""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
|
spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
|
||||||
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
|
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
|
||||||
durs = report.durations()
|
durs = report.durations()
|
||||||
assert durs["a"] == 0.0
|
assert durs["a"] == 0.0
|
||||||
|
|
||||||
@@ -257,8 +257,8 @@ class TestRunReportToDict:
|
|||||||
def test_to_dict_no_timestamps(self) -> None:
|
def test_to_dict_no_timestamps(self) -> None:
|
||||||
"""未开始/未结束的任务 started_at/finished_at/duration 应为 None。"""
|
"""未开始/未结束的任务 started_at/finished_at/duration 应为 None。"""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
|
spec: TaskSpec[Any] = TaskSpec("a", _fn) # type: ignore[arg-type]
|
||||||
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
|
report.results["a"] = TaskResult(spec=spec, status=TaskStatus.PENDING)
|
||||||
d = report.to_dict()
|
d = report.to_dict()
|
||||||
entry = d["results"][0]
|
entry = d["results"][0]
|
||||||
assert entry["started_at"] is None
|
assert entry["started_at"] is None
|
||||||
@@ -439,7 +439,7 @@ class TestRunReportToHtml:
|
|||||||
"""任务名含 HTML 特殊字符应被转义。"""
|
"""任务名含 HTML 特殊字符应被转义。"""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
malicious = "<script>evil()</script>"
|
malicious = "<script>evil()</script>"
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any](malicious, _fn)
|
spec: TaskSpec[Any] = TaskSpec(malicious, _fn)
|
||||||
result: TaskResult[Any] = TaskResult(
|
result: TaskResult[Any] = TaskResult(
|
||||||
spec=spec,
|
spec=spec,
|
||||||
status=TaskStatus.SUCCESS,
|
status=TaskStatus.SUCCESS,
|
||||||
@@ -538,7 +538,7 @@ class TestRunReportToHtml:
|
|||||||
"""无时间戳时应渲染占位符。"""
|
"""无时间戳时应渲染占位符。"""
|
||||||
report = px.RunReport()
|
report = px.RunReport()
|
||||||
# 构造无时间戳的结果
|
# 构造无时间戳的结果
|
||||||
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn)
|
spec: TaskSpec[Any] = TaskSpec("a", _fn)
|
||||||
result: TaskResult[Any] = TaskResult(
|
result: TaskResult[Any] = TaskResult(
|
||||||
spec=spec,
|
spec=spec,
|
||||||
status=TaskStatus.SUCCESS,
|
status=TaskStatus.SUCCESS,
|
||||||
@@ -652,12 +652,10 @@ class TestRunReportSerializationIntegration:
|
|||||||
def double(extract: list[int]) -> list[int]:
|
def double(extract: list[int]) -> list[int]:
|
||||||
return [x * 2 for x in extract]
|
return [x * 2 for x in extract]
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("extract", extract, tags=("ingest",)),
|
||||||
px.TaskSpec("extract", extract, tags=("ingest",)),
|
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
|
||||||
px.TaskSpec("double", double, depends_on=("extract",), tags=("transform",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
|
|
||||||
# 序列化 → 反序列化
|
# 序列化 → 反序列化
|
||||||
@@ -753,12 +751,10 @@ class TestRunId:
|
|||||||
def task_b(a: int) -> int:
|
def task_b(a: int) -> int:
|
||||||
return a * 2
|
return a * 2
|
||||||
|
|
||||||
graph = px.Graph.from_specs(
|
graph = px.Graph.from_specs([
|
||||||
[
|
px.TaskSpec("a", task_a),
|
||||||
px.TaskSpec("a", task_a),
|
px.TaskSpec("b", task_b, depends_on=("a",)),
|
||||||
px.TaskSpec("b", task_b, depends_on=("a",)),
|
])
|
||||||
]
|
|
||||||
)
|
|
||||||
report = px.run(graph, strategy="sequential")
|
report = px.run(graph, strategy="sequential")
|
||||||
profile = report.profile(graph)
|
profile = report.profile(graph)
|
||||||
assert profile.total_duration >= 0
|
assert profile.total_duration >= 0
|
||||||
|
|||||||
@@ -133,6 +133,7 @@ class TestShMocked:
|
|||||||
|
|
||||||
def test_sh_called_process_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
|
def test_sh_called_process_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
|
||||||
"""CalledProcessError 被捕获,打印中文错误并返回 None。"""
|
"""CalledProcessError 被捕获,打印中文错误并返回 None。"""
|
||||||
|
|
||||||
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||||
raise subprocess.CalledProcessError(returncode=1, cmd=cmd)
|
raise subprocess.CalledProcessError(returncode=1, cmd=cmd)
|
||||||
|
|
||||||
@@ -145,6 +146,7 @@ class TestShMocked:
|
|||||||
|
|
||||||
def test_sh_file_not_found_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
|
def test_sh_file_not_found_error_returns_none(self, monkeypatch: Any, capsys: Any) -> None:
|
||||||
"""FileNotFoundError 被捕获,打印中文错误并返回 None。"""
|
"""FileNotFoundError 被捕获,打印中文错误并返回 None。"""
|
||||||
|
|
||||||
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||||
raise FileNotFoundError("[Errno 2] No such file")
|
raise FileNotFoundError("[Errno 2] No such file")
|
||||||
|
|
||||||
@@ -156,6 +158,75 @@ class TestShMocked:
|
|||||||
assert "命令未找到" in captured.out
|
assert "命令未找到" in captured.out
|
||||||
|
|
||||||
|
|
||||||
|
class TestShCheckFalse:
|
||||||
|
"""check=False 模式:非零返回码返回 CompletedProcess(不返回 None)。"""
|
||||||
|
|
||||||
|
def test_check_false_nonzero_returns_completed_process(self) -> None:
|
||||||
|
"""check=False 时非零返回码返回 CompletedProcess 而非 None。"""
|
||||||
|
result = sh(["false"], check=False, label="容忍命令")
|
||||||
|
assert result is not None
|
||||||
|
assert result.returncode != 0
|
||||||
|
|
||||||
|
def test_check_false_success_returns_completed_process(self) -> None:
|
||||||
|
"""check=False 时成功也返回 CompletedProcess。"""
|
||||||
|
result = sh(["true"], check=False)
|
||||||
|
assert result is not None
|
||||||
|
assert result.returncode == 0
|
||||||
|
|
||||||
|
def test_check_false_file_not_found_returns_none(self, capsys: Any) -> None:
|
||||||
|
"""check=False 时命令未找到仍返回 None(FileNotFoundError 非返回码)。"""
|
||||||
|
result = sh(["this_command_does_not_exist_xyz"], check=False, label="缺失")
|
||||||
|
assert result is None
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "缺失失败" in captured.out
|
||||||
|
assert "命令未找到" in captured.out
|
||||||
|
|
||||||
|
def test_check_false_capture_stdout(self) -> None:
|
||||||
|
"""check=False + capture=True 可读取 stdout(含非零返回码)。"""
|
||||||
|
result = sh("echo out; exit 3", capture=True, check=False, label="捕获")
|
||||||
|
assert result is not None
|
||||||
|
assert result.returncode == 3
|
||||||
|
assert result.stdout == "out\n"
|
||||||
|
|
||||||
|
|
||||||
|
class TestShTimeout:
|
||||||
|
"""timeout 参数:超时返回 None。"""
|
||||||
|
|
||||||
|
def test_timeout_expired_returns_none(self, capsys: Any) -> None:
|
||||||
|
"""超时时返回 None + 打印中文错误。"""
|
||||||
|
# sleep 5 但 timeout=0.1,必超时
|
||||||
|
result = sh(["sleep", "5"], timeout=0.1, label="睡眠")
|
||||||
|
assert result is None
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "睡眠超时" in captured.out
|
||||||
|
assert "timeout=0.1" in captured.out
|
||||||
|
|
||||||
|
def test_timeout_not_expired_returns_completed_process(self) -> None:
|
||||||
|
"""未超时正常返回 CompletedProcess。"""
|
||||||
|
result = sh(["true"], timeout=10, label="快速")
|
||||||
|
assert result is not None
|
||||||
|
assert result.returncode == 0
|
||||||
|
|
||||||
|
def test_timeout_passed_to_subprocess(self, monkeypatch: Any) -> None:
|
||||||
|
"""timeout 参数应传递给 subprocess.run。"""
|
||||||
|
captured: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def fake_run(cmd: Any, **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||||
|
captured["timeout"] = kwargs.get("timeout")
|
||||||
|
return subprocess.CompletedProcess(cmd, 0)
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
sh(["ls"], timeout=30)
|
||||||
|
assert captured["timeout"] == 30
|
||||||
|
|
||||||
|
def test_timeout_with_check_false(self, capsys: Any) -> None:
|
||||||
|
"""check=False + timeout 超时仍返回 None。"""
|
||||||
|
result = sh(["sleep", "5"], check=False, timeout=0.1, label="限时")
|
||||||
|
assert result is None
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "限时超时" in captured.out
|
||||||
|
|
||||||
|
|
||||||
class TestShExported:
|
class TestShExported:
|
||||||
"""验证 px.sh 顶层导出。"""
|
"""验证 px.sh 顶层导出。"""
|
||||||
|
|
||||||
|
|||||||
@@ -354,7 +354,7 @@ class TestTaskSpecVerbose:
|
|||||||
|
|
||||||
def test_verbose_default_is_false(self) -> None:
|
def test_verbose_default_is_false(self) -> None:
|
||||||
"""verbose 默认应为 False."""
|
"""verbose 默认应为 False."""
|
||||||
spec: px.TaskSpec[Any] = px.TaskSpec[Any]("a", cmd=[*ECHO_CMD, "hi"])
|
spec: px.TaskSpec[Any] = px.TaskSpec("a", cmd=[*ECHO_CMD, "hi"])
|
||||||
assert spec.verbose is False
|
assert spec.verbose is False
|
||||||
|
|
||||||
def test_verbose_true_prints_command(self, capsys: pytest.CaptureFixture[str]) -> None:
|
def test_verbose_true_prints_command(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
@@ -367,7 +367,7 @@ class TestTaskSpecVerbose:
|
|||||||
|
|
||||||
def test_verbose_false_silent(self, capsys: pytest.CaptureFixture[str]) -> None:
|
def test_verbose_false_silent(self, capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
"""verbose=False 时不应打印命令信息."""
|
"""verbose=False 时不应打印命令信息."""
|
||||||
graph = px.Graph.from_specs([px.TaskSpec[Any]("echo", cmd=[*ECHO_CMD, "silent"], verbose=False)])
|
graph = px.Graph.from_specs([px.TaskSpec("echo", cmd=[*ECHO_CMD, "silent"], verbose=False)])
|
||||||
_ = px.run(graph, strategy="sequential")
|
_ = px.run(graph, strategy="sequential")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "执行命令" not in captured.out
|
assert "执行命令" not in captured.out
|
||||||
@@ -390,7 +390,7 @@ class TestTaskSpecVerbose:
|
|||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
graph = px.Graph.from_specs([px.TaskSpec[Any]("ls", cmd=ECHO_CMD, cwd=Path(tmpdir), verbose=True)])
|
graph = px.Graph.from_specs([px.TaskSpec("ls", cmd=ECHO_CMD, cwd=Path(tmpdir), verbose=True)])
|
||||||
_ = px.run(graph, strategy="sequential")
|
_ = px.run(graph, strategy="sequential")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "工作目录" in captured.out
|
assert "工作目录" in captured.out
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ skipsdist = true
|
|||||||
[testenv]
|
[testenv]
|
||||||
uv_sync = true
|
uv_sync = true
|
||||||
deps =
|
deps =
|
||||||
.[dev]
|
.[dev,docs,fast,office]
|
||||||
commands =
|
commands =
|
||||||
pytest -m "not slow" {posargs}
|
pytest -m "not slow" {posargs}
|
||||||
passenv =
|
passenv =
|
||||||
|
|||||||
Reference in New Issue
Block a user