57 Commits

Author SHA1 Message Date
zhou 3f9c52e6f1 bump version to 0.2.12
Release / build (push) Failing after 23m3s
Release / publish-pypi (push) Has been skipped
Release / release (push) Has been skipped
2026-06-28 18:56:42 +08:00
zhou 8fadf6edd8 fix(executors): 修复进程池退出阻塞问题
1. 新增_shutdown_process_pool函数,在run()结束时主动关闭进程池
2. 通过atexit注册兜底清理逻辑,防止进程池泄漏
3. 先调用shutdown(wait=False)通知管理线程退出,再强制kill工作进程,避免Python退出时threading._shutdown等待join导致数秒阻塞
4. 新增测试规范文档说明测试相关规则
2026-06-28 18:56:27 +08:00
zhou abc1152538 refactor(cli): 统一使用@px.task装饰器定义任务,重构任务注册和别名管理
1. 将folderzip/folderback/gittool中的旧TaskSpec定义替换为@px.task装饰器
2. 重构pymake模块,将maturin_build_cmd转为常量定义,合并别名配置
3. 精简测试文件中的冗余测试用例
2026-06-28 18:12:30 +08:00
zhou 5e561b4b3a refactor: 重构CliRunner,新增cmd工厂函数优化任务定义
1. 新增cmd工厂函数,简化TaskSpec创建并自动推导名称
2. 重构CliRunner,将graphs参数替换为tasks+aliases,支持扁平任务注册与别名映射
3. 替换所有cli工具中的旧版任务定义方式,使用新API简化代码
4. 补充对应测试用例,适配新的运行器API
2026-06-28 17:52:52 +08:00
zhou 40f641611b feat: 新增多项核心功能并优化默认执行策略
1.  将CliRunner默认执行策略从sequential改为dependency
2.  新增RunReport的任务状态查询和时长统计方法
3.  实现task装饰器并补充executor参数文档
4.  新增进程池执行器支持CPU密集型任务
5.  新增Graph.chain链式构建和add_subgraph子图合并功能
6.  新增流式任务传递、进程池执行、命名空间等多类测试用例
7.  补充tests目录路径导入配置
2026-06-28 15:10:15 +08:00
zhou 232e7293d9 refactor(system): 简化write_file实现,使用pathlib替代手动文件操作。 2026-06-28 11:20:58 +08:00
zhou a1bae58e56 refactor: 优化日志配置与代码细节
1. 统一使用__name__替代硬编码的logger名称
2. 使用pathlib替代os.path处理程序名
3. 细化异常捕获并优化日志打印格式
4. 收紧文件内容检查的异常捕获范围
2026-06-28 10:57:51 +08:00
zhou cbc7cc0a75 docs: 拆分测试规范到独立技能文档并更新主规范
将原python-standards.md中的测试章节迁移到新建的pyflowx-testing/SKILL.md,更新主规范指向新文档,同时整理优化了整体文档结构与内容。
2026-06-28 10:19:26 +08:00
zhou d0ff7d7b4d docs: 更新 README 与新增 Python 开发规范文档
本次提交大幅完善了 PyFlowX 的 README 文档,新增了四种执行策略、软依赖、并发限制、任务钩子等多项特性说明,补充了任务模板、图组合、缓存键等新功能的使用示例,同时更新了执行参数、执行策略对照表与模块结构文档。另外新增了 .trae/rules/python-standards.md 规范文档,统一了项目的代码风格、类型检查、测试编写等开发标准。
2026-06-28 09:34:45 +08:00
zhou d154f67ce0 +trae ignore 2026-06-28 08:44:23 +08:00
zhou 9999071119 refactor(executors): 重构执行器逻辑,移除重复mixin并优化分层排序
主要变更:
1.  将任务跳过/重试逻辑从类mixin改为模块级函数,减少代码重复
2.  优化_graph.layers()的前置校验逻辑,统一在run入口执行
3.  重构存储过期检查API,移除废弃的_expired方法
4.  优化TaskSpec.cache_key异常处理,增加指定异常捕获并记录警告
5.  修复verbose模式下的事件回调逻辑,正确触发RUNNING事件
6.  调整测试用例以适配新的API和行为变更
2026-06-28 08:25:15 +08:00
zhou bdd70e9c43 refactor: 重构项目代码结构,拆分职责模块
1. 抽离图组合逻辑到pyflowx.compose,原graph.py仅保留单图DAG逻辑
2. 抽离命令执行逻辑到pyflowx.command,移除task.py内的_run_command
3. 重构上下文签名缓存,优化性能
4. 移除废弃的utils.perf_timer相关代码
5. 为JSONBackend添加batch批量落盘优化
6. 调整导入路径与公开API,更新测试用例
7. 简化条件判断逻辑,移除冗余代码
2026-06-28 02:28:38 +08:00
zhou c15b38516a bump version to 0.2.11
Release / build (push) Failing after 29m15s
Release / publish-pypi (push) Has been skipped
Release / release (push) Has been skipped
2026-06-27 23:08:32 +08:00
zhou 7d4e8a40ce refactor(cli): 重构CLI模块结构,整理系统工具与开发工具
1. 将原cli根目录下的clearscreen、taskkill、which工具迁移到cli/system子目录
2. 新增cli/dev子目录并添加envdev环境配置工具
3. 更新pyproject.toml中的脚本入口点映射
4. 调整tests/cli下的测试文件导入路径
5. 整理tasks/system.py的__all__导出顺序
2026-06-27 22:01:02 +08:00
zhou 1b2d6d6a2c chore: 更新依赖配置并移除 pysnooper 2026-06-27 21:53:20 +08:00
zhou df890f0f16 chore: 移除独立的envpy和envrs命令,合并功能到envdev
将原来envpy和envrs的环境配置功能整合到envdev命令中,删除了冗余的独立CLI模块和测试文件,统一管理Python、Conda和Rust的环境配置。
2026-06-27 21:22:36 +08:00
zhou b62a544569 chore: 调整Python版本与依赖适配,新增性能报告测试与工具函数
1.  将Python版本从3.13降级到3.11
2.  为typing-extensions添加版本适配标记
3.  简化dev依赖组,移除pysnooper
4.  重构perf_timer,提取_generate_report独立函数
5.  新增性能报告生成与测试用例
2026-06-27 20:47:29 +08:00
zhou d58fc5536e chore: 发布 pyflowx 0.2.10,新增性能计时器与多项重构
1. 新增 perf_timer 工具与配套测试用例
2. 重构任务条件跳过逻辑,优化失败条件展示
3. 重构 Graph 子图生成逻辑,提取公共依赖修剪函数
4. 重构条件模块,统一条件名称与失败原因获取逻辑
5. 重构存储后端,提取 TTL 共享逻辑并优化实现
6. 重构执行器模块,使用 Mixin 复用代码,拆分任务与层执行逻辑
7. 删除冗余的 which 命令测试文件
8. 更新依赖锁文件
2026-06-27 20:15:35 +08:00
zhou c3b86b603d bump version to 0.2.10
Release / build (push) Failing after 11m58s
Release / publish-pypi (push) Has been skipped
Release / release (push) Has been skipped
2026-06-27 19:41:24 +08:00
zhou 327bd6e069 feat: 优化条件不满足时的报错信息展示
1. 新增格式化reason的工具函数统一处理报错信息
2. 支持从条件函数中提取自定义的失败原因
3. 完善NOT和OR条件的失败原因传递逻辑
4. 移除任务跳过的冗余打印输出
2026-06-27 19:40:51 +08:00
zhou 22f8d2110d chore: add pysnooper dev dependency and update configs
1. add pysnooper>=1.2.3 to dev dependencies in pyproject.toml and uv.lock
2. update type hints in task.py from Iterator to Generator
3. add more PyPI mirrors and update envdev.py comments and checks
4. fix trailing whitespace in executors.py
2026-06-27 19:35:11 +08:00
zhou 2a1f2f7175 refactor(envdev, conditions): 重构环境配置脚本,新增平台和文件条件检查
1. 移除废弃的envqt命令入口
2. 新增IS_WINDOWS、IS_LINUX等平台检测条件
3. 新增FILE_CONTENT_EXISTS文件内容检查条件
4. 使用内置条件替代硬编码的平台判断
5. 为任务添加条件控制,仅在符合场景时执行
2026-06-27 18:29:40 +08:00
zhou 9d033e1c0b refactor(system): add setenv_group and write_file task helpers
1. 为setenv和which函数添加正确的返回类型注解
2. 新增setenv_group批量设置环境变量的任务组
3. 新增write_file写入文件的任务工具函数
4. 更新__all__导出所有新增的工具函数

feat(cli/envdev): rewrite envdev cli with proper config and args
1. 重构环境开发CLI脚本,使用argparse替换原有TypedDict配置
2. 新增Python和Conda镜像源选择参数
3. 自动生成并写入Python pip和Conda配置文件
4. 优化任务依赖和命名,统一使用系统工具函数
2026-06-27 17:12:53 +08:00
zhou 336f7b7292 -envqt 2026-06-27 16:45:02 +08:00
zhou 65dcbcbf62 bump version to 0.2.9
Release / build (push) Failing after 16m3s
Release / publish-pypi (push) Has been skipped
Release / release (push) Has been skipped
2026-06-27 16:42:10 +08:00
zhou 7fa97a01e3 test(executors): add future annotations import to edge case test file
为测试文件添加from __future__ import annotations以支持更规范的类型注解写法
2026-06-27 16:33:24 +08:00
zhou 83da5135d0 test: add tests for graph all_deps and defaults inheritance
- add test_all_deps_combines_hard_and_soft to verify all_deps returns correct hard+soft deps in order
- add multiple tests for GraphDefaults field inheritance, including normal inheritance and non-override of custom values
2026-06-27 16:32:34 +08:00
zhou 7463a60649 test: 修复代码检查警告并优化测试用例
1. 为测试代码添加pyrefly忽略注释解决类型检查警告
2. 优化lambda参数命名为通配符符合PEP8规范
3. 增加断言检查任务函数非空并修正参数传递
4. 统一环境变量测试的命名和清理逻辑
2026-06-27 16:26:56 +08:00
zhou 87dd010342 test: add multiple new test cases and update python version
1. update .python-version from 3.11 to 3.13
2. add tests for IS_RUNNING and DIR_EXISTS conditions
3. add graph-related tests including string ref parsing, mermaid output, GraphComposer and compose function
4. add storage backend TTL tests for both MemoryBackend and JSONBackend
5. add new system task tests for clr, reset_icon_cache, setenv and which
6. add comprehensive task spec tests including soft dependencies, retry policy, context managers and task template
7. add executor edge case tests for various scenarios
2026-06-27 16:17:05 +08:00
zhou bdfee7bee4 ci: 简化CI/CD配置,移除冗余测试步骤和覆盖率上报
重构了GitHub Actions工作流,合并重复的CI任务,移除了预发布测试环节、多余的格式检查和安全审计任务,精简了 tox 测试命令与矩阵配置,同时删除了本地 tox 配置中的覆盖率和测试结果上报参数,优化整体流水线效率。
2026-06-27 16:00:44 +08:00
zhou b954fb1622 build(coverage): 调整coverage配置,新增cli目录到忽略白名单并提高达标阈值至95%
修改了pyproject.toml中的coverage配置:将src/pyflowx/cli/*加入omit排除列表,同时将测试覆盖率达标阈值从80提升至95
2026-06-27 15:57:00 +08:00
zhou a7b7a82dff ci: 完善CI/CD流程,添加测试覆盖率与并行测试配置
1. 为tox测试命令添加并行执行、覆盖率报告和JUnit结果输出
2. 拆分CI工作流为lint、格式检查、类型检查、安全审计、多矩阵测试和覆盖率汇总
3. 新增release前的预测试步骤,让build依赖测试通过
4. 移除低效的依赖策略测速测试用例
5. 配置多Python版本跨平台测试矩阵并上传测试 artifacts
2026-06-27 15:53:08 +08:00
zhou 40f0478146 bump version to 0.2.8
Release / build (push) Failing after 31s
Release / publish-pypi (push) Has been skipped
Release / release (push) Has been skipped
2026-06-27 15:44:09 +08:00
zhou b808b880f8 ci(github workflow): simplify release workflow
移除了冗余的预检查步骤、简化了工作流配置,更新了action版本并优化了版本提取和产物处理逻辑
2026-06-27 15:43:55 +08:00
zhou e073ff41ee ci: simplify and merge CI jobs
1. 合并lint和typecheck任务为一个job,减少重复的环境配置步骤
2. 精简测试矩阵,只保留Python3.8和3.13两个版本
3. 移除不必要的覆盖率上传和聚合检查job
4. 简化工作流触发条件,只保留push和手动触发
2026-06-27 15:43:24 +08:00
zhou ea0c51de5e build: 调整llm依赖条件并更新pyflowx版本
1. 为llm依赖添加linux平台限制
2. 移除uv.lock中的前置发布版本配置项
3. 将pyflowx版本从0.2.6升级到0.2.7
2026-06-27 15:33:33 +08:00
zhou 2b3f4b82d3 bump version to 0.2.7
Release / Pre-release Check (push) Failing after 35s
Release / Build Artifacts (push) Has been skipped
Release / Publish to PyPI (push) Has been skipped
Release / Publish Release (push) Has been skipped
2026-06-27 15:23:48 +08:00
zhou 1e23c48efc chore: 调整Python版本并修复类型注解语法
1. 将.python-version中的Python版本从3.13改为3.11
2. 移除过时的from __future__ import annotations导入
3. 把字符串形式的泛型类型注解替换为原生语法格式
2026-06-27 14:35:51 +08:00
zhou 5c8ec281ff refactor: 重构重试策略、条件函数与上下文注入逻辑
主要变更:
1. 替换旧retries参数为RetryPolicy配置
2. 重构条件函数,支持上下文参数与动态依赖判断
3. 更新上下文注入逻辑,支持软依赖与更清晰的注入描述
4. 新增sglang CLI命令与相关配置
5. 格式化代码统一列表与参数写法
6. 更新文档与测试用例适配新API
2026-06-27 14:33:54 +08:00
zhou 6f01cde8ac feat(cli): add ModelScopeHub model download command line tool
add new msdown CLI command powered by modelscope SDK via uvx, support downloading models/datasets/spaces from ModelScopeHub to local directory
2026-06-27 11:20:50 +08:00
zhou bcd189ae60 refactor(graph,runner): 重构引用解析逻辑,拆分GraphComposer
1.  抽离CliRunner中的引用解析逻辑为GraphComposer类,分离图数据与组合职责
2.  取消Graph的frozen修饰,简化内部属性修改逻辑
3.  重构任务执行与跳过逻辑,合并重复代码并优化条件求值时机
4.  调整TaskSpec为普通dataclass,移除不必要的replace重建
5.  修复测试用例中skip_if_missing的断言值
6.  重构命令执行逻辑,抽离为模块级函数避免闭包捕获参数
2026-06-27 10:13:52 +08:00
zhou 20c4fb87c5 feat: 添加上游任务跳过豁免、进程检查条件及相关优化
1. 新增allow_upstream_skip参数支持任务不跟随上游跳过
2. 新增IS_RUNNING内置条件检查进程运行状态
3. 调整skip_if_missing默认值为False
4. 补充跳过任务的事件上报和verbose打印
5. 优化reset_icon_cache示例任务使用新特性
6. 更新测试用例匹配默认参数变更
2026-06-27 09:24:22 +08:00
zhou a98eb6e344 feat(conditions): add DIR_EXISTS builtin condition, update system tasks
- add Path type import and DIR_EXISTS condition method
- update reset_icon_cache tasks to add directory existence checks
- simplify explorer restart command and add installation check
2026-06-27 09:04:58 +08:00
zhou 752ff618b2 refactor(system tasks): 格式化代码并新增重启资源管理器任务
将原有的单行TaskSpec调用拆分为多行格式化写法,同时补充restart_explorer任务到任务列表中
2026-06-27 09:00:22 +08:00
zhou f15f235ecf chore: 发布v0.2.6版本,新增重置图标缓存工具
1. 新增reseticon命令行工具用于重置Windows图标缓存
2. 重构平台常量导出逻辑,移除顶层直接导出的IS_*变量
3. 为系统任务相关的TaskSpec添加verbose输出
4. 优化测试用例的列表格式和平台条件写法
5. 更新依赖锁定文件和项目配置
2026-06-27 08:45:48 +08:00
zhou 9d79cddbd6 refactor(system cli): 统一命名风格并新增图标缓存重置工具
1. 将系统任务的大写命名改为蛇形命名:CLR→clr, SETENV→setenv, WHICH→which
2. 更新对应cli工具的导入和调用代码
3. 新增restart_icon_cache命令行工具和reset_icon_cache系统任务
2026-06-27 08:29:30 +08:00
zhou af9aab395a bump version to 0.2.6
Release / Pre-release Check (push) Failing after 31s
Release / Build Artifacts (push) Has been skipped
Release / Publish to PyPI (push) Has been skipped
Release / Publish Release (push) Has been skipped
2026-06-27 00:58:05 +08:00
zhou 6f334fde73 refactor(cli/hfdownload): 重构下载工具,改用SETENV和modelscope命令
1.  移除本地setenvs函数,改用封装好的SETENV任务
2.  替换hf下载命令为modelscope下载命令
3.  优化参数命名和默认下载目录逻辑
4.  简化任务编排代码
2026-06-27 00:39:17 +08:00
zhou 2ccd84ac3b chore(tasks): remove unused task module doc and export code
d
2026-06-27 00:14:07 +08:00
zhou ec30af3edb refactor(system tasks): 重构系统任务模块并完善功能
1. 为CLR、SETENV、WHICH三个函数添加完整的类型注解和文档字符串
2. 重构SETENV支持两种环境变量设置模式
3. 优化WHICH的跨平台适配和输出格式
4. 新增模块级文档说明并导出所有任务函数
2026-06-26 23:34:53 +08:00
zhou 10bbc07118 refactor(cli): 重构清屏和which命令实现
1. 提取清屏、设置环境变量、命令查找逻辑到system任务模块
2. 统一命令行工具的任务实现方式,减少重复代码
3. 修正pyproject.toml中的cli命令名拼写错误
4. 移除过时的测试用例代码
2026-06-26 23:27:45 +08:00
zhou 194cf3c343 chore(pyflowx): 升级pyflowx版本到0.2.5
仅更新了依赖锁定文件中的pyflowx版本号
2026-06-26 22:49:03 +08:00
zhou 1880cd7a34 bump version to 0.2.5
Release / Build Artifacts (push) Has been skipped
Release / Publish to PyPI (push) Has been skipped
Release / Publish Release (push) Has been skipped
Release / Pre-release Check (push) Failing after 31s
2026-06-26 21:59:45 +08:00
zhou d43c9e4044 bump version to 0.2.4 2026-06-26 21:57:53 +08:00
zhou 22ac9fc4dd test: 完善多份测试用例的类型标注与校验逻辑
1. 为多个测试函数补充pytest.CaptureFixture[str]类型注解
2. 为graphlib类型声明文件补全方法参数类型
3. 为pdftool测试的mock函数添加Any类型标注
4. 新增数据库连接非空校验断言
5. 优化emlmanager测试的字典展开格式与修复decode测试bug
6. 为gittool测试添加命令类型列表校验
7. 为envrs测试添加pyrefly忽略注释
2026-06-26 21:57:44 +08:00
zhou 7ded8df05e refactor: 整理代码格式并修复部分类型和依赖问题
1. 调整task.py的TypeVar导入和默认值
2. 格式化多处列表和参数写法,统一括号风格
3. 为pdftool.py添加pyrefly忽略注释修复类型警告
4. 为emlmanager.py添加数据库连接断言和检查
5. 修正hfdownload.py的depends_on参数为元组格式
2026-06-26 21:52:44 +08:00
zhou fd282db28f refactor: 整理代码格式与项目结构,修复命令检查bug
1. 重构多处列表展开写法,统一代码格式风格
2. 修复executors.py中命令不存在时的类型判断bug
3. 删除废弃的envlinux.py并替换为envdev.py,更新CLI入口配置
4. 为storage.py的后端方法添加override装饰器
5. 移除空的cli/__init__.py冗余导入
6. 更新pyproject.toml依赖与配置项
7. 精简测试用例代码
2026-06-26 21:45:06 +08:00
87 changed files with 13961 additions and 3895 deletions
+17 -96
View File
@@ -3,127 +3,48 @@ name: CI
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# ─────────────────────────────────────────────────────────────
# lint:代码风格与格式检查(单平台即可)
# ─────────────────────────────────────────────────────────────
lint:
name: Lint (ruff)
lint-and-typecheck:
name: Lint & Typecheck
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/checkout@v4
- name: 安装 uv
uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@v5
with:
version: latest
enable-cache: true
cache-dependency-glob: uv.lock
- name: 设置 Python 3.13
uses: actions/setup-python@v5
- uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: 安装依赖
run: uv sync --extra dev --frozen
- run: uv sync
- run: uv run ruff check src tests
- run: uv run pyrefly check .
- name: Ruff 检查
run: uv run ruff check src tests
# ─────────────────────────────────────────────────────────────
# typecheckpyrefly 严格类型检查
# ─────────────────────────────────────────────────────────────
typecheck:
name: Typecheck (pyrefly)
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: 安装 uv
uses: astral-sh/setup-uv@v5
with:
version: latest
enable-cache: true
cache-dependency-glob: uv.lock
- name: 设置 Python 3.13
uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: 安装依赖
run: uv sync --extra dev --frozen
- name: pyrefly 严格类型检查
run: uv run pyrefly check .
# ─────────────────────────────────────────────────────────────
# test:多平台 × 多 Python 版本矩阵测试 + 覆盖率
# ─────────────────────────────────────────────────────────────
test:
name: Test (${{ matrix.os }} / py${{ matrix.python-version }})
name: Test (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ ubuntu-latest, windows-latest, macos-latest ]
python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12', '3.13' ]
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/checkout@v4
- name: 安装 uv
uses: astral-sh/setup-uv@v5
- uses: astral-sh/setup-uv@v5
with:
version: latest
enable-cache: true
cache-dependency-glob: uv.lock
- name: 设置 Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
python-version: |
3.8
3.13
- name: 安装依赖
run: uv sync --extra dev --frozen
- name: 运行测试
run: uv run pytest -v --cov=pyflowx --cov-report=xml --cov-report=term-missing
- name: 上传覆盖率
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13'
uses: actions/upload-artifact@v4
with:
name: coverage-${{ matrix.os }}-py${{ matrix.python-version }}
path: coverage.xml
retention-days: 7
# ─────────────────────────────────────────────────────────────
# 聚合:所有检查通过后才标记完成
# ─────────────────────────────────────────────────────────────
ci-pass:
name: CI Pass
runs-on: ubuntu-latest
needs: [ lint, typecheck, test ]
if: always()
steps:
- name: 检查依赖任务结果
if: ${{ needs.lint.result != 'success' || needs.typecheck.result != 'success' || needs.test.result != 'success' }}
run: |
echo "lint: ${{ needs.lint.result }}"
echo "typecheck: ${{ needs.typecheck.result }}"
echo "test: ${{ needs.test.result }}"
exit 1
- name: 全部通过
run: echo "✅ 所有 CI 检查通过"
- run: uvx tox run -e py38,py313
+21 -153
View File
@@ -2,192 +2,60 @@ name: Release
on:
push:
tags:
- 'v*.*.*'
workflow_dispatch:
inputs:
tag:
description: '发布版本号(如 v0.1.0'
required: true
type: string
tags: ['v*.*.*']
permissions:
contents: write
# Trusted Publishing (OIDC) 上传 PyPI 所需
id-token: write
jobs:
# ─────────────────────────────────────────────────────────────
# 预检:版本号校验 + 与 pyproject.toml 一致性检查
# ─────────────────────────────────────────────────────────────
pre-check:
name: Pre-release Check
build:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.meta.outputs.version }}
tag: ${{ steps.meta.outputs.tag }}
version: ${{ steps.version.outputs.version }}
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
fetch-depth: 0
- name: 解析版本号
id: meta
run: |
if [ -n "${{ inputs.tag }}" ]; then
TAG="${{ inputs.tag }}"
else
TAG="${GITHUB_REF#refs/tags/}"
fi
# 去除前缀 v
VERSION="${TAG#v}"
echo "tag=$TAG" >> $GITHUB_OUTPUT
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "发布版本: $VERSION (tag: $TAG)"
- name: 校验版本号格式
run: |
VERSION="${{ steps.meta.outputs.version }}"
if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$'; then
echo "❌ 版本号格式错误: $VERSION(应为 x.y.z 或 x.y.z-rc.n"
exit 1
fi
- name: 校验 pyproject.toml 版本一致
run: |
# 精确提取 [project] 段的 version 字段(避免匹配到依赖的 version)
PY_VERSION=$(awk '/^\[project\]/{f=1} f&&/^version[[:space:]]*=/{gsub(/[" ]/,"",$3); print $3; exit}' pyproject.toml)
echo "pyproject.toml version: $PY_VERSION"
if [ "$PY_VERSION" != "${{ steps.meta.outputs.version }}" ]; then
echo "❌ pyproject.toml 版本($PY_VERSION) 与 tag 版本(${{ steps.meta.outputs.version }}) 不一致"
echo "请先更新 pyproject.toml 中的 version 字段"
exit 1
fi
# ─────────────────────────────────────────────────────────────
# 构建:wheel + sdist(纯 Python,单平台即可)
# ─────────────────────────────────────────────────────────────
build:
name: Build Artifacts
needs: pre-check
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: 安装 uv
uses: astral-sh/setup-uv@v5
with:
version: latest
enable-cache: true
- name: 设置 Python 3.13
uses: actions/setup-python@v5
- uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: 安装依赖
run: uv sync --extra dev --frozen
- run: uv build
- name: 构建 wheel + sdist
run: uv build
- id: version
run: echo "version=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
- name: 校验产物
run: |
echo "待上传产物:"
ls -la dist/
if [ -z "$(ls -A dist/*.whl dist/*.tar.gz 2>/dev/null)" ]; then
echo "❌ 未找到 wheel 或 sdist 产物"
exit 1
fi
- name: 上传构建产物
uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v7
with:
name: dist
path: dist/*
retention-days: 30
path: dist/
# ─────────────────────────────────────────────────────────────
# 发布:上传到 PyPITrusted Publishing / OIDC
# ─────────────────────────────────────────────────────────────
publish-pypi:
name: Publish to PyPI
needs: [pre-check, build]
needs: build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/project/pyflowx/${{ needs.pre-check.outputs.version }}
permissions:
id-token: write
environment: pypi
steps:
- name: 下载构建产物
uses: actions/download-artifact@v4
- uses: actions/download-artifact@v8
with:
name: dist
path: dist
- name: 上传到 PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
attestations: true
- uses: pypa/gh-action-pypi-publish@release/v1
# ─────────────────────────────────────────────────────────────
# 发布:创建 GitHub Release
# ─────────────────────────────────────────────────────────────
release:
name: Publish Release
needs: [pre-check, build, publish-pypi]
needs: [build, publish-pypi]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: 下载构建产物
uses: actions/download-artifact@v4
- uses: actions/download-artifact@v8
with:
name: dist
path: assets
path: dist
- name: 整理发布产物
run: |
ls -la assets/
- name: 生成 Release Notes
id: notes
run: |
{
echo "## pyflowx ${{ needs.pre-check.outputs.version }}"
echo ""
echo "### 下载"
echo ""
echo "- **Wheel**: \`pyflowx-${{ needs.pre-check.outputs.version }}-py3-none-any.whl\`"
echo "- **源码包**: \`pyflowx-${{ needs.pre-check.outputs.version }}.tar.gz\`"
echo ""
echo "### 安装"
echo ""
echo '```bash'
echo "pip install pyflowx==${{ needs.pre-check.outputs.version }}"
echo '```'
echo ""
echo "### 完整变更日志"
} > RELEASE_NOTES.md
{
echo "content<<EOF"
cat RELEASE_NOTES.md
echo "EOF"
} >> $GITHUB_OUTPUT
- name: 创建 GitHub Release
uses: softprops/action-gh-release@v2
- uses: softprops/action-gh-release@v2
with:
tag_name: ${{ needs.pre-check.outputs.tag }}
name: pyflowx ${{ needs.pre-check.outputs.version }}
body: ${{ steps.notes.outputs.content }}
files: assets/*
draft: false
prerelease: ${{ contains(needs.pre-check.outputs.version, '-') }}
files: dist/*
generate_release_notes: true
+1 -1
View File
@@ -1 +1 @@
3.13
3.11
+15
View File
@@ -0,0 +1,15 @@
# PYTHON
.coverage
.pytest_cache/
.ruff_cache/
.tox/
.venv/
__pycache__/
# NODEJS
node_modules/
# IDE
.idea
.trae
.vscode
+11
View File
@@ -0,0 +1,11 @@
---
alwaysApply: true
scene: git_message
---
在此处编写规则,自定义 AI 生成提交信息的风格。
## 提交信息格式
- 提交信息必须使用中文。
- 提交信息必须包含变更的类型(例如 "fix"、"feat"、"refactor" 等)。
- 提交信息必须尽简洁明了,不要超过一段落。
+157
View File
@@ -0,0 +1,157 @@
# Python 开发规范
本规范结合 Python 最佳实践,作为编写与审查 Python 代码的统一标准。
详细操作指南见 `.agents/skills/` 下相应技能。
## 工具链(以 pyproject.toml 为准)
| 工具 | 用途 | 配置要点 |
|------|------|---------|
| **ruff** | lint + format | `line-length=120``target-version="py38"` |
| **pyrefly** | 类型检查 | `preset="strict"``python-version="3.8"` |
| **pytest** | 测试 | `asyncio_default_fixture_loop_scope="function"`marker `slow` |
| **coverage** | 覆盖率 | `branch=true``fail_under=95``concurrency=["thread"]` |
| **pre-commit** | 提交前检查 | ruff `--fix` + trailing-whitespace + end-of-file-fixer |
验证(每次修改后必做):
```bash
uvx --from pyflowx pymake tc
uvx --from pyflowx pymake cov
```
## 兼容性
- **最低 Python 3.8**:用 `from __future__ import annotations` 延迟注解求值;
按版本用 `typing.List`(3.8) → 内置泛型(3.9) → `X | Y`(3.10) → `typing.override`(3.12)。
- **版本守卫**`if sys.version_info >= (3, X):` 引入高版本 API;低版本回退分支加 `# pragma: no cover`
- **零运行时依赖**:仅依赖标准库(3.8 需 `graphlib_backport``typing-extensions`)。
新增依赖须审慎,优先用标准库。
## 类型注解
- **公共 API 必须有完整类型注解**,包括返回类型;私有函数也应有注解。
- 泛型用 `TypeVar`PEP 696 `default=` 仅 3.13+ 标准库支持,3.83.12 用 `typing_extensions.TypeVar`
- `Mapping`/`Sequence` 用于只读参数,`dict`/`list` 用于可变返回。
- `Any` 仅用于真正动态场景(如 `Context` 跨任务异构映射);任务内部类型必须完全静态。
- 禁用裸 `# type: ignore`;确需时加具体规则码(如 `# type: ignore[union-attr]`)。
- **`TYPE_CHECKING` 守卫**:仅类型检查需要的导入放 `if TYPE_CHECKING:` 块内,避免循环依赖。
- **类型收窄**:用 `assert isinstance(x, Y)` 辅助 pyrefly 推断;`cast()` 仅用于类型系统无法表达的场景。
## 数据结构
- **不可变优先**:配置/描述类用 `@dataclass(frozen=True)`;可变类属性标注 `RUF012` 豁免。
- **缓存**:实例级用 `functools.cached_property`,按参数键控用 `functools.lru_cache`
不可哈希参数需 try/except 回退。修改被缓存数据源后必须手动清空缓存。
- **抽象基类**:接口用 `abc.ABC` + `@abstractmethod`(如 `StateBackend`)。
- **枚举**:状态/标志值用 `enum.Enum`(如 `TaskStatus`),禁止裸字符串/魔术数字;枚举值用 `UPPER_SNAKE`
- **`__repr__`**:可变类实现 `__repr__`(含关键字段);`frozen=True` dataclass 自动生成。
## 模块与导入
- **单一职责**:每模块只做一件事(`task.py` 数据结构、`executors.py` 执行、`command.py` 命令、`compose.py` 组合)。禁止跨职责边界。
- **导入顺序**ruff isort):`__future__` → 标准库 → 第三方 → 本地,各组间空行。
- **惰性导入**:仅为打破循环依赖时使用,函数体内导入并注释说明;顶层导入是默认。
- **`__all__`**:定义 `__all__` 显式声明导出符号,位置仅次于 `__future__` 之后。
- **禁用 star imports**`from x import *` 污染命名空间、破坏类型检查(`__init__.py` 聚合经 `__all__` 控制为例外)。
- **避免 `utils.py`/`helpers.py`**:按职责归入对应模块。
## 函数设计
- **模块级函数优于 Mixin**:共享逻辑用模块级函数,类只持有状态与薄方法。
- **静态方法慎用**:纯函数直接放模块级。
- **参数 ≤ 5 个**为宜;超出用 dataclass 封装参数对象。
- **单一职责**:一个函数做一件事;过长函数考虑拆分。
- **异常范围要窄**:只捕获预期异常(如 `(TypeError, ValueError, KeyError, AttributeError)`),
**禁止** `except Exception` 掩盖 bug;捕获后至少 `logger.warning` 记录。
- **可变默认参数**`def f(x=[])` 是经典坑;用 `None` 哨兵或 `field(default_factory=list)`
## 异常处理
- **自定义异常家族**:继承公共基类(如 `PyFlowXError`),按错误场景分类。
- **异常包装**`raise NewError(...) from exc` 保留因果链。
- **不要吞异常**:捕获后必须处理(记录/包装/重抛),禁止空 `except: pass`
- **钩子/回调异常**:第三方回调异常仅记录,不影响主流程。
## 并发与线程安全
- **进程全局状态**`os.environ`/`os.chdir`)在并发场景下必须用全局锁(`threading.RLock`)序列化。
- **条件评估不可有可变状态**:组合条件(NOT/AND/OR)不得修改共享 `_reason`,避免竞态。
- **批量 I/O**:循环内多次写盘改为批量一次(`contextmanager` 包裹延迟落盘)。
- **信号量限流**`concurrency_key` + `Semaphore` 按组限流。
## 测试
详细操作指南见 `.agents/skills/pyflowx-testing` 技能。硬约束:
- **覆盖率 ≥ 95%**branch coverage),不得下降。
- **公共 API 优先测试**:用公共接口(`has`/`get`),不访问私有方法;
故障注入等场景可临时访问私有属性,docstring 注明原因。
- **命名**`test_<被测对象>_<场景>`
- **断言**:原生 `assert x == 1`,禁用 `self.assertEqual``pytest.raises` 必填 `match=`
- **Mock 优先级**`monkeypatch` > 内联 stub > `unittest.mock` > `pytest-mock`
禁用 `@patch` 装饰器、`mock.patch.object` 上下文、`pytest-mock``mocker` fixture。
- **fixture**`tmp_path`/`monkeypatch`/`capsys` 优先;autouse 仅全局必需时用。
- **slow 标记**:耗时测试加 `@pytest.mark.slow`CI 可 `-m "not slow"` 跳过。
- **测试代码也跑 ruff**`tests/**` 忽略 `ARG001`/`ARG002`
## 代码风格
- **行宽 120**ruff formatter 处理)。
- **docstring**:公共 API 必须有;中文叙述 + 中文注释是本项目既有风格。
- **打印和日志**:使用中文打印和日志,避免使用英文。
- **命名**`snake_case` 函数/变量,`PascalCase` 类,`UPPER_SNAKE` 常量,`_leading_underscore` 私有。
- **字符串引号**:ruff 默认双引号。
- **末尾单 `\n`**、**无尾随空格**pre-commit 强制)。
- **不用 emoji**:除非用户明确要求。
## Pythonic 风格
- **`is` 比较 `None`/`True`/`False`**:单例用 `is`,值用 `==`PEP 8 E711/E712)。
- **EAFP 优于 LBYL**:先尝试再处理异常,而非先检查再执行(避免竞态窗口)。
- **truthiness**`if items:` 优于 `if len(items) > 0:`
- **字符串格式化**:首选 f-string;`%` 仅用于 `logging` 延迟格式化。
- **推导式**优于 `map`+`filter`> 2 层拆为显式循环。
- **`enumerate`** 替代 `range(len())`**`zip`** 并行迭代(3.10+ 用 `strict=True`)。
- **解包** `a, b = pair` 优于索引访问;忽略值用 `_`
- **海象运算符 `:=`**(3.8+):赋值+判断合一,但不滥用。
## 日志
- **`logging.getLogger(__name__)`**:每模块独立 logger,禁用 `print` 调试残留。
- **结构化上下文**`extra={...}` 传字段;`logger.warning("task %r failed: %s", name, exc)` 优于 f-string(延迟格式化)。
- **日志级别**`DEBUG` 诊断 / `INFO` 关键流程 / `WARNING` 可恢复异常 / `ERROR` 需人工介入。
- **禁止日志密码/密钥**:脱敏后再记录。
## 路径与资源
- **优先 `pathlib.Path`**`Path("a") / "b"` 而非 `os.path.join`ruff `PTH` 强制);
禁止字符串拼接路径。类型注解用 `Path`,边界 `str` 立即包装。
- **`with` 语句**:文件、锁、连接、临时目录一律用 `with``contextlib.contextmanager`
多资源用 `contextlib.ExitStack`
- **显式关闭**:长生命周期对象(连接池、线程池)实现 `close()`,但优先 `with`
- **批量操作**:循环内多次 acquire/release 改为批量一次。
## 安全
- **禁用 `eval`/`exec`**:处理不可信输入时绝不使用;用 `ast.literal_eval` 或专用解析器。
- **`subprocess`**:禁用 `shell=True` 除非命令完全可信;优先 `list[str]` 形式。
- **凭证不入仓**:密钥/token/密码放 `.env` 或环境变量,`.gitignore` 必须包含 `.env`
- **日志脱敏**:记录请求/响应时移除 `Authorization``password` 等字段。
- **依赖审计**`uv lock` 后审阅新增依赖,避免引入已知 CVE 的包。
## 性能要点
- **避免重复计算**:循环内查询应缓存或预构建映射(如 `{name: spec}`)。
- **避免双重查找**`has(k)` + `get(k)` 改为单次 `get(k)` + `KeyError` 回退。
- **统一校验**:入口校验一次,下游路径不重复(如 `run()` 统一 `validate()``layers()` 不再重复)。
- **事件 emit**:任务生命周期必须 emit `RUNNING``SUCCESS`/`FAILED`/`SKIPPED`
不要留死分支(`# pragma: no cover` 是清理信号,应激活或删除)。
## Git 与提交
- **不自动提交/push**:除非用户明确要求。
- **不修改 git config**。
- **不运行破坏性命令**`push --force`/`reset --hard`/`clean -f`)除非用户明确要求。
- **staging**:按文件名添加,不用 `git add -A`/`git add .`,避免误加敏感文件。
- **commit message**:简洁,聚焦"为什么"而非"是什么";遵循仓库既有风格。
+135
View File
@@ -0,0 +1,135 @@
---
name: "pyflowx-testing"
description: "PyFlowX 项目的测试编写规范与 mock 使用指南。在编写或审查测试、选择 mock 工具、设计 fixture、处理 asyncio 测试时调用。"
---
# PyFlowX 测试规范
本技能是 `.trae/rules/python-standards.md` 测试章节的详细展开。
规则文件仅保留硬约束指针,本文件提供完整操作指南。
## 总则
- **覆盖率 ≥ 95%**branch coverage),不得下降。
- **公共 API 优先测试**:测试用公共接口(`has`/`get`),不访问私有方法
(如 `_expired`)。兼容旧测试的私有方法应删除并迁移测试。
例外:`_store`/`_flush` 等内部状态在无法用公共 API 触发时(如模拟过期、
故障注入),可临时访问私有属性,并在 docstring 注明原因。
- **命名**`test_<被测对象>_<场景>`,如 `test_storage_key_cache_key_exception_returns_name`
- **每个测试一个断言重点**;多个断言要语义相关。
- **slow 标记**:耗时测试加 `@pytest.mark.slow`CI 可 `-m "not slow"` 跳过。
- **测试代码也跑 ruff**`tests/**` 忽略 `ARG001`/`ARG002`(未用 fixture 参数)。
- **断言风格**:用原生 `assert` + 比较运算符(`assert x == 1`),
不用 `self.assertEqual`pytest 会生成更清晰的 diff。
## Mock 工具选择(强制)
**优先级**`monkeypatch` > 内联 stub > `unittest.mock` > `pytest-mock`
| 场景 | 工具 | 示例 |
|------|------|------|
| 替换模块属性 / 环境变量 / 工作目录 | `monkeypatch` | `monkeypatch.setattr(subprocess, "run", fake_run)` |
| `os.environ["KEY"]` 临时设置 | `monkeypatch.setenv` | `monkeypatch.setenv("LOCALAPPDATA", "C:\\...")` |
| 切换 cwd | `monkeypatch.chdir` | `monkeypatch.chdir(tmp_path)` |
| 一次性 stub 函数 | 内联 lambda / 闭包 | `ran = []; monkeypatch.setattr(subprocess, "run", lambda *c, **__: ran.append(c))` |
| 复杂 spy(记录调用次数/参数/返回序列) | `unittest.mock.MagicMock` | 仅当 lambda 不足以表达时 |
| `with patch(...)` 上下文 | **禁用**(用 monkeypatch | monkeypatch 自动 teardown 更安全 |
**禁止**
- 不用 `pytest-mock``mocker` fixture(项目虽在 dev 依赖声明,但实际
测试代码未使用;为保持风格统一,新代码继续用 `monkeypatch`)。
- 不用 `unittest.mock.patch` 装饰器(`@patch("x.y")`),它隐藏依赖且
与 pytest fixture 模式不兼容;用 `monkeypatch.setattr` 替代。
- 不用 `mock.patch.object` 作为上下文管理器,除非被测代码本身就是
contextmanager(此时用 `monkeypatch.setattr` 仍更简单)。
## monkeypatch 使用规范
- **类型注解**fixture 参数标注 `monkeypatch: pytest.MonkeyPatch`
- **作用域**monkeypatch 自动在测试结束时撤销,**禁止**手动
`monkeypatch.setattr(x, "y", original)` 恢复(多余且容易遗漏)。
例外:在单个测试内需要中途恢复时,用 `monkeypatch.undo()` 全量撤销。
- **替换目标**:替换"被测代码看到的对象",而非全局对象本身。
- 错误:`monkeypatch.setattr("os.path.exists", fake)` —— 替换全局,影响其他模块。
- 正确:`monkeypatch.setattr(pyflowx.command.shutil, "which", fake)` ——
替换被测模块引用的 `shutil.which`
- **属性 vs 字符串路径**:优先属性访问形式 `monkeypatch.setattr(obj, "attr", val)`
而非字符串路径 `monkeypatch.setattr("pkg.mod.obj.attr", val)`
前者有 IDE 跳转与重构支持。
- **记录调用**:用闭包 `ran: list[tuple] = []` + `lambda *a, **k: ran.append((a, k))`
替代 `MagicMock`,可读性更好且无需导入。
## Stub 与 Spy 模式
- **轻量 stub**:内联定义 `class MockResult: returncode = 0; stdout = ""`
替代 `MagicMock(return_value=...)`,类型明确且不引入 mock 依赖。
- **状态收集**:闭包 + list 比 `mock.call_args_list` 更易断言:
```python
calls: list[list[str]] = []
def fake_run(cmd: list[str], **_: Any) -> MockResult:
calls.append(cmd)
return MockResult()
monkeypatch.setattr(subprocess, "run", fake_run)
assert calls == [["clear"]]
```
- **副作用序列**:需要按调用次数返回不同值时,用 `itertools.cycle` 或
手动计数器,而非 `side_effect=[...]`mock 专有 API)。
- **异常注入**`def raise_oserror(*a, **k): raise OSError("...")`
用 `pytest.raises(OSError)` 验证,而非 `side_effect=OSError`。
## 异常断言
- **`pytest.raises`**:必填 `match=` 正则(除非异常消息完全不可预测),
避免误捕获同类异常:
```python
with pytest.raises(StorageError, match="cannot write"):
b.save("a", 1)
```
- **异常链**:验证 `__cause__` 时用 `exc_info.value.__cause__`
确认 `raise X from Y` 因果链完整。
- **禁止** `try/except + assert False`:用 `pytest.raises` 替代。
## Fixture 规范
- **`tmp_path`**:处理临时文件,自动清理,禁止 `tempfile.mkdtemp()` 手动管理。
- **`monkeypatch`**:环境变量、cwd、模块属性 mock(见上)。
- **`capsys`/`capfd`**:捕获 stdout/stderr,验证日志或命令输出。
- **autouse fixture**:仅在全局必需时用(如 `conftest.py` 的
`packtool_tmp_workdir` 自动切到 tmp_path);否则显式声明参数。
- **fixture 命名**`snake_case`,描述"提供什么"而非"测试什么"
`sample_graph` 优于 `test_data`)。
- **fixture 作用域**:默认 `function``module`/`session` 仅当构造昂贵且
只读时,并加注释说明无副作用。
## asyncio 测试
- **fixture `loop_scope="function"`**pyproject 已配置默认值)。
- **async 测试**`async def test_x():`pytest-asyncio 自动驱动。
- **await 检查**:测试异步函数必须 `await` 结果,禁止仅验证返回 coroutine 对象。
- **异步 mock**:用 `AsyncMock`3.8+ 在 `unittest.mock`)或
`async def fake(): return value`,禁用 `MagicMock(return_value=coro)`。
## 参数化
- **`@pytest.mark.parametrize`**:用 `ids` 参数提供可读标识:
```python
@pytest.mark.parametrize(
("strategy", "expected_workers"),
[("sequential", 1), ("thread", 8), ("async", 1)],
ids=["seq", "thread-8", "async"],
)
```
- **参数命名**:参数元组用有意义名称,而非 `("a", "b")`。
- **组合爆炸**:参数组合 > 20 时拆分测试,避免单个测试函数臃肿。
## 测试组织
- **文件命名**`test_<被测模块>.py``test_storage.py` 对应 `storage.py`)。
- **类分组**:仅在测试逻辑强相关时用 `class TestXxx:` 分组;默认用模块级函数。
- **docstring**:每个测试函数一句话说明"测试什么场景",复杂场景补充"为什么"。
- **setup/teardown**:优先 fixture`setup_method`/`teardown_method` 仅在
无法用 fixture 表达时(罕见)。
+143 -20
View File
@@ -14,18 +14,25 @@ PyFlowX 把"任务依赖"这件事做到极致简单:**参数名就是依赖
## 特性
- **零样板** —— 参数名即依赖,框架自动注入上游结果
- **种执行策略** —— `sequential`(调试)/ `thread`I/O 密集同步)/ `async`I/O 密集异步)
- **种执行策略** —— `sequential`(调试)/ `thread`I/O 密集同步)/ `async`I/O 密集异步)/ `dependency`(依赖驱动,最大化并行)
- **类型安全** —— `TaskSpec[T]` 把返回类型一路传到 `RunReport`mypy strict 通过
- **DAG 校验** —— 构建时即时校验重名、缺失依赖、环
- **自动分层** —— Kahn 算法分组,同层任务可并行
- **重试与超时** —— 每个任务独立配置 `retries` `timeout`
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用
- **重试与超时** —— 每个任务独立配置 `RetryPolicy`max_attempts/delay/backoff/jitter/retry_on`timeout`
- **软依赖** —— `soft_depends_on` 仅用于上下文注入,不参与拓扑分层
- **并发限制** —— `concurrency_key` + `concurrency_limits` 按组限流
- **任务钩子** —— `TaskHooks`pre_run/post_run/on_failure)生命周期回调
- **断点续跑** —— `MemoryBackend` / `JSONBackend`,成功结果可缓存复用;`batch()` 批量落盘
- **缓存键** —— `cache_key` 函数基于输入计算稳定键,使不同输入产生独立缓存
- **命令任务** —— `cmd` 参数直接执行外部命令,支持列表/shell/可调用对象
- **条件执行** —— `conditions` 参数按平台、环境变量、应用安装等条件跳过任务
- **图组合** —— `compose` / `GraphComposer` 编程式展开多图字符串引用
- **任务模板** —— `task_template` 工厂批量生成相似 TaskSpec
- **图级默认值** —— `GraphDefaults` 统一配置 retry/timeout/concurrency 等
- **CLI 运行器** —— `CliRunner` 把多个图映射为命令行子命令,替代 Makefile
- **可观测** —— `on_event` 回调、`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
- **可观测** —— `on_event` 回调RUNNING/SUCCESS/FAILED/SKIPPED`dry_run` 预览、`verbose` 生命周期日志、Mermaid 可视化
- **零运行时依赖** —— 仅依赖标准库(3.8 需 `graphlib_backport`
- **95% 测试覆盖** —— 分支覆盖率>= 95%
- **97% 测试覆盖** —— 分支覆盖率 >= 95%
## 安装
@@ -67,23 +74,31 @@ print(report["double"]) # [2, 4, 6]
### TaskSpec —— 任务描述
`TaskSpec` 是不可变的任务描述符,是唯一需要配置的东西:
`TaskSpec` 是不可变的任务描述符`Generic[T]`,返回类型一路传到 `RunReport`,是唯一需要配置的东西:
```python
px.TaskSpec(
name="fetch_user", # 唯一标识
fn=fetch_user, # 同步或异步函数
cmd=["curl", "..."], # 或: 执行命令(覆盖 fn
depends_on=("auth",), # 依赖的任务名
depends_on=("auth",), # 依赖(参与拓扑分层)
soft_depends_on=("cache",), # 软依赖(仅注入,不参与分层)
args=(uid,), # 静态位置参数(追加在注入参数后)
kwargs={"timeout": 30}, # 静态关键字参数
retries=3, # 失败重试次数(0 = 仅一次)
retry=px.RetryPolicy(max_attempts=3, delay=1.0, backoff=2.0), # 重试策略
timeout=30.0, # 超时秒数(None = 不限制)
tags=("api", "user"), # 自由标签,用于子图过滤
conditions=(is_prod,), # 条件函数列表(全部为 True 才执行)
priority=10, # 同层内优先级(高优先执行,默认 0)
concurrency_key="db", # 并发分组键(配合 concurrency_limits 限流)
cache_key=lambda ctx: str(ctx.get("uid")), # 缓存键函数(不同输入独立缓存)
hooks=px.TaskHooks(pre_run=..., post_run=..., on_failure=...), # 生命周期钩子
cwd=Path("/tmp"), # 命令工作目录(仅 cmd 模式)
env={"DEBUG": "1"}, # 环境变量覆盖(fn 与 cmd 模式均生效)
verbose=True, # 打印命令输出(仅 cmd 模式)
skip_if_missing=True, # 命令不存在时自动跳过(仅 list[str] cmd
allow_upstream_skip=False, # 上游 SKIPPED/FAILED 时是否仍执行
continue_on_error=False, # 本任务失败是否不中断整体
)
```
@@ -97,18 +112,54 @@ px.TaskSpec(
### Graph —— DAG 构建
```python
graph = px.Graph.from_specs([...]) # 整批校验(推荐)
# 图级默认值:TaskSpec 字段为 None 时回退
defaults = px.GraphDefaults(retry=px.RetryPolicy(max_attempts=2), timeout=60.0)
graph = px.Graph.from_specs([...], defaults=defaults) # 整批校验(推荐)
# 或增量构建
graph = px.Graph()
graph = px.Graph(defaults=defaults)
graph.add(px.TaskSpec("a", fn_a))
graph.add(px.TaskSpec("b", fn_b, ("a",)))
graph.validate() # 显式校验(环检测)
graph.layers() # 拓扑分层
graph.layers() # 拓扑分层(run() 入口已统一校验,直接调用需自行先 validate)
graph.to_mermaid() # Mermaid 可视化
graph.describe() # 人类可读摘要
graph.subgraph(("api",)) # 按标签切片
graph.subgraph_by_names(("a", "b")) # 按名称切片
graph.map("fetch", [1, 2, 3], lambda i: TaskSpec(f"fetch_{i}", ...)) # 批量 fan-out
```
### 图组合 —— compose
`compose` / `GraphComposer` 把带字符串引用的多个图展开为纯 `Graph`
```python
graphs = {
"build": px.Graph.from_specs([px.TaskSpec("b", cmd=["echo", "b"])]),
"all": px.Graph.from_specs(["build", px.TaskSpec("t", cmd=["echo", "t"])]),
}
resolved = px.compose(graphs) # "all" 图中的 "build" 引用被展开
```
引用格式:`"command_name"`(整个图)或 `"command_name.task_name"`(特定任务)。
`CliRunner` 内部自动调用 `compose`
### 任务模板 —— task_template
`task_template` 工厂批量生成相似 TaskSpec:
```python
fetch = px.task_template(
fn=fetch_url,
retry=px.RetryPolicy(max_attempts=5),
timeout=30.0,
tags=("api",),
)
graph = px.Graph.from_specs([
fetch("users", url="https://api.example.com/users"),
fetch("posts", url="https://api.example.com/posts"),
])
```
### run —— 执行
@@ -116,12 +167,14 @@ graph.subgraph_by_names(("a", "b")) # 按名称切片
```python
report = px.run(
graph,
strategy="async", # sequential | thread | async
strategy="async", # sequential | thread | async | dependency
max_workers=8, # thread 策略的线程池大小
concurrency_limits={"db": 2}, # 按 concurrency_key 限流
dry_run=False, # True = 仅打印计划
verbose=False, # True = 打印任务生命周期日志
on_event=callback, # 状态转换回调
on_event=callback, # 状态转换回调RUNNING/SUCCESS/FAILED/SKIPPED
state=px.JSONBackend("state.json"), # 断点续跑后端
continue_on_error=False, # True = 单任务失败不中断整体
)
```
@@ -141,7 +194,7 @@ report.describe() # 人类可读报告
按顺序求值:
1. **标注为 `Context`** 的参数 → 接收完整上游结果映射
2. **名称匹配依赖** 的参数 → 接收该依赖的结果
2. **名称匹配依赖** 的参数 → 接收该依赖的结果(含软依赖,缺失时注入默认值)
3. **`**kwargs`** 参数 → 接收所有依赖结果(dict)
4. **`TaskSpec.args` / `kwargs`** → 为非依赖参数提供静态值
@@ -170,8 +223,11 @@ def fetch_user(uid: int) -> dict: # uid 来自 TaskSpec.args
| `sequential` | 串行 | 调试、CPU 密集 | 直接调用 | 事件循环 |
| `thread` | 线程池 | I/O 密集同步 | 线程池 | 不支持 |
| `async` | 事件循环 | I/O 密集异步 | 卸载到线程池 | 事件循环 |
| `dependency` | 依赖驱动 | 最大化并行度 | 卸载到线程池 | 事件循环 |
所有策略都遵循 `retries``timeout`、上下文注入、状态后端,并发出 `TaskEvent`
所有策略都遵循 `RetryPolicy``timeout`、上下文注入、状态后端`concurrency_limits`
并发出 `TaskEvent`RUNNING/SUCCESS/FAILED/SKIPPED)。`dependency` 策略无层屏障:
任务在其所有硬依赖完成后立即启动。
## 命令任务
@@ -275,12 +331,25 @@ python examples/async_aggregation.py
from pyflowx import JSONBackend
# 第一次运行:成功结果写入 state.json
backend = JSONBackend("state.json")
backend = JSONBackend("state.json", ttl=3600) # ttl 秒数,过期条目自动忽略
report = px.run(graph, strategy="sequential", state=backend)
# 第二次运行:已缓存任务自动跳过
# 第二次运行:已缓存任务自动跳过(状态为 SKIPPED
report = px.run(graph, strategy="sequential", state=backend)
# report.results 中缓存任务状态为 SKIPPED
```
`run()` 内部以 `backend.batch()` 包裹整个执行:所有 `save` 延迟到运行结束时统一落盘一次
`JSONBackend` 从 O(N²) 降为 O(N) 磁盘写入;`MemoryBackend` 为 no-op)。
**缓存键**:默认存储键为任务名。配置 `cache_key` 函数后,键为 `"name:cache_key_value"`
使不同输入产生独立缓存条目:
```python
px.TaskSpec(
"fetch_user",
fn=fetch_user,
cache_key=lambda ctx: str(ctx.get("uid")), # 不同 uid 独立缓存
)
```
## 错误处理
@@ -321,14 +390,52 @@ except px.PyFlowXError:
PyFlowX 专注于**单机 DAG 调度**的极致简洁,适合 ETL、数据处理、CI 流水线等场景。
## 高级特性
### 并发限制
`concurrency_key` 分组限流,避免压垮下游资源:
```python
graph = px.Graph.from_specs([
px.TaskSpec("q1", fn=query_db, concurrency_key="db"),
px.TaskSpec("q2", fn=query_db, concurrency_key="db"),
px.TaskSpec("q3", fn=query_db, concurrency_key="db"),
])
# 同一时刻最多 2 个 "db" 组任务运行
px.run(graph, strategy="async", concurrency_limits={"db": 2})
```
### 任务钩子
`TaskHooks` 在任务生命周期触发(异常仅记录,不影响任务状态):
```python
hooks = px.TaskHooks(
pre_run=lambda spec: print(f"start {spec.name}"),
post_run=lambda spec, value: print(f"done {spec.name}"),
on_failure=lambda spec, exc: alert(spec.name, exc),
)
px.TaskSpec("task", fn=work, hooks=hooks)
```
### 优先级
同层内按 `priority` 降序执行(稳定排序):
```python
px.TaskSpec("low", fn=work, priority=0)
px.TaskSpec("high", fn=work, priority=10) # 同层内先执行
```
## 开发
```bash
# 安装开发依赖
uv sync --extra dev
# 运行测试(含覆盖率)
uv run pytest --cov=pyflowx --cov-fail-under=100
# 运行测试(含覆盖率,阈值 95%
uv run pytest --cov=pyflowx --cov-fail-under=95
# 类型检查
uv run mypy
@@ -338,6 +445,22 @@ uv run ruff check src tests examples
uv run ruff format --check src tests examples
```
## 模块结构
| 模块 | 职责 |
|------|------|
| `task.py` | 纯数据结构:`TaskSpec``RetryPolicy``TaskHooks``TaskStatus` |
| `graph.py` | DAG 构建、校验、分层、可视化 |
| `compose.py` | 多图组合:`GraphComposer` / `compose` |
| `context.py` | 上下文注入:参数名→依赖解析 |
| `command.py` | 命令执行:`run_command`list/shell/Callable |
| `conditions.py` | 条件执行:内置条件与组合器 |
| `executors.py` | 执行器与 `run` 入口:四种策略共享模块级辅助 |
| `storage.py` | 状态后端:`MemoryBackend` / `JSONBackend`batch flush |
| `runner.py` | CLI 运行器:`CliRunner` |
| `report.py` | 运行结果:`RunReport` / `TaskResult` |
| `errors.py` | 错误家族:`PyFlowXError` 子类 |
## 许可证
MIT
+22 -14
View File
@@ -6,43 +6,48 @@ classifiers = [
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Topic :: Software Development :: Libraries :: Application Frameworks",
]
dependencies = ["graphlib_backport >= 1.0.0; python_version < '3.9'"]
dependencies = [
"graphlib_backport >= 1.0.0; python_version < '3.9'",
"typing-extensions>=4.13.2; python_version < '3.10'",
]
description = "Lightweight, type-safe DAG task scheduler with multi-strategy execution."
keywords = ["async", "dag", "scheduler", "task", "workflow"]
license = { text = "MIT" }
name = "pyflowx"
readme = "README.md"
requires-python = ">=3.8"
version = "0.2.3"
version = "0.2.12"
[project.scripts]
autofmt = "pyflowx.cli.autofmt:main"
bumpversion = "pyflowx.cli.bumpversion:main"
clr = "pyflowx.cli.clearscreen:main"
emlman = "pyflowx.cli.emlmanager:main"
envlinux = "pyflowx.cli.envlinux:main"
envpy = "pyflowx.cli.envpy:main"
envqt = "pyflowx.cli.envqt:main"
envrs = "pyflowx.cli.envrs:main"
filedate = "pyflowx.cli.filedate:main"
filelvl = "pyflowx.cli.filelevel:main"
foldback = "pyflowx.cli.folderback:main"
foldzip = "pyflowx.cli.folderzip:main"
gitt = "pyflowx.cli.gittool:main"
hfdown = "pyflowx.cli.hfdownload:main"
lscalc = "pyflowx.cli.lscalc:main"
msdown = "pyflowx.cli.llm.msdownload:main"
packtool = "pyflowx.cli.packtool:main"
pdftool = "pyflowx.cli.pdftool:main"
piptool = "pyflowx.cli.piptool:main"
pymake = "pyflowx.cli.pymake:main"
reseticon = "pyflowx.cli.reseticoncache:main"
scrcap = "pyflowx.cli.screenshot:main"
sglang = "pyflowx.cli.llm.sglang:main"
sshcopy = "pyflowx.cli.sshcopyid:main"
taskk = "pyflowx.cli.taskkill:main"
wch = "pyflowx.cli.which:main"
# dev
envdev = "pyflowx.cli.dev.envdev:main"
# system
clr = "pyflowx.cli.system.clearscreen:main"
taskk = "pyflowx.cli.system.taskkill:main"
wch = "pyflowx.cli.system.which:main"
[project.optional-dependencies]
dev = [
@@ -60,6 +65,9 @@ dev = [
"tox-uv>=1.13.1",
"tox>=4.25.0",
]
llm = [
"sglang[all]==0.5.10rc0; python_version >= '3.10' and sys_platform == 'linux'",
]
office = [
"pillow>=10.4.0",
"pymupdf>=1.24.11",
@@ -85,12 +93,12 @@ packages = ["src/pyflowx"]
pyflowx = { workspace = true }
[dependency-groups]
dev = ["pyflowx[dev,office]"]
dev = ["pyflowx[dev,office,llm]"]
[tool.coverage.run]
branch = true
concurrency = ["thread"]
omit = ["src/pyflowx/examples/*", "tests/*"]
omit = ["src/pyflowx/cli/*", "src/pyflowx/examples/*", "tests/*"]
source = ["pyflowx"]
[tool.coverage.report]
@@ -100,7 +108,7 @@ exclude_lines = [
"pragma: no cover",
"raise NotImplementedError",
]
fail_under = 80
fail_under = 95
show_missing = true
[tool.pytest.ini_options]
@@ -146,6 +154,6 @@ select = [
"**/tests/**" = ["ARG001", "ARG002"]
[tool.pyrefly]
preset = "basic"
preset = "strict"
project-includes = ["**/*.ipynb", "**/*.py*"]
python-version = "3.8"
+35 -17
View File
@@ -4,9 +4,15 @@
--------
* :class:`TaskSpec` —— 不可变任务描述符(唯一需要配置的东西)。
* :class:`Graph` —— 由一组 spec 构建的 DAG;负责校验、分层、可视化。
* :func:`run` —— 以 ``sequential`` / ``thread`` / ``async`` 策略执行图。
* :func:`run` ——以 ``sequential`` / ``thread`` / ``async`` / ``dependency``
策略执行图。
* :class:`RunReport` —— 类型化、可查询的运行结果。
* :class:`Context` —— 整体上下文注入的标注标记。
* :class:`RetryPolicy` —— 重试策略(max_attempts/delay/backoff/jitter/retry_on)。
* :class:`TaskHooks` —— 任务生命周期钩子(pre_run/post_run/on_failure)。
* :class:`GraphDefaults` —— 图级默认值。
* :func:`compose` —— 编程式组合多图。
* :func:`task_template` —— 批量生成相似 TaskSpec 的工厂。
* 状态后端::class:`StateBackend`、:class:`MemoryBackend`、:class:`JSONBackend`。
快速上手
@@ -18,7 +24,7 @@
graph = px.Graph.from_specs([
px.TaskSpec("extract", extract),
px.TaskSpec("double", double, ("extract",)),
px.TaskSpec("double", double, depends_on=("extract",)),
])
report = px.run(graph, strategy="sequential")
print(report["double"]) # [2, 4, 6]
@@ -29,23 +35,18 @@
from pyflowx.conditions import IS_WINDOWS, BuiltinConditions
graph = px.Graph.from_specs([
# 使用命令列表
px.TaskSpec("list_files", cmd=["ls", "-la"]),
# 使用 shell 命令
px.TaskSpec("check_git", cmd="git status"),
# 条件执行:仅在 Windows 上运行
px.TaskSpec(
"win_only",
cmd=["dir"],
conditions=(IS_WINDOWS,)
),
# 条件执行:仅在 git 已安装时运行
px.TaskSpec(
"git_check",
cmd=["git", "--version"],
conditions=(BuiltinConditions.HAS_INSTALLED("git"),)
),
# 命令不存在时自动跳过(而非失败)
px.TaskSpec(
"optional_build",
cmd=["maturin", "build"],
@@ -57,6 +58,8 @@
from __future__ import annotations
from .command import run_command
from .compose import GraphComposer, compose
from .conditions import (
IS_LINUX,
IS_MACOS,
@@ -78,13 +81,25 @@ from .errors import (
TaskTimeoutError,
)
from .executors import Strategy, run
from .graph import Graph
from .graph import Graph, GraphDefaults
from .report import RunReport
from .runner import CliExitCode, CliRunner
from .storage import JSONBackend, MemoryBackend, StateBackend
from .task import TaskCmd, TaskEvent, TaskResult, TaskSpec, TaskStatus
from .task import (
CacheKeyFn,
RetryPolicy,
TaskCmd,
TaskEvent,
TaskHooks,
TaskResult,
TaskSpec,
TaskStatus,
cmd,
task,
task_template,
)
__version__ = "0.2.3"
__version__ = "0.3.6"
__all__ = [
"IS_LINUX",
@@ -92,38 +107,41 @@ __all__ = [
"IS_POSIX",
"IS_WINDOWS",
"BuiltinConditions",
"CacheKeyFn",
"CliExitCode",
# CLI 运行器
"CliRunner",
# 条件判断
"Condition",
"Constants",
"Context",
"CycleError",
"DuplicateTaskError",
"Graph",
"GraphComposer",
"GraphDefaults",
"InjectionError",
"JSONBackend",
"MemoryBackend",
"MissingDependencyError",
# 错误
"PyFlowXError",
"RetryPolicy",
"RunReport",
# 状态后端
"StateBackend",
"StorageError",
"Strategy",
"TaskCmd",
"TaskEvent",
"TaskFailedError",
"TaskHooks",
"TaskResult",
# 核心类型
"TaskSpec",
"TaskStatus",
"TaskTimeoutError",
# 辅助(高级)
"build_call_args",
"cmd",
"compose",
"describe_injection",
# 执行
"run",
"run_command",
"task",
"task_template",
]
-78
View File
@@ -1,78 +0,0 @@
"""CLI 工具模块.
提供各种命令行工具的入口点.
"""
from __future__ import annotations
# 自动格式化工具
from pyflowx.cli.autofmt import main as autofmt_main
from pyflowx.cli.bumpversion import main as bumpversion_main
from pyflowx.cli.clearscreen import main as clearscreen_main
# EML 邮件管理工具
from pyflowx.cli.emlmanager import main as emlmanager_main
# EML 邮件管理工具
from pyflowx.cli.emlmanager import main as emlmanager_web_main
from pyflowx.cli.envpy import main as envpy_main
from pyflowx.cli.envqt import main as envqt_main
from pyflowx.cli.envrs import main as envrs_main
# 文件工具
from pyflowx.cli.filedate import main as filedate_main
from pyflowx.cli.filelevel import main as filelevel_main
from pyflowx.cli.folderback import main as folderback_main
from pyflowx.cli.folderzip import main as folderzip_main
# Git 工具
from pyflowx.cli.gittool import main as gittool_main
# 仿真工具
from pyflowx.cli.lscalc import main as lscalc_main
# 打包工具
from pyflowx.cli.packtool import main as packtool_main
# PDF 工具
from pyflowx.cli.pdftool import main as pdftool_main
# 开发工具
from pyflowx.cli.piptool import main as piptool_main
from pyflowx.cli.pymake import main as pymake_main
from pyflowx.cli.screenshot import main as screenshot_main
from pyflowx.cli.sshcopyid import main as sshcopyid_main
__all__ = [
# 自动格式化工具
"autofmt_main",
"bumpversion_main",
"clearscreen_main",
# EML 邮件管理工具
"emlmanager_main",
"emlmanager_web_main",
"envpy_main",
"envqt_main",
"envrs_main",
# 文件工具
"filedate_main",
"filelevel_main",
"folderback_main",
"folderzip_main",
# Git 工具
"gittool_main",
# 仿真工具
"lscalc_main",
# 打包工具
"packtool_main",
# PDF 工具
"pdftool_main",
# 开发工具
"piptool_main",
"pymake_main",
"screenshot_main",
"sshcopyid_main",
# 系统工具
"taskkill_main",
"which_main",
]
+6 -6
View File
@@ -268,13 +268,13 @@ def main() -> None:
cmd.extend(["--fix", "--unsafe-fixes"])
graph = px.Graph.from_specs([px.TaskSpec("ruff_check", cmd=cmd, verbose=True)])
elif args.command == "doc":
graph = px.Graph.from_specs(
[px.TaskSpec("auto_docstring", fn=auto_add_docstrings, args=(Path(args.root_dir),), verbose=True)]
)
graph = px.Graph.from_specs([
px.TaskSpec("auto_docstring", fn=auto_add_docstrings, args=(Path(args.root_dir),), verbose=True)
])
elif args.command == "sync":
graph = px.Graph.from_specs(
[px.TaskSpec("sync_config", fn=sync_pyproject_config, args=(Path(args.root_dir),), verbose=True)]
)
graph = px.Graph.from_specs([
px.TaskSpec("sync_config", fn=sync_pyproject_config, args=(Path(args.root_dir),), verbose=True)
])
else:
parser.print_help()
return
+29 -27
View File
@@ -212,16 +212,14 @@ def main() -> None:
# 更新所有文件的版本号(使用顺序执行避免竞争条件)
# 使用相对于 cwd 的路径作为任务名,确保唯一性
graph = px.Graph.from_specs(
[
px.TaskSpec(
f"bump_{file.relative_to(Path.cwd())}".replace("\\", "_").replace("/", "_").replace(".", "_"),
fn=bump_file_version,
args=(file, part),
)
for file in all_files
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
f"bump_{file.relative_to(Path.cwd())}".replace("\\", "_").replace("/", "_").replace(".", "_"),
fn=bump_file_version,
args=(file, part),
)
for file in all_files
])
report = px.run(graph, strategy="sequential")
# 收集新版本号(取第一个成功的结果)
@@ -238,24 +236,28 @@ def main() -> None:
print(f"版本号已更新为: {new_version}")
# 提交修改
graph = px.Graph.from_specs(
[
px.TaskSpec("git_add", cmd=["git", "add", "."]),
px.TaskSpec(
"git_commit", cmd=["git", "commit", "-m", f"bump version to {new_version}"], depends_on=["git_add"]
),
]
)
px.run(graph, strategy="sequential")
# 提交修改并创建标签
tasks = [
px.TaskSpec("git_add", cmd=["git", "add", "."]),
px.TaskSpec(
"git_commit",
cmd=["git", "commit", "-m", f"bump version to {new_version}"],
depends_on=("git_add",),
),
]
# 创建 git tag
if not args.no_tag:
tag_name = f"v{new_version}"
graph = px.Graph.from_specs(
[
px.TaskSpec("git_tag", cmd=["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"]),
]
tasks.append(
px.TaskSpec(
"git_tag",
cmd=["git", "tag", "-a", tag_name, "-m", f"Release {tag_name}"],
depends_on=("git_commit",),
)
)
px.run(graph, strategy="sequential")
print(f"已创建标签: {tag_name}")
graph = px.Graph.from_specs(tasks)
px.run(graph, strategy="sequential")
if not args.no_tag:
print(f"已创建标签: v{new_version}")
-27
View File
@@ -1,27 +0,0 @@
"""清屏工具.
跨平台清屏工具, 支持终端和控制台清屏.
"""
from __future__ import annotations
import subprocess
import pyflowx as px
from pyflowx.conditions import Constants
def clear_screen() -> None:
"""使用系统命令清屏."""
if Constants.IS_WINDOWS:
subprocess.run(["cmd", "/c", "cls"], check=False)
else:
subprocess.run(["clear"], check=False)
print("\033[2J\033[H", end="")
def main() -> None:
"""清屏工具主函数."""
graph = px.Graph.from_specs([px.TaskSpec("clearscreen", fn=clear_screen)])
px.run(graph, strategy="thread")
View File
+331
View File
@@ -0,0 +1,331 @@
from __future__ import annotations
import argparse
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
from pyflowx.conditions import BuiltinConditions
from pyflowx.tasks.system import setenv_group, write_file
# ============================================================================
# Mirror 配置
# ============================================================================
DOWNLOAD_MIRROR_SCRIPT: str = "curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh"
INSTALL_MIRROR_SCRIPT: str = "sudo bash /tmp/linuxmirrors.sh"
# ============================================================================
# Python 配置
# ============================================================================
PyMirrorType = Literal["tsinghua", "aliyun", "huaweicloud", "ustc", "zju"]
PIP_INDEX_URLS: dict[PyMirrorType, str] = {
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
"huaweicloud": "https://mirrors.huaweicloud.com/repository/pypi/simple/",
"ustc": "https://pypi.mirrors.ustc.edu.cn/simple/",
"zju": "https://mirrors.zju.edu.cn/pypi/simple/",
}
PIP_TRUSTED_HOSTS: dict[PyMirrorType, str] = {
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
"aliyun": "mirrors.aliyun.com",
"huaweicloud": "mirrors.huaweicloud.com",
"ustc": "pypi.mirrors.ustc.edu.cn",
"zju": "mirrors.zju.edu.cn",
}
PIP_CONFIG_PATH = Path.home() / ".pip" / "pip.conf" if BuiltinConditions.IS_LINUX() else Path.home() / "pip" / "pip.ini"
UV_INDEX_URLS = PIP_INDEX_URLS
UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
# ============================================================================
# Conda 配置
# ============================================================================
CondaMirrorType = Literal["tsinghua", "ustc", "bsfu", "aliyun"]
CONDA_MIRROR_URLS: dict[CondaMirrorType, list[str]] = {
"tsinghua": [
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/",
],
"ustc": [
"https://mirrors.ustc.edu.cn/anaconda/pkgs/main/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/free/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/r/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.ustc.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.ustc.edu.cn/anaconda/cloud/pytorch/",
],
"bsfu": [
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/main/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/free/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/r/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/msys2/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/pro/",
"https://mirrors.bsfu.edu.cn/anaconda/pkgs/dev/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/conda-forge/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/bioconda/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/menpo/",
"https://mirrors.bsfu.edu.cn/anaconda/cloud/pytorch/",
],
"aliyun": [
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
"https://mirrors.aliyun.com/anaconda/pkgs/r/",
"https://mirrors.aliyun.com/anaconda/pkgs/msys2/",
"https://mirrors.aliyun.com/anaconda/pkgs/pro/",
"https://mirrors.aliyun.com/anaconda/pkgs/dev/",
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
"https://mirrors.aliyun.com/anaconda/cloud/bioconda/",
"https://mirrors.aliyun.com/anaconda/cloud/menpo/",
"https://mirrors.aliyun.com/anaconda/cloud/pytorch/",
],
}
CONDA_CONFIG_PATH = Path.home() / ".condarc"
# ============================================================================
# Qt 配置
# ============================================================================
QT_LIBS: list[str] = [
"build-essential",
"libgl1",
"libegl1",
"libglib2.0-0",
"libfontconfig1",
"libfreetype6",
"libxkbcommon0",
"libdbus-1-3",
"libxcb-xinerama0",
"libxcb-icccm4",
"libxcb-image0",
"libxcb-keysyms1",
"libxcb-randr0",
"libxcb-render-util0",
"libxcb-shape0",
"libxcb-xfixes0",
"libxcb-cursor0",
]
CHINESE_FONTS: list[str] = [
"fonts-noto-cjk",
"fonts-wqy-microhei",
"fonts-wqy-zenhei",
"fonts-noto-color-emoji",
]
# ============================================================================
# Rust 配置
# ============================================================================
RustMirrorType = Literal["tsinghua", "ustc", "aliyun"]
RustVersionType = Literal["stable", "nightly", "beta"]
DEFAULT_RUST_VERSION: RustVersionType = "stable"
DEFAULT_MIRROR: RustMirrorType = "tsinghua"
RUSTUP_MIRRORS: dict[RustMirrorType, dict[str, str]] = {
"tsinghua": {
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
},
"aliyun": {
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
},
"ustc": {
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
},
}
RUSTUP_DOWNLOAD_URL_LINUX = "https://mirrors.aliyun.com/repo/rust/rustup-init.sh"
RUSTUP_DOWNLOAD_URL_WINDOWS = "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"
RUST_CONFIG_PATH = Path.home() / ".cargo" / "config.toml"
RUST_SCCACHE_DIR: Path = Path.home() / ".cargo" / "sccache"
RUST_SCCACHE_CACHE_SIZE: str = "20G"
def main() -> None:
"""主函数."""
parser = argparse.ArgumentParser(description="环境开发工具")
parser.add_argument(
"--python-mirror",
nargs="?",
type=str,
default="tsinghua",
choices=get_args(PyMirrorType),
help="Python 镜像源",
)
parser.add_argument(
"--conda-mirror",
nargs="?",
type=str,
default="tsinghua",
choices=get_args(CondaMirrorType),
help="Conda 镜镜像源",
)
parser.add_argument(
"--rust-mirror",
nargs="?",
type=str,
default=DEFAULT_MIRROR,
choices=get_args(RustMirrorType),
help="Rust 镜像源",
)
parser.add_argument(
"--rust-version",
nargs="?",
type=str,
default=DEFAULT_RUST_VERSION,
choices=get_args(RustVersionType),
help=f"Rust 版本, 推荐: {get_args(RustVersionType)}",
)
args = parser.parse_args()
python_mirror = args.python_mirror
conda_mirror_urls = CONDA_MIRROR_URLS[args.conda_mirror]
rust_mirror = args.rust_mirror
rust_version = args.rust_version
# 确保配置文件目录存在
PIP_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
CONDA_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
RUST_CONFIG_PATH.parent.mkdir(parents=True, exist_ok=True)
RUST_SCCACHE_DIR.mkdir(parents=True, exist_ok=True)
# 使用 conditions 自动控制任务执行
graph = px.Graph.from_specs([
# 系统镜像配置(仅 Linux 且未配置国内镜像)
px.TaskSpec(
"download_mirror",
cmd=DOWNLOAD_MIRROR_SCRIPT,
conditions=(
BuiltinConditions.IS_LINUX(),
BuiltinConditions.NOT(
BuiltinConditions.OR(
*[
BuiltinConditions.FILE_CONTENT_EXISTS(f, m)
for f in [
"/etc/apt/sources.list",
"/etc/apt/sources.list.d/ubuntu.sources",
]
for m in get_args(PyMirrorType)
],
)
),
),
verbose=True,
),
px.TaskSpec(
"install_mirror",
cmd=INSTALL_MIRROR_SCRIPT,
depends_on=("download_mirror",),
verbose=True,
),
# 安装 Qt 依赖(仅 Linux
px.TaskSpec(
"install_qt_libs",
cmd=["sudo", "apt", "install", "-y", *QT_LIBS],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_mirror",),
allow_upstream_skip=True,
verbose=True,
),
# 安装中文字体(仅 Linux
px.TaskSpec(
"install_fonts",
cmd=["sudo", "apt", "install", "-y", *CHINESE_FONTS],
conditions=(BuiltinConditions.IS_LINUX(),),
depends_on=("install_mirror",),
allow_upstream_skip=True,
verbose=True,
),
# 设置 Python 环境变量
*setenv_group({
"PIP_INDEX_URL": PIP_INDEX_URLS[python_mirror],
"PIP_TRUSTED_HOSTS": PIP_TRUSTED_HOSTS[python_mirror],
"UV_INDEX_URL": UV_INDEX_URLS[python_mirror],
"UV_PYTHON_INSTALL_MIRROR": UV_PYTHON_INSTALL_MIRROR,
"UV_HTTP_TIMEOUT": "600",
"UV_LINK_MODE": "copy",
}),
# 写入 Python 配置(仅当未配置)
write_file(
str(PIP_CONFIG_PATH),
f"[global]\nindex-url = {PIP_INDEX_URLS[python_mirror]}\ntrusted-host = {PIP_TRUSTED_HOSTS[python_mirror]}",
),
# 写入 Conda 配置(仅当未配置)
write_file(
str(CONDA_CONFIG_PATH),
"show_channel_urls: true\nchannels:\n - " + "\n - ".join(conda_mirror_urls) + "\n - defaults",
),
# 设置 Rust 镜像源
*setenv_group({
"RUSTUP_DIST_SERVER": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_DIST_SERVER"],
"RUSTUP_UPDATE_ROOT": RUSTUP_MIRRORS[rust_mirror]["RUSTUP_UPDATE_ROOT"],
"RUST_SCCACHE_DIR": str(RUST_SCCACHE_DIR),
"RUST_SCCACHE_CACHE_SIZE": RUST_SCCACHE_CACHE_SIZE,
}),
# 写入 Rust 配置(仅当未配置)
write_file(
str(RUST_CONFIG_PATH),
f"""
[source.crates-io]
replace-with = '{rust_mirror}'
[source.{rust_mirror}]
registry = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
[registries.{rust_mirror}]
index = "sparse+{RUSTUP_MIRRORS[rust_mirror]["TOML_REGISTRY"]}"
""",
),
# 下载 Rustup 安装脚本
px.TaskSpec(
"download_rustup",
cmd=["curl", "-fsSL", RUSTUP_DOWNLOAD_URL_LINUX, "-o", "rustup-init.sh"],
conditions=(BuiltinConditions.IS_LINUX(), BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup"))),
verbose=True,
),
px.TaskSpec(
"download_rustup_win",
cmd=[
"powershell",
"-Command",
"Invoke-WebRequest",
"-Uri",
RUSTUP_DOWNLOAD_URL_WINDOWS,
"-OutFile",
"rustup-init.exe",
],
conditions=(
BuiltinConditions.IS_WINDOWS(),
BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("rustup")),
),
verbose=True,
),
# 安装 Rust 工具链
px.TaskSpec(
"install_rust",
cmd=["rustup", "toolchain", "install", rust_version],
conditions=(BuiltinConditions.HAS_INSTALLED("rustup"),),
depends_on=("setenv_rustup_dist_server",),
allow_upstream_skip=True,
verbose=True,
),
])
px.run(graph, strategy="thread", verbose=True)
+35 -15
View File
@@ -88,6 +88,8 @@ class EmailDatabase:
def insert_email(self, email_data: dict[str, Any]) -> bool:
"""插入邮件数据."""
assert self.conn, "数据库连接未初始化"
try:
with self._lock:
cursor = self.conn.cursor()
@@ -123,6 +125,8 @@ class EmailDatabase:
self, keyword: str = "", field: str = "all", limit: int = 100, offset: int = 0
) -> list[dict[str, Any]]:
"""搜索邮件."""
assert self.conn, "数据库连接未初始化"
with self._lock:
cursor = self.conn.cursor()
@@ -154,6 +158,8 @@ class EmailDatabase:
def get_grouped_emails(self) -> dict[str, list[dict[str, Any]]]:
"""获取按主题分组的邮件."""
assert self.conn, "数据库连接未初始化"
with self._lock:
cursor = self.conn.cursor()
cursor.execute(f"SELECT * FROM {TABLE_NAME} ORDER BY subject, date_parsed DESC")
@@ -183,6 +189,8 @@ class EmailDatabase:
def get_email_count(self) -> int:
"""获取邮件总数."""
assert self.conn, "数据库连接未初始化"
with self._lock:
cursor = self.conn.cursor()
cursor.execute(f"SELECT COUNT(*) FROM {TABLE_NAME}")
@@ -190,6 +198,8 @@ class EmailDatabase:
def clear_all(self) -> None:
"""清空所有邮件数据."""
assert self.conn, "数据库连接未初始化"
with self._lock:
cursor = self.conn.cursor()
cursor.execute(f"DELETE FROM {TABLE_NAME}")
@@ -230,7 +240,7 @@ def _parse_email_date(date_str: str) -> str:
try:
dt = parsedate_to_datetime(date_str)
return dt.isoformat()
except Exception:
except (ValueError, TypeError, OverflowError):
return date_str
@@ -267,11 +277,11 @@ def _extract_email_body_part(part: Any) -> str:
decoded_text = payload.decode(charset, errors="replace")
except (UnicodeDecodeError, LookupError) as decode_error:
# 如果指定编码失败,尝试常见编码
logger.warning(f"字符编码 {charset} 解码失败: {decode_error}")
logger.warning("字符编码 %s 解码失败: %s", charset, decode_error)
for fallback_charset in ["utf-8", "gbk", "gb2312", "latin-1"]:
try:
decoded_text = payload.decode(fallback_charset, errors="replace")
logger.info(f"成功使用备用编码 {fallback_charset} 解码")
logger.info("成功使用备用编码 %s 解码", fallback_charset)
break
except (UnicodeDecodeError, LookupError):
continue
@@ -283,15 +293,15 @@ def _extract_email_body_part(part: Any) -> str:
# 限制长度并返回
result = decoded_text[:MAX_BODY_LENGTH]
if len(decoded_text) > MAX_BODY_LENGTH:
logger.debug(f"正文内容过长,截取前{MAX_BODY_LENGTH}字符")
logger.debug("正文内容过长,截取前%d字符", MAX_BODY_LENGTH)
return result
except AttributeError as attr_error:
logger.error(f"邮件部分对象属性错误: {attr_error}")
logger.error("邮件部分对象属性错误: %s", attr_error)
return ""
except Exception as unexpected_error:
logger.error(f"提取邮件正文时发生未知错误: {unexpected_error}")
logger.error("提取邮件正文时发生未知错误: %s", unexpected_error)
return ""
@@ -557,15 +567,13 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
emails = self.db.search_emails(keyword, field, limit, offset)
total_count = self.db.get_email_count()
self._send_json_response(
{
"emails": emails,
"count": len(emails),
"total": total_count,
"limit": limit,
"offset": offset,
}
)
self._send_json_response({
"emails": emails,
"count": len(emails),
"total": total_count,
"limit": limit,
"offset": offset,
})
def _api_get_email(self, query_params: dict[str, list[str]]) -> None:
"""API: 获取单个邮件详情."""
@@ -578,6 +586,10 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
self._send_json_response({"error": "缺少邮件ID"}, 400)
return
if not self.db.conn:
self._send_json_response({"error": "数据库连接未初始化"}, 500)
return
with self.db._lock:
cursor = self.db.conn.cursor()
cursor.execute(f"SELECT * FROM {TABLE_NAME} WHERE id = ?", (int(email_id),))
@@ -630,6 +642,10 @@ class EmlManagerHandler(BaseHTTPRequestHandler):
if not eml_files:
return
if not self.db.conn:
self._send_json_response({"error": "数据库连接未初始化"}, 500)
return
# 先批量查询所有已存在的文件
with self.db._lock:
cursor = self.db.conn.cursor()
@@ -1268,6 +1284,10 @@ def main() -> None:
if eml_files:
print(f"发现 {len(eml_files)} 个 EML 文件,开始导入...")
if not EmlManagerHandler.db.conn:
print("数据库连接未初始化,无法导入邮件")
return
# 先批量查询所有已存在的文件
with EmlManagerHandler.db._lock:
cursor = EmlManagerHandler.db.conn.cursor()
-11
View File
@@ -1,11 +0,0 @@
import pyflowx as px
def main() -> None:
"""主函数."""
# 使用更安全的分步执行方式,便于调试和捕获错误
graph = px.Graph.from_specs([
px.TaskSpec("download", cmd="curl -sSL https://linuxmirrors.cn/main.sh -o /tmp/linuxmirrors.sh", verbose=True),
px.TaskSpec("install", cmd="sudo bash /tmp/linuxmirrors.sh", verbose=True, depends_on=("download",)),
])
px.run(graph, strategy="thread")
-122
View File
@@ -1,122 +0,0 @@
"""Python 环境配置工具.
用于设置 pip 镜像源, 支持清华和阿里云等国内镜像源,
同时配置 UV 和 Conda 的镜像源.
"""
from __future__ import annotations
import argparse
import os
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import Constants
# ============================================================================
# 配置
# ============================================================================
PIP_INDEX_URLS: dict[str, str] = {
"tsinghua": "https://pypi.tuna.tsinghua.edu.cn/simple",
"aliyun": "https://mirrors.aliyun.com/pypi/simple/",
}
PIP_TRUSTED_HOSTS: dict[str, str] = {
"tsinghua": "pypi.tuna.tsinghua.edu.cn",
"aliyun": "mirrors.aliyun.com",
}
UV_INDEX_URL: str = "https://mirrors.aliyun.com/pypi/simple/"
UV_PYTHON_INSTALL_MIRROR: str = "https://registry.npmmirror.com/-/binary/python-build-standalone"
CONDA_MIRROR_URLS: dict[str, list[str]] = {
"tsinghua": [
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/",
"https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/",
],
"aliyun": [
"https://mirrors.aliyun.com/anaconda/pkgs/main/",
"https://mirrors.aliyun.com/anaconda/pkgs/free/",
"https://mirrors.aliyun.com/anaconda/cloud/conda-forge/",
],
}
# ============================================================================
# 辅助函数
# ============================================================================
def set_pip_mirror(mirror: str = "tsinghua", token: str | None = None) -> None:
"""设置 pip 镜像源.
Parameters
----------
mirror : str
镜像源名称: tsinghua, aliyun
token : str | None
PyPI token for publishing
"""
index_url = PIP_INDEX_URLS.get(mirror, PIP_INDEX_URLS["tsinghua"])
trusted_host = PIP_TRUSTED_HOSTS.get(mirror, "")
# 设置环境变量
os.environ["PIP_INDEX_URL"] = index_url
os.environ["UV_INDEX_URL"] = UV_INDEX_URL
os.environ["UV_DEFAULT_INDEX"] = UV_INDEX_URL
os.environ["UV_PYTHON_INSTALL_MIRROR"] = UV_PYTHON_INSTALL_MIRROR
# 写入 pip 配置文件
pip_dir = Path.home() / "pip"
pip_dir.mkdir(exist_ok=True)
pip_conf = pip_dir / ("pip.ini" if Constants.IS_WINDOWS else "pip.conf")
pip_conf.write_text(f"[global]\nindex-url = {index_url}\n[install]\ntrusted-host = {trusted_host}\n")
# 写入 conda 配置文件
condarc = Path.home() / ".condarc"
conda_urls = CONDA_MIRROR_URLS.get(mirror, CONDA_MIRROR_URLS["tsinghua"])
condarc.write_text(
"show_channel_urls: true\nchannels:\n" + "\n".join(f" - {url}" for url in conda_urls) + "\n - defaults\n"
)
# 写入 pypirc 配置文件 (如果有 token)
if token:
pypirc = Path.home() / ".pypirc"
pypirc.write_text(
f"[pypi]\nrepository: https://upload.pypi.org/legacy/\nusername: __token__\npassword: {token}\n"
)
print(f"已设置 pip 镜像源: {mirror} ({index_url})")
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""Python 环境配置工具主函数."""
parser = argparse.ArgumentParser(
description="EnvPy - Python 环境配置工具",
usage="envpy <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 设置镜像源命令
mirror_parser = subparsers.add_parser("mirror", help="设置 pip 镜像源")
mirror_parser.add_argument("name", choices=["tsinghua", "aliyun"], help="镜像源名称")
mirror_parser.add_argument("--token", type=str, help="PyPI token for publishing")
args = parser.parse_args()
if args.command == "mirror":
graph = px.Graph.from_specs(
[px.TaskSpec("set_pip_mirror", fn=set_pip_mirror, args=(args.name,), kwargs={"token": args.token})]
)
else:
parser.print_help()
return
px.run(graph, strategy="thread")
-57
View File
@@ -1,57 +0,0 @@
"""PyQt 环境配置工具.
用于设置 PyQt 相关环境变量, 安装依赖环境.
"""
from __future__ import annotations
import pyflowx as px
from pyflowx.conditions import Constants
QT_LIBS: list[str] = [
"build-essential",
"libgl1",
"libegl1",
"libglib2.0-0",
"libfontconfig1",
"libfreetype6",
"libxkbcommon0",
"libdbus-1-3",
"libxcb-xinerama0",
"libxcb-icccm4",
"libxcb-image0",
"libxcb-keysyms1",
"libxcb-randr0",
"libxcb-render-util0",
"libxcb-shape0",
"libxcb-xfixes0",
"libxcb-cursor0",
]
CHINESE_FONTS: list[str] = [
"fonts-noto-cjk",
"fonts-wqy-microhei",
"fonts-wqy-zenhei",
"fonts-noto-color-emoji",
]
def main() -> None:
"""PyQt 环境配置工具主函数."""
graph = px.Graph.from_specs(
[
px.TaskSpec(
"envqt_install",
cmd=["sudo", "apt", "install", "-y", *QT_LIBS],
conditions=(lambda: Constants.IS_LINUX,),
verbose=True,
),
px.TaskSpec(
"envqt_fonts",
cmd=["sudo", "apt", "install", "-y", *CHINESE_FONTS],
conditions=(lambda: Constants.IS_LINUX,),
verbose=True,
),
],
)
px.run(graph, strategy="thread", verbose=True)
-150
View File
@@ -1,150 +0,0 @@
"""Rust 环境配置工具.
配置 Rustup 和 Cargo 的国内镜像源,
加速 Rust 工具链和依赖包的下载.
"""
from __future__ import annotations
import argparse
import os
import subprocess
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
# ============================================================================
# 配置
# ============================================================================
RUSTUP_MIRRORS: dict[str, dict[str, str]] = {
"aliyun": {
"RUSTUP_DIST_SERVER": "https://mirrors.aliyun.com/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.aliyun.com/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.aliyun.com/crates.io-index/",
},
"ustc": {
"RUSTUP_DIST_SERVER": "https://mirrors.ustc.edu.cn/rust-static",
"RUSTUP_UPDATE_ROOT": "https://mirrors.ustc.edu.cn/rust-static/rustup",
"TOML_REGISTRY": "https://mirrors.ustc.edu.cn/crates.io-index/",
},
"tsinghua": {
"RUSTUP_DIST_SERVER": "https://mirrors.tuna.tsinghua.edu.cn/rustup",
"RUSTUP_UPDATE_ROOT": "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup",
"TOML_REGISTRY": "https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/",
},
}
UsableRustVersion = Literal["stable", "nightly", "beta"]
UsableMirror = Literal["aliyun", "ustc", "tsinghua"]
DEFAULT_RUST_VERSION: str = "stable"
DEFAULT_MIRROR: UsableMirror = "tsinghua"
# ============================================================================
# 辅助函数
# ============================================================================
def set_rust_mirror(mirror: UsableMirror = DEFAULT_MIRROR) -> None:
"""设置 Rust 镜像源.
Parameters
----------
mirror : str
镜像源名称: aliyun, ustc, tsinghua
"""
mirror_dict = RUSTUP_MIRRORS.get(mirror, RUSTUP_MIRRORS[DEFAULT_MIRROR])
server = mirror_dict["RUSTUP_DIST_SERVER"]
update_root = mirror_dict["RUSTUP_UPDATE_ROOT"]
toml_registry = mirror_dict["TOML_REGISTRY"]
# 设置环境变量
os.environ["RUSTUP_DIST_SERVER"] = server
os.environ["RUSTUP_UPDATE_ROOT"] = update_root
# 写入 cargo 配置
cargo_dir = Path.home() / ".cargo"
cargo_dir.mkdir(exist_ok=True)
cargo_config = cargo_dir / "config.toml"
cargo_config.write_text(
f"""[source.crates-io]
replace-with = '{mirror}'
[source.{mirror}]
registry = "sparse+{toml_registry}"
[registries.{mirror}]
index = "sparse+{toml_registry}"
"""
)
print(f"已设置 Rust 镜像源: {mirror}")
def install_rust(version: UsableRustVersion = DEFAULT_RUST_VERSION) -> None:
"""安装 Rust 工具链.
Parameters
----------
version : str
Rust 版本: stable, nightly, beta
"""
try:
subprocess.run(["rustup", "toolchain", "install", version], check=True)
print(f"已安装 Rust {version}")
except FileNotFoundError:
print("未找到 rustup,请先安装 Rust: https://rustup.rs")
raise
# ============================================================================
# CLI Runner
# ============================================================================
def main() -> None:
"""Rust 环境配置工具主函数."""
parser = argparse.ArgumentParser(
description="EnvRs - Rust 环境配置工具",
usage="envrs <command> [options]",
)
subparsers = parser.add_subparsers(dest="command", help="可用命令")
# 设置镜像源命令
mirror_parser = subparsers.add_parser("mirror", help="设置 Rust 镜像源")
mirror_parser.add_argument(
"name",
nargs="?",
default=DEFAULT_MIRROR,
choices=get_args(UsableMirror),
help=f"镜像源名称 ({get_args(UsableMirror)})",
)
# 安装 Rust 命令
install_parser = subparsers.add_parser("install", help="安装 Rust 工具链")
install_parser.add_argument(
"version",
nargs="?",
default=DEFAULT_RUST_VERSION,
choices=get_args(UsableRustVersion),
help=f"Rust 版本 ({get_args(UsableRustVersion)})",
)
args = parser.parse_args()
if args.command == "mirror":
graph = px.Graph.from_specs(
[px.TaskSpec("set_rust_mirror", fn=set_rust_mirror, args=(args.name,), verbose=True)]
)
elif args.command == "install":
graph = px.Graph.from_specs(
[px.TaskSpec("install_rust", cmd=["rustup", "toolchain", "install", args.version], verbose=True)]
)
else:
parser.print_help()
return
px.run(graph, strategy="thread", verbose=True)
+16 -20
View File
@@ -113,27 +113,23 @@ def main() -> None:
args = parser.parse_args()
if args.command == "add":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": False},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": False},
)
])
elif args.command == "clear":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": True},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"process_files_date",
fn=process_files_date,
args=([Path(f) for f in args.files],),
kwargs={"clear": True},
)
])
else:
parser.print_help()
return
+6 -15
View File
@@ -66,19 +66,10 @@ def backup_folder(src: str, dst: str, max_zip: int = 5) -> None:
zip_target(src_path, dst_path, max_zip)
# ============================================================================
# TaskSpec 定义
# ============================================================================
folderback_default: px.TaskSpec = px.TaskSpec(
"folderback_default",
fn=lambda: backup_folder(".", "./backup", 5),
)
# ============================================================================
# CLI Runner
# ============================================================================
@px.task
def folderback_default() -> None:
"""备份当前目录到 ./backup."""
backup_folder(".", "./backup", 5)
def main() -> None:
@@ -86,9 +77,9 @@ def main() -> None:
runner = px.CliRunner(
strategy="thread",
description="FolderBack - 文件夹备份工具",
graphs={
aliases={
# 备份当前目录到 ./backup
"b": px.Graph.from_specs([folderback_default]),
"b": folderback_default,
},
)
runner.run_cli()
+6 -12
View File
@@ -57,16 +57,10 @@ def zip_folders(cwd: str = ".") -> None:
archive_folder(dir_path)
# ============================================================================
# TaskSpec 定义
# ============================================================================
folderzip_default: px.TaskSpec = px.TaskSpec("folderzip_default", fn=lambda: zip_folders("."))
# ============================================================================
# CLI Runner
# ============================================================================
@px.task
def folderzip_default() -> None:
"""压缩当前目录下的所有文件夹."""
zip_folders(".")
def main() -> None:
@@ -74,9 +68,9 @@ def main() -> None:
runner = px.CliRunner(
strategy="thread",
description="FolderZip - 文件夹压缩工具",
graphs={
aliases={
# 压缩当前目录下的所有文件夹
"z": px.Graph.from_specs([folderzip_default]),
"z": folderzip_default,
},
)
runner.run_cli()
+39 -41
View File
@@ -33,24 +33,25 @@ def init_sub_dirs() -> None:
sub_dirs = [subdir for subdir in Path.cwd().iterdir() if subdir.is_dir()]
for subdir in sub_dirs:
px.run(
px.Graph.from_specs(
[
px.TaskSpec(
"init",
cmd=["git", "init"],
conditions=[not_has_git_repo],
cwd=str(subdir),
),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=["init"], cwd=str(subdir)),
px.TaskSpec(
"commit", cmd=["git", "commit", "-m", "init commit"], depends_on=["add"], cwd=str(subdir)
),
]
),
px.Graph.from_specs([
px.TaskSpec(
"init",
cmd=["git", "init"],
conditions=(lambda _: not_has_git_repo(),),
cwd=subdir,
),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",)),
px.TaskSpec("commit", cmd=["git", "commit", "-m", "init commit"], depends_on=("add",)),
]),
)
isub: px.TaskSpec = px.TaskSpec("isub", fn=init_sub_dirs)
@px.task(name="isub")
def isub() -> None:
"""初始化子目录的Git仓库."""
init_sub_dirs()
push: px.TaskSpec = px.TaskSpec("push", cmd=["git", "push"])
pull: px.TaskSpec = px.TaskSpec("pull", cmd=["git", "pull"])
kill_tgit: px.TaskSpec = px.TaskSpec("task_kill", cmd=["taskkill", "/f", "/t", "/im", "tgitcache.exe"])
@@ -71,39 +72,36 @@ def main() -> None:
runner = px.CliRunner(
strategy="thread",
description="Gittool - Git 执行工具.",
graphs={
aliases={
# 添加并提交
"a": px.Graph.from_specs(
[
px.TaskSpec("add", cmd=["git", "add", "."], conditions=[has_files]),
px.TaskSpec("commit", cmd=["git", "commit", "-m", "chore: update"], depends_on=["add"]),
]
),
# 清理
"c": px.Graph.from_specs(
[
px.TaskSpec("clean", cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS]),
px.TaskSpec("status", cmd=["git", "status", "--porcelain"], depends_on=["clean"]),
]
"a": px.Graph.from_specs([
px.TaskSpec("add", cmd=["git", "add", "."], conditions=(lambda _: has_files(),)),
px.TaskSpec("commit", cmd=["git", "commit", "-m", "chore: update"], depends_on=("add",)),
]),
# 清理(chain: clean → status
"c": px.Graph().chain(
px.TaskSpec("clean", cmd=["git", "clean", "-xfd", *EXCLUDE_CMDS]),
px.TaskSpec("status", cmd=["git", "status", "--porcelain"]),
),
# 初始化、添加并提交
"i": px.Graph.from_specs(
[
px.TaskSpec("init", cmd=["git", "init"], conditions=[not_has_git_repo]),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=["init"], conditions=[has_files]),
px.TaskSpec(
"commit", cmd=["git", "commit", "-m", "init commit"], depends_on=["add"], conditions=[has_files]
),
]
),
"i": px.Graph.from_specs([
px.TaskSpec("init", cmd=["git", "init"], conditions=(lambda _: not_has_git_repo(),)),
px.TaskSpec("add", cmd=["git", "add", "."], depends_on=("init",), conditions=(lambda _: has_files(),)),
px.TaskSpec(
"commit",
cmd=["git", "commit", "-m", "init commit"],
depends_on=("add",),
conditions=(lambda _: has_files(),),
),
]),
# 初始化子目录
"isub": px.Graph.from_specs([isub]),
"isub": isub,
# 推送
"p": px.Graph.from_specs([push]),
"p": push,
# 拉取
"pl": px.Graph.from_specs([pull]),
"pl": pull,
# 重启TGit缓存
"r": px.Graph.from_specs([kill_tgit]),
"r": kill_tgit,
},
)
runner.run_cli()
-86
View File
@@ -1,86 +0,0 @@
import argparse
import os
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
HFDownloadType = Literal["model", "dataset", "space"]
def setenvs():
"""设置 HuggingFace mirror 环境变量."""
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
def main():
parser = argparse.ArgumentParser(description="Download a model from HuggingFace.")
parser.add_argument("dataset_name", type=str, help="HuggingFace dataset name.")
parser.add_argument(
"--type",
type=str,
nargs="?",
default="dataset",
choices=get_args(HFDownloadType),
help="HuggingFace dataset type.",
)
parser.add_argument("--use-hfd", action="store_true", help="Use HFD tool to download dataset.")
args = parser.parse_args()
if not args.dataset_name:
parser.error("dataset_name is required")
dataset_name = args.dataset_name
# 创建下载目录
download_dir = Path.cwd() / dataset_name
download_dir.mkdir(parents=True, exist_ok=True)
if args.use_hfd:
graph = px.Graph.from_specs(
[
px.TaskSpec(name="setenvs", fn=setenvs, verbose=True),
px.TaskSpec(
name="download_hfd",
cmd=["wget", "https://hf-mirror.com/hfd/hfd.sh"],
depends_on=["setenvs"],
verbose=True,
),
px.TaskSpec(
name="chmod_hfd",
cmd=["chmod", "a+x", "hfd.sh"],
depends_on=["download_hfd"],
verbose=True,
),
px.TaskSpec(
name="run_hfd",
cmd=["./hfd.sh", dataset_name, args.type],
depends_on=["chmod_hfd"],
verbose=True,
),
]
)
else:
graph = px.Graph.from_specs(
[
px.TaskSpec(name="setenvs", fn=setenvs, verbose=True),
px.TaskSpec(
name="download",
cmd=[
"uvx",
"hf",
"download",
"--repo-type",
args.type,
"--force-download",
dataset_name,
"--local-dir",
str(Path.cwd() / dataset_name),
],
depends_on=["setenvs"],
verbose=True,
),
]
)
px.run(graph, strategy="thread", verbose=True)
View File
+41
View File
@@ -0,0 +1,41 @@
"""Download from ModelScopeHub."""
import argparse
from pathlib import Path
from typing import Literal, get_args
import pyflowx as px
DownloadType = Literal["model", "dataset", "space"]
def main():
parser = argparse.ArgumentParser(description="Download a model from ModelScopeHub.")
parser.add_argument("name", help="Target name.")
parser.add_argument("--type", "-t", nargs="?", default="model", choices=get_args(DownloadType), help="Target type.")
parser.add_argument("--dir", default=None, help="Download directory.")
args = parser.parse_args()
if not args.name:
parser.error("name is required")
download_dir: Path = Path(args.dir) if args.dir else Path.home() / ".models" / args.name.split("/")[-1]
download_dir.mkdir(parents=True, exist_ok=True)
graph = px.Graph.from_specs([
px.TaskSpec(
name="download",
cmd=[
"uvx",
"modelscope",
"download",
f"--{args.type}",
args.name,
"--local_dir",
str(download_dir),
],
verbose=True,
),
])
px.run(graph, strategy="thread", verbose=True)
+63
View File
@@ -0,0 +1,63 @@
"""使用 SGLang 运行本地模型."""
import argparse
from pathlib import Path
import pyflowx as px
from pyflowx.conditions import BuiltinConditions, Constants
def main():
parser = argparse.ArgumentParser(description="启动 SGLang 服务")
parser.add_argument("--model", default="~/.models/Qwen2.5-Coder-32B-Instruct-AWQ", help="模型路径")
parser.add_argument("--port", type=int, default=8000, help="服务端口")
parser.add_argument("--ctx-len", type=int, default=28672, help="最大上下文长度")
parser.add_argument("--mem", type=float, default=0.75, help="显存占比 (0-1)")
parser.add_argument("--host", default="0.0.0.0", help="主机地址")
parser.add_argument("--log-level", default="info", help="日志级别")
args = parser.parse_args()
if not args.model:
parser.error("model is required")
model_dir = Path(args.model).expanduser()
if not model_dir.exists():
parser.error(f"Model directory {model_dir} does not exist.")
graph = px.Graph.from_specs([
px.TaskSpec(
name="download",
cmd=[
"uv",
"install",
"sglang[all]",
],
conditions=(BuiltinConditions.NOT(BuiltinConditions.HAS_INSTALLED("sglang")),),
verbose=True,
),
px.TaskSpec(
name="run",
cmd=[
"python" if Constants.IS_WINDOWS else "python3",
"-m",
"sglang.launch_server",
"--model-path",
str(model_dir),
"--host",
str(args.host),
"--port",
"8000",
"--mem-fraction-static",
str(args.mem),
"--context-length",
"32768",
"--tool-call-parser",
"qwen",
"--log-level",
str(args.log_level),
],
verbose=True,
),
])
px.run(graph, strategy="sequential", verbose=True)
+67 -68
View File
@@ -146,7 +146,7 @@ def pdf_extract_text(input_path: Path, output_path: Path) -> None:
doc = fitz.open(str(input_path))
text = ""
for page in doc:
text += page.get_text() + "\n\n"
text += str(page.get_text()) + "\n\n"
doc.close()
output_path.parent.mkdir(parents=True, exist_ok=True)
@@ -164,6 +164,7 @@ def pdf_extract_images(input_path: Path, output_dir: Path) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
image_count = 0
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
images = page.get_images(full=True)
for img_idx, img in enumerate(images):
@@ -249,9 +250,13 @@ def pdf_info(input_path: Path) -> None:
doc = fitz.open(str(input_path))
print(f"文件: {input_path}")
print(f"页数: {doc.page_count}")
# pyrefly: ignore [missing-attribute]
print(f"标题: {doc.metadata.get('title', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"作者: {doc.metadata.get('author', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"创建日期: {doc.metadata.get('creationDate', 'N/A')}")
# pyrefly: ignore [missing-attribute]
print(f"修改日期: {doc.metadata.get('modDate', 'N/A')}")
print(f"文件大小: {input_path.stat().st_size / 1024:.1f} KB")
doc.close()
@@ -281,6 +286,7 @@ def pdf_ocr(input_path: Path, output_path: Path, lang: str = "chi_sim+eng") -> N
new_page = new_doc.new_page(width=page.rect.width, height=page.rect.height)
new_page.insert_image(new_page.rect, pixmap=pix)
text_rect = fitz.Rect(0, 0, page.rect.width, page.rect.height)
# pyrefly: ignore [bad-argument-type]
new_page.insert_textbox(text_rect, ocr_text)
output_path.parent.mkdir(parents=True, exist_ok=True)
@@ -319,6 +325,7 @@ def pdf_to_images(input_path: Path, output_dir: Path, dpi: int = 300) -> None:
doc = fitz.open(str(input_path))
output_dir.mkdir(parents=True, exist_ok=True)
# pyrefly: ignore [bad-argument-type]
for page_num, page in enumerate(doc):
pix = page.get_pixmap(dpi=dpi)
image_path = output_dir / f"{input_path.stem}_page_{page_num + 1}.png"
@@ -436,87 +443,79 @@ def main() -> None: # noqa: PLR0912
args = parser.parse_args()
if args.command == "m":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_merge", fn=pdf_merge, args=([Path(p) for p in args.inputs], Path(args.output)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_merge", fn=pdf_merge, args=([Path(p) for p in args.inputs], Path(args.output)))
])
elif args.command == "s":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_split", fn=pdf_split, args=(Path(args.input), Path(args.output_dir)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_split", fn=pdf_split, args=(Path(args.input), Path(args.output_dir)))
])
elif args.command == "c":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_compress", fn=pdf_compress, args=(Path(args.input), Path(args.output)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_compress", fn=pdf_compress, args=(Path(args.input), Path(args.output)))
])
elif args.command == "e":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_encrypt", fn=pdf_encrypt, args=(Path(args.input), Path(args.output), args.password))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_encrypt", fn=pdf_encrypt, args=(Path(args.input), Path(args.output), args.password))
])
elif args.command == "d":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_decrypt", fn=pdf_decrypt, args=(Path(args.input), Path(args.output), args.password))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_decrypt", fn=pdf_decrypt, args=(Path(args.input), Path(args.output), args.password))
])
elif args.command == "xt":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_extract_text", fn=pdf_extract_text, args=(Path(args.input), Path(args.output)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_extract_text", fn=pdf_extract_text, args=(Path(args.input), Path(args.output)))
])
elif args.command == "xi":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_extract_images", fn=pdf_extract_images, args=(Path(args.input), Path(args.output_dir)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_extract_images", fn=pdf_extract_images, args=(Path(args.input), Path(args.output_dir)))
])
elif args.command == "w":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pdf_watermark",
fn=pdf_add_watermark,
args=(Path(args.input), Path(args.output)),
kwargs={"text": args.text},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_watermark",
fn=pdf_add_watermark,
args=(Path(args.input), Path(args.output)),
kwargs={"text": args.text},
)
])
elif args.command == "r":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pdf_rotate",
fn=pdf_rotate,
args=(Path(args.input), Path(args.output)),
kwargs={"rotation": args.rotation},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_rotate",
fn=pdf_rotate,
args=(Path(args.input), Path(args.output)),
kwargs={"rotation": args.rotation},
)
])
elif args.command == "crop":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pdf_crop",
fn=pdf_crop,
args=(Path(args.input), Path(args.output)),
kwargs={"margins": (args.left, args.top, args.right, args.bottom)},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_crop",
fn=pdf_crop,
args=(Path(args.input), Path(args.output)),
kwargs={"margins": (args.left, args.top, args.right, args.bottom)},
)
])
elif args.command == "i":
graph = px.Graph.from_specs([px.TaskSpec("pdf_info", fn=pdf_info, args=(Path(args.input),))])
elif args.command == "ocr":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_ocr", fn=pdf_ocr, args=(Path(args.input), Path(args.output)), kwargs={"lang": args.lang})]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_ocr", fn=pdf_ocr, args=(Path(args.input), Path(args.output)), kwargs={"lang": args.lang})
])
elif args.command == "img":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pdf_to_images",
fn=pdf_to_images,
args=(Path(args.input), Path(args.output_dir)),
kwargs={"dpi": args.dpi},
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pdf_to_images",
fn=pdf_to_images,
args=(Path(args.input), Path(args.output_dir)),
kwargs={"dpi": args.dpi},
)
])
elif args.command == "repair":
graph = px.Graph.from_specs(
[px.TaskSpec("pdf_repair", fn=pdf_repair, args=(Path(args.input), Path(args.output)))]
)
graph = px.Graph.from_specs([
px.TaskSpec("pdf_repair", fn=pdf_repair, args=(Path(args.input), Path(args.output)))
])
else:
parser.print_help()
return
+28 -34
View File
@@ -21,12 +21,10 @@ PACKAGE_DIR = "packages"
REQUIREMENTS_FILE = "requirements.txt"
# 受保护的包名集合
_PROTECTED_PACKAGES: frozenset[str] = frozenset(
{
"pyflowx",
"bitool",
}
)
_PROTECTED_PACKAGES: frozenset[str] = frozenset({
"pyflowx",
"bitool",
})
# ============================================================================
@@ -161,37 +159,33 @@ def main() -> None:
if args.command == "i":
graph = px.Graph.from_specs([px.TaskSpec("pip_install", cmd=["pip", "install", *args.packages], verbose=True)])
elif args.command == "u":
graph = px.Graph.from_specs(
[px.TaskSpec("pip_uninstall", fn=pip_uninstall, args=(args.packages,), verbose=True)]
)
graph = px.Graph.from_specs([
px.TaskSpec("pip_uninstall", fn=pip_uninstall, args=(args.packages,), verbose=True)
])
elif args.command == "r":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pip_reinstall",
fn=pip_reinstall,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pip_reinstall",
fn=pip_reinstall,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
])
elif args.command == "d":
graph = px.Graph.from_specs(
[
px.TaskSpec(
"pip_download",
fn=pip_download,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"pip_download",
fn=pip_download,
args=(args.packages,),
kwargs={"offline": args.offline},
verbose=True,
)
])
elif args.command == "up":
graph = px.Graph.from_specs(
[px.TaskSpec("pip_upgrade", cmd=["python", "-m", "pip", "install", "--upgrade", "pip"], verbose=True)]
)
graph = px.Graph.from_specs([
px.TaskSpec("pip_upgrade", cmd=["python", "-m", "pip", "install", "--upgrade", "pip"], verbose=True)
])
elif args.command == "f":
graph = px.Graph.from_specs([px.TaskSpec("pip_freeze", fn=pip_freeze, verbose=True)])
else:
+58 -67
View File
@@ -9,48 +9,65 @@ from __future__ import annotations
import pyflowx as px
from pyflowx.conditions import Constants
MATURIN_BUILD_COMMAND = ["maturin", "build", "-r"]
if Constants.IS_WINDOWS:
MATURIN_BUILD_COMMAND.extend(["--target", "x86_64-win7-windows-msvc", "-Zbuild-std", "-i", "python3.8"])
def maturin_build_cmd() -> list[str]:
"""获取 maturin 构建命令(根据平台自动添加参数).
# 扁平注册所有任务(px.cmd 自动从命令前两段推导 name)
tasks: list[px.TaskSpec] = [
px.cmd(["uv", "build"]),
px.cmd(MATURIN_BUILD_COMMAND),
px.cmd(["uv", "sync"]),
px.cmd(["gitt", "c"], name="git_clean"),
px.cmd(
["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"],
name="test",
),
px.cmd(
["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"],
name="test_fast",
),
px.cmd(
["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
name="test_coverage",
),
px.cmd(["pyrefly", "check", "."]),
px.cmd(["git", "add", "-A"], name="git_add_all"),
px.cmd(["bumpversion"]),
px.cmd(["bumpversion", "minor"]),
px.cmd(["git", "push"]),
px.cmd(["git", "push", "--tags"], name="git_push_tags"),
px.cmd(["hatch", "publish"], name="publish_python"),
px.cmd(["twine", "upload", "--disable-progress-bar"], name="twine_publish"),
]
Returns
-------
list[str]
完整的 maturin 构建命令列表.
"""
command = ["maturin", "build", "-r"].copy()
if Constants.IS_WINDOWS:
command.extend(["--target", "x86_64-win7-windows-msvc", "-Zbuild-std", "-i", "python3.8"])
return command
# 单任务别名(alias 名与任务名相同):直接内联 TaskSpec,避免 str 自引用
aliases: dict[str, str | list[str | px.TaskSpec] | px.TaskSpec | px.Graph] = {
# 构建命令
"b": "uv_build",
"bc": "maturin_build",
"ba": ["b", "bc"],
# 安装命令
"sync": "uv_sync",
# 清理命令
"c": "git_clean",
# 开发工具
"bump": ["c", "tc", "git_add_all", "bumpversion"],
"bumpmi": "bumpversion_minor",
"cov": ["git_clean", "test_coverage"],
"doc": px.cmd(["sphinx-build", "-b", "html", "docs", "docs/_build"], name="doc"),
"lint": px.cmd(["ruff", "check", "--fix", "--unsafe-fixes"], name="lint"),
"pb": ["twine_publish", "publish_python"],
"t": "test",
"tf": "test_fast",
"tc": ["pyrefly_check", "lint"],
"tox": px.cmd(["tox", "-p", "auto"], name="tox"),
# 发布命令
"p": ["git_clean", "git_push", "git_push_tags"],
}
uv_build: px.TaskSpec = px.TaskSpec("uv_build", cmd=["uv", "build"])
maturin_build: px.TaskSpec = px.TaskSpec("maturin_build", cmd=maturin_build_cmd())
uv_sync: px.TaskSpec = px.TaskSpec("uv_sync", cmd=["uv", "sync"])
git_clean: px.TaskSpec = px.TaskSpec("git_clean", cmd=["gitt", "c"])
test: px.TaskSpec = px.TaskSpec(
"test", cmd=["pytest", "-m", "not slow", "-n", "8", "--dist", "loadfile", "--color=yes", "--durations=10"]
)
test_fast: px.TaskSpec = px.TaskSpec(
"test_fast", cmd=["pytest", "-m", "not slow", "--dist", "loadfile", "--color=yes", "--durations=10"]
)
test_coverage: px.TaskSpec = px.TaskSpec(
"test_coverage",
cmd=["pytest", "--cov", "-n", "8", "--dist", "loadfile", "--tb=short", "-v", "--color=yes", "--durations=10"],
)
ruff_lint: px.TaskSpec = px.TaskSpec("lint", cmd=["ruff", "check", "--fix", "--unsafe-fixes"])
typecheck: px.TaskSpec = px.TaskSpec("pyrefly_check", cmd=["pyrefly", "check", "."])
git_add_all: px.TaskSpec = px.TaskSpec("git_add_all", cmd=["git", "add", "-A"])
bump: px.TaskSpec = px.TaskSpec("bumpversion", cmd=["bumpversion"])
doc: px.TaskSpec = px.TaskSpec("doc", cmd=["sphinx-build", "-b", "html", "docs", "docs/_build"])
git_push: px.TaskSpec = px.TaskSpec("git_push", cmd=["git", "push"])
git_push_tags: px.TaskSpec = px.TaskSpec("git_push_tags", cmd=["git", "push", "--tags"])
hatch_publish: px.TaskSpec = px.TaskSpec("publish_python", cmd=["hatch", "publish"])
twine_publish: px.TaskSpec = px.TaskSpec("twine_publish", cmd=["twine", "upload", "--disable-progress-bar"])
tox: px.TaskSpec = px.TaskSpec("tox", cmd=["tox", "-p", "auto"])
def main():
def main() -> None:
"""pymake 构建工具.
🔨 构建命令:
@@ -78,10 +95,10 @@ def main():
📦 发布命令:
pymake pb - 发布到 PyPI (twine + hatch)
版本管理:
🔖 版本管理:
pymake bump - 自动升级版本号并提交修改 (清理 + 检查 + 格式化 + git add + bumpversion)
💡 常用工作流:
💡 常用工作流:
1. 日常开发: pymake lint && pymake t
2. 构建发布包: pymake ba
3. 多版本兼容性测试: pymake tox
@@ -95,31 +112,5 @@ def main():
pymake lint # 格式化代码
pymake type # 类型检查
"""
runner = px.CliRunner(
strategy="sequential",
description="PyMake - Python 构建工具",
graphs={
# 构建命令
"b": px.Graph.from_specs([uv_build]),
"bc": px.Graph.from_specs([maturin_build]),
"ba": px.Graph.from_specs(["b", "bc"]),
# 安装命令
"sync": px.Graph.from_specs([uv_sync]),
# 清理命令
"c": px.Graph.from_specs([git_clean]),
# 开发工具
"bump": px.Graph.from_specs(["c", "tc", git_add_all, bump]),
"bumpmi": px.Graph.from_specs([px.TaskSpec("bumpversion_minor", cmd=["bumpversion", "minor"])]),
"cov": px.Graph.from_specs([git_clean, test_coverage]),
"doc": px.Graph.from_specs([doc]),
"lint": px.Graph.from_specs([ruff_lint]),
"pb": px.Graph.from_specs([twine_publish, hatch_publish]),
"t": px.Graph.from_specs([test]),
"tf": px.Graph.from_specs([test_fast]),
"tc": px.Graph.from_specs([typecheck, "lint"]),
"tox": px.Graph.from_specs([tox]),
# 发布命令
"p": px.Graph.from_specs([git_clean, git_push, git_push_tags]),
},
)
runner = px.CliRunner(strategy="sequential", description="PyMake - Python 构建工具", tasks=tasks, aliases=aliases)
runner.run_cli()
+10
View File
@@ -0,0 +1,10 @@
from __future__ import annotations
import pyflowx as px
from pyflowx.tasks.system import reset_icon_cache
def main() -> None:
"""重启图标缓存工具主函数."""
graph = px.Graph.from_specs(reset_icon_cache())
px.run(graph, strategy="thread")
View File
+15
View File
@@ -0,0 +1,15 @@
"""清屏工具.
跨平台清屏工具, 支持终端和控制台清屏.
"""
from __future__ import annotations
import pyflowx as px
from pyflowx.tasks.system import clr
def main() -> None:
"""清屏工具主函数."""
graph = px.Graph.from_specs([clr()])
px.run(graph, strategy="thread")
@@ -35,6 +35,6 @@ def main() -> None:
[
px.TaskSpec(f"kill_{proc_name}", cmd=[*cmd, f"{proc_name}*"], verbose=True)
for proc_name in args.process_names
]
],
)
px.run(graph, strategy="thread")
+21
View File
@@ -0,0 +1,21 @@
"""命令查找工具.
跨平台查找可执行命令路径, 类似 Unix 的 which 命令.
"""
from __future__ import annotations
import argparse
import pyflowx as px
from pyflowx.tasks.system import which
def main() -> None:
"""命令查找工具主函数."""
parser = argparse.ArgumentParser(description="Which - 命令查找工具")
parser.add_argument("commands", nargs="+", help="要查找的命令名称, 如: python ls ps gcc...")
args = parser.parse_args()
graph = px.Graph.from_specs([which(cmd) for cmd in args.commands])
px.run(graph, strategy="thread")
-51
View File
@@ -1,51 +0,0 @@
"""命令查找工具.
跨平台查找可执行命令路径, 类似 Unix 的 which 命令.
"""
from __future__ import annotations
import argparse
import shutil
from pathlib import Path
import pyflowx as px
def which_command(command: str) -> Path | None:
"""查找命令路径.
Parameters
----------
command : str
命令名称
Returns
-------
Path | None
命令路径, 如果未找到则返回 None
"""
cmd_path = shutil.which(command)
if cmd_path:
print(f"匹配路径: - {cmd_path}")
return Path(cmd_path)
else:
print(f"{command}: 未找到")
return None
def main() -> None:
"""命令查找工具主函数."""
parser = argparse.ArgumentParser(
description="Which - 命令查找工具",
usage="which <command> [command ...]",
)
parser.add_argument(
"commands",
type=str,
nargs="+",
help="要查找的命令名称 (如: python pip node npm git uv rustc cargo)",
)
args = parser.parse_args()
graph = px.Graph.from_specs([px.TaskSpec(f"which_{cmd}", fn=which_command, args=(cmd,)) for cmd in args.commands])
px.run(graph, strategy="thread")
+98
View File
@@ -0,0 +1,98 @@
"""命令执行器:把 :class:`~pyflowx.task.TaskSpec` 的 ``cmd`` 字段(list /
shell 字符串 / 可调用对象)转换为统一执行入口。
历史背景:原 ``task.py`` 的模块文档声明其为"纯数据结构",但 ``_run_command``
属于命令执行逻辑,违反单一职责。此处将其抽离,``TaskSpec`` 仅持有配置,
执行逻辑集中于本模块,便于独立测试与维护。
"""
from __future__ import annotations
import os
import subprocess
from typing import Any, List, Union, cast
from .task import TaskSpec
__all__ = ["run_command"]
def run_command(spec: TaskSpec[Any]) -> Any: # noqa: PLR0912
"""执行 ``spec.cmd`` 指定的命令(list / shell 字符串 / 可调用对象)。
与原 ``TaskSpec._run_command`` 行为一致:
- 可调用对象:直接调用,异常包装为 :class:`RuntimeError`。
- list / str:通过 :func:`subprocess.run` 执行,非零返回码抛
:class:`RuntimeError```verbose=False`` 时附 stderr)。
- ``verbose=True`` 时打印执行信息与返回码到 stdout。
- ``cwd`` / ``env`` 通过 subprocess 参数隔离(进程级状态仅在 fn 任务路径
使用,cmd 路径不依赖 ``os.chdir`` / ``os.environ``)。
"""
cmd = spec.cmd
verbose = spec.verbose
cwd = spec.cwd
timeout = spec.timeout
env_override = spec.env
# 可调用对象:直接调用,返回其结果。
if callable(cmd) and not isinstance(cmd, (list, str)):
name = getattr(cmd, "__name__", "callable")
if verbose:
print(f"[verbose] 执行可调用命令: {name}", flush=True)
if cwd is not None:
print(f"[verbose] 工作目录: {cwd}", flush=True)
try:
return cmd()
except Exception as e:
raise RuntimeError(f"可调用命令执行异常: {name}: {e}") from e
is_list = isinstance(cmd, list)
if is_list:
cmd_str = " ".join(arg for arg in cmd) # type: ignore[union-attr]
verb = "执行命令"
label = "命令"
else:
cmd_str = cast(str, cmd)
verb = "执行 Shell"
label = "Shell 命令"
if verbose:
print(f"[verbose] {verb}: {cmd_str}", flush=True)
if cwd is not None:
print(f"[verbose] 工作目录: {cwd}", flush=True)
# 合并环境变量
run_env: dict[str, str] | None = None
if env_override:
run_env = dict(os.environ)
run_env.update(env_override)
try:
result = subprocess.run(
cast(Union[str, List[str]], cmd),
shell=not is_list,
cwd=cwd,
env=run_env,
timeout=timeout,
capture_output=not verbose,
text=True,
check=False,
)
except FileNotFoundError:
raise RuntimeError(f"{label}未找到: {cmd_str}") from None
except subprocess.TimeoutExpired:
raise RuntimeError(f"{label}执行超时: {cmd_str} ({timeout}s)") from None
except OSError as e:
raise RuntimeError(f"{label}执行异常: {cmd_str}: {e}") from e
if verbose:
print(f"[verbose] 返回码: {result.returncode}", flush=True)
if result.returncode == 0:
return None
err_msg = f"{label}执行失败: `{cmd_str}`, 返回码: {result.returncode}"
if not verbose and result.stderr.strip():
err_msg += f"\n{result.stderr.strip()}"
raise RuntimeError(err_msg)
+115
View File
@@ -0,0 +1,115 @@
"""图组合:将带字符串引用的多个图展开为纯 :class:`~pyflowx.graph.Graph`。
历史背景:原 ``graph.py`` 同时承载 DAG 构建/校验/分层与多图组合逻辑,
职责过载。组合逻辑(:class:`GraphComposer` / :func:`compose`)与单图 DAG
模型正交,此处抽离为独立模块,便于按需导入与独立演进。
"""
from __future__ import annotations
from dataclasses import replace
from typing import Any
from .graph import Graph
from .task import TaskSpec
__all__ = ["GraphComposer", "compose"]
class GraphComposer:
"""将带字符串引用的图展开为纯 :class:`TaskSpec` 图。
引用格式:
* ``"command_name"`` —— 引用整个命令图。
* ``"command_name.task_name"`` —— 引用特定任务。
引用按顺序展开,后续引用的任务依赖前面引用的最后一个任务;
原始 ``TaskSpec`` 之间也按出现顺序串行依赖。
"""
def __init__(self, graphs: dict[str, Graph]) -> None:
self.graphs = graphs
def resolve_all(self) -> dict[str, Graph]:
"""解析所有图的字符串引用,返回展开后的新图映射。"""
resolved: dict[str, Graph] = {}
for cmd_name, graph in self.graphs.items():
resolved[cmd_name] = self.expand_refs(graph, cmd_name)
return resolved
def expand_refs(self, graph: Graph, current_cmd: str) -> Graph:
"""展开图中的字符串引用。若无 ``_pending_refs``,原样返回。"""
pending_refs = graph._pending_refs
if not pending_refs:
return graph
all_specs: list[TaskSpec[Any]] = []
previous_ref_last_task: str | None = None
for ref in pending_refs:
expanded_specs = self.parse_ref(ref, current_cmd)
if previous_ref_last_task and expanded_specs:
for i, task in enumerate(expanded_specs):
if i == 0 or not task.depends_on:
expanded_specs[i] = replace(task, depends_on=tuple({*task.depends_on, previous_ref_last_task}))
if expanded_specs:
previous_ref_last_task = expanded_specs[-1].name
all_specs.extend(expanded_specs)
original_specs = list(graph.all_specs().values())
if original_specs:
if previous_ref_last_task:
first = original_specs[0]
all_specs.append(replace(first, depends_on=tuple({*first.depends_on, previous_ref_last_task})))
else:
all_specs.append(original_specs[0])
for i in range(1, len(original_specs)):
current_task = original_specs[i]
previous_task_name = original_specs[i - 1].name
all_specs.append(
replace(current_task, depends_on=tuple({*current_task.depends_on, previous_task_name}))
)
return Graph.from_specs(all_specs, defaults=graph.defaults)
def parse_ref(self, ref: str, current_cmd: str) -> list[TaskSpec[Any]]:
"""解析单个字符串引用,返回对应的 TaskSpec 列表。"""
if ref == current_cmd:
raise ValueError(f"循环引用: 命令 '{current_cmd}' 引用了自己")
if "." in ref:
cmd_name, task_name = ref.split(".", 1)
if cmd_name not in self.graphs:
raise ValueError(f"引用的命令 '{cmd_name}' 不存在")
ref_graph = self.graphs[cmd_name]
if task_name not in ref_graph.all_specs():
raise ValueError(f"任务 '{task_name}' 不存在于命令 '{cmd_name}'")
return [ref_graph.all_specs()[task_name]]
else:
cmd_name = ref
if cmd_name not in self.graphs:
raise ValueError(f"引用的命令 '{cmd_name}' 不存在")
ref_graph = self.graphs[cmd_name]
ref_graph = self.expand_refs(ref_graph, cmd_name)
return list(ref_graph.all_specs().values())
def compose(
graphs: dict[str, Graph],
) -> dict[str, Graph]:
"""编程式解析多图的字符串引用,返回展开后的新图映射。
与 :class:`GraphComposer` 等价,但作为独立函数暴露,供不使用
:class:`~pyflowx.runner.CliRunner` 的编程式用户调用。
Examples
--------
>>> graphs = {
... "build": px.Graph.from_specs([px.TaskSpec("b", cmd=["echo", "b"])]),
... "all": px.Graph.from_specs(["build", px.TaskSpec("t", cmd=["echo", "t"])]),
... }
>>> resolved = px.compose(graphs)
>>> "b" in resolved["all"].all_specs()
True
"""
return GraphComposer(graphs).resolve_all()
+190 -163
View File
@@ -1,18 +1,29 @@
"""条件判断模块.
提供平台条件、应用安装条件等预定义条件判断函数,
用于 TaskSpec 的条件执行功能.
所有条件均为 ``Callable[[Context], bool]``,接收依赖上下文映射(可能为空)。
这使得条件可基于上游任务的运行时返回值做决策,实现动态分支。
内置条件分两类:
1. *静态条件* —— 不依赖上下文(平台/环境变量/安装检查),通过 ``_static``
包装忽略传入的 context,便于作为模块级常量使用。
2. *上下文条件* —— 基于上游结果判断,如 :meth:`BuiltinConditions.DEP_EQUALS`。
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from typing import Callable
from pathlib import Path
from typing import Any, Callable
# 条件判断函数类型
Condition = Callable[[], bool]
from .task import Condition, Context
logger = logging.getLogger(__name__)
__all__ = ["BuiltinConditions", "Condition", "Constants"]
class Constants:
@@ -24,200 +35,216 @@ class Constants:
IS_POSIX: bool = sys.platform != "win32"
def _static(predicate: Callable[[], bool], name: str) -> Condition:
"""将无参谓词包装为忽略上下文的 :class:`Condition`。"""
def _cond(_ctx: Context) -> bool:
return predicate()
_cond.__name__ = name
return _cond
def _cond_name(cond: Condition) -> str:
"""获取条件的可读名称。"""
return getattr(cond, "__name__", repr(cond))
# ---------------------------------------------------------------------- #
# 模块级静态条件常量
# ---------------------------------------------------------------------- #
IS_WINDOWS: Condition = _static(lambda: Constants.IS_WINDOWS, "IS_WINDOWS")
IS_LINUX: Condition = _static(lambda: Constants.IS_LINUX, "IS_LINUX")
IS_MACOS: Condition = _static(lambda: Constants.IS_MACOS, "IS_MACOS")
IS_POSIX: Condition = _static(lambda: Constants.IS_POSIX, "IS_POSIX")
class BuiltinConditions:
"""内置条件判断函数集合."""
"""内置条件判断函数集合.
静态条件工厂返回忽略上下文的 :class:`Condition`;上下文条件工厂返回
会读取依赖结果的 :class:`Condition`。
"""
# ------------------------------------------------------------------ #
# 静态条件
# ------------------------------------------------------------------ #
@staticmethod
def IS_WINDOWS() -> Condition:
"""检查是否为 Windows 平台."""
return IS_WINDOWS
@staticmethod
def IS_WINDOWS() -> bool:
"""是否为 Windows 平台."""
return Constants.IS_WINDOWS
def IS_LINUX() -> Condition:
"""检查是否为 Linux 平台."""
return IS_LINUX
@staticmethod
def IS_LINUX() -> bool:
bool = Constants.IS_LINUX
return bool
def IS_MACOS() -> Condition:
"""检查是否为 macOS 平台."""
return IS_MACOS
@staticmethod
def IS_MACOS() -> bool:
"""是否为 macOS 平台."""
return Constants.IS_MACOS
def IS_POSIX() -> Condition:
"""检查是否为 POSIX 平台."""
return IS_POSIX
@staticmethod
def IS_POSIX() -> bool:
"""是否为 POSIX 系统 (Linux/macOS)."""
return Constants.IS_POSIX
@staticmethod
def PYTHON_VERSION(major: int, minor: int | None = None) -> bool:
"""检查 Python 版本是否匹配.
Parameters
----------
major : int
主版本号.
minor : int | None
次版本号, 若为 None 则仅检查主版本.
Returns
-------
bool
版本是否匹配.
"""
def PYTHON_VERSION(major: int, minor: int | None = None) -> Condition:
"""检查 Python 版本是否匹配."""
if minor is None:
return sys.version_info.major == major
return sys.version_info.major == major and sys.version_info.minor == minor
return _static(lambda: sys.version_info.major == major, f"PYTHON_VERSION({major})")
return _static(
lambda: sys.version_info.major == major and sys.version_info.minor == minor,
f"PYTHON_VERSION({major},{minor})",
)
@staticmethod
def PYTHON_VERSION_AT_LEAST(major: int, minor: int = 0) -> bool:
"""检查 Python 版本是否 >= 指定版本.
def PYTHON_VERSION_AT_LEAST(major: int, minor: int = 0) -> Condition:
"""检查 Python 版本是否 >= 指定版本."""
return _static(lambda: sys.version_info >= (major, minor), f"PYTHON_VERSION_AT_LEAST({major},{minor})")
Parameters
----------
major : int
主版本号.
minor : int
次版本号.
@staticmethod
def IS_RUNNING(app_name: str) -> Condition:
"""检查指定应用是否正在运行."""
Returns
-------
bool
当前版本是否 >= 指定版本.
"""
return sys.version_info >= (major, minor)
def _check() -> bool:
if Constants.IS_WINDOWS:
result = subprocess.run(
["tasklist", "/nh", "/fi", f"imagename eq {app_name}"],
capture_output=True,
text=True,
check=False,
)
return app_name.lower() in result.stdout.lower()
else:
result = subprocess.run(["pgrep", "-x", app_name], capture_output=True, check=False)
return result.returncode == 0
return _static(_check, f"IS_RUNNING({app_name!r})")
@staticmethod
def HAS_INSTALLED(app_name: str) -> Condition:
"""检查指定应用是否已安装.
"""检查指定应用是否已安装."""
return _static(lambda: shutil.which(app_name) is not None, f"HAS_INSTALLED({app_name!r})")
Parameters
----------
app_name : str
应用名称 (如 "git", "python", "pytest").
Returns
-------
Condition
条件判断函数.
"""
def _check() -> bool:
return shutil.which(app_name) is not None
_check.__name__ = f"HAS_INSTALLED({app_name!r})"
return _check
@staticmethod
def DIR_EXISTS(path: Path) -> Condition:
"""路径是否存在."""
return _static(path.exists, f"DIR_EXISTS({path!r})")
@staticmethod
def ENV_VAR_EXISTS(var_name: str) -> Condition:
"""检查环境变量是否存在.
Parameters
----------
var_name : str
环境变量名.
Returns
-------
Condition
条件判断函数.
"""
def _check() -> bool:
return var_name in os.environ
_check.__name__ = f"ENV_VAR_EXISTS({var_name!r})"
return _check
"""检查环境变量是否存在."""
return _static(lambda: var_name in os.environ, f"ENV_VAR_EXISTS({var_name!r})")
@staticmethod
def ENV_VAR_EQUALS(var_name: str, value: str) -> Condition:
"""检查环境变量是否等于指定值.
Parameters
----------
var_name : str
环境变量名.
value : str
期望的值.
Returns
-------
Condition
条件判断函数.
"""
def _check() -> bool:
return os.environ.get(var_name) == value
_check.__name__ = f"ENV_VAR_EQUALS({var_name!r}, {value!r})"
return _check
"""检查环境变量是否等于指定值."""
return _static(
lambda: os.environ.get(var_name) == value,
f"ENV_VAR_EQUALS({var_name!r},{value!r})",
)
@staticmethod
def NOT(condition: Condition) -> Condition:
"""对条件取反.
Parameters
----------
condition : Condition
原始条件.
Returns
-------
Condition
取反后的条件.
"""
def FILE_CONTENT_EXISTS(path: Path | str, content: str) -> Condition:
"""检查文件是否包含指定内容."""
def _check() -> bool:
return not condition()
p = Path(path)
if not p.exists():
return False
try:
return content in p.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
return False
_check.__name__ = f"NOT({getattr(condition, '__name__', repr(condition))})"
return _check
return _static(_check, f"FILE_CONTENT_EXISTS({path!r},{content!r})")
# ------------------------------------------------------------------ #
# 上下文条件:基于上游依赖结果
# ------------------------------------------------------------------ #
@staticmethod
def DEP_EQUALS(dep_name: str, value: Any) -> Condition:
"""上游任务 ``dep_name`` 的返回值等于 ``value`` 时为真。
若依赖未在上下文中(被跳过或未执行),返回 ``False``。
"""
def _cond(ctx: Context) -> bool:
return dep_name in ctx and ctx[dep_name] == value
_cond.__name__ = f"DEP_EQUALS({dep_name!r},{value!r})"
return _cond
@staticmethod
def DEP_MATCHES(dep_name: str, predicate: Callable[[Any], bool]) -> Condition:
"""上游任务 ``dep_name`` 的返回值满足 ``predicate`` 时为真。
依赖不存在时返回 ``False``。
"""
def _cond(ctx: Context) -> bool:
if dep_name not in ctx:
return False
try:
return predicate(ctx[dep_name])
except Exception as exc:
logger.warning("DEP_MATCHES predicate %r raised: %r", dep_name, exc)
return False
_cond.__name__ = f"DEP_MATCHES({dep_name!r},{getattr(predicate, '__name__', 'pred')})"
return _cond
@staticmethod
def DEP_PRESENT(dep_name: str) -> Condition:
"""上游任务 ``dep_name`` 存在于上下文(即已成功执行)时为真。"""
def _cond(ctx: Context) -> bool:
return dep_name in ctx and ctx[dep_name] is not None
_cond.__name__ = f"DEP_PRESENT({dep_name!r})"
return _cond
@staticmethod
def DEP_TRUTHY(dep_name: str) -> Condition:
"""上游任务 ``dep_name`` 的返回值为真值时为真。"""
def _cond(ctx: Context) -> bool:
return bool(ctx.get(dep_name))
_cond.__name__ = f"DEP_TRUTHY({dep_name!r})"
return _cond
# ------------------------------------------------------------------ #
# 逻辑组合
# ------------------------------------------------------------------ #
@staticmethod
def NOT(condition: Condition) -> Condition:
"""对条件取反."""
def _cond(ctx: Context) -> bool:
return not condition(ctx)
_cond.__name__ = f"NOT({_cond_name(condition)})"
return _cond
@staticmethod
def AND(*conditions: Condition) -> Condition:
"""多个条件的逻辑与.
"""多个条件的逻辑与."""
Parameters
----------
*conditions : Condition
条件列表.
def _cond(ctx: Context) -> bool:
return all(c(ctx) for c in conditions)
Returns
-------
Condition
组合条件.
"""
def _check() -> bool:
return all(c() for c in conditions)
names = [getattr(c, "__name__", repr(c)) for c in conditions]
_check.__name__ = f"AND({', '.join(names)})"
return _check
_cond.__name__ = f"AND({', '.join(_cond_name(c) for c in conditions)})"
return _cond
@staticmethod
def OR(*conditions: Condition) -> Condition:
"""多个条件的逻辑或.
"""多个条件的逻辑或."""
Parameters
----------
*conditions : Condition
条件列表.
def _cond(ctx: Context) -> bool:
return any(c(ctx) for c in conditions)
Returns
-------
Condition
组合条件.
"""
def _check() -> bool:
return any(c() for c in conditions)
names = [getattr(c, "__name__", repr(c)) for c in conditions]
_check.__name__ = f"OR({', '.join(names)})"
return _check
# 导出常用条件
IS_WINDOWS: Callable[[], bool] = BuiltinConditions.IS_WINDOWS
IS_LINUX: Callable[[], bool] = BuiltinConditions.IS_LINUX
IS_MACOS: Callable[[], bool] = BuiltinConditions.IS_MACOS
IS_POSIX: Callable[[], bool] = BuiltinConditions.IS_POSIX
_cond.__name__ = f"OR({', '.join(_cond_name(c) for c in conditions)})"
return _cond
+35 -60
View File
@@ -1,23 +1,22 @@
"""上下文注入:把上游结果转换为函数参数。
本机制让用户可以编写普通函数,其参数名*就是*依赖声明,从而消除其他
DAG 库中泛滥的样板包装器(如 ``def wrapper(): return fn(workflow.get_task_result('x'))``
DAG 库中泛滥的样板包装器。
注入规则(按顺序求值)
----------------------
1. **标注为** :class:`Context` 的参数接收完整结果映射。适用于需要遍历
所有输入的任务
2. **名称匹配某个依赖**的参数接收该依赖的结果。
1. **标注为** :class:`Context` 的参数接收完整结果映射(含硬依赖与软依赖)。
2. **名称匹配某个依赖**(硬或软)的参数接收该依赖的结果
3. ``**kwargs`` 参数以 dict 形式接收*所有*依赖结果。
4. ``TaskSpec.args`` / ``TaskSpec.kwargs`` 为*非依赖*参数提供静态值。
若某参数无法解析且无默认值,则抛出 :class:`~pyflowx.errors.InjectionError`
并附带精确错误信息。
若某参数无法解析且无默认值,则抛出 :class:`~pyflowx.errors.InjectionError`
"""
from __future__ import annotations
import inspect
from functools import lru_cache
from typing import Any, Mapping
from .errors import InjectionError
@@ -26,22 +25,30 @@ from .task import Context, TaskSpec
__all__ = ["Context", "_is_context_annotation", "build_call_args", "describe_injection"]
def _is_context_annotation(annotation: Any) -> bool:
"""判断参数标注是否为(或指向)``Context``。
@lru_cache(maxsize=1024)
def _cached_signature(fn: Any) -> inspect.Signature:
"""缓存 ``inspect.signature`` 结果(按 fn 对象键控)。
处理三种形式:
* ``Context`` 别名对象本身;
* ``__name__``/``_name`` 为 ``Context`` 或 ``Mapping`` 的 typing 别名;
* *字符串*标注(``from __future__ import annotations`` 会在运行时
把所有标注变为字符串),如 ``"Context"`` 或 ``"px.Context"``。
``fn`` 对象在 :meth:`TaskSpec.effective_fn` 缓存后稳定,签名重复内省
属纯开销。对不可哈希的可调用对象,调用方回退到直接内省。
"""
return inspect.signature(fn)
def _signature(fn: Any) -> inspect.Signature:
"""获取签名,优先走缓存;``fn`` 不可哈希时回退到直接内省。"""
try:
return _cached_signature(fn)
except TypeError:
return inspect.signature(fn)
def _is_context_annotation(annotation: Any) -> bool:
"""判断参数标注是否为(或指向)``Context``。"""
if annotation is Context:
return True
# `from __future__ import annotations` 产生的字符串标注。
if isinstance(annotation, str):
# 匹配 "Context"、"px.Context"、"pyflowx.Context" 等。
return annotation == "Context" or annotation.endswith(".Context")
# 按限定名匹配,支持 ``from pyflowx import Context`` 再导出。
name = getattr(annotation, "__name__", None) or getattr(annotation, "_name", None)
return name in ("Context", "Mapping")
@@ -52,39 +59,22 @@ def build_call_args(
) -> tuple[tuple[Any, ...], dict[str, Any]]:
"""解析用于调用 ``spec.fn`` 的 ``(args, kwargs)``。
参数
----
spec:
任务 spec,提供 ``fn``、``depends_on``、``args``、``kwargs``。
context:
依赖名 -> 结果值的映射。仅保证本任务自身的 ``depends_on`` 条目
存在;其他任务的结果被排除,以保持注入的确定性。
返回
----
(args, kwargs)
可直接展开为 ``spec.fn(*args, **kwargs)``。
抛出
----
InjectionError
若必需参数无法满足,或静态 ``kwargs`` 与注入依赖名冲突。
``context`` 必须已包含所有硬依赖与软依赖的结果(软依赖被跳过时由
执行器填入 :attr:`TaskSpec.defaults` 中的默认值)。
"""
# 使用 effective_fn 而不是 fn,以支持 cmd 参数
fn = spec.effective_fn
sig = inspect.signature(fn)
sig = _signature(fn)
params = sig.parameters
# 检测特殊参数类型。
var_keyword = next(
(p for p in params.values() if p.kind == inspect.Parameter.VAR_KEYWORD),
None,
)
# 本任务相关的上下文子集。
dep_context: dict[str, Any] = {name: context[name] for name in spec.depends_on if name in context}
# 本任务相关的上下文子集:硬依赖 + 软依赖
all_deps = set(spec.depends_on) | set(spec.soft_depends_on)
dep_context: dict[str, Any] = {name: context[name] for name in all_deps if name in context}
# 检测静态 kwargs 与依赖名的冲突。
collisions = set(spec.kwargs) & set(dep_context)
if collisions:
raise InjectionError(
@@ -96,8 +86,6 @@ def build_call_args(
injected_kwargs: dict[str, Any] = {}
leftover_dep_results: dict[str, Any] = dict(dep_context)
# 被 spec.args 消费的位置参数。记录哪些参数名已被位置填充,
# 以便在基于名称的注入(依赖 / Context / 静态 kwargs)时跳过。
positional_params: list[str] = []
positional_kinds = (
inspect.Parameter.POSITIONAL_ONLY,
@@ -106,33 +94,25 @@ def build_call_args(
for pname, param in params.items():
if param.kind in positional_kinds:
positional_params.append(pname)
# 前 len(spec.args) 个位置参数由 spec.args 填充。
args_filled: set[str] = set(positional_params[: len(spec.args)])
for pname, param in params.items():
# 跳过已被位置 spec.args 填充的参数。
if pname in args_filled:
continue
# 规则 1:标注为 Context -> 完整映射。
if _is_context_annotation(param.annotation):
injected_kwargs[pname] = dep_context
continue
# 规则 2:名称匹配某个依赖。
if pname in dep_context:
injected_kwargs[pname] = dep_context[pname]
leftover_dep_results.pop(pname, None)
continue
# 规则 3:在循环后通过 **kwargs 处理。
# 规则 4:静态 kwargs 填充其余参数。
if pname in spec.kwargs:
injected_kwargs[pname] = spec.kwargs[pname]
continue
# 该参数无来源:必须有默认值,否则报错。
if param.default is inspect.Parameter.empty and param.kind not in (
inspect.Parameter.VAR_POSITIONAL,
inspect.Parameter.VAR_KEYWORD,
@@ -142,9 +122,7 @@ def build_call_args(
f"parameter {pname!r} has no dependency, static value, or default.",
)
# 规则 3:**kwargs 吞掉剩余依赖结果。
if var_keyword is not None and leftover_dep_results:
# 先合并静态 kwargs,再合并依赖结果(冲突已在上方拒绝)。
merged = dict(spec.kwargs)
merged.update(injected_kwargs)
merged.update(leftover_dep_results)
@@ -154,14 +132,9 @@ def build_call_args(
def describe_injection(spec: TaskSpec[Any]) -> str:
"""生成任务参数注入方式的人类可读描述。
供 ``dry_run`` 使用,在不执行的情况下展示执行计划。
"""
# 使用 effective_fn 而不是 fn,以支持 cmd 参数
"""生成任务参数注入方式的人类可读描述。供 ``dry_run`` 使用。"""
fn = spec.effective_fn
sig = inspect.signature(fn)
# 确定哪些位置参数由 spec.args 填充。
sig = _signature(fn)
positional_params = [
p
for p, param in sig.parameters.items()
@@ -172,6 +145,7 @@ def describe_injection(spec: TaskSpec[Any]) -> str:
)
]
args_filled = set(positional_params[: len(spec.args)])
all_deps = set(spec.depends_on) | set(spec.soft_depends_on)
parts = []
for pname, param in sig.parameters.items():
if pname in args_filled:
@@ -179,8 +153,9 @@ def describe_injection(spec: TaskSpec[Any]) -> str:
parts.append(f"{pname}={spec.args[idx]!r}")
elif _is_context_annotation(param.annotation):
parts.append(f"{pname}=<Context>")
elif pname in spec.depends_on:
parts.append(f"{pname}=<result:{pname}>")
elif pname in all_deps:
tag = "soft" if pname in spec.soft_depends_on else "dep"
parts.append(f"{pname}=<{tag}:{pname}>")
elif pname in spec.kwargs:
parts.append(f"{pname}={spec.kwargs[pname]!r}")
elif param.default is not inspect.Parameter.empty:
+6 -8
View File
@@ -31,14 +31,12 @@ def aggregate(ctx: px.Context) -> dict[str, Any]:
def main() -> None:
graph = px.Graph.from_specs(
[
# Static positional args parameterise the same function twice.
px.TaskSpec("fetch_user", fetch_user, args=(1,)),
px.TaskSpec("fetch_posts", fetch_posts, args=(1,)),
px.TaskSpec("aggregate", aggregate, depends_on=("fetch_user", "fetch_posts")),
]
)
graph = px.Graph.from_specs([
# Static positional args parameterise the same function twice.
px.TaskSpec("fetch_user", fetch_user, args=(1,)),
px.TaskSpec("fetch_posts", fetch_posts, args=(1,)),
px.TaskSpec("aggregate", aggregate, depends_on=("fetch_user", "fetch_posts")),
])
print("=== Dry run ===")
_ = px.run(graph, strategy="async", dry_run=True)
+21 -19
View File
@@ -10,19 +10,21 @@ Demonstrates the core PyFlowX workflow:
from __future__ import annotations
from typing import Any
import pyflowx as px
# --- task functions: pure, testable, no framework coupling ------------- #
def extract_customers() -> list[dict]:
def extract_customers() -> list[dict[str, Any]]:
return [
{"id": "C001", "name": "Alice"},
{"id": "C002", "name": "Bob"},
]
def extract_orders() -> list[dict]:
def extract_orders() -> list[dict[str, Any]]:
return [
{"id": "O001", "customer_id": "C001", "amount": 150.0},
{"id": "O002", "customer_id": "C002", "amount": 200.5},
@@ -31,32 +33,32 @@ def extract_orders() -> list[dict]:
# Parameter names match dependency names → automatic injection.
def transform(
extract_customers: list[dict],
extract_orders: list[dict],
) -> list[dict]:
extract_customers: list[dict[str, Any]],
extract_orders: list[dict[str, Any]],
) -> list[dict[str, Any]]:
cmap = {c["id"]: c for c in extract_customers}
return [{**o, "customer_name": cmap[o["customer_id"]]["name"]} for o in extract_orders if o["customer_id"] in cmap]
def load(transform: list[dict]) -> int:
def load(transform: list[dict[str, Any]]) -> int:
print(f" loaded {len(transform)} records")
return len(transform)
def main() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("extract_customers", extract_customers, tags=("extract",)),
px.TaskSpec("extract_orders", extract_orders, tags=("extract",)),
px.TaskSpec(
"transform",
transform,
depends_on=("extract_customers", "extract_orders"),
tags=("transform",),
),
px.TaskSpec("load", load, depends_on=("transform",), retries=1, tags=("load",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("extract_customers", extract_customers, tags=("extract",)),
px.TaskSpec("extract_orders", extract_orders, tags=("extract",)),
px.TaskSpec(
"transform",
transform,
depends_on=("extract_customers", "extract_orders"),
tags=("transform",),
),
px.TaskSpec(
"load", load, depends_on=("transform",), retry=px.RetryPolicy(max_attempts=1, delay=1.0), tags=("load",)
),
])
print("=== Execution plan ===")
print(graph.describe())
+5 -7
View File
@@ -29,13 +29,11 @@ def merge(fetch_a: str, fetch_b: str) -> str:
def main() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("fetch_a", fetch_a),
px.TaskSpec("fetch_b", fetch_b),
px.TaskSpec("merge", merge, depends_on=("fetch_a", "fetch_b")),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("fetch_a", fetch_a),
px.TaskSpec("fetch_b", fetch_b),
px.TaskSpec("merge", merge, depends_on=("fetch_a", "fetch_b")),
])
print("=== Mermaid diagram ===")
print(graph.to_mermaid("LR"))
+666 -350
View File
File diff suppressed because it is too large Load Diff
+349 -127
View File
@@ -2,31 +2,139 @@
使用标准库的 :mod:`graphlib`3.9+ :mod:`graphlib_backport`3.8
进行拓扑排序图以增量方式构建并即时校验使配置错误在构建时而非执行时快速失败
支持
* 图级默认值 :class:`GraphDefaults`TaskSpec 字段为 ``None`` 时回退
* :meth:`Graph.map` 工厂批量生成 fan-out 任务
* 字符串引用与 :func:`compose` 编程式组合多个图
* 软依赖仅用于上下文注入不参与拓扑分层
"""
from __future__ import annotations
__all__ = [
"Graph",
"GraphDefaults",
]
import inspect
import sys
from dataclasses import dataclass, field
from typing import Any, Iterable, Mapping, Sequence
from dataclasses import dataclass, field, replace
from typing import Any, Callable, Iterable, Mapping, Sequence
from .errors import CycleError, DuplicateTaskError, MissingDependencyError
from .task import TaskSpec
from .task import Context, RetryPolicy, TaskSpec
# graphlib 自 3.9 起进入标准库;3.8 回退到 backport。
if sys.version_info >= (3, 9): # pragma: no cover
import graphlib # pyright: ignore[reportUnreachable]
_TopologicalSorter = graphlib.TopologicalSorter
else: # pragma: no cover
import graphlib # type: ignore[import-untyped] # pragma: no cover
import graphlib # type: ignore[import-untyped]
_TopologicalSorter = graphlib.TopologicalSorter # pragma: no cover
@dataclass(frozen=True)
@dataclass
class GraphDefaults:
"""图级默认值。TaskSpec 对应字段为 ``None`` 时回退到此处。
仅对可空字段生效retry/timeout/strategy/env/cwd/tags/priority/
continue_on_error/concurrency_key非空字段name/fn/cmd不回退
"""
retry: RetryPolicy | None = None
timeout: float | None = None
strategy: str | None = None
tags: tuple[str, ...] = ()
env: Mapping[str, str] | None = None
cwd: Any = None # Path | None
priority: int = 0
continue_on_error: bool = False
concurrency_key: str | None = None
verbose: bool = False
def _prune_deps(spec: TaskSpec[Any], keep: Callable[[str], bool]) -> TaskSpec[Any]:
"""返回新 spec,其 ``depends_on`` / ``soft_depends_on`` 仅保留 ``keep(dep)`` 为真的依赖。"""
return replace(
spec,
depends_on=tuple(d for d in spec.depends_on if keep(d)),
soft_depends_on=tuple(d for d in spec.soft_depends_on if keep(d)),
)
def _make_namespaced_fn(orig_fn: Any, ns: str, dep_names: set[str]) -> Any:
"""包装 fn,使其能接收带 ``ns:`` 前缀的依赖名,调用时映射回原参数名。
命名空间合并后依赖名带前缀 ``build:extract`` Python 参数名
不能含 ``:``wrapper ``**kwargs`` 接收所有依赖内部把带前缀的依赖名
映射回原参数名后调用原 fn
无依赖参数时直接返回原 fn
"""
if not dep_names or orig_fn is None:
return orig_fn
try:
orig_sig = inspect.signature(orig_fn)
except (TypeError, ValueError):
return orig_fn
# 带前缀依赖名 -> 原参数名
name_map: dict[str, str] = {f"{ns}:{orig}": orig for orig in dep_names}
prefix = f"{ns}:"
# 检查原 fn 是否有 Context 标注参数
context_param_name: str | None = None
for p in orig_sig.parameters.values():
ann = p.annotation
if ann is not Context and not (isinstance(ann, str) and ann.endswith("Context")):
continue
context_param_name = p.name
break
if context_param_name is not None:
def wrapper(ctx: Any = None, **kwargs: Any) -> Any:
# ctx 是 dep_context,键为带前缀的依赖名;映射回原始键
orig_ctx: dict[str, Any] = {}
for k, v in (ctx or {}).items():
orig_ctx[name_map.get(k, k)] = v
# kwargs 中带前缀的依赖也映射回原参数名
for k, v in kwargs.items():
if k in name_map:
orig_ctx[name_map[k]] = v
return orig_fn(**{context_param_name: orig_ctx})
ctx_param = inspect.Parameter("ctx", inspect.Parameter.POSITIONAL_OR_KEYWORD, annotation=Context)
kw_param = inspect.Parameter("kwargs", inspect.Parameter.VAR_KEYWORD)
wrapper.__signature__ = inspect.Signature( # type: ignore[attr-defined]
parameters=[ctx_param, kw_param],
return_annotation=orig_sig.return_annotation,
)
else:
def wrapper(**kwargs: Any) -> Any: # type: ignore[no-redef]
orig_kwargs: dict[str, Any] = {}
for k, v in kwargs.items():
if k.startswith(prefix):
orig_kwargs[k[len(prefix) :]] = v
return orig_fn(**orig_kwargs)
kw_param = inspect.Parameter("kwargs", inspect.Parameter.VAR_KEYWORD)
wrapper.__signature__ = inspect.Signature( # type: ignore[attr-defined]
parameters=[kw_param],
return_annotation=orig_sig.return_annotation,
)
wrapper.__name__ = f"{ns}_{getattr(orig_fn, '__name__', 'fn')}"
wrapper.__doc__ = getattr(orig_fn, "__doc__", None)
return wrapper
@dataclass
class Graph:
"""校验后不可变的有向无环任务图。
"""校验后的有向无环任务图。
通过添加 :class:`~pyflowx.task.TaskSpec` 实例构建每次 ``add``
执行即时校验重名缺失依赖:meth:`validate` / :meth:`layers`
@@ -38,106 +146,155 @@ class Graph:
specs: dict[str, TaskSpec[Any]] = field(default_factory=dict)
deps: dict[str, tuple[str, ...]] = field(default_factory=dict)
defaults: GraphDefaults = field(default_factory=GraphDefaults)
namespace: str | None = None
# 待解析的字符串引用列表(由 GraphComposer 消费);为空表示无引用。
_pending_refs: list[str] = field(default_factory=list)
# resolved_spec 缓存:避免执行期每个任务多次重复 dataclasses.replace 判断。
# 在 specs / defaults 变更时失效。
_resolved_cache: dict[str, TaskSpec[Any]] = field(default_factory=dict)
# ------------------------------------------------------------------ #
# 构建
# ------------------------------------------------------------------ #
def add(self, spec: TaskSpec[Any]) -> Graph:
"""注册一个任务 spec,并即时校验。
返回 ``self`` 以支持链式调用但推荐入口是 :meth:`from_specs`
它会整批校验允许单次调用中的前向引用
"""
if spec.name in self.specs:
raise DuplicateTaskError(spec.name)
self.specs[spec.name] = spec
self.deps[spec.name] = spec.depends_on
# 为增量 API 即时检查重名与缺失依赖。
"""注册一个任务 spec,并即时校验。返回 ``self`` 支持链式调用。"""
self._register(spec)
self._validate_references()
return self
@classmethod
def from_specs(cls, specs: Iterable[TaskSpec[Any] | str]) -> Graph:
"""从可迭代的 task spec 构建图.
def chain(self, *specs: TaskSpec[Any]) -> Graph:
"""链式注册任务:每个 spec 自动依赖前一个。
先收集所有 spec再统一校验这意味着任务可以引用*后出现*
依赖顺序无关就像声明式配置文件的读取方式
支持字符串引用允许引用其他命令图中的任务
字符串引用将在CliRunner中解析展开
Parameters
----------
specs : Iterable[TaskSpec[Any] | str]
TaskSpec对象或字符串引用的列表
Returns
-------
Graph
构建完成的图
Note
-----
字符串引用格式
- "command_name" - 引用整个命令图
- "command_name.task_name" - 引用特定任务
``chain(a, b, c)`` 等价于 ``b`` 依赖 ``a````c`` 依赖 ``b``
spec 已带 ``depends_on``则前驱名追加到现有依赖前
返回 ``self`` 支持链式调用
Examples
--------
>>> graph = Graph.from_specs([
... TaskSpec("build", cmd=["uv", "build"]),
... "test", # 引用test命令图
... ])
>>> graph = px.Graph().chain(extract, transform, load)
"""
graph = cls()
prev_name: str | None = None
for s in specs:
current = s
if prev_name is not None:
# 将前驱追加到 depends_on 最前(保持显式依赖优先)
new_deps = (prev_name, *s.depends_on) if prev_name not in s.depends_on else s.depends_on
current = replace(s, depends_on=new_deps)
self.add(current)
prev_name = current.name
return self
def _register(self, spec: TaskSpec[Any]) -> None:
if spec.name in self.specs:
raise DuplicateTaskError(spec.name)
self.specs[spec.name] = spec
# 拓扑依赖仅含硬依赖;软依赖仅用于注入,不影响分层。
self.deps[spec.name] = spec.depends_on
self._resolved_cache.clear()
@classmethod
def from_specs(
cls,
specs: Iterable[TaskSpec[Any] | str],
defaults: GraphDefaults | None = None,
*,
namespace: str | None = None,
) -> Graph:
"""从可迭代的 task spec 构建图。
先收集所有 spec再统一校验允许前向引用支持字符串引用
:func:`compose` :class:`GraphComposer` 解析展开
Parameters
----------
specs:
TaskSpec 对象或字符串引用的列表
defaults:
图级默认值``None`` 使用空 :class:`GraphDefaults`
namespace:
可选命名空间用于 :meth:`add_subgraph` 合并时加前缀
"""
graph = cls(defaults=defaults or GraphDefaults(), namespace=namespace)
pending_refs: list[str] = []
for spec in specs:
if isinstance(spec, str):
# 字符串引用,稍后解析
pending_refs.append(spec)
elif isinstance(spec, TaskSpec):
if spec.name in graph.specs:
raise DuplicateTaskError(spec.name)
graph.specs[spec.name] = spec
graph.deps[spec.name] = spec.depends_on
graph._register(spec)
else:
raise TypeError(f"from_specs只接受TaskSpecstr,收到: {type(spec)}")
raise TypeError(f"from_specs 只接受 TaskSpecstr,收到: {type(spec)}")
# 存储待解析的引用
if pending_refs:
# 使用特殊属性存储引用,稍后在CliRunner中解析
# 由于Graph是frozen dataclass,我们需要特殊处理
object.__setattr__(graph, "_pending_refs", pending_refs)
graph._pending_refs = pending_refs
graph._validate_references()
graph.validate()
return graph
def add_subgraph(self, sub: Graph, *, namespace: str | None = None) -> Graph:
"""将子图合并到当前图,任务名加命名空间前缀避免冲突。
参数
----
sub:
待合并的子图
namespace:
命名空间前缀``None`` 时使用 ``sub.namespace``若子图也无命名空间
则抛出 ``ValueError``最终任务名为 ``f"{ns}:{original_name}"``
合并后子图内任务的依赖名也会被加前缀与子图外部任务的依赖保持原样
返回 ``self`` 支持链式调用
"""
ns = namespace or sub.namespace
if not ns:
raise ValueError("add_subgraph 需要 namespace 或子图自带 namespace")
def _rename(name: str) -> str:
# 仅对子图内部任务名加前缀;外部依赖保持原样
return f"{ns}:{name}" if name in sub.specs else name
sub_names = set(sub.specs.keys())
for spec in sub.specs.values():
# 子图内部依赖名需加前缀,对应的 fn 参数也需包装
internal_deps = (set(spec.depends_on) | set(spec.soft_depends_on)) & sub_names
new_fn = _make_namespaced_fn(spec.fn, ns, internal_deps) if spec.fn else spec.fn
new_spec = replace(
spec,
name=_rename(spec.name),
fn=new_fn,
depends_on=tuple(_rename(d) for d in spec.depends_on),
soft_depends_on=tuple(_rename(d) for d in spec.soft_depends_on),
)
self._register(new_spec)
self._validate_references()
self.validate()
return self
# ------------------------------------------------------------------ #
# 校验
# ------------------------------------------------------------------ #
def _validate_references(self) -> None:
"""确保每个依赖名都存在于图中。"""
for name, deps in self.deps.items():
for dep in deps:
"""确保每个依赖名都存在于图中。硬依赖与软依赖都校验。"""
for name, spec in self.specs.items():
for dep in spec.depends_on:
if dep not in self.specs:
raise MissingDependencyError(name, dep)
for dep in spec.soft_depends_on:
if dep not in self.specs:
raise MissingDependencyError(name, dep)
def validate(self) -> None:
"""执行完整 DAG 校验。
存在环时抛出 :class:`~pyflowx.errors.CycleError`
依赖存在性由 :meth:`_validate_references` 检查
"""
"""执行完整 DAG 校验。存在环时抛出 :class:`CycleError`。"""
self._validate_references()
sorter = _TopologicalSorter(self.deps)
try:
# prepare() 在有环时抛出 CycleError;此处不需要
# static_order() 的结果,仅利用其校验副作用。
sorter.prepare()
except graphlib.CycleError as exc:
# exc.args[1] 是构成环的节点列表。
except graphlib.CycleError as exc: # type: ignore[name-defined]
cycle: Sequence[str] = exc.args[1] if len(exc.args) > 1 else []
raise CycleError(list(cycle)) from exc
@@ -153,10 +310,54 @@ class Graph:
"""返回 ``name`` 的 spec;不存在则 ``KeyError``。"""
return self.specs[name]
def resolved_spec(self, name: str) -> TaskSpec[Any]:
"""返回应用图级默认值后的 spec(不修改原图)。
对于 ``retry``/``timeout``/``strategy``/``env``/``cwd`` 等可空
字段 spec 字段为默认空值且图级默认值非空则用
:func:`dataclasses.replace` 生成带默认值的副本
结果按 ``name`` 缓存specs / defaults 变更时缓存失效
"""
cached = self._resolved_cache.get(name)
if cached is not None:
return cached
spec = self.specs[name]
d = self.defaults
overrides: dict[str, Any] = {}
if spec.retry == RetryPolicy() and d.retry is not None:
overrides["retry"] = d.retry
if spec.timeout is None and d.timeout is not None:
overrides["timeout"] = d.timeout
if spec.strategy is None and d.strategy is not None:
overrides["strategy"] = d.strategy
if spec.env is None and d.env is not None:
overrides["env"] = d.env
if spec.cwd is None and d.cwd is not None:
overrides["cwd"] = d.cwd
if spec.priority == 0 and d.priority != 0:
overrides["priority"] = d.priority
if not spec.continue_on_error and d.continue_on_error:
overrides["continue_on_error"] = True
if spec.concurrency_key is None and d.concurrency_key is not None:
overrides["concurrency_key"] = d.concurrency_key
if not spec.verbose and d.verbose:
overrides["verbose"] = True
if not spec.tags and d.tags:
overrides["tags"] = d.tags
resolved = spec if not overrides else replace(spec, **overrides)
self._resolved_cache[name] = resolved
return resolved
def dependencies(self, name: str) -> tuple[str, ...]:
"""``name`` 的直接前驱。"""
"""``name`` 的直接硬依赖前驱。"""
return self.deps[name]
def all_deps(self, name: str) -> tuple[str, ...]:
"""``name`` 的硬依赖 + 软依赖。"""
spec = self.specs[name]
return tuple(spec.depends_on) + tuple(spec.soft_depends_on)
def all_specs(self) -> Mapping[str, TaskSpec[Any]]:
"""name -> spec 的只读视图。"""
return self.specs
@@ -164,18 +365,18 @@ class Graph:
def layers(self) -> list[list[str]]:
"""将任务分组为可并行执行的层(Kahn 算法)。
同层任务无相互依赖可并发执行层按执行顺序返回
同层任务无相互依赖可并发执行软依赖不参与分层
层按执行顺序返回图有环时抛出 :class:`CycleError`
图有环时抛出 :class:`~pyflowx.errors.CycleError`
.. note::
本方法假定图已通过 :meth:`validate` 校验 :func:`pyflowx.run`
在入口统一执行一次若直接调用本方法需自行先校验
"""
self.validate()
sorter = _TopologicalSorter(self.deps)
result: list[list[str]] = []
# ``get_ready`` + ``done`` 每次给出一层,正好是并行执行所需的分组。
sorter.prepare()
while sorter.is_active():
ready = list(sorter.get_ready())
# 排序以保证确定性、可复现的执行计划。
ready.sort()
result.append(ready)
for node in ready:
@@ -186,35 +387,16 @@ class Graph:
# 子图 / 标签过滤
# ------------------------------------------------------------------ #
def subgraph(self, tags: Iterable[str]) -> Graph:
"""返回仅包含匹配任意标签的任务的新图。
依赖会被修剪仅保留被保留任务之间的边指向被丢弃任务的边
会被移除被保留的任务不再等待它们用于调试时运行大型
DAG 的切片
"""
"""返回仅包含匹配任意标签的任务的新图。依赖边被修剪。"""
wanted: set[str] = set(tags)
kept: list[TaskSpec[Any]] = []
for spec in self.specs.values():
if wanted & set(spec.tags):
pruned_deps = tuple(
d for d in spec.depends_on if d in self.specs and (wanted & set(self.specs[d].tags))
)
kept.append(
TaskSpec[Any](
name=spec.name,
fn=spec.fn,
cmd=spec.cmd,
depends_on=pruned_deps,
args=spec.args,
kwargs=spec.kwargs,
retries=spec.retries,
timeout=spec.timeout,
tags=spec.tags,
conditions=spec.conditions,
cwd=spec.cwd,
)
)
return Graph.from_specs(kept)
def _dep_kept(dep: str) -> bool:
return dep in self.specs and bool(wanted & set(self.specs[dep].tags))
kept: list[TaskSpec[Any]] = [
_prune_deps(spec, _dep_kept) for spec in self.specs.values() if wanted & set(spec.tags)
]
return Graph.from_specs(kept, defaults=self.defaults)
def subgraph_by_names(self, names: Iterable[str]) -> Graph:
"""返回限定于 ``names`` 的新图(边已修剪)。"""
@@ -222,36 +404,72 @@ class Graph:
for n in wanted:
if n not in self.specs:
raise KeyError(f"Unknown task name: {n!r}")
kept: list[TaskSpec[Any]] = []
for spec in self.specs.values():
if spec.name in wanted:
pruned_deps = tuple(d for d in spec.depends_on if d in wanted)
kept.append(
TaskSpec[Any](
name=spec.name,
fn=spec.fn,
cmd=spec.cmd,
depends_on=pruned_deps,
args=spec.args,
kwargs=spec.kwargs,
retries=spec.retries,
timeout=spec.timeout,
tags=spec.tags,
conditions=spec.conditions,
cwd=spec.cwd,
)
)
return Graph.from_specs(kept)
kept: list[TaskSpec[Any]] = [
_prune_deps(spec, lambda d: d in wanted) for spec in self.specs.values() if spec.name in wanted
]
return Graph.from_specs(kept, defaults=self.defaults)
# ------------------------------------------------------------------ #
# Fan-out / map-reduce
# ------------------------------------------------------------------ #
def map(
self,
name_fn: Callable[[int], str],
spec: TaskSpec[Any],
items: Sequence[Any],
arg_factory: Callable[[Any], tuple[Any, ...]] | None = None,
depends_on_per: Callable[[int], tuple[str, ...]] | None = None,
) -> list[TaskSpec[Any]]:
"""为 ``items`` 中每个元素生成一个 TaskSpec 并加入图。
用于 fan-out / map-reduce 模式返回生成的 spec 列表便于
后续 reduce 任务依赖
Parameters
----------
name_fn:
接受索引 ``i``返回任务名需保证唯一
spec:
模板 spec ``name`` ``args`` 会被覆盖
items:
待分发的数据序列
arg_factory:
接受一个 item返回位置参数元组覆盖 spec.args
``None`` 则将单个 item 作为唯一位置参数
depends_on_per:
接受索引 ``i``返回该任务的额外硬依赖``None`` 则继承 spec.depends_on
Returns
-------
list[TaskSpec]
生成的 spec 列表已加入图
Examples
--------
>>> fetch_tmpl = px.TaskSpec("", fn=fetch_user)
>>> specs = graph.map(lambda i: f"fetch_{i}", fetch_tmpl, [1, 2, 3])
>>> reduce_spec = px.TaskSpec("reduce", fn=reduce_fn, depends_on=tuple(s.name for s in specs))
"""
generated: list[TaskSpec[Any]] = []
for i, item in enumerate(items):
name = name_fn(i)
args = arg_factory(item) if arg_factory is not None else (item,)
extra_deps = depends_on_per(i) if depends_on_per is not None else ()
new_spec = replace(
spec,
name=name,
args=tuple(args),
depends_on=tuple(spec.depends_on) + tuple(extra_deps),
)
self.add(new_spec)
generated.append(new_spec)
return generated
# ------------------------------------------------------------------ #
# 可视化
# ------------------------------------------------------------------ #
def to_mermaid(self, orientation: str = "TD") -> str:
"""将 DAG 渲染为 Mermaid ``graph`` 定义字符串。
无外部依赖输出可粘贴到 Markdown VS Code Mermaid 预览
渲染或保存为文件
"""
"""将 DAG 渲染为 Mermaid ``graph`` 定义字符串。"""
valid = {"TD", "TB", "BT", "LR", "RL"}
orientation = orientation.upper()
if orientation not in valid:
@@ -262,6 +480,10 @@ class Graph:
for name, deps in self.deps.items():
for dep in deps:
lines.append(f" {dep} --> {name}")
# 软依赖用虚线
for name, spec in self.specs.items():
for dep in spec.soft_depends_on:
lines.append(f" {dep} -.-> {name}")
return "\n".join(lines) + "\n"
# ------------------------------------------------------------------ #
+16
View File
@@ -69,6 +69,22 @@ class RunReport:
"""以 FAILED 状态结束的任务名列表。"""
return [name for name, r in self.results.items() if r.status == TaskStatus.FAILED]
def succeeded_tasks(self) -> list[str]:
"""以 SUCCESS 状态结束的任务名列表。"""
return [name for name, r in self.results.items() if r.status == TaskStatus.SUCCESS]
def skipped_tasks(self) -> list[str]:
"""以 SKIPPED 状态结束的任务名列表。"""
return [name for name, r in self.results.items() if r.status == TaskStatus.SKIPPED]
def tasks_by_status(self, status: TaskStatus) -> list[str]:
"""返回指定状态的任务名列表。"""
return [name for name, r in self.results.items() if r.status == status]
def durations(self) -> dict[str, float]:
"""任务名 -> 执行时长(秒)。无时长记录的为 0.0。"""
return {name: (r.duration or 0.0) for name, r in self.results.items()}
def describe(self) -> str:
"""用于调试的人类可读多行报告。"""
lines: list[str] = [f"RunReport(success={self.success})"]
+101 -182
View File
@@ -15,8 +15,10 @@ import argparse
import enum
import sys
from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any, Sequence, get_args
from .compose import GraphComposer
from .errors import PyFlowXError
from .executors import Strategy, run
from .graph import Graph
@@ -39,6 +41,12 @@ def _apply_verbose_to_graph(graph: Graph, verbose: bool) -> Graph:
使用 ``dataclasses.replace`` 在不可变的 TaskSpec 上创建带 verbose 标记的副本.
依赖关系标签等元数据全部保留.
Note
-----
``_wrap_cmd`` 不再闭包捕获 ``verbose`` 此函数不再是必需的
直接翻转 ``spec.verbose`` 即可生效保留是为了向后兼容现有调用与测试
TaskSpec 仍是 frozen dataclass故仍用 ``replace`` 创建副本
Parameters
----------
graph : Graph
@@ -60,226 +68,137 @@ def _apply_verbose_to_graph(graph: Graph, verbose: bool) -> Graph:
return Graph.from_specs(new_specs)
@dataclass(frozen=True)
@dataclass
class CliRunner:
"""命令行运行器: 根据用户输入执行对应的任务流图.
将命令名映射到 Graph 实例.
通过 ``sys.argv`` 解析用户输入的命令, 执行对应的图.
将命令名映射到 Graph 实例. 通过 ``sys.argv`` 解析用户输入的命令,
执行对应的图.
Parameters
----------
aliases : dict[str, str | list[str] | Graph]
命令别名到任务引用的映射. 每个值可以是:
* ``str`` 单个任务名 (引用 ``tasks`` 中注册的任务),
生成单任务图.
* ``list[str]`` 任务名列表, 自动 :meth:`Graph.chain` 建立链式依赖,
即后一个任务依赖前一个.
* :class:`~pyflowx.graph.Graph` 直接使用该图 (用于复杂场景,
自定义 ``conditions``并行分支等).
tasks : list[TaskSpec]
扁平注册的任务列表. ``aliases`` 中的字符串引用这些任务名.
未被任何 alias 引用的任务不会被执行.
strategy : str | Strategy
默认执行策略 (``Strategy.SEQUENTIAL`` / ``Strategy.THREAD`` /
``Strategy.ASYNC`` 或对应字符串). 可被命令行 ``--strategy`` 覆盖.
默认执行策略. 可被命令行 ``--strategy`` 覆盖.
description : str
CLI 帮助文本.
verbose : bool
是否显示详细执行过程. ``True`` 时打印任务生命周期和 subprocess 输出.
默认 ``True``. 可被命令行 ``--quiet`` 关闭.
**graphs : Graph
命令名到图的映射. 每个 key 是一个命令名, value 是对应的
:class:`~pyflowx.graph.Graph`.
是否显示详细执行过程. 默认 ``True``, 可被命令行 ``--quiet`` 关闭.
Examples
--------
基本用法::
简单场景 (tasks + aliases)::
runner = px.CliRunner(
clean=px.Graph.from_specs(
[
px.TaskSpec("cargo_clean", cmd=["cargo", "clean"]),
]
),
build=px.Graph.from_specs(
[
px.TaskSpec("uv_build", cmd=["uv", "build"]),
]
),
tasks=[
px.cmd(["uv", "build"]), # name="uv_build"
px.cmd(["maturin", "build"], name="maturin_build"),
px.cmd(["ruff", "check", "--fix"], name="lint"),
],
aliases={
"b": "uv_build",
"ba": ["uv_build", "maturin_build"], # chain: maturin 依赖 uv
"lint": "lint",
},
)
runner.run() # 解析 sys.argv
runner.run()
指定策略与描述::
复杂场景 (直接用 Graph)::
runner = px.CliRunner(
strategy=px.Strategy.THREAD,
aliases={
"a": px.Graph.from_specs([
px.TaskSpec("add", cmd=["git", "add", "."], conditions=(...)),
px.TaskSpec("commit", cmd=["git", "commit"], depends_on=("add",)),
]),
},
)
runner.run(["test", "--strategy", "sequential"])
"""
graphs: dict[str, Graph] = field(default_factory=dict)
strategy: Strategy = field(default="sequential")
aliases: dict[str, str | list[str | TaskSpec[Any]] | TaskSpec[Any] | Graph] = field(default_factory=dict)
tasks: list[TaskSpec[Any]] = field(default_factory=list)
strategy: Strategy = field(default="dependency")
description: str = field(default_factory=str)
verbose: bool = field(default_factory=lambda: True)
# 解析后的命令→图映射,__post_init__ 填充
graphs: dict[str, Graph] = field(default_factory=dict, init=False)
def __post_init__(self) -> None:
if not self.graphs:
raise ValueError("CliRunner 至少需要一个命令 (通过关键字参数提供)")
if not self.aliases:
raise ValueError("CliRunner 至少需要一个别名 (通过 aliases= 提供)")
# 解析并展开字符串引用
self._resolve_graph_refs()
# 1. 把 tasks 注册为虚拟命令图(每个 task 一个图),加入 raw_graphs
# 使 GraphComposer 能解析对它们的字符串引用
raw_graphs: dict[str, Graph] = {}
for spec in self.tasks:
if spec.name in raw_graphs:
raise ValueError(f"任务名重复: {spec.name!r}")
raw_graphs[spec.name] = Graph.from_specs([spec])
def _resolve_graph_refs(self) -> None:
"""解析并展开图中的字符串引用.
# 2. 把每个 alias 转为 Graphalias 名可与 task 名相同,覆盖 task 注册)
for alias, value in self.aliases.items():
raw_graphs[alias] = self._alias_to_graph(alias, value)
支持两种引用格式
1. "command_name" - 引用整个命令图
2. "command_name.task_name" - 引用特定任务
# 3. 解析图间字符串引用(str / list[str] 引用其他 alias 或任务)
self.graphs = GraphComposer(raw_graphs).resolve_all()
递归解析所有引用直到所有图都只包含TaskSpec对象
@staticmethod
def _alias_to_graph(
alias: str,
value: str | list[str | TaskSpec[Any]] | TaskSpec[Any] | Graph,
) -> Graph:
"""把 alias 的值转换为 Graph.
* ``str`` 对其他 alias 或已注册任务名的引用, GraphComposer 展开.
* ``TaskSpec`` 单个内联任务, 生成单任务图.
* ``list[str | TaskSpec]`` 引用/任务混合列表, GraphComposer 展开时
自动让后续引用依赖前面 (chain 语义). 元素为 alias 任务名或
:class:`TaskSpec` 对象 (内联任务).
* ``Graph`` 原样返回 (用于复杂场景: conditions并行分支等).
"""
resolved_graphs: dict[str, Graph] = {}
for cmd_name, graph in self.graphs.items():
resolved_graph = self._expand_refs(graph, cmd_name)
resolved_graphs[cmd_name] = resolved_graph
# 更新graphs字典
object.__setattr__(self, "graphs", resolved_graphs)
def _expand_refs(self, graph: Graph, current_cmd: str) -> Graph:
"""展开图中的字符串引用.
Parameters
----------
graph : Graph
包含可能的字符串引用的图
current_cmd : str
当前命令名用于避免循环引用
Returns
-------
Graph
展开后的图只包含TaskSpec对象
Note
-----
引用按顺序展开后续引用的任务依赖于前面引用的任务完成
例如["c", "tc", bump] 会展开为
- c的所有任务无依赖
- tc的所有任务依赖于c的最后一个任务
- bump任务依赖于tc的最后一个任务
"""
# 检查是否有待解析的引用
pending_refs = getattr(graph, "_pending_refs", None)
if not pending_refs:
return graph
# 收集所有TaskSpec(按正确顺序:先引用,后原始TaskSpec)
all_specs: list[TaskSpec[Any]] = []
# 记录每个引用展开后的所有任务名,用于建立依赖链
previous_ref_last_task: str | None = None
# 先解析每个引用,并建立依赖关系
for ref in pending_refs:
expanded_specs = self._parse_ref(ref, current_cmd)
# 如果有前面的引用,让当前引用的所有任务依赖于前面引用的最后一个任务
if previous_ref_last_task and expanded_specs:
# 为当前引用的每个任务添加依赖
for i, task in enumerate(expanded_specs):
# 只为没有依赖的任务添加依赖,或者为第一个任务添加依赖
if i == 0 or not task.depends_on:
updated_task = replace(task, depends_on=tuple({*task.depends_on, previous_ref_last_task}))
expanded_specs[i] = updated_task
# 记录当前引用的最后一个任务名
if expanded_specs:
previous_ref_last_task = expanded_specs[-1].name
all_specs.extend(expanded_specs)
# 然后添加原始图中的TaskSpec,并让它们按顺序执行
original_specs = list(graph.all_specs().values())
if original_specs:
# 第一个原始TaskSpec依赖于最后一个引用的任务
if previous_ref_last_task:
first_original = original_specs[0]
updated_first = replace(
first_original, depends_on=tuple({*first_original.depends_on, previous_ref_last_task})
)
all_specs.append(updated_first)
else:
# 如果没有引用,直接添加第一个原始TaskSpec
all_specs.append(original_specs[0])
# 后续的原始TaskSpec依赖于前一个原始TaskSpec
for i in range(1, len(original_specs)):
current_task = original_specs[i]
previous_task_name = original_specs[i - 1].name
# 更新依赖,确保顺序执行
updated_task = replace(current_task, depends_on=tuple({*current_task.depends_on, previous_task_name}))
all_specs.append(updated_task)
# 创建新的图(不包含引用)
return Graph.from_specs(all_specs)
def _parse_ref(self, ref: str, current_cmd: str) -> list[TaskSpec[Any]]:
"""解析单个字符串引用.
Parameters
----------
ref : str
引用字符串"tc""tc.lint"
current_cmd : str
当前命令名用于避免循环引用
Returns
-------
list[TaskSpec[Any]]
解析后的TaskSpec列表
Raises
------
ValueError
如果引用无效或存在循环引用
"""
# 避免循环引用
if ref == current_cmd:
raise ValueError(f"循环引用: 命令 '{current_cmd}' 引用了自己")
# 解析引用格式
if "." in ref:
# 特定任务引用: "command_name.task_name"
cmd_name, task_name = ref.split(".", 1)
if cmd_name not in self.graphs:
raise ValueError(f"引用的命令 '{cmd_name}' 不存在")
# 获取特定任务
ref_graph = self.graphs[cmd_name]
if task_name not in ref_graph.all_specs():
raise ValueError(f"任务 '{task_name}' 不存在于命令 '{cmd_name}'")
return [ref_graph.all_specs()[task_name]]
else:
# 整个命令图引用: "command_name"
cmd_name = ref
if cmd_name not in self.graphs:
raise ValueError(f"引用的命令 '{cmd_name}' 不存在")
# 获取整个图的所有任务
ref_graph = self.graphs[cmd_name]
# 递归展开引用(如果引用的图也有引用)
ref_graph = self._expand_refs(ref_graph, cmd_name)
return list(ref_graph.all_specs().values())
if isinstance(value, Graph):
return value
if isinstance(value, TaskSpec):
return Graph.from_specs([value])
if isinstance(value, str):
# 字符串引用,用 _pending_refs 占位,GraphComposer 后续展开
return Graph.from_specs([value]) # type: ignore[arg-type]
if isinstance(value, list):
if not value:
raise ValueError(f"别名 {alias!r} 的任务列表为空")
for item in value:
if not isinstance(item, (str, TaskSpec)):
raise TypeError(f"别名 {alias!r} 的列表元素类型无效: {type(item).__name__}, 预期 str 或 TaskSpec")
# str/TaskSpec 混合列表,由 GraphComposer 展开(自动建立 chain 依赖)
return Graph.from_specs(value)
raise TypeError(
f"别名 {alias!r} 的值类型无效: {type(value).__name__}, 预期 str/TaskSpec/list[str|TaskSpec]/Graph"
)
# ------------------------------------------------------------------ #
# 内省
# ------------------------------------------------------------------ #
@property
def commands(self) -> list[str]:
"""可用的命令列表 (按插入顺序)."""
return list(self.graphs.keys())
"""可用的命令列表 (按 aliases 定义顺序, 不含 tasks 中未引用的任务)."""
return list(self.aliases.keys())
# ------------------------------------------------------------------ #
# 参数解析
# ------------------------------------------------------------------ #
def _prog_name(self) -> str:
"""从 sys.argv[0] 推导程序名."""
import os
return os.path.basename(sys.argv[0]) if sys.argv else "pyflowx"
return Path(sys.argv[0]).name if sys.argv else "pyflowx"
def create_parser(self) -> argparse.ArgumentParser:
"""创建参数解析器.
@@ -365,9 +284,9 @@ class CliRunner:
parser.print_help()
return CliExitCode.FAILURE.value
# 验证命令
if parsed.command not in self.graphs:
available = ", ".join(self.graphs.keys())
# 验证命令(必须是已注册的 alias,不接受裸任务名)
if parsed.command not in self.aliases:
available = ", ".join(self.commands)
print(
f"错误: 未知命令 {parsed.command!r} (可用命令: {available})",
file=sys.stderr,
+197 -47
View File
@@ -4,83 +4,197 @@
执行器向后端查询某任务是否已有存储结果若有则跳过该任务并将其
存储值注入下游任务
本模块刻意保持最小化仅持久化*成功*结果失败任务会重跑存储
形态为扁平的 ``{task_name: result}`` 映射内置两个后端
存储键由 :meth:`TaskSpec.storage_key` 计算默认为任务名若任务配置
``cache_key``则键为 ``"name:cache_key_value"``使不同输入产生
独立缓存条目
* :class:`MemoryBackend` 快速进程内 I/O默认
* :class:`JSONBackend` 持久化到 JSON 文件支持跨进程续跑
两者均零依赖``json`` 为标准库用户可子类化
:class:`StateBackend` 接入 SQLiteRedis
支持 TTL``has`` 在条目过期时返回 ``False``
"""
from __future__ import annotations
import json
import sys
import time
from abc import ABC, abstractmethod
from collections.abc import Iterator
from contextlib import contextmanager, nullcontext
from pathlib import Path
from typing import Any, Mapping
from typing import Any, ContextManager, Mapping
if sys.version_info >= (3, 12):
from typing import override
else:
from typing_extensions import override # pragma: no cover
from .errors import StorageError
class StateBackend(ABC):
"""可续跑状态存储的抽象基类。"""
"""可续跑状态存储的抽象基类。
所有方法以 ``key`` 为参数通常为任务名或 ``name:cache_key``
"""
@abstractmethod
def load(self) -> Mapping[str, Any]:
"""返回完整的存储映射(可能为空)。"""
@abstractmethod
def save(self, name: str, value: Any) -> None:
def save(self, key: str, value: Any) -> None:
"""持久化单个任务的成功结果。"""
@abstractmethod
def has(self, name: str) -> bool:
"""``name`` 是否已有存储结果。"""
def has(self, key: str) -> bool:
"""``key`` 是否已有未过期的存储结果。"""
@abstractmethod
def get(self, name: str) -> Any:
"""返回 ``name`` 的存储结果(不存在则抛 ``KeyError``)。"""
def get(self, key: str) -> Any:
"""返回 ``key`` 的存储结果(不存在则抛 ``KeyError``)。"""
@abstractmethod
def clear(self) -> None:
"""清除所有存储状态。"""
def flush(self) -> None: # noqa: B027
"""将内存中暂存的状态持久化到外部介质。
class MemoryBackend(StateBackend):
"""进程内 dict 后端。进程退出即丢失。"""
默认无操作 :class:`MemoryBackend` 无需落盘
:class:`JSONBackend` :meth:`batch` 期间会延迟落盘需在退出时调用
"""
def __init__(self) -> None:
self._store: dict[str, Any] = {}
def batch(self) -> ContextManager[None]:
"""返回一个上下文管理器,期间 :meth:`save` 可延迟 :meth:`flush`。
默认实现为 no-op :class:`MemoryBackend`:class:`JSONBackend`
覆盖为进入时标记延迟退出时统一 flush 一次将每任务一次落盘
N 次写入降为整次运行一次O(N) 而非 O()
"""
return nullcontext()
class _TTLStateBackendMixin(StateBackend):
"""TTL 状态后端共享逻辑。
``has`` / ``get`` / ``load`` / ``save`` / ``clear`` 的统一实现
委托给四个原始存取原语:meth:`_get_raw`:meth:`_put_raw`
:meth:`_iter_raw`:meth:`_clear_raw`并基于 :meth:`_now`
``self._ttl`` 提供统一的过期判断 :meth:`_is_expired`
子类需设置 ``self._ttl`` 并实现上述四个原语如需自定义时间源
``time.monotonic``可覆盖 :meth:`_now`
"""
_ttl: float | None
# ---- 原语:由子类实现 ---- #
@abstractmethod
def _get_raw(self, key: str) -> tuple[Any, float] | None:
"""返回 ``(value, ts)``;键不存在时返回 ``None``。"""
@abstractmethod
def _put_raw(self, key: str, value: Any, ts: float) -> None:
"""写入一条记录。"""
@abstractmethod
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
"""迭代所有记录(不做过期过滤),yield ``(key, value, ts)``。"""
@abstractmethod
def _clear_raw(self) -> None:
"""清空所有记录。"""
# ---- 共享实现 ---- #
def _now(self) -> float:
"""当前时间戳,默认为 wall-clock 秒。"""
return time.time()
def _is_expired(self, ts: float) -> bool:
"""时间戳 ``ts`` 是否已过期。"""
if self._ttl is None:
return False
return (self._now() - ts) > self._ttl
@override
def load(self) -> Mapping[str, Any]:
return dict(self._store)
return {k: v for k, v, ts in self._iter_raw() if not self._is_expired(ts)}
def save(self, name: str, value: Any) -> None:
self._store[name] = value
@override
def save(self, key: str, value: Any) -> None:
self._put_raw(key, value, self._now())
def has(self, name: str) -> bool:
return name in self._store
@override
def has(self, key: str) -> bool:
entry = self._get_raw(key)
return entry is not None and not self._is_expired(entry[1])
def get(self, name: str) -> Any:
return self._store[name]
@override
def get(self, key: str) -> Any:
entry = self._get_raw(key)
if entry is None or self._is_expired(entry[1]):
raise KeyError(key)
return entry[0]
@override
def clear(self) -> None:
self._clear_raw()
class MemoryBackend(_TTLStateBackendMixin):
"""进程内 dict 后端。进程退出即丢失。
Parameters
----------
ttl:
条目存活秒数``None`` 表示永不过期``has`` 在条目超过 ttl
返回 ``False``但不主动删除下次 ``save`` 覆盖
"""
def __init__(self, ttl: float | None = None) -> None:
self._store: dict[str, tuple[Any, float]] = {}
self._ttl = ttl
@override
def _now(self) -> float:
return time.monotonic()
@override
def _get_raw(self, key: str) -> tuple[Any, float] | None:
return self._store.get(key)
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
self._store[key] = (value, ts)
@override
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
for k, (v, ts) in self._store.items():
yield k, v, ts
@override
def _clear_raw(self) -> None:
self._store.clear()
class JSONBackend(StateBackend):
class JSONBackend(_TTLStateBackendMixin):
"""基于文件的 JSON 存储,用于跨进程续跑。
结果必须可 JSON 序列化不可序列化的值会抛出
:class:`~pyflowx.errors.StorageError`运行本身不会中止仅该条
结果的持久化失败
存储格式``{key: {"value": v, "ts": epoch_seconds}}``
``ts`` 用于 TTL 判断结果必须可 JSON 序列化
Parameters
----------
path:
JSON 文件路径
ttl:
条目存活秒数``None`` 表示永不过期
"""
def __init__(self, path: str) -> None:
def __init__(self, path: str, ttl: float | None = None) -> None:
self._path: str = path
self._store: dict[str, Any] = {}
self._ttl = ttl
self._store: dict[str, dict[str, Any]] = {}
self._defer_flush: bool = False
self._load()
def _load(self) -> None:
@@ -90,7 +204,13 @@ class JSONBackend(StateBackend):
with open(self._path, encoding="utf-8") as fh:
data: Any = json.load(fh)
if isinstance(data, dict):
self._store = data
# 兼容纯值格式与带元数据格式
self._store = {}
for k, v in data.items():
if isinstance(v, dict) and "value" in v and "ts" in v:
self._store[k] = v
else:
self._store[k] = {"value": v, "ts": time.time()}
except (OSError, json.JSONDecodeError) as exc:
raise StorageError(f"cannot read state file {self._path!r}", exc) from exc
@@ -99,32 +219,62 @@ class JSONBackend(StateBackend):
try:
with open(tmp, "w", encoding="utf-8") as fh:
json.dump(self._store, fh, ensure_ascii=False, indent=2)
_ = Path(tmp).replace(Path(self._path))
except (OSError, TypeError) as exc:
raise StorageError(f"cannot write state file {self._path!r}", exc) from exc
def load(self) -> Mapping[str, Any]:
return dict(self._store)
@override
def _get_raw(self, key: str) -> tuple[Any, float] | None:
entry = self._store.get(key)
if entry is None:
return None
return entry["value"], float(entry.get("ts", 0))
def save(self, name: str, value: Any) -> None:
# 在修改内存状态前先校验可序列化性。
@override
def _put_raw(self, key: str, value: Any, ts: float) -> None:
self._store[key] = {"value": value, "ts": ts}
@override
def _iter_raw(self) -> Iterator[tuple[str, Any, float]]:
for k, entry in self._store.items():
yield k, entry["value"], float(entry.get("ts", 0))
@override
def _clear_raw(self) -> None:
self._store.clear()
@override
def clear(self) -> None:
super().clear()
self._flush()
@override
def save(self, key: str, value: Any) -> None:
try:
_ = json.dumps(value)
except (TypeError, ValueError) as exc:
raise StorageError(f"result of task {name!r} is not JSON-serialisable", exc) from exc
self._store[name] = value
raise StorageError(f"result of key {key!r} is not JSON-serialisable", exc) from exc
super().save(key, value)
if not self._defer_flush:
self._flush()
@override
def flush(self) -> None:
self._flush()
def has(self, name: str) -> bool:
return name in self._store
@override
@contextmanager
def batch(self) -> Iterator[None]:
"""进入批量模式:``save`` 暂不落盘,退出时统一 flush 一次。
def get(self, name: str) -> Any:
return self._store[name]
def clear(self) -> None:
self._store.clear()
self._flush()
将整次运行 N 个任务的 N 次全量落盘降为 1
"""
self._defer_flush = True
try:
yield
finally:
self._defer_flush = False
self._flush()
def resolve_backend(backend: StateBackend | None) -> StateBackend:
+479 -183
View File
@@ -15,25 +15,38 @@
* ``TaskStatus`` 是封闭枚举执行器绝不发明临时字符串
"""
from __future__ import annotations
import logging
import os
import shutil
import sys
import threading
from contextlib import contextmanager
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from functools import cached_property
from pathlib import Path
from typing import (
Any,
Callable,
ContextManager,
Coroutine,
Generator,
Generic,
List,
Mapping,
Optional,
Tuple,
TypeVar,
Union,
cast,
)
T = TypeVar("T")
if sys.version_info >= (3, 13):
from typing import TypeVar
else:
from typing_extensions import TypeVar # pragma: no cover
T = TypeVar("T", default=Any)
# 任务可调用对象可以是同步或异步的。显式保留联合类型,让 mypy 理解两种形态。
TaskFn = Union[
@@ -52,8 +65,104 @@ TaskCmd = Union[
Callable[..., Any], # Python 函数
]
# 条件判断函数类型
Condition = Callable[[], bool]
# 执行策略:sequential/thread/async 为层屏障模型,dependency 为依赖驱动模型。
Strategy = Union[str, "StrategyKind"]
StrategyKind = Any # 占位,避免循环;executors 模块用 Literal 约束
logger = logging.getLogger(__name__)
# 条件判断函数类型:接收依赖上下文(可能为空映射),返回是否应执行。
Condition = Callable[[Context], bool]
# 缓存键计算函数:基于依赖上下文计算稳定字符串键。
CacheKeyFn = Callable[[Context], str]
def _format_skip_reason(failed_conditions: list[str]) -> str:
"""格式化跳过原因:≤2 个全展示,>2 个仅展示前 2 个并附总数。"""
if len(failed_conditions) <= 2:
return f"条件不满足: {', '.join(failed_conditions)}"
return f"条件不满足: {', '.join(failed_conditions[:2])}{len(failed_conditions)}个条件"
# ---------------------------------------------------------------------- #
# 重试策略
# ---------------------------------------------------------------------- #
@dataclass(frozen=True)
class RetryPolicy:
"""任务失败重试策略。
参数
----
max_attempts:
最大尝试次数含首次``1`` 表示仅尝试一次不重试
delay:
两次尝试之间的初始等待秒数
backoff:
退避倍率 n 次重试等待 ``delay * backoff ** (n-1)``
jitter:
抖动上限秒数每次等待加上 ``[0, jitter)`` 的随机量避免惊群
retry_on:
仅对这些异常类型重试默认 ``(Exception,)`` 重试所有异常
传入空元组等价于不重试
Note
-----
替代旧版 ``retries: int````retries=2`` 等价于
``RetryPolicy(max_attempts=3)``
"""
max_attempts: int = 1
delay: float = 0.0
backoff: float = 1.0
jitter: float = 0.0
retry_on: tuple[type[BaseException], ...] = (Exception,)
def __post_init__(self) -> None:
if self.max_attempts < 1:
raise ValueError(f"RetryPolicy.max_attempts must be >= 1, got {self.max_attempts}.")
if self.delay < 0:
raise ValueError(f"RetryPolicy.delay must be >= 0, got {self.delay}.")
if self.backoff < 0:
raise ValueError(f"RetryPolicy.backoff must be >= 0, got {self.backoff}.")
if self.jitter < 0:
raise ValueError(f"RetryPolicy.jitter must be >= 0, got {self.jitter}.")
@property
def retries(self) -> int:
"""重试次数(不含首次),等价于 ``max_attempts - 1``。"""
return self.max_attempts - 1
def should_retry(self, exc: BaseException) -> bool:
"""异常是否属于可重试类型。"""
return isinstance(exc, self.retry_on)
def wait_seconds(self, attempt: int) -> float:
"""第 ``attempt`` 次失败后应等待的秒数(attempt 从 1 开始)。"""
if attempt < 1:
return 0.0
import random
base = self.delay * (self.backoff ** max(0, attempt - 1))
jitter = random.uniform(0, self.jitter) if self.jitter > 0 else 0.0
return base + jitter
# ---------------------------------------------------------------------- #
# 任务钩子
# ---------------------------------------------------------------------- #
@dataclass(frozen=True)
class TaskHooks:
"""任务生命周期钩子。
所有钩子均为可选``pre_run`` 在任务实际执行前调用``post_run``
在成功后调用并接收返回值``on_failure`` 在最终失败后调用并接收异常
钩子异常不会影响任务状态仅记录日志
"""
pre_run: Callable[[TaskSpec[Any]], None] | None = None
post_run: Callable[[TaskSpec[Any], Any], None] | None = None
on_failure: Callable[[TaskSpec[Any], BaseException], None] | None = None
class TaskStatus(Enum):
@@ -83,235 +192,426 @@ class TaskSpec(Generic[T]):
- ``list[str]``: 命令及参数列表 ``["ls", "-la"]``
- ``str``: shell 命令字符串 ``"pip freeze > requirements.txt"``
- ``Callable``: Python 函数 ``fn`` 参数等效
若提供此参数会自动包装为执行函数覆盖 ``fn`` 参数
depends_on:
必须先完成才运行本任务的任务名列表顺序无关框架会做
拓扑排序
硬依赖任务名必须全部成功完成才运行本任务
上游被 SKIPPED 本任务也会被 SKIPPED除非
``allow_upstream_skip=True``
soft_depends_on:
软依赖任务名会等待其完成但其结果不影响本任务是否执行
- 上游成功注入其返回值
- 上游 SKIPPED 或失败注入 :attr:`defaults` 中提供的默认值
适用于"可选输入"场景
defaults:
软依赖的默认值映射 ``{dep_name: default_value}``
软依赖未提供结果时使用未在 defaults 中出现的软依赖默认为 ``None``
args:
静态位置参数追加在注入参数*之后*适用于参数化任务
``fetch_user(uid)``
静态位置参数追加在注入参数*之后*
kwargs:
静态关键字参数若与注入名冲突则抛出
:class:`~pyflowx.errors.InjectionError`
retries:
失败后的重试次数``0`` 表示仅尝试一次
retry:
:class:`RetryPolicy` 重试策略默认仅尝试一次
timeout:
最大执行时长``None`` 表示不限制异步任务使用
:func:`asyncio.wait_for`线程/异步执行器中的同步任务会
取消 worker future
:func:`asyncio.wait_for`同步任务通过线程 future 取消
tags:
自由标签 :meth:`Graph.subgraph` 做选择性执行与调试
自由标签 :meth:`Graph.subgraph` 做选择性执行与调试
也可用于并发限制分组
conditions:
条件判断函数列表只有所有条件都返回 ``True`` 时才执行任务
任一条件返回 ``False``任务被标记为 SKIPPED
用于平台判断环境变量检查等场景
条件判断函数列表接收依赖上下文全部返回 ``True`` 时才执行任务
任一返回 ``False`` 任务被标记为 SKIPPED
cwd:
命令执行的工作目录仅在使用 ``cmd`` 参数时有效
``None`` 表示当前目录
工作目录 ``cmd`` 任务作为子进程工作目录 ``fn`` 任务
通过临时切换当前目录生效
env:
环境变量覆盖映射 ``cmd`` 任务合并到子进程环境 ``fn``
任务在执行期间临时设置
verbose:
是否在命令执行时显示详细输出``True`` 打印执行的命令
及其标准输出/标准错误仅在使用 ``cmd`` 参数时有效
``False`` 时静默捕获输出失败时仍会包含在错误信息中
是否打印详细输出``True`` 时打印执行的命令返回码与输出
``cmd``以及任务生命周期
skip_if_missing:
仅对 ``cmd`` ``list[str]`` 的任务有效``True`` 自动检查
命令是否存在通过 :func:`shutil.which`不存在则跳过任务
标记为 SKIPPED而非失败适用于构建工具场景避免因未安装
某些工具 maturintox而导致整个图执行失败
对于 ``str`` (shell) ``Callable`` 类型的 ``cmd``此参数无效
仅对 ``cmd`` ``list[str]`` 有效``True`` 通过
:func:`shutil.which` 检查命令是否存在不存在则跳过
allow_upstream_skip:
若为 ``True``硬依赖被 SKIPPED 时本任务仍执行软依赖不影响
适用于清理类任务
strategy:
单任务执行策略覆盖``None`` 表示继承图级策略
``"sequential"`` 同步直接调用``"thread"``/``"async"`` 将同步
任务卸载到线程池异步任务跑在事件循环上
priority:
同层任务调度优先级数值越大越先启动仅影响同层内启动顺序
不打破层屏障默认 ``0``
concurrency_key:
并发限制分组键具有相同键的任务共享一个信号量限制同时
运行的实例数具体限额由 :func:`run` ``concurrency_limits``
参数提供 ``{key: limit}`` 映射``None`` 表示不限制
continue_on_error:
若为 ``True``任务最终失败时不中止整图仅标记本任务 FAILED
其硬依赖下游被 SKIPPED其余任务继续默认 ``False``
cache_key:
缓存键计算函数若提供则用其基于依赖上下文计算的字符串键
存取状态后端使不同输入产生独立缓存条目``None`` 表示用任务名
hooks:
:class:`TaskHooks` 生命周期钩子
executor:
同步任务的执行器``"thread"``默认线程池/ ``"process"``
进程池绕过 GIL适合 CPU 密集型``fn`` 须可 pickle/
``"inline"``直接在事件循环线程调用最快但会阻塞循环
"""
name: str
fn: Optional[TaskFn[T]] = None
cmd: Optional[TaskCmd] = None
depends_on: Tuple[str, ...] = ()
args: Tuple[Any, ...] = ()
fn: TaskFn[T] | None = None
cmd: TaskCmd | None = None
depends_on: tuple[str, ...] = ()
soft_depends_on: tuple[str, ...] = ()
defaults: Mapping[str, Any] = field(default_factory=dict)
args: tuple[Any, ...] = ()
kwargs: Mapping[str, Any] = field(default_factory=dict)
retries: int = 0
timeout: Optional[float] = None
tags: Tuple[str, ...] = ()
conditions: Tuple[Condition, ...] = ()
cwd: Optional[Path] = None
retry: RetryPolicy = field(default_factory=RetryPolicy)
timeout: float | None = None
tags: tuple[str, ...] = ()
conditions: tuple[Condition, ...] = ()
cwd: Path | None = None
env: Mapping[str, str] | None = None
verbose: bool = False
skip_if_missing: bool = True
skip_if_missing: bool = False
allow_upstream_skip: bool = False
strategy: str | None = None
priority: int = 0
concurrency_key: str | None = None
continue_on_error: bool = False
cache_key: CacheKeyFn | None = None
hooks: TaskHooks = field(default_factory=TaskHooks)
executor: str = "thread" # "thread" | "process" | "inline"
def __post_init__(self) -> None:
if not self.name:
raise ValueError("TaskSpec.name must be a non-empty string.")
if self.retries < 0:
raise ValueError(f"TaskSpec '{self.name}': retries must be >= 0.")
if self.retry.max_attempts < 1:
raise ValueError(f"TaskSpec '{self.name}': retry.max_attempts must be >= 1.")
if self.timeout is not None and self.timeout <= 0:
raise ValueError(f"TaskSpec '{self.name}': timeout must be > 0.")
if self.name in self.depends_on:
if self.name in self.depends_on or self.name in self.soft_depends_on:
raise ValueError(f"TaskSpec '{self.name}' cannot depend on itself.")
overlap = set(self.depends_on) & set(self.soft_depends_on)
if overlap:
raise ValueError(f"TaskSpec '{self.name}': depends_on 与 soft_depends_on 不能重叠: {sorted(overlap)}")
if self.fn is None and self.cmd is None:
raise ValueError(f"TaskSpec '{self.name}': 必须提供 fn 或 cmd 参数。")
@property
@cached_property
def effective_fn(self) -> TaskFn[T]:
"""获取有效的执行函数.
"""获取有效的执行函数
若提供 ``cmd`` 参数返回包装后的命令执行函数
否则返回 ``fn`` 参数
若提供 ``cmd``返回包装后的命令执行函数否则返回 ``fn``
包装函数在每次调用时从 ``self`` 读取 ``verbose``/``cwd``/``env``/
``timeout``避免闭包捕获运行期参数使翻转字段无需重建 spec
结果按实例缓存:func:`functools.cached_property`frozen dataclass
字段不可变``_wrap_cmd`` 生成的闭包稳定无需每次访问重建
"""
if self.cmd is not None:
return self._wrap_cmd()
if self.fn is not None:
return self.fn
raise ValueError(f"TaskSpec '{self.name}': 没有可执行的函数或命令。") # pragma: no cover
def _wrap_cmd(self) -> TaskFn[Any]:
"""将 cmd 包装为可执行函数.
"""将 cmd 包装为可执行函数
实际执行逻辑位于 :mod:`pyflowx.command`避免 :class:`TaskSpec`
作为纯数据结构混入命令执行逻辑
"""
from .command import run_command
spec = self
def _run() -> T:
return cast(T, run_command(spec))
_run.__name__ = spec.name
return _run # type: ignore[return-value]
def should_execute(self, context: Context) -> tuple[bool, str | None]:
"""检查任务是否应执行。
Returns
-------
TaskFn[Any]
包装后的执行函数.
(should_run, skip_reason)
``should_run`` False ``skip_reason`` 描述跳过原因
失败条件超过 2 个时仅展示前 2 个并附总数
"""
cmd = self.cmd
cwd = self.cwd
timeout = self.timeout
verbose = self.verbose
if isinstance(cmd, list):
cmd_list = cast(List[str], cmd)
def _run_list() -> T:
import subprocess
cmd_str = " ".join(str(arg) for arg in cmd_list)
if verbose:
print(f"[verbose] 执行命令: {cmd_str}", flush=True)
if cwd is not None:
print(f"[verbose] 工作目录: {cwd}", flush=True)
try:
result = subprocess.run(
cmd_list,
cwd=cwd,
timeout=timeout,
capture_output=not verbose,
text=True,
check=False,
# 逐个求值条件,记录失败项。
failed_conditions: list[str] = []
for condition in self.conditions:
try:
ok = condition(context)
except Exception:
ok = False
failed_conditions.append("匿名条件(执行错误)")
continue
if not ok:
reason = getattr(condition, "_reason", None)
if reason is not None:
failed_conditions.append(
", ".join(str(r) for r in reason) if isinstance(reason, list) else str(reason),
)
except FileNotFoundError:
raise RuntimeError(f"命令未找到: {cmd_str}") from None
except subprocess.TimeoutExpired:
raise RuntimeError(f"命令执行超时: {cmd_str} ({timeout}s)") from None
except OSError as e:
raise RuntimeError(f"命令执行异常: {cmd_str}: {e}") from e
else:
failed_conditions.append(getattr(condition, "__name__", None) or "匿名条件")
if verbose:
print(f"[verbose] 返回码: {result.returncode}", flush=True)
if failed_conditions:
return False, _format_skip_reason(failed_conditions)
if result.returncode == 0:
return cast(T, None) # type: ignore[return-value]
if self.skip_if_missing and not self._is_cmd_available():
cmd_name = self.cmd[0] if isinstance(self.cmd, list) and self.cmd else "unknown"
return False, f"命令不存在: {cmd_name}"
err_msg = f"命令执行失败: `{cmd_str}`, 返回码: {result.returncode}"
if not verbose and result.stderr.strip():
err_msg += f"\n{result.stderr.strip()}"
raise RuntimeError(err_msg)
_run_list.__name__ = self.name
return _run_list # type: ignore[return-value]
if isinstance(cmd, str):
def _run_shell() -> T:
import subprocess
if verbose:
print(f"[verbose] 执行 Shell: {cmd}", flush=True)
if cwd is not None:
print(f"[verbose] 工作目录: {cwd}", flush=True)
try:
result = subprocess.run(
cmd,
shell=True,
cwd=cwd,
timeout=timeout,
capture_output=not verbose,
text=True,
check=False,
)
except FileNotFoundError:
raise RuntimeError(f"Shell 命令未找到: {cmd}") from None
except subprocess.TimeoutExpired:
raise RuntimeError(f"Shell 命令执行超时: {cmd} ({timeout}s)") from None
except OSError as e:
raise RuntimeError(f"Shell 命令执行异常: {cmd}: {e}") from e
if verbose:
print(f"[verbose] 返回码: {result.returncode}", flush=True)
if result.returncode == 0:
return cast(T, None) # type: ignore[return-value]
err_msg = f"Shell 命令执行失败: `{cmd}`, 返回码: {result.returncode}"
if not verbose and result.stderr.strip():
err_msg += f"\n{result.stderr.strip()}"
raise RuntimeError(err_msg)
_run_shell.__name__ = self.name
return _run_shell # type: ignore[return-value]
if callable(cmd):
return cmd # type: ignore[return-value]
raise TypeError(f"TaskSpec '{self.name}': 不支持的 cmd 类型 {type(cmd).__name__}") # pragma: no cover
def should_execute(self) -> bool:
"""检查任务是否应该执行.
Returns
-------
bool
若所有条件都返回 ``True`` ``skip_if_missing`` 检查通过
则返回 ``True``否则返回 ``False``
"""
if not all(condition() for condition in self.conditions):
return False
return not (self.skip_if_missing and not self._is_cmd_available())
return True, None
def _is_cmd_available(self) -> bool:
"""检查 ``cmd`` 是否可用.
仅对 ``list[str]`` 类型的 ``cmd`` 进行检查通过 :func:`shutil.which`
对于 ``str`` (shell) ``Callable`` 类型始终返回 ``True``
Returns
-------
bool
命令可用返回 ``True``否则返回 ``False``
"""
import shutil
"""检查 ``cmd`` 是否可用(仅 list[str])。"""
cmd = self.cmd
if isinstance(cmd, list) and cmd:
first_arg = cast(str, cmd[0])
return shutil.which(first_arg) is not None
return shutil.which(cmd[0]) is not None
return True
def env_context(self) -> ContextManager[None]:
"""返回临时应用 ``env`` 与 ``cwd`` 的上下文管理器。
``fn`` 任务生效``cmd`` 任务在 :func:`_run_command` 中直接
传给子进程
"""
return _env_and_cwd(self.env, self.cwd)
def storage_key(self, context: Context) -> str:
"""计算状态后端存储键。"""
if self.cache_key is None:
return self.name
try:
return f"{self.name}:{self.cache_key(context)}"
except (TypeError, ValueError, KeyError, AttributeError) as exc:
# cache_key 抛出预期内的数据/类型异常时回退到 name,但仍记录警告
# 以便用户发现 cache_key 实现中的 bug。
logger.warning(
"task %r: cache_key 回退到 name%s: %s",
self.name,
type(exc).__name__,
exc,
)
return self.name
# 全局锁:序列化对进程级状态(os.environ / os.chdir)的临时修改。
# ``fn`` 任务在 thread/async 策略下并发执行时,若各自配置了不同的
# ``cwd``/``env``,会相互覆盖(os.chdir 与 os.environ 均为进程全局)。
# 该锁仅包裹"切换→执行→恢复"区间,保证正确性;不使用 cwd/env 的任务不受影响。
_env_cwd_lock = threading.RLock()
@contextmanager
def _env_and_cwd(
env: Mapping[str, str] | None,
cwd: Path | None,
) -> Generator[None, None, None]:
"""临时设置环境变量与工作目录。
``os.environ`` ``os.chdir`` 是进程级全局状态 thread/async 策略下
并发执行多个带 ``env``/``cwd`` ``fn`` 任务时会相互覆盖本函数通过
模块级 :data:`_env_cwd_lock` 串行化"切换→执行→恢复"区间确保正确性
``env`` 且无 ``cwd`` 时直接 yield不获取锁
"""
if not env and cwd is None:
yield
return
with _env_cwd_lock:
saved_env: dict[str, str] = {}
saved_cwd: str | None = None
if env:
for k, v in env.items():
if k in os.environ:
saved_env[k] = os.environ[k]
os.environ[k] = v
if cwd is not None:
saved_cwd = str(Path.cwd())
os.chdir(cwd)
try:
yield
finally:
if saved_cwd is not None:
os.chdir(saved_cwd)
# 恢复环境变量
if env:
for k in env:
if k in saved_env:
os.environ[k] = saved_env[k]
else:
os.environ.pop(k, None)
# ---------------------------------------------------------------------- #
# 任务模板:批量生成相似 TaskSpec 的工厂
# ---------------------------------------------------------------------- #
def _task_noop() -> None:
"""task(cmd=...) 形式下的占位 fn(cmd 任务执行期不调用 fn)。"""
return None
def task(
fn: TaskFn[Any] | None = None,
*,
cmd: TaskCmd | None = None,
depends_on: tuple[str, ...] = (),
soft_depends_on: tuple[str, ...] = (),
defaults: Mapping[str, Any] | None = None,
args: tuple[Any, ...] = (),
kwargs: Mapping[str, Any] | None = None,
retry: RetryPolicy | None = None,
timeout: float | None = None,
tags: tuple[str, ...] = (),
conditions: tuple[Condition, ...] = (),
cwd: str | Path | None = None,
env: Mapping[str, str] | None = None,
verbose: bool = False,
skip_if_missing: bool = False,
allow_upstream_skip: bool = False,
strategy: str | None = None,
priority: int = 0,
concurrency_key: str | None = None,
continue_on_error: bool = False,
cache_key: CacheKeyFn | None = None,
hooks: TaskHooks | None = None,
name: str | None = None,
) -> Any:
"""装饰器:将函数转为 :class:`TaskSpec`。
``name`` 默认取 ``fn.__name__``可直接装饰函数或带参数使用
Examples
--------
>>> @px.task
... def extract(): return [1, 2, 3]
>>> @px.task(depends_on=("extract",))
... def double(extract): return [x * 2 for x in extract]
>>> graph = px.Graph.from_specs([extract, double])
"""
def _decorate(func: TaskFn[Any]) -> TaskSpec[Any]:
spec_name = name or func.__name__
return TaskSpec(
name=spec_name,
fn=func,
cmd=cmd,
depends_on=depends_on,
soft_depends_on=soft_depends_on,
defaults=dict(defaults) if defaults else {},
args=args,
kwargs=dict(kwargs) if kwargs else {},
retry=retry if retry is not None else RetryPolicy(),
timeout=timeout,
tags=tags,
conditions=conditions,
cwd=Path(cwd) if isinstance(cwd, str) else cwd,
env=dict(env) if env else None,
verbose=verbose,
skip_if_missing=skip_if_missing,
allow_upstream_skip=allow_upstream_skip,
strategy=strategy,
priority=priority,
concurrency_key=concurrency_key,
continue_on_error=continue_on_error,
cache_key=cache_key,
hooks=hooks if hooks is not None else TaskHooks(),
)
if fn is None and cmd is None:
# 带参数调用:@task(depends_on=...),等待被装饰函数
return _decorate
if fn is None:
# task(cmd=..., name=...) 直接构造,无被装饰函数
if name is None:
raise ValueError("task(cmd=...) 需要显式提供 name")
return _decorate(_task_noop)
return _decorate(fn)
def cmd(
command: list[str],
*,
name: str | None = None,
depends_on: tuple[str, ...] = (),
**kwargs: Any,
) -> TaskSpec[Any]:
"""从命令列表快速创建 :class:`TaskSpec`。
``name`` 默认为 ``"_".join(command[:2])`` ``["uv", "build"]`` ``"uv_build"``
若命令不足两个元素则用 ``"_".join(command)``
其余关键字参数透传给 :class:`TaskSpec` ``depends_on````tags``
Examples
--------
>>> uv_build = px.cmd(["uv", "build"])
>>> uv_build.name
'uv_build'
>>> lint = px.cmd(["ruff", "check", "--fix"], name="lint")
>>> lint.name
'lint'
"""
spec_name = name or "_".join(command[:2]) if len(command) >= 2 else "_".join(command)
return TaskSpec(
name=spec_name,
cmd=command,
depends_on=depends_on,
**kwargs,
)
def task_template(
fn: TaskFn[Any] | None = None,
cmd: TaskCmd | None = None,
**defaults: Any,
) -> Callable[..., TaskSpec[Any]]:
"""创建任务模板工厂。
返回的工厂接受 ``name`` 与任意覆盖字段生成 :class:`TaskSpec`
适用于批量创建相似任务 fan-out
Examples
--------
>>> Fetch = px.task_template(fn=fetch_user, retry=px.RetryPolicy(max_attempts=3))
>>> specs = [Fetch(f"fetch_{uid}", args=(uid,)) for uid in range(5)]
"""
base = dict(defaults)
if fn is not None:
base["fn"] = fn
if cmd is not None:
base["cmd"] = cmd
def _factory(name: str, **overrides: Any) -> TaskSpec[Any]:
merged = dict(base)
merged.update(overrides)
return TaskSpec(name, **merged)
_factory.__name__ = "task_template_factory"
return _factory
@dataclass
class TaskResult(Generic[T]):
"""运行期间产生的可变单任务记录。
每次运行都会创建全新的 :class:`TaskResult`spec 本身保持不可变
这让同一个图可以安全地重复运行
"""
"""运行期间产生的可变单任务记录。"""
spec: TaskSpec[T]
status: TaskStatus = TaskStatus.PENDING
value: Optional[T] = None
error: Optional[BaseException] = None
value: T | None = None
error: BaseException | None = None
attempts: int = 0
started_at: Optional[datetime] = None
finished_at: Optional[datetime] = None
reason: Optional[str] = None # 跳过原因
started_at: datetime | None = None
finished_at: datetime | None = None
reason: str | None = None # 跳过原因
@property
def duration(self) -> Optional[float]:
def duration(self) -> float | None:
"""从开始到结束的耗时(秒),未开始/未结束则为 ``None``。"""
if self.started_at is None or self.finished_at is None:
return None
@@ -320,15 +620,11 @@ class TaskResult(Generic[T]):
@dataclass(frozen=True)
class TaskEvent:
"""执行期间向观察者发出的不可变事件。
传递给 :func:`pyflowx.run` ``on_event`` 回调让调用者无需耦合
执行器内部即可构建进度条指标或结构化日志
"""
"""执行期间向观察者发出的不可变事件。"""
task: str
status: TaskStatus
attempts: int = 0
error: Optional[str] = None
duration: Optional[float] = None
reason: Optional[str] = None # 跳过原因,如 "条件不满足"、"上游任务被跳过"、"缓存"
error: str | None = None
duration: float | None = None
reason: str | None = None
+1
View File
@@ -0,0 +1 @@
+119
View File
@@ -0,0 +1,119 @@
"""系统操作任务模块.
提供常用的系统操作任务封装, 包括清屏环境变量设置命令查找等.
遵循实用主义原则, 仅提供核心功能, 无过度设计.
"""
from __future__ import annotations
__all__ = [
"clr",
"reset_icon_cache",
"setenv",
"setenv_group",
"which",
"write_file",
]
import os
import subprocess
from pathlib import Path
import pyflowx as px
from pyflowx import BuiltinConditions
from pyflowx.conditions import Constants
def clr():
"""清屏任务."""
cmd = ["cls"] if Constants.IS_WINDOWS else ["clear"]
return px.TaskSpec("clear_screen", fn=lambda: subprocess.run(cmd, check=False))
def reset_icon_cache() -> list[px.TaskSpec]:
"""重置图标缓存任务."""
if not Constants.IS_WINDOWS:
print("reset_icon_cache: 仅在 Windows 上支持")
return []
local_app_data = os.environ.get("LOCALAPPDATA", "")
icon_cache_db = Path(local_app_data) / "IconCache.db"
explorer_cache_dir = Path(local_app_data) / "Microsoft" / "Windows" / "Explorer"
return [
px.TaskSpec(
"kill_explorer",
cmd=["taskkill", "/f", "/im", "explorer.exe"],
conditions=(BuiltinConditions.IS_RUNNING("explorer.exe"),),
verbose=True,
),
px.TaskSpec(
"delete_icon_cache",
cmd=["cmd", "/c", "del", "/a", "/q", str(icon_cache_db)],
conditions=(BuiltinConditions.DIR_EXISTS(icon_cache_db),),
depends_on=("kill_explorer",),
verbose=True,
),
px.TaskSpec(
"delete_icon_cache_all",
cmd=["cmd", "/c", "del", "/a", "/q", str(explorer_cache_dir / "iconcache*")],
conditions=(BuiltinConditions.DIR_EXISTS(explorer_cache_dir),),
depends_on=("kill_explorer",),
verbose=True,
),
px.TaskSpec(
"restart_explorer",
cmd=["cmd", "/c", "start", "explorer.exe"],
conditions=(
BuiltinConditions.HAS_INSTALLED("explorer.exe"),
BuiltinConditions.NOT(BuiltinConditions.IS_RUNNING("explorer.exe")),
),
depends_on=("delete_icon_cache", "delete_icon_cache_all"),
allow_upstream_skip=True,
verbose=True,
),
]
def setenv(name: str, value: str, default: bool = False) -> px.TaskSpec:
"""设置环境变量任务."""
def set_env():
if default:
os.environ.setdefault(name, value)
else:
os.environ[name] = value
return px.TaskSpec(f"setenv_{name.lower()}", fn=set_env, verbose=True)
def setenv_group(envs: dict[str, str], default: bool = False) -> list[px.TaskSpec]:
"""设置环境变量组任务."""
return [setenv(name, value, default) for name, value in envs.items()]
def which(cmd: str) -> px.TaskSpec:
"""查找命令路径任务."""
which_cmd = "where" if Constants.IS_WINDOWS else "which"
def find_command():
result = subprocess.run([which_cmd, cmd], capture_output=True, text=True, check=False)
if result.returncode == 0:
# Windows 的 where 可能返回多行, 取第一个
path = result.stdout.strip().split("\n")[0].strip()
print(f"{cmd} -> {path}")
else:
print(f"{cmd} -> 未找到")
return px.TaskSpec(f"which_{cmd}", fn=find_command)
def write_file(path: str, content: str, encoding: str = "utf-8") -> px.TaskSpec:
"""写入文件任务."""
def write():
p = Path(path)
p.write_text(content, encoding=encoding)
return px.TaskSpec(f"write_file_{path}", fn=write, verbose=True)
+26
View File
@@ -0,0 +1,26 @@
"""进程池测试辅助:模块级函数(须可 pickle)。"""
from __future__ import annotations
import time
def cpu_heavy(n: int) -> int:
"""CPU 密集型计算(求平方和)。"""
return sum(i * i for i in range(n))
def add(a: int, b: int) -> int:
"""简单加法。"""
return a + b
def sub(a: int, b: int) -> int:
"""简单减法。"""
return a - b
def slow_sleep(seconds: float) -> int:
"""睡眠指定秒数,用于测试超时。"""
time.sleep(seconds)
return int(seconds)
+8 -7
View File
@@ -7,6 +7,7 @@ from unittest.mock import patch
import pytest
import pyflowx as px
from pyflowx.cli import bumpversion
@@ -76,7 +77,7 @@ class TestBumpFileVersion:
content = test_file.read_text(encoding="utf-8")
assert "build" not in content
def test_no_version_found(self, tmp_path: Path, capsys) -> None:
def test_no_version_found(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should return None when no version pattern found."""
test_file = tmp_path / "test.txt"
test_file.write_text("no version here", encoding="utf-8")
@@ -149,7 +150,7 @@ dependencies = ["lib >= 2.0.0", "other >= 3.0.0"]
assert "lib >= 2.0.0" in updated
assert "other >= 3.0.0" in updated
def test_file_read_error(self, tmp_path: Path, capsys) -> None:
def test_file_read_error(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should handle file read errors."""
# 创建一个目录而不是文件
test_file = tmp_path / "test_dir"
@@ -158,7 +159,7 @@ dependencies = ["lib >= 2.0.0", "other >= 3.0.0"]
with pytest.raises(Exception): # noqa: B017
bumpversion.bump_file_version(test_file, "patch")
def test_file_write_error(self, tmp_path: Path, capsys) -> None:
def test_file_write_error(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should handle file write errors."""
# 在只读目录中创建文件(这个测试在某些系统上可能不适用)
test_file = tmp_path / "readonly.toml"
@@ -224,7 +225,7 @@ class TestVersionPattern:
class TestEdgeCases:
"""Test edge cases and error handling."""
def test_empty_file(self, tmp_path: Path, capsys) -> None:
def test_empty_file(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should handle empty file."""
test_file = tmp_path / "empty.txt"
test_file.write_text("", encoding="utf-8")
@@ -280,7 +281,7 @@ class TestBumpVersionCli:
# Mock px.run: 只真正执行第一次调用(版本更新),其余返回空 dict
with patch("sys.argv", ["bumpversion", "minor", "--no-tag"]), patch("pyflowx.run") as mock_run:
def run_side_effect(graph, strategy=None):
def run_side_effect(graph: px.Graph, strategy: str | None = None):
# 执行实际版本更新任务
results = {}
for spec in graph.specs.values():
@@ -294,14 +295,14 @@ class TestBumpVersionCli:
# 验证版本号已更新
assert test_file.read_text(encoding="utf-8") == '__version__ = "1.1.0"'
def test_no_valid_files(self, tmp_path: Path, capsys) -> None:
def test_no_valid_files(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should handle no valid files."""
test_file = tmp_path / "test.txt"
test_file.write_text("这是一个测试文件", encoding="utf-8")
with patch("sys.argv", ["bumpversion", "minor", "--no-tag"]), patch("pyflowx.run") as mock_run:
def run_side_effect(graph, strategy=None):
def run_side_effect(graph: px.Graph, strategy: str | None = None):
# 执行实际版本更新任务
results = {}
for spec in graph.specs.values():
+2 -25
View File
@@ -2,33 +2,10 @@
from __future__ import annotations
from unittest.mock import MagicMock, patch
from unittest.mock import patch
import pyflowx as px
from pyflowx.cli import clearscreen
from pyflowx.conditions import Constants
# ---------------------------------------------------------------------- #
# clear_screen
# ---------------------------------------------------------------------- #
class TestClearScreen:
"""Test clear_screen function."""
def test_clear_screen_windows(self) -> None:
"""Should clear screen on Windows."""
if Constants.IS_WINDOWS:
with patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
clearscreen.clear_screen()
assert mock_run.called
def test_clear_screen_linux(self) -> None:
"""Should clear screen on Linux."""
with patch.object(Constants, "IS_WINDOWS", False), patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
clearscreen.clear_screen()
assert mock_run.called
from pyflowx.cli.system import clearscreen
# ---------------------------------------------------------------------- #
+190 -211
View File
@@ -30,6 +30,8 @@ class TestEmailDatabase:
db_path = tmp_path / "test.db"
db = emlmanager.EmailDatabase(db_path)
assert db.conn is not None
cursor = db.conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='emails'")
result = cursor.fetchone()
@@ -41,6 +43,8 @@ class TestEmailDatabase:
db_path = tmp_path / "test.db"
db = emlmanager.EmailDatabase(db_path)
assert db.conn is not None
cursor = db.conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='index' AND name='idx_subject'")
result = cursor.fetchone()
@@ -68,6 +72,7 @@ class TestEmailDatabase:
result = db.insert_email(email_data)
assert result is True
assert db.conn is not None
cursor = db.conn.cursor()
cursor.execute("SELECT COUNT(*) FROM emails")
@@ -101,6 +106,8 @@ class TestEmailDatabase:
email_data["file_hash"] = "xyz789"
db.insert_email(email_data)
assert db.conn is not None
cursor = db.conn.cursor()
cursor.execute("SELECT COUNT(*) FROM emails")
count = cursor.fetchone()[0]
@@ -118,21 +125,19 @@ class TestEmailDatabase:
# Insert test emails
for i in range(5):
db.insert_email(
{
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
})
results = db.search_emails(limit=3)
assert len(results) == 3
@@ -143,37 +148,33 @@ class TestEmailDatabase:
db_path = tmp_path / "test.db"
db = emlmanager.EmailDatabase(db_path)
db.insert_email(
{
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Important Meeting",
"sender": "sender1@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Meeting body",
"body_html": "<p>Meeting body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Important Meeting",
"sender": "sender1@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Meeting body",
"body_html": "<p>Meeting body</p>",
"has_attachments": 0,
"file_size": 1024,
})
db.insert_email(
{
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Casual Chat",
"sender": "sender2@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Chat body",
"body_html": "<p>Chat body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Casual Chat",
"sender": "sender2@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Chat body",
"body_html": "<p>Chat body</p>",
"has_attachments": 0,
"file_size": 1024,
})
results = db.search_emails(keyword="Meeting", field="subject")
assert len(results) == 1
@@ -185,37 +186,33 @@ class TestEmailDatabase:
db_path = tmp_path / "test.db"
db = emlmanager.EmailDatabase(db_path)
db.insert_email(
{
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Test",
"sender": "alice@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Body",
"body_html": "<p>Body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Test",
"sender": "alice@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Body",
"body_html": "<p>Body</p>",
"has_attachments": 0,
"file_size": 1024,
})
db.insert_email(
{
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Test",
"sender": "bob@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Body",
"body_html": "<p>Body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Test",
"sender": "bob@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Body",
"body_html": "<p>Body</p>",
"has_attachments": 0,
"file_size": 1024,
})
results = db.search_emails(keyword="alice", field="sender")
assert len(results) == 1
@@ -227,21 +224,19 @@ class TestEmailDatabase:
db_path = tmp_path / "test.db"
db = emlmanager.EmailDatabase(db_path)
db.insert_email(
{
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Project Update",
"sender": "manager@example.com",
"recipients": "team@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Please review the quarterly report",
"body_html": "<p>Please review the quarterly report</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Project Update",
"sender": "manager@example.com",
"recipients": "team@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Please review the quarterly report",
"body_html": "<p>Please review the quarterly report</p>",
"has_attachments": 0,
"file_size": 1024,
})
# Search for keyword in subject
results = db.search_emails(keyword="Project", field="all")
@@ -258,53 +253,47 @@ class TestEmailDatabase:
db = emlmanager.EmailDatabase(db_path)
# Insert emails with same subject (different prefixes)
db.insert_email(
{
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Meeting Tomorrow",
"sender": "sender1@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Body 1",
"body_html": "<p>Body 1</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path1.eml",
"file_hash": "hash1",
"subject": "Meeting Tomorrow",
"sender": "sender1@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Body 1",
"body_html": "<p>Body 1</p>",
"has_attachments": 0,
"file_size": 1024,
})
db.insert_email(
{
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Re: Meeting Tomorrow",
"sender": "sender2@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Body 2",
"body_html": "<p>Body 2</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path2.eml",
"file_hash": "hash2",
"subject": "Re: Meeting Tomorrow",
"sender": "sender2@example.com",
"recipients": "recipient@example.com",
"date": "Tue, 2 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-02T12:00:00",
"body_text": "Body 2",
"body_html": "<p>Body 2</p>",
"has_attachments": 0,
"file_size": 1024,
})
db.insert_email(
{
"file_path": "/test/path3.eml",
"file_hash": "hash3",
"subject": "Different Topic",
"sender": "sender3@example.com",
"recipients": "recipient@example.com",
"date": "Wed, 3 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-03T12:00:00",
"body_text": "Body 3",
"body_html": "<p>Body 3</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path3.eml",
"file_hash": "hash3",
"subject": "Different Topic",
"sender": "sender3@example.com",
"recipients": "recipient@example.com",
"date": "Wed, 3 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-03T12:00:00",
"body_text": "Body 3",
"body_html": "<p>Body 3</p>",
"has_attachments": 0,
"file_size": 1024,
})
grouped = db.get_grouped_emails()
# Should have 2 groups: "Meeting Tomorrow" and "Different Topic"
@@ -333,21 +322,19 @@ class TestEmailDatabase:
assert db.get_email_count() == 0
for i in range(3):
db.insert_email(
{
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
})
assert db.get_email_count() == 3
db.close()
@@ -359,21 +346,19 @@ class TestEmailDatabase:
# Insert some emails
for i in range(3):
db.insert_email(
{
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
})
assert db.get_email_count() == 3
@@ -411,7 +396,7 @@ class TestDecodeMimeWords:
def test_decode_none(self) -> None:
"""Should handle None input."""
result = emlmanager.decode_mime_words(None)
result = emlmanager.decode_mime_words("")
assert result == ""
def test_decode_mixed_encoding(self) -> None:
@@ -702,21 +687,19 @@ class TestEmlManagerHandler:
# Insert some emails
for i in range(3):
db.insert_email(
{
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": f"/test/path{i}.eml",
"file_hash": f"hash{i}",
"subject": f"Subject {i}",
"sender": f"sender{i}@example.com",
"recipients": "recipient@example.com",
"date": f"Mon, {i + 1} Jan 2024 12:00:00 +0000",
"date_parsed": f"2024-01-0{i + 1}T12:00:00",
"body_text": f"Body {i}",
"body_html": f"<p>Body {i}</p>",
"has_attachments": 0,
"file_size": 1024,
})
# Create a mock handler instance without calling __init__
handler = Mock(spec=emlmanager.EmlManagerHandler)
@@ -738,21 +721,19 @@ class TestEmlManagerHandler:
db = emlmanager.EmailDatabase(db_path)
# Insert test email
db.insert_email(
{
"file_path": "/test/path.eml",
"file_hash": "hash",
"subject": "Test Subject",
"sender": "sender@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Test body",
"body_html": "<p>Test body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path.eml",
"file_hash": "hash",
"subject": "Test Subject",
"sender": "sender@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Test body",
"body_html": "<p>Test body</p>",
"has_attachments": 0,
"file_size": 1024,
})
# Create a mock handler instance without calling __init__
handler = Mock(spec=emlmanager.EmlManagerHandler)
@@ -775,21 +756,19 @@ class TestEmlManagerHandler:
db = emlmanager.EmailDatabase(db_path)
# Insert test email
db.insert_email(
{
"file_path": "/test/path.eml",
"file_hash": "hash",
"subject": "Test Subject",
"sender": "sender@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Test body",
"body_html": "<p>Test body</p>",
"has_attachments": 0,
"file_size": 1024,
}
)
db.insert_email({
"file_path": "/test/path.eml",
"file_hash": "hash",
"subject": "Test Subject",
"sender": "sender@example.com",
"recipients": "recipient@example.com",
"date": "Mon, 1 Jan 2024 12:00:00 +0000",
"date_parsed": "2024-01-01T12:00:00",
"body_text": "Test body",
"body_html": "<p>Test body</p>",
"has_attachments": 0,
"file_size": 1024,
})
assert db.get_email_count() == 1
-110
View File
@@ -1,110 +0,0 @@
"""Tests for cli.envpy module."""
from __future__ import annotations
from pathlib import Path
from unittest.mock import patch
import pytest
import pyflowx as px
from pyflowx.cli import envpy
# ---------------------------------------------------------------------- #
# set_pip_mirror
# ---------------------------------------------------------------------- #
class TestSetPipMirror:
"""Test set_pip_mirror function."""
def test_set_pip_mirror_tsinghua(self, tmp_path: Path) -> None:
"""Should set tsinghua mirror."""
with patch.object(Path, "home", return_value=tmp_path):
envpy.set_pip_mirror("tsinghua")
# Check pip config
pip_config = tmp_path / "pip" / "pip.ini"
if envpy.Constants.IS_WINDOWS:
assert pip_config.exists() or (tmp_path / "pip" / "pip.conf").exists()
def test_set_pip_mirror_aliyun(self, tmp_path: Path) -> None:
"""Should set aliyun mirror."""
with patch.object(Path, "home", return_value=tmp_path):
envpy.set_pip_mirror("aliyun")
# Check pip config
pip_dir = tmp_path / "pip"
assert pip_dir.exists()
def test_set_pip_mirror_with_token(self, tmp_path: Path) -> None:
"""Should set mirror with token."""
with patch.object(Path, "home", return_value=tmp_path):
envpy.set_pip_mirror("tsinghua", token="test_token")
# Check that token is set
def test_set_pip_mirror_creates_pip_dir(self, tmp_path: Path) -> None:
"""Should create pip directory if it doesn't exist."""
pip_dir = tmp_path / "pip"
with patch.object(Path, "home", return_value=tmp_path):
envpy.set_pip_mirror("tsinghua")
assert pip_dir.exists()
assert pip_dir.is_dir()
# ---------------------------------------------------------------------- #
# main function
# ---------------------------------------------------------------------- #
class TestMain:
"""Test main function."""
def test_main_mirror_tsinghua(self) -> None:
"""main() should handle mirror tsinghua command."""
with patch("sys.argv", ["envpy", "mirror", "tsinghua"]), patch.object(px, "run") as mock_run, patch.object(
envpy, "set_pip_mirror"
):
envpy.main()
assert mock_run.called
def test_main_mirror_aliyun(self) -> None:
"""main() should handle mirror aliyun command."""
with patch("sys.argv", ["envpy", "mirror", "aliyun"]), patch.object(px, "run") as mock_run, patch.object(
envpy, "set_pip_mirror"
):
envpy.main()
assert mock_run.called
def test_main_mirror_with_token(self) -> None:
"""main() should handle mirror with token."""
with patch("sys.argv", ["envpy", "mirror", "tsinghua", "--token", "test_token"]), patch.object(
px, "run"
) as mock_run, patch.object(envpy, "set_pip_mirror"):
envpy.main()
assert mock_run.called
def test_main_with_no_args_shows_help(self) -> None:
"""main() with no args should show help and return."""
with patch("sys.argv", ["envpy"]):
envpy.main()
# Should print help and return
def test_main_invalid_mirror_shows_error(self) -> None:
"""main() with invalid mirror should show error."""
with patch("sys.argv", ["envpy", "mirror", "invalid"]), pytest.raises(SystemExit) as exc_info:
envpy.main()
assert exc_info.value.code == 2
def test_main_creates_task_spec_with_correct_name(self) -> None:
"""main() should create TaskSpec with correct name."""
with patch("sys.argv", ["envpy", "mirror", "tsinghua"]), patch.object(px, "run") as mock_run, patch.object(
envpy, "set_pip_mirror"
):
envpy.main()
graph = mock_run.call_args[0][0]
task_names = list(graph.all_specs().keys())
assert "set_pip_mirror" in task_names
def test_main_uses_thread_strategy(self) -> None:
"""main() should use thread strategy."""
with patch("sys.argv", ["envpy", "mirror", "tsinghua"]), patch.object(px, "run") as mock_run, patch.object(
envpy, "set_pip_mirror"
):
envpy.main()
assert mock_run.call_args[1]["strategy"] == "thread"
-209
View File
@@ -1,209 +0,0 @@
"""Tests for cli.envrs module."""
from __future__ import annotations
import os
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
import pyflowx as px
from pyflowx.cli import envrs
# ---------------------------------------------------------------------- #
# set_rust_mirror
# ---------------------------------------------------------------------- #
class TestSetRustMirror:
"""Test set_rust_mirror function."""
def test_set_rust_mirror_aliyun(self, tmp_path: Path) -> None:
"""Should set aliyun mirror."""
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("aliyun")
# Check environment variables
assert os.environ.get("RUSTUP_DIST_SERVER") == "https://mirrors.aliyun.com/rustup"
assert os.environ.get("RUSTUP_UPDATE_ROOT") == "https://mirrors.aliyun.com/rustup/rustup"
# Check cargo config
cargo_config = tmp_path / ".cargo" / "config.toml"
assert cargo_config.exists()
content = cargo_config.read_text()
assert "aliyun" in content
def test_set_rust_mirror_ustc(self, tmp_path: Path) -> None:
"""Should set ustc mirror."""
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("ustc")
assert os.environ.get("RUSTUP_DIST_SERVER") == "https://mirrors.ustc.edu.cn/rust-static"
assert os.environ.get("RUSTUP_UPDATE_ROOT") == "https://mirrors.ustc.edu.cn/rust-static/rustup"
def test_set_rust_mirror_tsinghua(self, tmp_path: Path) -> None:
"""Should set tsinghua mirror."""
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("tsinghua")
assert os.environ.get("RUSTUP_DIST_SERVER") == "https://mirrors.tuna.tsinghua.edu.cn/rustup"
assert os.environ.get("RUSTUP_UPDATE_ROOT") == "https://mirrors.tuna.tsinghua.edu.cn/rustup/rustup"
def test_set_rust_mirror_unknown_uses_default(self, tmp_path: Path) -> None:
"""Should use default mirror for unknown mirror name."""
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("unknown")
# Should use default mirror (tsinghua)
assert os.environ.get("RUSTUP_DIST_SERVER") == "https://mirrors.tuna.tsinghua.edu.cn/rustup"
def test_set_rust_mirror_creates_cargo_dir(self, tmp_path: Path) -> None:
"""Should create .cargo directory if it doesn't exist."""
cargo_dir = tmp_path / ".cargo"
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("aliyun")
assert cargo_dir.exists()
assert cargo_dir.is_dir()
def test_set_rust_mirror_prints_message(self, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
"""Should print mirror name."""
with patch.object(Path, "home", return_value=tmp_path):
envrs.set_rust_mirror("aliyun")
captured = capsys.readouterr()
assert "已设置 Rust 镜像源: aliyun" in captured.out
# ---------------------------------------------------------------------- #
# install_rust
# ---------------------------------------------------------------------- #
class TestInstallRust:
"""Test install_rust function."""
def test_install_rust_stable(self) -> None:
"""Should install stable Rust."""
with patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
envrs.install_rust("stable")
mock_run.assert_called_once_with(["rustup", "toolchain", "install", "stable"], check=True)
def test_install_rust_nightly(self) -> None:
"""Should install nightly Rust."""
with patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
envrs.install_rust("nightly")
mock_run.assert_called_once_with(["rustup", "toolchain", "install", "nightly"], check=True)
def test_install_rust_beta(self) -> None:
"""Should install beta Rust."""
with patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
envrs.install_rust("beta")
mock_run.assert_called_once_with(["rustup", "toolchain", "install", "beta"], check=True)
def test_install_rust_file_not_found(self) -> None:
"""Should raise FileNotFoundError when rustup not found."""
with patch("subprocess.run", side_effect=FileNotFoundError), pytest.raises(FileNotFoundError):
envrs.install_rust("stable")
def test_install_rust_prints_message(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Should print installation message."""
with patch("subprocess.run") as mock_run:
mock_run.return_value = MagicMock(returncode=0)
envrs.install_rust("stable")
captured = capsys.readouterr()
assert "已安装 Rust stable" in captured.out
# ---------------------------------------------------------------------- #
# main function
# ---------------------------------------------------------------------- #
class TestMain:
"""Test main function."""
def test_main_mirror_aliyun(self) -> None:
"""main() should handle mirror aliyun command."""
with patch("sys.argv", ["envrs", "mirror", "aliyun"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
assert mock_run.called
def test_main_mirror_ustc(self) -> None:
"""main() should handle mirror ustc command."""
with patch("sys.argv", ["envrs", "mirror", "ustc"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
assert mock_run.called
def test_main_mirror_tsinghua(self) -> None:
"""main() should handle mirror tsinghua command."""
with patch("sys.argv", ["envrs", "mirror", "tsinghua"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
assert mock_run.called
def test_main_mirror_default(self) -> None:
"""main() should use default mirror when not specified."""
with patch("sys.argv", ["envrs", "mirror"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
assert mock_run.called
def test_main_install_stable(self) -> None:
"""main() should handle install stable command."""
with patch("sys.argv", ["envrs", "install", "stable"]), patch.object(px, "run") as mock_run:
envrs.main()
assert mock_run.called
def test_main_install_nightly(self) -> None:
"""main() should handle install nightly command."""
with patch("sys.argv", ["envrs", "install", "nightly"]), patch.object(px, "run") as mock_run:
envrs.main()
assert mock_run.called
def test_main_install_beta(self) -> None:
"""main() should handle install beta command."""
with patch("sys.argv", ["envrs", "install", "beta"]), patch.object(px, "run") as mock_run:
envrs.main()
assert mock_run.called
def test_main_install_default(self) -> None:
"""main() should use default version when not specified."""
with patch("sys.argv", ["envrs", "install"]), patch.object(px, "run") as mock_run:
envrs.main()
assert mock_run.called
def test_main_with_no_args_shows_help(self) -> None:
"""main() with no args should show help and return."""
with patch("sys.argv", ["envrs"]):
envrs.main()
# Should print help and return
def test_main_invalid_version_shows_error(self) -> None:
"""main() with invalid version should show error."""
with patch("sys.argv", ["envrs", "install", "invalid"]), pytest.raises(SystemExit) as exc_info:
envrs.main()
assert exc_info.value.code == 2
def test_main_invalid_mirror_shows_error(self) -> None:
"""main() with invalid mirror should show error."""
with patch("sys.argv", ["envrs", "mirror", "invalid"]), pytest.raises(SystemExit) as exc_info:
envrs.main()
assert exc_info.value.code == 2
def test_main_creates_task_spec_with_verbose(self) -> None:
"""main() should create TaskSpec with verbose=True."""
with patch("sys.argv", ["envrs", "mirror", "aliyun"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
graph = mock_run.call_args[0][0]
specs = graph.all_specs()
for spec in specs.values():
assert spec.verbose is True
def test_main_uses_thread_strategy(self) -> None:
"""main() should use thread strategy."""
with patch("sys.argv", ["envrs", "mirror", "aliyun"]), patch.object(px, "run") as mock_run, patch.object(
envrs, "set_rust_mirror"
):
envrs.main()
assert mock_run.call_args[1]["strategy"] == "thread"
+1
View File
@@ -107,6 +107,7 @@ class TestTaskSpecDefinitions:
def test_kill_tgit_spec(self) -> None:
"""kill_tgit spec should be properly defined."""
assert gittool.kill_tgit.name == "task_kill"
assert isinstance(gittool.kill_tgit.cmd, list)
assert "taskkill" in gittool.kill_tgit.cmd
+3 -1
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
@@ -71,7 +72,7 @@ class TestPdfCompress:
mock_fitz_open.return_value = mock_doc
# Mock save to actually create the file
def mock_save(*args, **kwargs):
def mock_save(*args: Any, **kwargs: Any):
output_file.write_bytes(b"Compressed PDF")
mock_doc.save = mock_save
@@ -237,6 +238,7 @@ class TestPdfInfo:
class TestPdfOcr:
"""Test pdf_ocr function."""
@pytest.mark.slow
def test_pdf_ocr_file(self, tmp_path: Path) -> None:
"""Should OCR PDF."""
pytest.importorskip("fitz")
+74 -108
View File
@@ -7,155 +7,121 @@ from unittest.mock import patch
import pytest
from pyflowx.cli import pymake
from pyflowx.conditions import Constants
# ---------------------------------------------------------------------- #
# maturin_build_cmd
# ---------------------------------------------------------------------- #
class TestMaturinBuildCmd:
"""Test maturin_build_cmd function."""
def test_returns_list(self) -> None:
"""Should return a list."""
cmd = pymake.maturin_build_cmd()
assert isinstance(cmd, list)
def test_contains_maturin_build(self) -> None:
"""Should contain 'maturin' and 'build'."""
cmd = pymake.maturin_build_cmd()
assert "maturin" in cmd
assert "build" in cmd
def test_contains_release_flag(self) -> None:
"""Should contain release flag '-r'."""
cmd = pymake.maturin_build_cmd()
assert "-r" in cmd
def test_windows_includes_target(self) -> None:
"""On Windows, should include target-specific flags."""
cmd = pymake.maturin_build_cmd()
if Constants.IS_WINDOWS:
assert "--target" in cmd
assert "x86_64-win7-windows-msvc" in cmd
assert "-Zbuild-std" in cmd
assert "-i" in cmd
assert "python3.8" in cmd
else:
# On non-Windows, should not include Windows-specific flags
assert "--target" not in cmd
def test_does_not_mutate_on_multiple_calls(self) -> None:
"""Multiple calls should return independent lists."""
cmd1 = pymake.maturin_build_cmd()
cmd2 = pymake.maturin_build_cmd()
assert cmd1 == cmd2
# Mutating one should not affect the other
cmd1.append("extra")
assert "extra" not in cmd2
def test_non_windows_excludes_target_flags(self) -> None:
"""On non-Windows, should not include Windows-specific flags (覆盖 22->32 分支)."""
from unittest.mock import patch
with patch.object(pymake.Constants, "IS_WINDOWS", False):
cmd = pymake.maturin_build_cmd()
assert "maturin" in cmd
assert "build" in cmd
assert "-r" in cmd
assert "--target" not in cmd
assert "-Zbuild-std" not in cmd
# ---------------------------------------------------------------------- #
# TaskSpec definitions
# ---------------------------------------------------------------------- #
def _find_task(name: str) -> pymake.px.TaskSpec:
"""从 pymake.tasks 或 aliases 中查找指定名称的 TaskSpec."""
for spec in pymake.tasks:
if spec.name == name:
return spec
# 单任务别名(doc/lint/tox)内联在 aliases dict 中
value = pymake.aliases.get(name)
if isinstance(value, pymake.px.TaskSpec):
return value
raise KeyError(f"任务 {name!r} 未找到")
class TestTaskSpecDefinitions:
"""Test that all TaskSpec definitions are valid."""
def test_uv_build_spec(self) -> None:
"""uv_build spec should be properly defined."""
assert pymake.uv_build.name == "uv_build"
assert pymake.uv_build.cmd == ["uv", "build"]
assert pymake.uv_build.skip_if_missing is True
spec = _find_task("uv_build")
assert spec.name == "uv_build"
assert spec.cmd == ["uv", "build"]
assert spec.skip_if_missing is False
def test_maturin_build_spec(self) -> None:
"""maturin_build spec should be properly defined."""
assert pymake.maturin_build.name == "maturin_build"
assert isinstance(pymake.maturin_build.cmd, list)
assert pymake.maturin_build.skip_if_missing is True
spec = _find_task("maturin_build")
assert spec.name == "maturin_build"
assert isinstance(spec.cmd, list)
assert spec.skip_if_missing is False
def test_uv_sync_spec(self) -> None:
"""uv_sync spec should be properly defined."""
assert pymake.uv_sync.name == "uv_sync"
assert pymake.uv_sync.cmd == ["uv", "sync"]
assert pymake.uv_sync.skip_if_missing is True
spec = _find_task("uv_sync")
assert spec.name == "uv_sync"
assert spec.cmd == ["uv", "sync"]
assert spec.skip_if_missing is False
def test_git_clean_spec(self) -> None:
"""git_clean spec should be properly defined."""
assert pymake.git_clean.name == "git_clean"
assert pymake.git_clean.cmd == ["gitt", "c"]
assert pymake.git_clean.skip_if_missing is True
spec = _find_task("git_clean")
assert spec.name == "git_clean"
assert spec.cmd == ["gitt", "c"]
assert spec.skip_if_missing is False
def test_test_spec(self) -> None:
"""test spec should be properly defined."""
assert pymake.test.name == "test"
assert isinstance(pymake.test.cmd, list)
assert "pytest" in pymake.test.cmd
assert "-m" in pymake.test.cmd
assert "not slow" in pymake.test.cmd
assert pymake.test.skip_if_missing is True
spec = _find_task("test")
assert spec.name == "test"
assert isinstance(spec.cmd, list)
assert "pytest" in spec.cmd
assert "-m" in spec.cmd
assert "not slow" in spec.cmd
assert spec.skip_if_missing is False
def test_test_fast_spec(self) -> None:
"""test_fast spec should be properly defined."""
assert pymake.test_fast.name == "test_fast"
assert isinstance(pymake.test_fast.cmd, list)
assert "pytest" in pymake.test_fast.cmd
assert "-n" not in pymake.test_fast.cmd # test_fast doesn't use parallel
assert pymake.test_fast.skip_if_missing is True
spec = _find_task("test_fast")
assert spec.name == "test_fast"
assert isinstance(spec.cmd, list)
assert "pytest" in spec.cmd
assert "-n" not in spec.cmd # test_fast doesn't use parallel
assert spec.skip_if_missing is False
def test_test_coverage_spec(self) -> None:
"""test_coverage spec should be properly defined."""
assert pymake.test_coverage.name == "test_coverage"
assert isinstance(pymake.test_coverage.cmd, list)
assert "pytest" in pymake.test_coverage.cmd
assert "--cov" in pymake.test_coverage.cmd
assert pymake.test_coverage.skip_if_missing is True
spec = _find_task("test_coverage")
assert spec.name == "test_coverage"
assert isinstance(spec.cmd, list)
assert "pytest" in spec.cmd
assert "--cov" in spec.cmd
assert spec.skip_if_missing is False
def test_ruff_lint_spec(self) -> None:
"""ruff_lint spec should be properly defined."""
assert pymake.ruff_lint.name == "lint"
assert isinstance(pymake.ruff_lint.cmd, list)
assert "ruff" in pymake.ruff_lint.cmd
assert "check" in pymake.ruff_lint.cmd
assert pymake.ruff_lint.skip_if_missing is True
"""lint spec should be properly defined."""
spec = _find_task("lint")
assert spec.name == "lint"
assert isinstance(spec.cmd, list)
assert "ruff" in spec.cmd
assert "check" in spec.cmd
assert spec.skip_if_missing is False
def test_doc_spec(self) -> None:
"""doc spec should be properly defined."""
assert pymake.doc.name == "doc"
assert isinstance(pymake.doc.cmd, list)
assert "sphinx-build" in pymake.doc.cmd
assert pymake.doc.skip_if_missing is True
spec = _find_task("doc")
assert spec.name == "doc"
assert isinstance(spec.cmd, list)
assert "sphinx-build" in spec.cmd
assert spec.skip_if_missing is False
def test_hatch_publish_spec(self) -> None:
"""hatch_publish spec should be properly defined."""
assert pymake.hatch_publish.name == "publish_python"
assert pymake.hatch_publish.cmd == ["hatch", "publish"]
assert pymake.hatch_publish.skip_if_missing is True
"""publish_python spec should be properly defined."""
spec = _find_task("publish_python")
assert spec.name == "publish_python"
assert spec.cmd == ["hatch", "publish"]
assert spec.skip_if_missing is False
def test_twine_publish_spec(self) -> None:
"""twine_publish spec should be properly defined."""
assert pymake.twine_publish.name == "twine_publish"
assert isinstance(pymake.twine_publish.cmd, list)
assert "twine" in pymake.twine_publish.cmd
assert "upload" in pymake.twine_publish.cmd
assert pymake.twine_publish.skip_if_missing is True
spec = _find_task("twine_publish")
assert spec.name == "twine_publish"
assert isinstance(spec.cmd, list)
assert "twine" in spec.cmd
assert "upload" in spec.cmd
assert spec.skip_if_missing is False
def test_tox_spec(self) -> None:
"""tox spec should be properly defined."""
assert pymake.tox.name == "tox"
assert pymake.tox.cmd == ["tox", "-p", "auto"]
assert pymake.tox.skip_if_missing is True
spec = _find_task("tox")
assert spec.name == "tox"
assert spec.cmd == ["tox", "-p", "auto"]
assert spec.skip_if_missing is False
# ---------------------------------------------------------------------- #
+1 -1
View File
@@ -7,7 +7,7 @@ from unittest.mock import patch
import pytest
import pyflowx as px
from pyflowx.cli import taskkill
from pyflowx.cli.system import taskkill
from pyflowx.conditions import Constants
-106
View File
@@ -1,106 +0,0 @@
"""Tests for cli.which module."""
from __future__ import annotations
import shutil
from pathlib import Path
from unittest.mock import patch
import pytest
import pyflowx as px
from pyflowx.cli import which
# ---------------------------------------------------------------------- #
# which_command
# ---------------------------------------------------------------------- #
class TestWhichCommand:
"""Test which_command function."""
def test_returns_path_when_command_found(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Should return Path when command is found."""
with patch.object(shutil, "which", return_value="/usr/bin/python"):
result = which.which_command("python")
assert result == Path("/usr/bin/python")
captured = capsys.readouterr()
assert "匹配路径" in captured.out
assert "/usr/bin/python" in captured.out
def test_returns_none_when_command_not_found(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Should return None when command is not found."""
with patch.object(shutil, "which", return_value=None):
result = which.which_command("nonexistent_cmd")
assert result is None
captured = capsys.readouterr()
assert "未找到" in captured.out
assert "nonexistent_cmd" in captured.out
def test_prints_match_path_on_success(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Should print '匹配路径: - <path>' on success."""
with patch.object(shutil, "which", return_value="C:\\Python\\python.exe"):
_ = which.which_command("python")
captured = capsys.readouterr()
assert "匹配路径: - C:\\Python\\python.exe" in captured.out
def test_prints_not_found_on_failure(self, capsys: pytest.CaptureFixture[str]) -> None:
"""Should print '<command>: 未找到' on failure."""
with patch.object(shutil, "which", return_value=None):
_ = which.which_command("missing")
captured = capsys.readouterr()
assert "missing: 未找到" in captured.out
# ---------------------------------------------------------------------- #
# main function
# ---------------------------------------------------------------------- #
class TestMain:
"""Test main function."""
def test_main_with_single_command(self) -> None:
"""main() should handle single command argument."""
with patch("sys.argv", ["which", "python"]), patch.object(
shutil, "which", return_value="/usr/bin/python"
), patch.object(px, "run") as mock_run:
which.main()
# Should create a graph with one task
assert mock_run.called
graph = mock_run.call_args[0][0]
assert isinstance(graph, px.Graph)
def test_main_with_multiple_commands(self) -> None:
"""main() should handle multiple command arguments."""
with patch("sys.argv", ["which", "python", "pip", "node"]), patch.object(
shutil, "which", return_value="/usr/bin/cmd"
), patch.object(px, "run") as mock_run:
which.main()
# Should create a graph with three tasks
assert mock_run.called
graph = mock_run.call_args[0][0]
assert isinstance(graph, px.Graph)
def test_main_with_no_args_shows_help(self) -> None:
"""main() with no args should show help and exit."""
with patch("sys.argv", ["which"]), pytest.raises(SystemExit) as exc_info:
which.main()
assert exc_info.value.code == 2
def test_main_creates_task_specs_with_correct_names(self) -> None:
"""main() should create TaskSpecs with correct names."""
with patch("sys.argv", ["which", "git", "npm"]), patch.object(
shutil, "which", return_value="/usr/bin/cmd"
), patch.object(px, "run") as mock_run:
which.main()
graph = mock_run.call_args[0][0]
# Check that task names are correct
task_names = list(graph.all_specs().keys())
assert "which_git" in task_names
assert "which_npm" in task_names
def test_main_uses_thread_strategy(self) -> None:
"""main() should use thread strategy."""
with patch("sys.argv", ["which", "python"]), patch.object(
shutil, "which", return_value="/usr/bin/python"
), patch.object(px, "run") as mock_run:
which.main()
assert mock_run.call_args[1]["strategy"] == "thread"
+7
View File
@@ -1,9 +1,16 @@
from __future__ import annotations
import sys
from pathlib import Path
import pytest
# 将 tests 目录加入 sys.path,使进程池测试能 import _proc_helper 模块级辅助函数。
# 进程池 pickle 要求被调用函数为模块级,conftest.py 在 xdist worker 中也会执行。
_TESTS_DIR = str(Path(__file__).resolve().parent)
if _TESTS_DIR not in sys.path:
sys.path.insert(0, _TESTS_DIR)
@pytest.fixture(autouse=True)
def packtool_tmp_workdir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
File diff suppressed because it is too large Load Diff
+101
View File
@@ -0,0 +1,101 @@
"""Tests for Graph.chain DSL."""
from __future__ import annotations
import pyflowx as px
from pyflowx.task import TaskSpec
def _fn() -> None:
return None
def test_chain_basic_linkage() -> None:
"""chain(a, b, c) 应建立 a->b->c 依赖."""
a = TaskSpec("a", _fn)
b = TaskSpec("b", _fn)
c = TaskSpec("c", _fn)
graph = px.Graph().chain(a, b, c)
assert graph.all_specs()["b"].depends_on == ("a",)
assert graph.all_specs()["c"].depends_on == ("b",)
assert graph.all_specs()["a"].depends_on == ()
def test_chain_single_spec() -> None:
"""chain(a) 应只注册 a,无依赖."""
a = TaskSpec("a", _fn)
graph = px.Graph().chain(a)
assert "a" in graph
assert graph.all_specs()["a"].depends_on == ()
def test_chain_preserves_existing_deps() -> None:
"""chain 应保留 spec 已有的 depends_on."""
a = TaskSpec("a", _fn)
b = TaskSpec("b", _fn)
c = TaskSpec("c", _fn, depends_on=("b",))
graph = px.Graph().chain(a, b, c)
# c 已有 depends_on=('b',),前驱是 b,已在依赖中,不重复添加
assert graph.all_specs()["c"].depends_on == ("b",)
def test_chain_merges_existing_deps() -> None:
"""chain 应将前驱追加到已有依赖前(若不存在)."""
a = TaskSpec("a", _fn)
x = TaskSpec("x", _fn)
c = TaskSpec("c", _fn, depends_on=("x",))
graph = px.Graph().chain(a, x, c)
# c 前驱是 x,但 c 已依赖 x,不重复
assert graph.all_specs()["c"].depends_on == ("x",)
def test_chain_returns_self() -> None:
"""chain 返回 self 支持链式调用."""
a = TaskSpec("a", _fn)
graph = px.Graph()
assert graph.chain(a) is graph
def test_chain_execution_order() -> None:
"""chain 应保证执行顺序."""
order: list[str] = []
def make(name: str):
def fn() -> str:
order.append(name)
return name
return fn
a = TaskSpec("a", make("a"))
b = TaskSpec("b", make("b"))
c = TaskSpec("c", make("c"))
graph = px.Graph().chain(a, b, c)
report = px.run(graph)
assert report.success
assert order == ["a", "b", "c"]
def test_chain_with_decorator_specs() -> None:
"""chain 应与 @task 装饰器配合."""
@px.task
def extract() -> int:
return 1
@px.task
def transform(extract: int) -> int:
return extract + 10
@px.task
def load(transform: int) -> int:
return transform + 100
graph = px.Graph().chain(extract, transform, load)
report = px.run(graph)
assert report.success
assert report["load"] == 111
+19 -19
View File
@@ -17,7 +17,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"build": px.Graph.from_specs([build_task]),
"test": px.Graph.from_specs([test_task]),
"all": px.Graph.from_specs([build_task, "test"]),
@@ -38,7 +38,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs([task2]),
"cmd3": px.Graph.from_specs([task3]),
@@ -57,7 +57,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"lint": px.Graph.from_specs([lint_task, format_task]),
"quick": px.Graph.from_specs(["lint.lint"]),
},
@@ -75,7 +75,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs(["cmd1", task2]),
"cmd3": px.Graph.from_specs(["cmd2", task3]),
@@ -93,7 +93,7 @@ class TestCommandReferences:
with pytest.raises(ValueError, match="循环引用"):
px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs(["cmd1", task1]),
},
)
@@ -105,7 +105,7 @@ class TestCommandReferences:
with pytest.raises(ValueError, match="引用的命令 'invalid' 不存在"):
px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs(["invalid", task1]),
},
)
@@ -117,7 +117,7 @@ class TestCommandReferences:
with pytest.raises(ValueError, match="任务 'invalid' 不存在于命令 'cmd1'"):
px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs(["cmd1.invalid"]),
},
@@ -130,7 +130,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1, task2]),
"cmd2": px.Graph.from_specs(["cmd1"]),
},
@@ -148,7 +148,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1, task2]),
"cmd2": px.Graph.from_specs(["cmd1", task3]),
},
@@ -168,7 +168,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs([task2, task3]),
"cmd3": px.Graph.from_specs([task4]),
@@ -205,7 +205,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs([task2]),
"all": px.Graph.from_specs(["cmd1", "cmd2", task3, task4, task5]),
@@ -242,7 +242,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1, task2]),
"cmd2": px.Graph.from_specs([task3]),
"all": px.Graph.from_specs(["cmd1", "cmd2", task4]),
@@ -279,7 +279,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"c": px.Graph.from_specs([git_clean]),
"tc": px.Graph.from_specs([typecheck, "lint"]),
"lint": px.Graph.from_specs([lint, format_task]),
@@ -319,7 +319,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs([task2]),
"cmd3": px.Graph.from_specs([task3]),
@@ -350,7 +350,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"all": px.Graph.from_specs([task1, task2, task3]),
},
)
@@ -373,7 +373,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1, task2]),
"all": px.Graph.from_specs(["cmd1"]),
},
@@ -399,7 +399,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1]),
"cmd2": px.Graph.from_specs(["cmd1", task2]),
"cmd3": px.Graph.from_specs(["cmd2", task3]),
@@ -430,7 +430,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"cmd1": px.Graph.from_specs([task1, task2]), # Parallel tasks
"cmd2": px.Graph.from_specs([task3, task4]), # Parallel tasks
"all": px.Graph.from_specs(["cmd1", "cmd2"]),
@@ -465,7 +465,7 @@ class TestCommandReferences:
runner = px.CliRunner(
strategy="sequential",
graphs={
aliases={
"clean": px.Graph.from_specs([clean]),
"build": px.Graph.from_specs([build1, build2]),
"test": px.Graph.from_specs([test1, test2]),
+283 -100
View File
@@ -1,9 +1,14 @@
"""Tests for conditions module."""
from __future__ import annotations
import os
import sys
from pathlib import Path
from unittest.mock import patch
import pytest
from pyflowx.conditions import (
IS_LINUX,
IS_MACOS,
@@ -13,164 +18,342 @@ from pyflowx.conditions import (
Constants,
)
_CTX: dict[str, object] = {}
def test_constants_is_windows():
"""Test Constants.IS_WINDOWS is correct."""
assert (sys.platform == "win32") == Constants.IS_WINDOWS
def test_constants_is_linux():
"""Test Constants.IS_LINUX is correct."""
assert (sys.platform == "linux") == Constants.IS_LINUX
def test_constants_is_macos():
"""Test Constants.IS_MACOS is correct."""
assert (sys.platform == "darwin") == Constants.IS_MACOS
def test_constants_is_posix():
"""Test Constants.IS_POSIX is correct."""
assert (sys.platform != "win32") == Constants.IS_POSIX
def test_builtin_conditions_is_windows():
"""Test BuiltinConditions.IS_WINDOWS."""
result = BuiltinConditions.IS_WINDOWS()
assert result == Constants.IS_WINDOWS
def test_module_level_static_conditions():
assert IS_WINDOWS(_CTX) == Constants.IS_WINDOWS
assert IS_LINUX(_CTX) == Constants.IS_LINUX
assert IS_MACOS(_CTX) == Constants.IS_MACOS
assert IS_POSIX(_CTX) == Constants.IS_POSIX
def test_builtin_conditions_is_linux():
"""Test BuiltinConditions.IS_LINUX."""
result = BuiltinConditions.IS_LINUX()
assert result == Constants.IS_LINUX
def test_builtin_conditions_is_macos():
"""Test BuiltinConditions.IS_MACOS."""
result = BuiltinConditions.IS_MACOS()
assert result == Constants.IS_MACOS
def test_builtin_conditions_is_posix():
"""Test BuiltinConditions.IS_POSIX."""
result = BuiltinConditions.IS_POSIX()
assert result == Constants.IS_POSIX
def test_builtin_conditions_python_version_major_only():
"""Test BuiltinConditions.PYTHON_VERSION with major only."""
# Test with current Python version
def test_python_version_major_only():
current_major = sys.version_info.major
assert BuiltinConditions.PYTHON_VERSION(current_major) is True
assert BuiltinConditions.PYTHON_VERSION(current_major + 1) is False
assert BuiltinConditions.PYTHON_VERSION(current_major)(_CTX) is True
assert BuiltinConditions.PYTHON_VERSION(current_major + 1)(_CTX) is False
def test_builtin_conditions_python_version_with_minor():
"""Test BuiltinConditions.PYTHON_VERSION with major and minor."""
def test_python_version_with_minor():
current_major = sys.version_info.major
current_minor = sys.version_info.minor
assert BuiltinConditions.PYTHON_VERSION(current_major, current_minor) is True
assert BuiltinConditions.PYTHON_VERSION(current_major, current_minor + 1) is False
assert BuiltinConditions.PYTHON_VERSION(current_major, current_minor)(_CTX) is True
assert BuiltinConditions.PYTHON_VERSION(current_major, current_minor + 1)(_CTX) is False
def test_builtin_conditions_python_version_at_least():
"""Test BuiltinConditions.PYTHON_VERSION_AT_LEAST."""
def test_python_version_at_least():
current_major = sys.version_info.major
current_minor = sys.version_info.minor
# Current version should be at least itself
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major, current_minor) is True
# Current version should be at least an older version
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major - 1, 0) is True
# Current version should NOT be at least a newer version
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major + 1, 0) is False
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major, current_minor)(_CTX) is True
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major - 1, 0)(_CTX) is True
assert BuiltinConditions.PYTHON_VERSION_AT_LEAST(current_major + 1, 0)(_CTX) is False
def test_builtin_conditions_HAS_INSTALLED_true():
"""Test BuiltinConditions.HAS_INSTALLED when app exists."""
# Python should always be available
condition = BuiltinConditions.HAS_INSTALLED("python")
assert condition() is True
def test_has_installed_true():
condition = BuiltinConditions.HAS_INSTALLED("python3")
assert condition(_CTX) is True
def test_builtin_conditions_HAS_INSTALLED_false():
"""Test BuiltinConditions.HAS_INSTALLED when app doesn't exist."""
def test_has_installed_false():
condition = BuiltinConditions.HAS_INSTALLED("nonexistent_app_12345")
assert condition() is False
assert condition(_CTX) is False
def test_builtin_conditions_env_var_exists_true():
"""Test BuiltinConditions.ENV_VAR_EXISTS when variable exists."""
def test_env_var_exists_true():
with patch.dict(os.environ, {"TEST_VAR": "value"}):
condition = BuiltinConditions.ENV_VAR_EXISTS("TEST_VAR")
assert condition() is True
assert condition(_CTX) is True
def test_builtin_conditions_env_var_exists_false():
"""Test BuiltinConditions.ENV_VAR_EXISTS when variable doesn't exist."""
def test_env_var_exists_false():
condition = BuiltinConditions.ENV_VAR_EXISTS("NONEXISTENT_VAR_12345")
assert condition() is False
assert condition(_CTX) is False
def test_builtin_conditions_env_var_equals_true():
"""Test BuiltinConditions.ENV_VAR_EQUALS when value matches."""
def test_env_var_equals_true():
with patch.dict(os.environ, {"TEST_VAR": "expected_value"}):
condition = BuiltinConditions.ENV_VAR_EQUALS("TEST_VAR", "expected_value")
assert condition() is True
assert condition(_CTX) is True
def test_builtin_conditions_env_var_equals_false():
"""Test BuiltinConditions.ENV_VAR_EQUALS when value doesn't match."""
def test_env_var_equals_false():
with patch.dict(os.environ, {"TEST_VAR": "different_value"}):
condition = BuiltinConditions.ENV_VAR_EQUALS("TEST_VAR", "expected_value")
assert condition() is False
assert condition(_CTX) is False
def test_builtin_conditions_not():
"""Test BuiltinConditions.NOT."""
true_condition = lambda: True # noqa: E731
false_condition = lambda: False # noqa: E731
def test_not():
true_cond = BuiltinConditions.HAS_INSTALLED("python3")
false_cond = BuiltinConditions.HAS_INSTALLED("nonexistent_app_12345")
not_true = BuiltinConditions.NOT(true_condition)
assert not_true() is False
not_false = BuiltinConditions.NOT(false_condition)
assert not_false() is True
assert BuiltinConditions.NOT(true_cond)(_CTX) is False
assert BuiltinConditions.NOT(false_cond)(_CTX) is True
def test_builtin_conditions_and_all_true():
"""Test BuiltinConditions.AND when all conditions are true."""
true_condition = lambda: True # noqa: E731
condition = BuiltinConditions.AND(true_condition, true_condition, true_condition)
assert condition() is True
def test_and_all_true():
cond = BuiltinConditions.AND(
BuiltinConditions.HAS_INSTALLED("python3"),
BuiltinConditions.HAS_INSTALLED("python3"),
)
assert cond(_CTX) is True
def test_builtin_conditions_and_one_false():
"""Test BuiltinConditions.AND when one condition is false."""
true_condition = lambda: True # noqa: E731
false_condition = lambda: False # noqa: E731
condition = BuiltinConditions.AND(true_condition, false_condition, true_condition)
assert condition() is False
def test_and_one_false():
cond = BuiltinConditions.AND(
BuiltinConditions.HAS_INSTALLED("python3"),
BuiltinConditions.HAS_INSTALLED("nonexistent_app"),
)
assert cond(_CTX) is False
def test_builtin_conditions_or_all_false():
"""Test BuiltinConditions.OR when all conditions are false."""
false_condition = lambda: False # noqa: E731
condition = BuiltinConditions.OR(false_condition, false_condition, false_condition)
assert condition() is False
def test_or_all_false():
cond = BuiltinConditions.OR(
BuiltinConditions.HAS_INSTALLED("nonexistent1"),
BuiltinConditions.HAS_INSTALLED("nonexistent2"),
)
assert cond(_CTX) is False
def test_builtin_conditions_or_one_true():
"""Test BuiltinConditions.OR when one condition is true."""
true_condition = lambda: True # noqa: E731
false_condition = lambda: False # noqa: E731
condition = BuiltinConditions.OR(false_condition, true_condition, false_condition)
assert condition() is True
def test_or_one_true():
cond = BuiltinConditions.OR(
BuiltinConditions.HAS_INSTALLED("nonexistent1"),
BuiltinConditions.HAS_INSTALLED("python3"),
)
assert cond(_CTX) is True
def test_exported_conditions():
"""Test exported condition functions."""
assert IS_WINDOWS() == Constants.IS_WINDOWS
assert IS_LINUX() == Constants.IS_LINUX
assert IS_MACOS() == Constants.IS_MACOS
assert IS_POSIX() == Constants.IS_POSIX
# ---------------------------------------------------------------------- #
# 上下文条件:基于上游依赖结果
# ---------------------------------------------------------------------- #
def test_dep_equals_true():
ctx = {"upstream": 42}
cond = BuiltinConditions.DEP_EQUALS("upstream", 42)
assert cond(ctx) is True
def test_dep_equals_false():
ctx = {"upstream": 99}
cond = BuiltinConditions.DEP_EQUALS("upstream", 42)
assert cond(ctx) is False
def test_dep_equals_missing_dep():
cond = BuiltinConditions.DEP_EQUALS("missing", 42)
assert cond({}) is False
def test_dep_matches_true():
ctx = {"upstream": [1, 2, 3]}
cond = BuiltinConditions.DEP_MATCHES("upstream", lambda v: len(v) == 3)
assert cond(ctx) is True
def test_dep_matches_false():
ctx = {"upstream": [1, 2]}
cond = BuiltinConditions.DEP_MATCHES("upstream", lambda v: len(v) == 3)
assert cond(ctx) is False
def test_dep_matches_exception_returns_false():
ctx = {"upstream": ""}
cond = BuiltinConditions.DEP_MATCHES("upstream", lambda v: v[0])
assert cond(ctx) is False
def test_dep_present_true():
ctx = {"upstream": "value"}
cond = BuiltinConditions.DEP_PRESENT("upstream")
assert cond(ctx) is True
def test_dep_present_false_none():
# pyrefly: ignore [implicit-any-empty-container]
ctx = {"upstream": None}
cond = BuiltinConditions.DEP_PRESENT("upstream")
assert cond(ctx) is False
def test_dep_present_false_missing():
cond = BuiltinConditions.DEP_PRESENT("missing")
assert cond({}) is False
def test_dep_truthy_true():
ctx = {"upstream": [1]}
cond = BuiltinConditions.DEP_TRUTHY("upstream")
assert cond(ctx) is True
def test_dep_truthy_false():
# pyrefly: ignore [implicit-any-empty-container]
ctx = {"upstream": []}
cond = BuiltinConditions.DEP_TRUTHY("upstream")
assert cond(ctx) is False
def test_dep_truthy_missing():
cond = BuiltinConditions.DEP_TRUTHY("missing")
assert cond({}) is False
def test_logical_combination_with_dep_conditions():
ctx = {"a": 1, "b": 0}
cond = BuiltinConditions.AND(
BuiltinConditions.DEP_EQUALS("a", 1),
BuiltinConditions.NOT(BuiltinConditions.DEP_TRUTHY("b")),
)
assert cond(ctx) is True
# ---------------------------------------------------------------------- #
# IS_RUNNING: 跨平台 subprocess 检测
# ---------------------------------------------------------------------- #
def test_is_running_windows_found(monkeypatch: pytest.MonkeyPatch):
"""Windows 上 tasklist 检测到进程."""
monkeypatch.setattr("pyflowx.conditions.Constants.IS_WINDOWS", True)
monkeypatch.setattr("pyflowx.conditions.Constants.IS_LINUX", False)
class MockResult:
stdout = "explorer.exe\nother.exe"
returncode = 0
monkeypatch.setattr(
"subprocess.run",
lambda *_, **__: MockResult(),
)
cond = BuiltinConditions.IS_RUNNING("explorer.exe")
assert cond({}) is True
def test_is_running_windows_not_found(monkeypatch: pytest.MonkeyPatch):
"""Windows 上 tasklist 未检测到进程."""
monkeypatch.setattr("pyflowx.conditions.Constants.IS_WINDOWS", True)
monkeypatch.setattr("pyflowx.conditions.Constants.IS_LINUX", False)
class MockResult:
stdout = "other.exe"
returncode = 0
monkeypatch.setattr(
"subprocess.run",
lambda *_, **__: MockResult(),
)
cond = BuiltinConditions.IS_RUNNING("explorer.exe")
assert cond({}) is False
def test_is_running_linux_found(monkeypatch: pytest.MonkeyPatch):
"""Linux 上 pgrep 检测到进程."""
monkeypatch.setattr("pyflowx.conditions.Constants.IS_WINDOWS", False)
monkeypatch.setattr("pyflowx.conditions.Constants.IS_LINUX", True)
class MockResult:
returncode = 0
monkeypatch.setattr(
"subprocess.run",
lambda *_, **__: MockResult(),
)
cond = BuiltinConditions.IS_RUNNING("nginx")
assert cond({}) is True
def test_is_running_linux_not_found(monkeypatch: pytest.MonkeyPatch):
"""Linux 上 pgrep 未检测到进程."""
monkeypatch.setattr("pyflowx.conditions.Constants.IS_WINDOWS", False)
monkeypatch.setattr("pyflowx.conditions.Constants.IS_LINUX", True)
class MockResult:
returncode = 1
monkeypatch.setattr(
"subprocess.run",
lambda *_, **__: MockResult(),
)
cond = BuiltinConditions.IS_RUNNING("nonexistent")
assert cond({}) is False
def test_dir_exists_true(tmp_path: Path):
"""DIR_EXISTS 检测路径存在."""
cond = BuiltinConditions.DIR_EXISTS(tmp_path)
assert cond({}) is True
def test_dir_exists_false(tmp_path: Path):
"""DIR_EXISTS 检测路径不存在."""
missing = tmp_path / "nonexistent"
cond = BuiltinConditions.DIR_EXISTS(missing)
assert cond({}) is False
def test_builtin_is_windows_returns_module_condition():
"""BuiltinConditions.IS_WINDOWS() 应返回模块级 IS_WINDOWS."""
assert BuiltinConditions.IS_WINDOWS() is IS_WINDOWS
def test_builtin_is_linux_returns_module_condition():
"""BuiltinConditions.IS_LINUX() 应返回模块级 IS_LINUX."""
assert BuiltinConditions.IS_LINUX() is IS_LINUX
def test_builtin_is_macos_returns_module_condition():
"""BuiltinConditions.IS_MACOS() 应返回模块级 IS_MACOS."""
assert BuiltinConditions.IS_MACOS() is IS_MACOS
def test_builtin_is_posix_returns_module_condition():
"""BuiltinConditions.IS_POSIX() 应返回模块级 IS_POSIX."""
assert BuiltinConditions.IS_POSIX() is IS_POSIX
def test_file_content_exists_missing_file(tmp_path: Path):
"""FILE_CONTENT_EXISTS 文件不存在时返回 False."""
cond = BuiltinConditions.FILE_CONTENT_EXISTS(tmp_path / "missing.txt", "x")
assert cond({}) is False
def test_file_content_exists_contains_content(tmp_path: Path):
"""FILE_CONTENT_EXISTS 文件包含内容时返回 True."""
f = tmp_path / "f.txt"
f.write_text("hello world", encoding="utf-8")
cond = BuiltinConditions.FILE_CONTENT_EXISTS(f, "world")
assert cond({}) is True
def test_file_content_exists_not_contains_content(tmp_path: Path):
"""FILE_CONTENT_EXISTS 文件不包含内容时返回 False."""
f = tmp_path / "f.txt"
f.write_text("hello", encoding="utf-8")
cond = BuiltinConditions.FILE_CONTENT_EXISTS(f, "missing")
assert cond({}) is False
def test_file_content_exists_decode_error_returns_false(tmp_path: Path):
"""FILE_CONTENT_EXISTS 读取非 UTF-8 文件应返回 False(解码异常被吞)."""
f = tmp_path / "bin.dat"
f.write_bytes(b"\xff\xfe\x00bad")
cond = BuiltinConditions.FILE_CONTENT_EXISTS(f, "x")
assert cond({}) is False
def test_dep_matches_missing_dep_returns_false():
"""DEP_MATCHES 依赖不存在时应返回 False(覆盖 if not in ctx 分支)."""
cond = BuiltinConditions.DEP_MATCHES("missing", lambda _v: True)
assert cond({}) is False
+1 -1
View File
@@ -141,7 +141,7 @@ class TestDescribeInjection:
spec = px.TaskSpec("t", fn, depends_on=("a",))
desc = describe_injection(spec)
assert "a=<result:a>" in desc
assert "a=<dep:a>" in desc
assert "ctx=<Context>" in desc
assert "flag=<default>" in desc
+62
View File
@@ -0,0 +1,62 @@
"""Tests for process executor (spec.executor='process')."""
from __future__ import annotations
import pytest
# pyrefly: ignore[missing-import]
from _proc_helper import add, cpu_heavy, slow_sleep, sub
import pyflowx as px
from pyflowx.errors import TaskFailedError
def test_process_executor_runs_cpu_task() -> None:
"""executor='process' 应在进程池中执行 CPU 密集型任务."""
spec = px.TaskSpec("cpu", fn=cpu_heavy, args=(1000,), executor="process")
graph = px.Graph.from_specs([spec])
report = px.run(graph)
assert report.success
assert report["cpu"] == sum(i * i for i in range(1000))
def test_process_executor_with_dependency() -> None:
"""进程池任务应支持依赖注入."""
spec1 = px.TaskSpec("a", fn=cpu_heavy, args=(100,), executor="process")
spec2 = px.TaskSpec("b", fn=add, args=(3, 4), executor="process", depends_on=("a",))
graph = px.Graph.from_specs([spec1, spec2])
report = px.run(graph)
assert report.success
assert report["b"] == 7
def test_process_executor_default_is_thread() -> None:
"""TaskSpec.executor 默认应为 'thread'."""
spec = px.TaskSpec("x", fn=lambda: None)
assert spec.executor == "thread"
def test_inline_executor_runs_in_event_loop() -> None:
"""executor='inline' 应直接在事件循环线程调用."""
spec = px.TaskSpec("inline", fn=add, args=(10, 20), executor="inline")
graph = px.Graph.from_specs([spec])
report = px.run(graph)
assert report.success
assert report["inline"] == 30
def test_process_executor_with_kwargs() -> None:
"""进程池任务应支持 kwargs 注入."""
spec = px.TaskSpec("kw", fn=sub, args=(10,), kwargs={"b": 3}, executor="process")
graph = px.Graph.from_specs([spec])
report = px.run(graph)
assert report.success
assert report["kw"] == 7
def test_process_executor_timeout() -> None:
"""进程池任务超时应抛 TaskFailedError."""
spec = px.TaskSpec("slow", fn=slow_sleep, args=(10.0,), executor="process", timeout=0.1)
graph = px.Graph.from_specs([spec])
with pytest.raises(TaskFailedError):
px.run(graph)
+57 -8
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import asyncio
import logging
import tempfile
import threading
import time
@@ -84,18 +85,62 @@ def test_retries_then_succeeds() -> None:
raise RuntimeError("not yet")
return "ok"
graph = px.Graph.from_specs([px.TaskSpec("flaky", flaky, retries=2)])
graph = px.Graph.from_specs([
px.TaskSpec("flaky", flaky, retry=px.RetryPolicy(max_attempts=3)),
])
report = px.run(graph, strategy="sequential")
assert report.success
assert report["flaky"] == "ok"
assert attempts["n"] == 3
def test_retries_with_delay() -> None:
"""测试带delay的重试会实际等待。"""
attempts = {"n": 0}
start_time = time.time()
def flaky() -> str:
attempts["n"] += 1
if attempts["n"] < 2:
raise RuntimeError("not yet")
return "ok"
graph = px.Graph.from_specs([
px.TaskSpec("flaky", flaky, retry=px.RetryPolicy(max_attempts=2, delay=0.1)),
])
report = px.run(graph, strategy="sequential")
elapsed = time.time() - start_time
assert report.success
assert elapsed >= 0.1 # 应有至少0.1秒的等待时间
assert attempts["n"] == 2
def test_timeout_then_retry_async(caplog: pytest.LogCaptureFixture) -> None:
"""测试超时后可以重试,并记录warning日志。"""
async def slow_task() -> str:
await asyncio.sleep(10) # 会触发超时
return "ok"
graph = px.Graph.from_specs([
px.TaskSpec("slow", slow_task, timeout=0.2, retry=px.RetryPolicy(max_attempts=2)),
])
with caplog.at_level(logging.WARNING, logger="pyflowx"):
with pytest.raises(px.TaskFailedError) as exc_info:
_ = px.run(graph, strategy="async")
assert exc_info.value.attempts == 2
assert "timed out" in str(exc_info.value.cause)
# 应有超时重试的warning日志
assert any("timed out" in r.message for r in caplog.records)
def test_retries_exhausted() -> None:
def always_fail() -> None:
raise RuntimeError("nope")
graph = px.Graph.from_specs([px.TaskSpec("f", always_fail, retries=2)])
graph = px.Graph.from_specs([
px.TaskSpec("f", always_fail, retry=px.RetryPolicy(max_attempts=3)),
])
with pytest.raises(TaskFailedError) as exc_info:
_ = px.run(graph, strategy="sequential")
assert exc_info.value.attempts == 3
@@ -332,7 +377,9 @@ def test_async_timeout_retry_then_succeed() -> None:
await asyncio.sleep(10) # 触发超时
return "ok"
graph = px.Graph.from_specs([px.TaskSpec("a", flaky, retries=2, timeout=0.05)])
graph = px.Graph.from_specs([
px.TaskSpec("a", flaky, retry=px.RetryPolicy(max_attempts=3), timeout=0.05),
])
report = px.run(graph, strategy="async")
assert report.success
assert report["a"] == "ok"
@@ -349,7 +396,9 @@ def test_async_failure_retry_branch(caplog: pytest.LogCaptureFixture) -> None:
raise RuntimeError("not yet")
return "ok"
graph = px.Graph.from_specs([px.TaskSpec("a", flaky, retries=2)])
graph = px.Graph.from_specs([
px.TaskSpec("a", flaky, retry=px.RetryPolicy(max_attempts=3)),
])
with caplog.at_level("WARNING", logger="pyflowx"):
report = px.run(graph, strategy="async")
assert report.success
@@ -489,7 +538,7 @@ def test_run_empty_graph() -> None:
# ---------------------------------------------------------------------- #
def test_downstream_skipped_when_upstream_skipped_sequential() -> None:
"""上游任务被 SKIPPED 后,下游任务也应被 SKIPPEDsequential 策略)."""
never_true = lambda: False # noqa: E731
never_true = lambda _ctx: False # noqa: E731
def downstream(upstream: str) -> str:
return upstream + "_processed"
@@ -506,7 +555,7 @@ def test_downstream_skipped_when_upstream_skipped_sequential() -> None:
def test_downstream_skipped_when_upstream_skipped_thread() -> None:
"""上游任务被 SKIPPED 后,下游任务也应被 SKIPPEDthread 策略)."""
never_true = lambda: False # noqa: E731
never_true = lambda _ctx: False # noqa: E731
def downstream(upstream: str) -> str:
return upstream + "_processed"
@@ -530,7 +579,7 @@ def test_downstream_skipped_when_upstream_skipped_async() -> None:
async def downstream(upstream: str) -> str:
return upstream + "_processed"
never_true = lambda: False # noqa: E731
never_true = lambda _ctx: False # noqa: E731
graph = px.Graph.from_specs([
px.TaskSpec("upstream", upstream, conditions=(never_true,)),
@@ -544,7 +593,7 @@ def test_downstream_skipped_when_upstream_skipped_async() -> None:
def test_downstream_executes_when_upstream_succeeds() -> None:
"""上游任务成功时,下游任务应正常执行."""
always_true = lambda: True # noqa: E731
always_true = lambda _ctx: True # noqa: E731
def upstream() -> str:
return "hello"
+334 -13
View File
@@ -1,7 +1,11 @@
"""Tests for executors module edge cases."""
from __future__ import annotations
import asyncio
import logging
import sys
from typing import Callable
import pytest
@@ -54,7 +58,7 @@ def test_verbose_event_callback_running():
assert report.success
def test_verbose_run_with_success_lifecycle(capsys):
def test_verbose_run_with_success_lifecycle(capsys: pytest.CaptureFixture[str]):
"""Test px.run with verbose=True prints SUCCESS lifecycle."""
spec = px.TaskSpec("test", fn=lambda: "result")
graph = px.Graph.from_specs([spec])
@@ -64,7 +68,7 @@ def test_verbose_run_with_success_lifecycle(capsys):
assert "成功" in captured.out
def test_verbose_run_with_failed_lifecycle(capsys):
def test_verbose_run_with_failed_lifecycle(capsys: pytest.CaptureFixture[str]):
"""Test px.run with verbose=True prints FAILED lifecycle with error."""
def raise_error():
@@ -80,12 +84,12 @@ def test_verbose_run_with_failed_lifecycle(capsys):
assert "test error" in captured.out
def test_verbose_run_with_skipped_lifecycle(capsys):
def test_verbose_run_with_skipped_lifecycle(capsys: pytest.CaptureFixture[str]):
"""Test px.run with verbose=True prints SKIPPED lifecycle."""
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
)
graph = px.Graph.from_specs([spec])
report = px.run(graph, strategy="sequential", verbose=True)
@@ -95,18 +99,22 @@ def test_verbose_run_with_skipped_lifecycle(capsys):
def test_verbose_run_with_user_callback():
"""Test px.run with verbose=True and user callback both called."""
"""Test px.run with verbose=True and user callback both called.
预期事件序列RUNNING开始 SUCCESS完成
"""
events = []
def on_event(event):
def on_event(event: px.TaskEvent):
events.append(event)
spec = px.TaskSpec("test", fn=lambda: "result")
graph = px.Graph.from_specs([spec])
report = px.run(graph, strategy="sequential", verbose=True, on_event=on_event)
assert report.success
assert len(events) == 1
assert events[0].status == px.TaskStatus.SUCCESS
assert len(events) == 2
assert events[0].status == px.TaskStatus.RUNNING
assert events[1].status == px.TaskStatus.SUCCESS
def test_verbose_event_callback_success():
@@ -140,7 +148,7 @@ def test_verbose_event_callback_skipped():
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
verbose=True,
)
graph = px.Graph.from_specs([spec])
@@ -161,7 +169,11 @@ def test_execute_sync_with_retries():
raise ValueError("temporary error")
return "success"
spec = px.TaskSpec("retry_test", fn=failing_function, retries=3)
spec = px.TaskSpec(
"retry_test",
fn=failing_function,
retry=px.RetryPolicy(max_attempts=3),
)
graph = px.Graph.from_specs([spec])
# Should succeed after retries
@@ -182,7 +194,11 @@ def test_execute_async_with_retries():
raise ValueError("temporary error")
return "success"
spec = px.TaskSpec("retry_async_test", fn=failing_async_function, retries=3)
spec = px.TaskSpec(
"retry_async_test",
fn=failing_async_function,
retry=px.RetryPolicy(max_attempts=3),
)
graph = px.Graph.from_specs([spec])
# Should succeed after retries
@@ -196,7 +212,7 @@ def test_execute_sync_skip_on_condition():
spec = px.TaskSpec(
"skip_test",
fn=lambda: "result",
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
)
graph = px.Graph.from_specs([spec])
@@ -210,7 +226,7 @@ def test_execute_async_skip_on_condition():
spec = px.TaskSpec(
"skip_async_test",
fn=lambda: "result",
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
)
graph = px.Graph.from_specs([spec])
@@ -243,3 +259,308 @@ def test_execute_async_with_error():
with pytest.raises(px.TaskFailedError):
px.run(graph, strategy="async")
# ---------------------------------------------------------------------- #
# _check_upstream_skipped 分支测试
# ---------------------------------------------------------------------- #
def test_allow_upstream_skip_allows_execution_after_skipped() -> None:
"""allow_upstream_skip=True 时上游被 SKIPPED 后本任务仍执行."""
never_true = lambda _ctx: False # noqa: E731
def downstream_task() -> str:
return "ran despite upstream skipped"
graph = px.Graph.from_specs([
px.TaskSpec("upstream", fn=lambda: "up", conditions=(never_true,)),
px.TaskSpec("downstream", fn=downstream_task, depends_on=("upstream",), allow_upstream_skip=True),
])
report = px.run(graph, strategy="sequential")
assert report.success
assert report.results["upstream"].status == TaskStatus.SKIPPED
assert report.results["downstream"].status == TaskStatus.SUCCESS
assert report["downstream"] == "ran despite upstream skipped"
def test_upstream_failed_skips_downstream() -> None:
"""上游 FAILED 时下游被 SKIPPED(除非 allow_upstream_skip=True."""
def boom():
raise ValueError("boom")
def downstream():
return "should not run"
graph = px.Graph.from_specs([
px.TaskSpec("upstream", fn=boom),
px.TaskSpec("downstream", fn=downstream, depends_on=("upstream",)),
])
with pytest.raises(px.TaskFailedError):
px.run(graph, strategy="sequential")
# ---------------------------------------------------------------------- #
# _evaluate_conditions 多条件分支测试
# ---------------------------------------------------------------------- #
def test_multiple_conditions_failure_truncation() -> None:
"""超过 2 个条件失败时应截断显示."""
spec = px.TaskSpec(
"multi_skip",
fn=lambda: "result",
conditions=(lambda _ctx: False, lambda _ctx: False, lambda _ctx: False, lambda _ctx: False, lambda _ctx: False),
)
graph = px.Graph.from_specs([spec])
report = px.run(graph, strategy="sequential", verbose=True)
assert report.success
assert report.results["multi_skip"].status == TaskStatus.SKIPPED
# reason 应显示 "条件不满足: <lambda>, <lambda> 等5个条件"
# ---------------------------------------------------------------------- #
# concurrency_key 测试
# ---------------------------------------------------------------------- #
def test_concurrency_key_sequential() -> None:
"""sequential 策略下 concurrency_key 无效果."""
spec = px.TaskSpec("a", fn=lambda: 1, concurrency_key="group1")
graph = px.Graph.from_specs([spec])
report = px.run(graph, strategy="sequential", concurrency_limits={"group1": 1})
assert report.success
def test_concurrency_key_thread() -> None:
"""thread 策略下 concurrency_key 应限制并发."""
import time
order = []
def make(name: str) -> Callable[[], str]:
def fn():
order.append(f"{name}-start")
time.sleep(0.1)
order.append(f"{name}-end")
return name
return fn
graph = px.Graph.from_specs([
px.TaskSpec("a", fn=make("a"), concurrency_key="group1"),
px.TaskSpec("b", fn=make("b"), concurrency_key="group1"),
px.TaskSpec("c", fn=make("c"), concurrency_key="group1"),
])
report = px.run(graph, strategy="thread", max_workers=10, concurrency_limits={"group1": 1})
assert report.success
# 由于 concurrency_key 限制为 1,任务应串行执行
# 验证顺序:每个任务的 start-end 应连续
# 可能顺序:a-start, a-end, b-start, b-end, c-start, c-end
def test_concurrency_key_async() -> None:
"""async 策略下 concurrency_key 应限制并发."""
import asyncio
async def task_a():
await asyncio.sleep(0.01)
return "a"
async def task_b():
await asyncio.sleep(0.01)
return "b"
graph = px.Graph.from_specs([
px.TaskSpec("a", fn=task_a, concurrency_key="group1"),
px.TaskSpec("b", fn=task_b, concurrency_key="group1"),
])
report = px.run(graph, strategy="async", concurrency_limits={"group1": 1})
assert report.success
# ---------------------------------------------------------------------- #
# dependency 策略测试
# ---------------------------------------------------------------------- #
def test_dependency_strategy_basic() -> None:
"""dependency 策略应正确执行."""
order = []
def make(name: str) -> Callable[[], str]:
def fn():
order.append(name)
return name
return fn
graph = px.Graph.from_specs([
px.TaskSpec("a", fn=make("a")),
px.TaskSpec("b", fn=make("b"), depends_on=("a",)),
px.TaskSpec("c", fn=make("c"), depends_on=("a",)),
px.TaskSpec("d", fn=make("d"), depends_on=("b", "c")),
])
report = px.run(graph, strategy="dependency")
assert report.success
assert "a" in order
assert "d" in order
def test_dependency_strategy_async() -> None:
"""dependency 策略下异步任务应正确执行."""
async def a():
return "a"
async def b(a: str):
return a + "b"
graph = px.Graph.from_specs([
px.TaskSpec("a", fn=a),
px.TaskSpec("b", fn=b, depends_on=("a",)),
])
report = px.run(graph, strategy="dependency")
assert report.success
assert report["b"] == "ab"
# ---------------------------------------------------------------------- #
# continue_on_error 测试
# ---------------------------------------------------------------------- #
def test_continue_on_error_marks_failed_but_continues() -> None:
"""continue_on_error=True 时任务失败不抛异常,但 report.success 为 True(无 TaskFailedError 抛出)。"""
def boom():
raise ValueError("boom")
graph = px.Graph.from_specs([
px.TaskSpec("fail", fn=boom, continue_on_error=True),
px.TaskSpec("other", fn=lambda: "ok"), # 无依赖,应继续
])
# continue_on_error=True 时 run 不抛异常,report.success 为 True
report = px.run(graph, strategy="sequential")
# report.success 为 True 因为没有抛 TaskFailedError
assert report.success # 因为 continue_on_error 阻止了 TaskFailedError
assert report.results["fail"].status == TaskStatus.FAILED
assert report.results["other"].status == TaskStatus.SUCCESS
def test_continue_on_error_downstream_skipped() -> None:
"""continue_on_error=True 时失败任务的下游被 SKIPPEDallow_upstream_skip=False 时)。"""
def boom():
raise ValueError("boom")
def downstream():
return "should not run"
graph = px.Graph.from_specs([
px.TaskSpec("fail", fn=boom, continue_on_error=True),
px.TaskSpec("dep", fn=downstream, depends_on=("fail",), allow_upstream_skip=False),
])
report = px.run(graph, strategy="sequential")
# report.success 为 True 因为 continue_on_error 阻止了 TaskFailedError
assert report.success
assert report.results["fail"].status == TaskStatus.FAILED
assert report.results["dep"].status == TaskStatus.SKIPPED
# ---------------------------------------------------------------------- #
# soft_depends_on 默认值注入测试
# ---------------------------------------------------------------------- #
def test_soft_depends_on_default_value_injection() -> None:
"""软依赖存在且成功时注入其结果值(参数名需与依赖名一致)。"""
def task_with_soft_dep(a: str | None = None) -> str:
return f"a={a}"
graph = px.Graph.from_specs([
px.TaskSpec("a", fn=lambda: "value"),
px.TaskSpec("b", fn=task_with_soft_dep, soft_depends_on=("a",)),
])
report = px.run(graph, strategy="sequential")
assert report.success
assert report["b"] == "a=value"
def test_soft_depends_on_skipped_injects_none() -> None:
"""软依赖被 SKIPPED 时注入 None(参数名需与依赖名一致)。"""
never_true = lambda _ctx: False # noqa: E731
def task_with_soft_dep(skipped: str | None = None) -> str:
return f"skipped={skipped}"
graph = px.Graph.from_specs([
px.TaskSpec("skipped", fn=lambda: "value", conditions=(never_true,)),
px.TaskSpec("b", fn=task_with_soft_dep, soft_depends_on=("skipped",)),
])
report = px.run(graph, strategy="sequential")
assert report.success
# 软依赖被 skipped 时注入 None(因为 global_context 中有 skipped,值为 None
assert report["b"] == "skipped=None"
# ---------------------------------------------------------------------- #
# hooks 异常处理测试
# ---------------------------------------------------------------------- #
def test_hooks_pre_run_exception_logged(caplog: pytest.LogCaptureFixture) -> None:
"""pre_run hook 抛异常应被记录但不影响任务."""
def bad_hook(_spec):
raise RuntimeError("hook error")
hooks = px.TaskHooks(pre_run=bad_hook)
spec = px.TaskSpec("a", fn=lambda: "ok", hooks=hooks)
graph = px.Graph.from_specs([spec])
with caplog.at_level(logging.WARNING, logger="pyflowx"):
report = px.run(graph, strategy="sequential")
assert report.success
assert any("hook" in r.message for r in caplog.records)
def test_hooks_post_run_exception_logged(caplog: pytest.LogCaptureFixture) -> None:
"""post_run hook 抛异常应被记录但不影响任务."""
def bad_hook(_spec, _value):
raise RuntimeError("post hook error")
hooks = px.TaskHooks(post_run=bad_hook)
spec = px.TaskSpec("a", fn=lambda: "ok", hooks=hooks)
graph = px.Graph.from_specs([spec])
with caplog.at_level(logging.WARNING, logger="pyflowx"):
report = px.run(graph, strategy="sequential")
assert report.success
assert any("hook" in r.message for r in caplog.records)
def test_hooks_on_failure_exception_logged(caplog: pytest.LogCaptureFixture) -> None:
"""on_failure hook 抛异常应被记录但不影响任务."""
def bad_hook(_spec, _exc):
raise RuntimeError("failure hook error")
hooks = px.TaskHooks(on_failure=bad_hook)
spec = px.TaskSpec("a", fn=lambda: (_ for _ in ()).throw(ValueError("task error")), hooks=hooks)
graph = px.Graph.from_specs([spec])
with caplog.at_level(logging.WARNING, logger="pyflowx"), pytest.raises(px.TaskFailedError):
px.run(graph, strategy="sequential")
assert any("hook" in r.message for r in caplog.records)
# ---------------------------------------------------------------------- #
# unknown strategy 测试
# ---------------------------------------------------------------------- #
def test_unknown_strategy_raises() -> None:
"""未知 strategy 应抛 ValueError."""
graph = px.Graph.from_specs([px.TaskSpec("a", fn=lambda: 1)])
with pytest.raises(ValueError, match="Unknown strategy"):
# pyrefly: ignore [bad-argument-type]
px.run(graph, strategy="unknown_strategy")
# ---------------------------------------------------------------------- #
# 空图测试
# ---------------------------------------------------------------------- #
def test_empty_graph_dependency_strategy() -> None:
"""dependency 策略下空图应正常返回."""
graph = px.Graph()
report = px.run(graph, strategy="dependency")
assert report.success
assert len(report) == 0
+257 -74
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import pytest
import pyflowx as px
from pyflowx.compose import GraphComposer, compose
from pyflowx.errors import CycleError, DuplicateTaskError, MissingDependencyError
@@ -13,13 +14,11 @@ def _fn() -> None:
def test_from_specs_builds_graph() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("a", "b")),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("a", "b")),
])
assert set(graph.names) == {"a", "b", "c"}
assert graph.dependencies("c") == ("a", "b")
assert len(graph) == 3
@@ -28,23 +27,19 @@ def test_from_specs_builds_graph() -> None:
def test_from_specs_allows_forward_references() -> None:
# b depends on a, but a is declared after b — order should not matter.
graph = px.Graph.from_specs(
[
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("a", _fn),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("a", _fn),
])
assert graph.layers() == [["a"], ["b"]]
def test_duplicate_task_raises() -> None:
with pytest.raises(DuplicateTaskError):
_ = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("a", _fn),
]
)
_ = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("a", _fn),
])
def test_missing_dependency_raises() -> None:
@@ -57,24 +52,20 @@ def test_missing_dependency_raises() -> None:
def test_cycle_detection() -> None:
with pytest.raises(CycleError):
_ = px.Graph.from_specs(
[
px.TaskSpec("a", _fn, depends_on=("c",)),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("b",)),
]
)
_ = px.Graph.from_specs([
px.TaskSpec("a", _fn, depends_on=("c",)),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("b",)),
])
def test_layers_grouping() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn),
px.TaskSpec("c", _fn, depends_on=("a", "b")),
px.TaskSpec("d", _fn, depends_on=("c",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn),
px.TaskSpec("c", _fn, depends_on=("a", "b")),
px.TaskSpec("d", _fn, depends_on=("c",)),
])
layers = graph.layers()
assert layers == [["a", "b"], ["c"], ["d"]]
@@ -85,12 +76,10 @@ def test_self_dependency_rejected() -> None:
def test_to_mermaid() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
])
mermaid = graph.to_mermaid()
assert mermaid.startswith("graph TD")
assert 'a["a"]' in mermaid
@@ -104,13 +93,11 @@ def test_to_mermaid_invalid_orientation() -> None:
def test_subgraph_by_tags() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn, tags=("ingest",)),
px.TaskSpec("b", _fn, depends_on=("a",), tags=("ingest",)),
px.TaskSpec("c", _fn, depends_on=("b",), tags=("report",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn, tags=("ingest",)),
px.TaskSpec("b", _fn, depends_on=("a",), tags=("ingest",)),
px.TaskSpec("c", _fn, depends_on=("b",), tags=("report",)),
])
sub = graph.subgraph(["ingest"])
assert set(sub.names) == {"a", "b"}
# Edge to dropped task c is removed; b no longer waits for anything
@@ -119,13 +106,11 @@ def test_subgraph_by_tags() -> None:
def test_subgraph_by_names() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("b",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
px.TaskSpec("c", _fn, depends_on=("b",)),
])
sub = graph.subgraph_by_names(["a", "b"])
assert set(sub.names) == {"a", "b"}
# c is dropped, so b's dep on c (none here) — but a->b edge preserved.
@@ -139,12 +124,10 @@ def test_subgraph_by_names_unknown() -> None:
def test_describe() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
])
desc = graph.describe()
assert "Layer 1" in desc
assert "Layer 2" in desc
@@ -179,6 +162,19 @@ def test_all_specs_returns_view() -> None:
assert view is graph.all_specs() or view == graph.all_specs()
def test_all_deps_combines_hard_and_soft() -> None:
"""all_deps 应返回硬依赖 + 软依赖的组合。"""
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn),
px.TaskSpec("c", _fn, depends_on=("a",), soft_depends_on=("b",)),
])
all_deps = graph.all_deps("c")
assert set(all_deps) == {"a", "b"}
# 硬依赖在前,软依赖在后
assert all_deps == ("a", "b")
def test_spec_accessor() -> None:
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)])
assert graph.spec("a").name == "a"
@@ -187,12 +183,10 @@ def test_spec_accessor() -> None:
def test_dependencies_accessor() -> None:
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, depends_on=("a",)),
])
assert graph.dependencies("a") == ()
assert graph.dependencies("b") == ("a",)
@@ -210,16 +204,20 @@ def test_empty_graph_layers() -> None:
def test_subgraph_preserves_metadata() -> None:
"""子图应保留原任务的 retries/timeout/tags 等元数据。"""
graph = px.Graph.from_specs(
[
px.TaskSpec("a", _fn, tags=("x",), retries=3, timeout=5.0),
px.TaskSpec("b", _fn, depends_on=("a",), tags=("y",)),
]
)
"""子图应保留原任务的 retry/timeout/tags 等元数据。"""
graph = px.Graph.from_specs([
px.TaskSpec(
"a",
_fn,
tags=("x",),
retry=px.RetryPolicy(max_attempts=3),
timeout=5.0,
),
px.TaskSpec("b", _fn, depends_on=("a",), tags=("y",)),
])
sub = graph.subgraph(["x"])
spec = sub.spec("a")
assert spec.retries == 3
assert spec.retry.max_attempts == 3
assert spec.timeout == 5.0
assert spec.tags == ("x",)
@@ -229,3 +227,188 @@ def test_subgraph_by_tags_no_match() -> None:
graph = px.Graph.from_specs([px.TaskSpec("a", _fn, tags=("x",))])
sub = graph.subgraph(["z"])
assert len(sub) == 0
# ---------------------------------------------------------------------- #
# from_specs str 类型分支测试
# ---------------------------------------------------------------------- #
def test_from_specs_with_string_ref() -> None:
"""from_specs 接受字符串引用并收集到 pending_refs."""
# 字符串引用被收集到 _pending_refs,而非尝试打开文件
graph = px.Graph.from_specs(["ref_cmd"])
assert graph._pending_refs == ["ref_cmd"]
def test_from_specs_with_invalid_type() -> None:
"""from_specs 接受不支持的类型时应抛 TypeError."""
with pytest.raises(TypeError, match="from_specs 只接受 TaskSpec 或 str"):
_ = px.Graph.from_specs([123]) # type: ignore[list-item]
# ---------------------------------------------------------------------- #
# to_mermaid 软依赖测试
# ---------------------------------------------------------------------- #
def test_to_mermaid_soft_depends_on() -> None:
"""to_mermaid 应正确绘制软依赖为虚线."""
graph = px.Graph.from_specs([
px.TaskSpec("a", _fn),
px.TaskSpec("b", _fn, soft_depends_on=("a",)),
])
mermaid = graph.to_mermaid()
assert "a -.-> b" in mermaid # 软依赖用虚线
# ---------------------------------------------------------------------- #
# GraphComposer 与 compose 测试
# ---------------------------------------------------------------------- #
def test_graph_composer_resolve_all() -> None:
"""GraphComposer.resolve_all 应展开所有图的字符串引用."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn), px.TaskSpec("a2", _fn, depends_on=("a1",))])
# 创建带 _pending_refs 的图
graph_b = px.Graph.from_specs([px.TaskSpec("b1", _fn)])
graph_b._pending_refs = ["cmd_a"] # 手动设置内部属性
composer = GraphComposer({"cmd_a": graph_a, "cmd_b": graph_b})
resolved = composer.resolve_all()
# graph_b 应包含 graph_a 的任务
assert "a1" in resolved["cmd_b"]
assert "a2" in resolved["cmd_b"]
def test_graph_composer_parse_ref_self_reference() -> None:
"""GraphComposer.parse_ref 应检测循环引用."""
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)])
composer = GraphComposer({"cmd": graph})
with pytest.raises(ValueError, match="循环引用"):
_ = composer.parse_ref("cmd", "cmd")
def test_graph_composer_parse_ref_cmd_not_found() -> None:
"""GraphComposer.parse_ref 应检测引用的命令不存在."""
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)])
composer = GraphComposer({"cmd": graph})
with pytest.raises(ValueError, match="引用的命令 'missing' 不存在"):
_ = composer.parse_ref("missing", "current")
def test_graph_composer_parse_ref_task_not_found() -> None:
"""GraphComposer.parse_ref 应检测任务不存在于引用的命令中."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn)])
graph_b = px.Graph.from_specs([px.TaskSpec("b1", _fn)])
composer = GraphComposer({"cmd_a": graph_a, "cmd_b": graph_b})
with pytest.raises(ValueError, match="任务 'missing' 不存在于命令 'cmd_a'"):
_ = composer.parse_ref("cmd_a.missing", "cmd_b")
def test_graph_composer_expand_refs_no_pending() -> None:
"""GraphComposer.expand_refs 无 pending_refs 时应原样返回."""
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)])
composer = GraphComposer({"cmd": graph})
expanded = composer.expand_refs(graph, "cmd")
assert expanded is graph
def test_compose_function() -> None:
"""compose() 函数应等同于 GraphComposer().resolve_all()。"""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn)])
graph_b = px.Graph.from_specs([px.TaskSpec("b1", _fn)])
graph_b._pending_refs = ["cmd_a"] # 手动设置内部属性
resolved = compose({"cmd_a": graph_a, "cmd_b": graph_b})
assert "a1" in resolved["cmd_b"]
def test_graph_composer_expand_refs_multiple_refs_chain() -> None:
"""expand_refs 多个 ref 应串联依赖:后一个 ref 首任务依赖前一个 ref 末任务."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn)])
graph_c = px.Graph.from_specs([px.TaskSpec("c1", _fn)])
graph_b = px.Graph.from_specs([px.TaskSpec("b1", _fn)])
graph_b._pending_refs = ["cmd_a", "cmd_c"]
composer = GraphComposer({"cmd_a": graph_a, "cmd_c": graph_c, "cmd_b": graph_b})
resolved = composer.resolve_all()
# c1 应依赖 a1(后 ref 首任务依赖前 ref 末任务)
assert "a1" in resolved["cmd_b"]
assert "c1" in resolved["cmd_b"]
assert "b1" in resolved["cmd_b"]
c1_spec = resolved["cmd_b"].all_specs()["c1"]
assert "a1" in c1_spec.depends_on
def test_graph_composer_expand_refs_ref_returns_empty() -> None:
"""expand_refs 引用空图时,previous_ref_last_task 保持 Noneoriginal_specs 走 else 分支."""
graph_empty = px.Graph.from_specs([])
graph_b = px.Graph.from_specs([px.TaskSpec("b1", _fn)])
graph_b._pending_refs = ["empty_cmd"]
composer = GraphComposer({"empty_cmd": graph_empty, "cmd_b": graph_b})
resolved = composer.resolve_all()
# b1 保留,无额外依赖
assert "b1" in resolved["cmd_b"]
b1_spec = resolved["cmd_b"].all_specs()["b1"]
assert b1_spec.depends_on == ()
def test_graph_composer_expand_refs_multiple_original_specs_serialized() -> None:
"""expand_refs 多个 original_specs 应串行依赖,且首个依赖 ref 末任务."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn)])
graph_b = px.Graph.from_specs([
px.TaskSpec("b1", _fn),
px.TaskSpec("b2", _fn),
px.TaskSpec("b3", _fn),
])
graph_b._pending_refs = ["cmd_a"]
composer = GraphComposer({"cmd_a": graph_a, "cmd_b": graph_b})
resolved = composer.resolve_all()
specs = resolved["cmd_b"].all_specs()
# b1 依赖 a1ref 末任务)
assert "a1" in specs["b1"].depends_on
# b2 依赖 b1,b3 依赖 b2(串行)
assert "b1" in specs["b2"].depends_on
assert "b2" in specs["b3"].depends_on
def test_graph_composer_parse_ref_dot_notation_success() -> None:
"""parse_ref 'cmd.task' 形式应返回对应单个 TaskSpec."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn), px.TaskSpec("a2", _fn)])
composer = GraphComposer({"cmd_a": graph_a})
result = composer.parse_ref("cmd_a.a2", "cmd_b")
assert len(result) == 1
assert result[0].name == "a2"
def test_graph_composer_parse_ref_dot_notation_cmd_not_found() -> None:
"""parse_ref 'missing.task' 形式应检测命令不存在."""
graph_a = px.Graph.from_specs([px.TaskSpec("a1", _fn)])
composer = GraphComposer({"cmd_a": graph_a})
with pytest.raises(ValueError, match="引用的命令 'missing' 不存在"):
_ = composer.parse_ref("missing.task", "cmd_b")
# ---------------------------------------------------------------------- #
# resolved_spec defaults 测试
# ---------------------------------------------------------------------- #
def test_resolved_spec_applies_defaults() -> None:
"""resolved_spec 应应用 Graph.defaults。"""
defaults = px.GraphDefaults(timeout=10.0, retry=px.RetryPolicy(max_attempts=2))
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)], defaults=defaults)
resolved = graph.resolved_spec("a")
assert resolved.timeout == 10.0
assert resolved.retry.max_attempts == 2
def test_resolved_spec_no_override() -> None:
"""resolved_spec 不应覆盖任务已有的设置。"""
defaults = px.GraphDefaults(timeout=10.0)
graph = px.Graph.from_specs([px.TaskSpec("a", _fn, timeout=5.0)], defaults=defaults)
resolved = graph.resolved_spec("a")
assert resolved.timeout == 5.0 # 保持原值,不被 defaults 覆盖
+152
View File
@@ -0,0 +1,152 @@
"""Tests for Graph namespace and add_subgraph."""
from __future__ import annotations
import pytest
import pyflowx as px
def _fn() -> None:
return None
def test_graph_namespace_field_default_none() -> None:
"""Graph 默认 namespace 为 None."""
graph = px.Graph()
assert graph.namespace is None
def test_graph_from_specs_with_namespace() -> None:
"""from_specs(namespace=...) 应设置 graph.namespace."""
graph = px.Graph.from_specs([px.TaskSpec("a", _fn)], namespace="ns1")
assert graph.namespace == "ns1"
def test_add_subgraph_prefixes_task_names() -> None:
"""add_subgraph 应给子图任务名加命名空间前缀."""
sub = px.Graph.from_specs(
[px.TaskSpec("extract", _fn), px.TaskSpec("build", _fn, depends_on=("extract",))],
namespace="build",
)
main = px.Graph.from_specs([px.TaskSpec("start", _fn)])
main.add_subgraph(sub)
assert "start" in main
assert "build:extract" in main
assert "build:build" in main
def test_add_subgraph_renames_internal_deps() -> None:
"""add_subgraph 应给子图内部依赖名加前缀."""
sub = px.Graph.from_specs(
[px.TaskSpec("a", _fn), px.TaskSpec("b", _fn, depends_on=("a",))],
namespace="ns",
)
main = px.Graph()
main.add_subgraph(sub)
b_spec = main.all_specs()["ns:b"]
assert b_spec.depends_on == ("ns:a",)
def test_add_subgraph_all_internal_deps_prefixed() -> None:
"""add_subgraph 子图内所有任务(含被依赖的)都加前缀."""
sub = px.Graph.from_specs(
[px.TaskSpec("ext", _fn), px.TaskSpec("b", _fn, depends_on=("ext",))],
namespace="ns",
)
main = px.Graph()
main.add_subgraph(sub)
b_spec = main.all_specs()["ns:b"]
assert b_spec.depends_on == ("ns:ext",)
assert "ns:ext" in main
def test_add_subgraph_requires_namespace() -> None:
"""add_subgraph 无 namespace 时应抛 ValueError."""
sub = px.Graph.from_specs([px.TaskSpec("a", _fn)]) # 无 namespace
main = px.Graph()
with pytest.raises(ValueError, match="namespace"):
main.add_subgraph(sub)
def test_add_subgraph_explicit_namespace_overrides() -> None:
"""add_subgraph(namespace=...) 应覆盖子图自带 namespace."""
sub = px.Graph.from_specs([px.TaskSpec("a", _fn)], namespace="original")
main = px.Graph()
main.add_subgraph(sub, namespace="override")
assert "override:a" in main
assert "original:a" not in main
def test_add_subgraph_internal_injection_works() -> None:
"""子图内部依赖注入应通过 wrapper 正常工作."""
sub = px.Graph.from_specs(
[
px.TaskSpec("extract", lambda: [1, 2, 3]),
px.TaskSpec("build", lambda extract: [x * 2 for x in extract], depends_on=("extract",)),
],
namespace="build",
)
main = px.Graph()
main.add_subgraph(sub)
report = px.run(main)
assert report.success
assert report["build:build"] == [2, 4, 6]
def test_add_subgraph_cross_namespace_ref_via_context() -> None:
"""跨命名空间引用应通过 Context 标注接收."""
def consumer(ctx: px.Context) -> str:
return f"got {ctx['ns:data']}"
sub = px.Graph.from_specs(
[px.TaskSpec("data", lambda: "data_value")],
namespace="ns",
)
main = px.Graph()
main.add_subgraph(sub)
main.add(px.TaskSpec("consumer", consumer, depends_on=("ns:data",)))
report = px.run(main)
assert report.success
assert report["consumer"] == "got data_value"
def test_add_subgraph_context_annotation_in_subgraph() -> None:
"""子图内部任务用 Context 标注时,wrapper 应正确传递."""
def sink(ctx: px.Context) -> int:
return ctx["src"]
sub = px.Graph.from_specs(
[
px.TaskSpec("src", lambda: 42),
px.TaskSpec("sink", sink, depends_on=("src",)),
],
namespace="ns",
)
main = px.Graph()
main.add_subgraph(sub)
report = px.run(main)
assert report.success
assert report["ns:sink"] == 42
def test_add_subgraph_chained() -> None:
"""多个子图可链式合并到主图."""
sub_a = px.Graph.from_specs([px.TaskSpec("a", _fn)], namespace="nsA")
sub_b = px.Graph.from_specs([px.TaskSpec("b", _fn)], namespace="nsB")
main = px.Graph()
main.add_subgraph(sub_a).add_subgraph(sub_b)
assert "nsA:a" in main
assert "nsB:b" in main
+47
View File
@@ -126,3 +126,50 @@ class TestRunReportDescribe:
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
desc = report.describe()
assert "-" in desc # duration 显示为 "-"
class TestRunReportQueries:
"""测试 RunReport 的新查询 API."""
def test_succeeded_tasks(self) -> None:
"""succeeded_tasks 返回 SUCCESS 状态的任务名."""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.SUCCESS)
report.results["b"] = _make_result("b", status=TaskStatus.FAILED)
report.results["c"] = _make_result("c", status=TaskStatus.SUCCESS)
assert report.succeeded_tasks() == ["a", "c"]
def test_skipped_tasks(self) -> None:
"""skipped_tasks 返回 SKIPPED 状态的任务名."""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.SKIPPED)
report.results["b"] = _make_result("b", status=TaskStatus.SUCCESS)
assert report.skipped_tasks() == ["a"]
def test_tasks_by_status(self) -> None:
"""tasks_by_status 按指定状态过滤."""
report = px.RunReport()
report.results["a"] = _make_result("a", status=TaskStatus.FAILED)
report.results["b"] = _make_result("b", status=TaskStatus.FAILED)
report.results["c"] = _make_result("c", status=TaskStatus.SUCCESS)
assert report.tasks_by_status(TaskStatus.FAILED) == ["a", "b"]
assert report.tasks_by_status(TaskStatus.SUCCESS) == ["c"]
assert report.tasks_by_status(TaskStatus.SKIPPED) == []
def test_durations(self) -> None:
"""durations 返回任务名 -> 时长映射."""
report = px.RunReport()
report.results["a"] = _make_result("a", duration=1.5)
report.results["b"] = _make_result("b", duration=2.0)
durs = report.durations()
assert durs["a"] == 1.5
assert durs["b"] == 2.0
def test_durations_no_duration(self) -> None:
"""无时长的任务应返回 0.0."""
report = px.RunReport()
spec: TaskSpec[Any] = TaskSpec[Any]("a", _fn) # type: ignore[arg-type]
report.results["a"] = TaskResult[Any](spec=spec, status=TaskStatus.PENDING)
durs = report.durations()
assert durs["a"] == 0.0
+198 -104
View File
@@ -29,24 +29,20 @@ def _echo_graph(name: str = "echo_task", msg: str = "hello") -> px.Graph:
def _failing_graph() -> px.Graph:
"""构造一个必定失败的单任务图."""
return px.Graph.from_specs(
[
px.TaskSpec(
"fail",
cmd=["python", "-c", "import sys; sys.exit(1)"],
)
]
)
return px.Graph.from_specs([
px.TaskSpec(
"fail",
cmd=["python", "-c", "import sys; sys.exit(1)"],
)
])
def _multi_task_graph() -> px.Graph:
"""构造一个带依赖的多任务图."""
return px.Graph.from_specs(
[
px.TaskSpec("a", cmd=[*ECHO_CMD, "a"]),
px.TaskSpec("b", cmd=[*ECHO_CMD, "b"], depends_on=("a",)),
]
)
return px.Graph.from_specs([
px.TaskSpec("a", cmd=[*ECHO_CMD, "a"]),
px.TaskSpec("b", cmd=[*ECHO_CMD, "b"], depends_on=("a",)),
])
# ---------------------------------------------------------------------- #
@@ -57,18 +53,18 @@ class TestCliRunnerConstruction:
def test_requires_at_least_one_command(self) -> None:
"""没有命令时应抛出 ValueError."""
with pytest.raises(ValueError, match="至少需要一个命令"):
with pytest.raises(ValueError, match="至少需要一个别名"):
_ = px.CliRunner()
def test_accepts_single_graph(self) -> None:
"""单个命令应正常构造."""
runner = px.CliRunner(graphs={"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
assert runner.commands == ["clean"]
def test_accepts_multiple_graphs(self) -> None:
"""多个命令应按插入顺序保留."""
runner = px.CliRunner(
graphs={
aliases={
"clean": _echo_graph("c", "clean"),
"build": _echo_graph("b", "build"),
"test": _echo_graph("t", "test"),
@@ -76,39 +72,39 @@ class TestCliRunnerConstruction:
)
assert runner.commands == ["clean", "build", "test"]
def test_default_strategy_is_sequential(self) -> None:
"""默认策略应为 Strategy.SEQUENTIAL."""
runner = px.CliRunner({"clean": _echo_graph()})
assert runner.strategy == "sequential"
def test_default_strategy_is_dependency(self) -> None:
"""默认策略应为 dependency(依赖驱动,最大并行度)."""
runner = px.CliRunner(aliases={"clean": _echo_graph()})
assert runner.strategy == "dependency"
def test_custom_strategy_string(self) -> None:
"""应支持通过字符串指定策略."""
runner = px.CliRunner({"clean": _echo_graph()}, strategy="thread")
runner = px.CliRunner(aliases={"clean": _echo_graph()}, strategy="thread")
assert runner.strategy == "thread"
def test_custom_strategy_enum(self) -> None:
"""应支持通过 Strategy 枚举指定策略."""
runner = px.CliRunner({"clean": _echo_graph()}, strategy="async")
runner = px.CliRunner(aliases={"clean": _echo_graph()}, strategy="async")
assert runner.strategy == "async"
def test_default_verbose_is_true(self) -> None:
"""默认 verbose 应为 True."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
assert runner.verbose is True
def test_custom_verbose_false(self) -> None:
"""应支持关闭 verbose."""
runner = px.CliRunner({"clean": _echo_graph()}, verbose=False)
runner = px.CliRunner(aliases={"clean": _echo_graph()}, verbose=False)
assert runner.verbose is False
def test_default_description_is_empty(self) -> None:
"""默认描述应为空字符串."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
assert runner.description == ""
def test_custom_description(self) -> None:
"""应支持自定义描述."""
runner = px.CliRunner({"clean": _echo_graph()}, description="My CLI")
runner = px.CliRunner(aliases={"clean": _echo_graph()}, description="My CLI")
assert runner.description == "My CLI"
@@ -120,13 +116,13 @@ class TestCliRunnerProperties:
def test_commands_returns_list(self) -> None:
"""commands 应返回列表."""
runner = px.CliRunner({"a": _echo_graph(), "b": _echo_graph()})
runner = px.CliRunner(aliases={"a": _echo_graph(), "b": _echo_graph()})
assert isinstance(runner.commands, list)
def test_graphs_contains_original_graphs(self) -> None:
"""graphs 应包含原始 Graph 实例."""
g = _echo_graph()
runner = px.CliRunner({"cmd": g})
runner = px.CliRunner(aliases={"cmd": g})
assert runner.graphs["cmd"] is g
@@ -140,69 +136,69 @@ class TestCliRunnerParser:
"""create_parser 应返回 ArgumentParser."""
from argparse import ArgumentParser
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
assert isinstance(parser, ArgumentParser)
def test_parser_has_command_argument(self) -> None:
"""解析器应有 command 位置参数."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean"])
assert parsed.command == "clean"
def test_parser_command_is_optional(self) -> None:
"""command 应为可选参数."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args([])
assert parsed.command is None
def test_parser_has_strategy_option(self) -> None:
"""解析器应有 --strategy 选项."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean", "--strategy", "thread"])
assert parsed.strategy == "thread"
def test_parser_strategy_default(self) -> None:
"""--strategy 默认值应与构造时一致."""
runner = px.CliRunner({"clean": _echo_graph()}, strategy="async")
runner = px.CliRunner(aliases={"clean": _echo_graph()}, strategy="async")
parser = runner.create_parser()
parsed = parser.parse_args(["clean"])
assert parsed.strategy == "async"
def test_parser_has_dry_run_flag(self) -> None:
"""解析器应有 --dry-run 标志."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean", "--dry-run"])
assert parsed.dry_run is True
def test_parser_dry_run_default_false(self) -> None:
"""--dry-run 默认为 False."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean"])
assert parsed.dry_run is False
def test_parser_has_list_flag(self) -> None:
"""解析器应有 --list 标志."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["--list"])
assert parsed.list is True
def test_parser_has_quiet_flag(self) -> None:
"""解析器应有 --quiet 标志."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean", "--quiet"])
assert parsed.quiet is True
def test_parser_quiet_default_false(self) -> None:
"""--quiet 默认为 False."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
parser = runner.create_parser()
parsed = parser.parse_args(["clean"])
assert parsed.quiet is False
@@ -226,7 +222,7 @@ class TestCliRunnerRunSuccess:
def test_run_valid_command_returns_zero(self) -> None:
"""有效命令执行成功应返回 0."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
exit_code = runner.run(["clean"])
assert exit_code == CliExitCode.SUCCESS.value
@@ -241,7 +237,7 @@ class TestCliRunnerRunSuccess:
executed.append("b")
runner = px.CliRunner(
{
aliases={
"a": px.Graph.from_specs([px.TaskSpec("a", track_a)]),
"b": px.Graph.from_specs([px.TaskSpec("b", track_b)]),
}
@@ -251,19 +247,19 @@ class TestCliRunnerRunSuccess:
def test_run_multi_task_graph(self) -> None:
"""应能执行带依赖的多任务图."""
runner = px.CliRunner({"multi": _multi_task_graph()})
runner = px.CliRunner(aliases={"multi": _multi_task_graph()})
exit_code = runner.run(["multi"])
assert exit_code == CliExitCode.SUCCESS.value
def test_run_with_strategy_override(self) -> None:
"""应支持通过 --strategy 覆盖默认策略."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
exit_code = runner.run(["echo", "--strategy", "thread"])
assert exit_code == CliExitCode.SUCCESS.value
def test_run_with_dry_run(self, capsys: pytest.CaptureFixture[str]) -> None:
"""--dry-run 应只打印计划不执行."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
exit_code = runner.run(["echo", "--dry-run"])
assert exit_code == CliExitCode.SUCCESS.value
captured = capsys.readouterr()
@@ -278,7 +274,7 @@ class TestCliRunnerVerbose:
def test_verbose_default_prints_lifecycle(self, capsys: pytest.CaptureFixture[str]) -> None:
"""默认 verbose=True 应打印任务生命周期."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
_ = runner.run(["echo"])
captured = capsys.readouterr()
# verbose 模式下应打印任务生命周期
@@ -286,7 +282,7 @@ class TestCliRunnerVerbose:
def test_quiet_flag_disables_verbose(self, capsys: pytest.CaptureFixture[str]) -> None:
"""--quiet 应关闭 verbose 输出."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
_ = runner.run(["echo", "--quiet"])
captured = capsys.readouterr()
# quiet 模式下不应有 [verbose] 前缀的输出
@@ -294,14 +290,14 @@ class TestCliRunnerVerbose:
def test_verbose_false_constructor_disables_verbose(self, capsys: pytest.CaptureFixture[str]) -> None:
"""构造时 verbose=False 应关闭 verbose 输出."""
runner = px.CliRunner({"echo": _echo_graph()}, verbose=False)
runner = px.CliRunner(aliases={"echo": _echo_graph()}, verbose=False)
_ = runner.run(["echo"])
captured = capsys.readouterr()
assert "[verbose]" not in captured.out
def test_verbose_prints_command_for_cmd_task(self, capsys: pytest.CaptureFixture[str]) -> None:
"""verbose 模式下 cmd 任务应打印执行的命令."""
runner = px.CliRunner({"echo": _echo_graph(msg="verbose-test")})
runner = px.CliRunner(aliases={"echo": _echo_graph(msg="verbose-test")})
_ = runner.run(["echo"])
captured = capsys.readouterr()
# 应打印执行的命令
@@ -311,30 +307,28 @@ class TestCliRunnerVerbose:
def test_verbose_prints_success_lifecycle(self, capsys: pytest.CaptureFixture[str]) -> None:
"""verbose 模式下成功任务应打印成功信息."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
_ = runner.run(["echo"])
captured = capsys.readouterr()
assert "成功" in captured.out
def test_verbose_prints_skip_lifecycle(self, capsys: pytest.CaptureFixture[str]) -> None:
"""verbose 模式下跳过的任务应打印跳过信息."""
graph = px.Graph.from_specs(
[
px.TaskSpec(
"skip_me",
cmd=[*ECHO_CMD, "skip"],
conditions=(lambda: False,),
),
]
)
runner = px.CliRunner({"skip": graph})
graph = px.Graph.from_specs([
px.TaskSpec(
"skip_me",
cmd=[*ECHO_CMD, "skip"],
conditions=(lambda _ctx: False,),
),
])
runner = px.CliRunner(aliases={"skip": graph})
_ = runner.run(["skip"])
captured = capsys.readouterr()
assert "跳过" in captured.out
def test_verbose_prints_failure_lifecycle(self, capsys: pytest.CaptureFixture[str]) -> None:
"""verbose 模式下失败任务应打印失败信息."""
runner = px.CliRunner({"fail": _failing_graph()})
runner = px.CliRunner(aliases={"fail": _failing_graph()})
_ = runner.run(["fail"])
captured = capsys.readouterr()
# 失败信息可能出现在 stdout (verbose) 或 stderr (PyFlowXError)
@@ -350,7 +344,7 @@ class TestCliRunnerRunFailure:
def test_run_unknown_command_returns_failure(self, capsys: pytest.CaptureFixture[str]) -> None:
"""未知命令应返回 1 并打印错误."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
exit_code = runner.run(["unknown"])
assert exit_code == CliExitCode.FAILURE.value
captured = capsys.readouterr()
@@ -359,7 +353,7 @@ class TestCliRunnerRunFailure:
def test_run_no_command_returns_failure(self, capsys: pytest.CaptureFixture[str]) -> None:
"""无命令时应返回 1 并打印帮助."""
runner = px.CliRunner({"clean": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph()})
exit_code = runner.run([])
assert exit_code == CliExitCode.FAILURE.value
captured = capsys.readouterr()
@@ -367,13 +361,13 @@ class TestCliRunnerRunFailure:
def test_run_failing_task_returns_failure(self) -> None:
"""任务失败时应返回 1."""
runner = px.CliRunner({"fail": _failing_graph()})
runner = px.CliRunner(aliases={"fail": _failing_graph()})
exit_code = runner.run(["fail"])
assert exit_code == CliExitCode.FAILURE.value
def test_run_failing_task_prints_error(self, capsys: pytest.CaptureFixture[str]) -> None:
"""任务失败时应打印错误信息."""
runner = px.CliRunner({"fail": _failing_graph()})
runner = px.CliRunner(aliases={"fail": _failing_graph()})
_ = runner.run(["fail"])
captured = capsys.readouterr()
# PyFlowXError 信息应输出到 stderr
@@ -388,14 +382,14 @@ class TestCliRunnerList:
def test_list_returns_success(self) -> None:
"""--list 应返回 0."""
runner = px.CliRunner({"clean": _echo_graph(), "build": _echo_graph()})
runner = px.CliRunner(aliases={"clean": _echo_graph(), "build": _echo_graph()})
exit_code = runner.run(["--list"])
assert exit_code == CliExitCode.SUCCESS.value
def test_list_prints_all_commands(self, capsys: pytest.CaptureFixture[str]) -> None:
"""--list 应打印所有命令."""
runner = px.CliRunner(
{
aliases={
"clean": _echo_graph("c", "clean"),
"build": _echo_graph("b", "build"),
"test": _echo_graph("t", "test"),
@@ -414,7 +408,7 @@ class TestCliRunnerList:
def track() -> None:
executed.append("ran")
runner = px.CliRunner({"a": px.Graph.from_specs([px.TaskSpec("a", track)])})
runner = px.CliRunner(aliases={"a": px.Graph.from_specs([px.TaskSpec("a", track)])})
_ = runner.run(["--list"])
assert executed == []
@@ -427,7 +421,7 @@ class TestCliRunnerErrorHandling:
def test_keyboard_interrupt_returns_130(self, capsys: pytest.CaptureFixture[str]) -> None:
"""KeyboardInterrupt 应返回 130."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
def raise_interrupt(*_args: Any, **_kwargs: Any) -> None:
raise KeyboardInterrupt
@@ -440,7 +434,7 @@ class TestCliRunnerErrorHandling:
def test_pyflowx_error_returns_failure(self, capsys: pytest.CaptureFixture[str]) -> None:
"""PyFlowXError 应返回 1."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
def raise_error(*_args: Any, **_kwargs: Any) -> None:
raise TaskFailedError("echo", RuntimeError("boom"), 1)
@@ -457,7 +451,7 @@ class TestCliRunnerErrorHandling:
class CustomError(Exception):
pass
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
def raise_custom(*_args: Any, **_kwargs: Any) -> None:
raise CustomError("unexpected")
@@ -474,14 +468,14 @@ class TestCliRunnerRunCli:
def test_run_cli_calls_sys_exit(self) -> None:
"""run_cli 应调用 sys.exit."""
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
with pytest.raises(SystemExit) as exc_info:
runner.run_cli(["echo"])
assert exc_info.value.code == CliExitCode.SUCCESS.value
def test_run_cli_exit_code_on_failure(self) -> None:
"""run_cli 失败时应以非零码退出."""
runner = px.CliRunner({"fail": _failing_graph()})
runner = px.CliRunner(aliases={"fail": _failing_graph()})
with pytest.raises(SystemExit) as exc_info:
runner.run_cli(["fail"])
assert exc_info.value.code == CliExitCode.FAILURE.value
@@ -489,7 +483,7 @@ class TestCliRunnerRunCli:
def test_run_cli_no_args_uses_sys_argv(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""run_cli 无参数时应使用 sys.argv."""
monkeypatch.setattr(sys, "argv", ["pymake", "echo"])
runner = px.CliRunner({"echo": _echo_graph()})
runner = px.CliRunner(aliases={"echo": _echo_graph()})
with pytest.raises(SystemExit) as exc_info:
runner.run_cli()
assert exc_info.value.code == CliExitCode.SUCCESS.value
@@ -523,31 +517,27 @@ class TestCliRunnerIntegration:
def test_condition_skipped_command_succeeds(self) -> None:
"""条件不满足时任务跳过, 整体仍成功."""
graph = px.Graph.from_specs(
[
px.TaskSpec(
"skip_me",
cmd=[*ECHO_CMD, "should not run"],
conditions=(lambda: False,),
),
]
)
runner = px.CliRunner({"skip": graph})
graph = px.Graph.from_specs([
px.TaskSpec(
"skip_me",
cmd=[*ECHO_CMD, "should not run"],
conditions=(lambda _ctx: False,),
),
])
runner = px.CliRunner(aliases={"skip": graph})
exit_code = runner.run(["skip"])
assert exit_code == CliExitCode.SUCCESS.value
def test_condition_met_command_succeeds(self) -> None:
"""条件满足时任务执行, 整体成功."""
graph = px.Graph.from_specs(
[
px.TaskSpec(
"run_me",
cmd=[*ECHO_CMD, "should run"],
conditions=(lambda: True,),
),
]
)
runner = px.CliRunner({"run": graph})
graph = px.Graph.from_specs([
px.TaskSpec(
"run_me",
cmd=[*ECHO_CMD, "should run"],
conditions=(lambda _ctx: True,),
),
])
runner = px.CliRunner(aliases={"run": graph})
exit_code = runner.run(["run"])
assert exit_code == CliExitCode.SUCCESS.value
@@ -562,15 +552,13 @@ class TestCliRunnerIntegration:
return fn
graph = px.Graph.from_specs(
[
px.TaskSpec("a", make("a")),
px.TaskSpec("b", make("b"), depends_on=("a",)),
px.TaskSpec("c", make("c"), depends_on=("a",)),
px.TaskSpec("d", make("d"), depends_on=("b", "c")),
]
)
runner = px.CliRunner({"diamond": graph})
graph = px.Graph.from_specs([
px.TaskSpec("a", make("a")),
px.TaskSpec("b", make("b"), depends_on=("a",)),
px.TaskSpec("c", make("c"), depends_on=("a",)),
px.TaskSpec("d", make("d"), depends_on=("b", "c")),
])
runner = px.CliRunner(aliases={"diamond": graph})
exit_code = runner.run(["diamond"])
assert exit_code == CliExitCode.SUCCESS.value
assert order == ["a", "b", "c", "d"]
@@ -578,7 +566,7 @@ class TestCliRunnerIntegration:
def test_mixed_fn_and_cmd_commands(self) -> None:
"""混合 fn 和 cmd 的命令应都能执行."""
runner = px.CliRunner(
{
aliases={
"fn_cmd": px.Graph.from_specs([px.TaskSpec("fn", fn=lambda: "fn-result")]),
"cmd_cmd": px.Graph.from_specs([px.TaskSpec("cmd", cmd=[*ECHO_CMD, "cmd-result"])]),
}
@@ -598,7 +586,7 @@ class TestCliRunnerIntegration:
ls_cmd = ["ls"]
graph = px.Graph.from_specs([px.TaskSpec("ls", cmd=ls_cmd, cwd=Path(tmpdir))])
runner = px.CliRunner({"ls": graph})
runner = px.CliRunner(aliases={"ls": graph})
exit_code = runner.run(["ls"])
assert exit_code == CliExitCode.SUCCESS.value
@@ -630,3 +618,109 @@ class TestApplyVerboseToGraph:
new_graph = _apply_verbose_to_graph(graph, verbose=True)
new_spec = new_graph.spec("a")
assert new_spec.verbose is True
# ---------------------------------------------------------------------- #
# 新 API: tasks + aliases
# ---------------------------------------------------------------------- #
class TestCliRunnerNewApi:
"""测试 CliRunner 的 tasks + aliases 新 API."""
def test_tasks_plus_aliases_single_str(self) -> None:
"""tasks 注册 + aliases str 引用单任务."""
runner = px.CliRunner(
tasks=[px.cmd([*ECHO_CMD, "a"], name="task_a")],
aliases={"a": "task_a"},
)
assert runner.commands == ["a"]
assert runner.run(["a"]) == CliExitCode.SUCCESS.value
def test_aliases_list_str_builds_chain(self) -> None:
"""aliases list[str] 应建立 chain 依赖(后一个依赖前一个)."""
runner = px.CliRunner(
tasks=[
px.cmd([*ECHO_CMD, "a"], name="task_a"),
px.cmd([*ECHO_CMD, "b"], name="task_b"),
],
aliases={"ab": ["task_a", "task_b"]},
)
graph = runner.graphs["ab"]
specs = graph.all_specs()
assert specs["task_b"].depends_on == ("task_a",)
def test_aliases_taskspec_value(self) -> None:
"""aliases 值为 TaskSpec 时直接生成单任务图."""
spec = px.cmd([*ECHO_CMD, "x"], name="inline_x")
runner = px.CliRunner(aliases={"x": spec})
assert runner.run(["x"]) == CliExitCode.SUCCESS.value
def test_aliases_graph_value(self) -> None:
"""aliases 值为 Graph 时原样使用(复杂场景:conditions 等)."""
graph = px.Graph.from_specs([
px.TaskSpec("a", cmd=[*ECHO_CMD, "a"]),
px.TaskSpec("b", cmd=[*ECHO_CMD, "b"], depends_on=("a",)),
])
runner = px.CliRunner(aliases={"g": graph})
assert set(runner.graphs["g"].all_specs().keys()) == {"a", "b"}
def test_alias_name_same_as_task_name_via_taskspec(self) -> None:
"""alias 名与 task 名相同时,用 TaskSpec 避免自引用循环."""
spec = px.cmd([*ECHO_CMD, "same"], name="same")
runner = px.CliRunner(aliases={"same": spec})
assert runner.run(["same"]) == CliExitCode.SUCCESS.value
def test_alias_str_reference_to_other_alias(self) -> None:
"""alias 值为 str 引用其他 alias."""
runner = px.CliRunner(
aliases={
"base": px.cmd([*ECHO_CMD, "base"], name="base"),
"wrapper": "base",
},
)
assert runner.run(["wrapper"]) == CliExitCode.SUCCESS.value
def test_empty_aliases_raises(self) -> None:
"""空 aliases 应抛 ValueError."""
with pytest.raises(ValueError, match="至少需要一个别名"):
_ = px.CliRunner()
def test_empty_list_value_raises(self) -> None:
"""空 list 作为 alias 值应抛 ValueError."""
with pytest.raises(ValueError, match="任务列表为空"):
_ = px.CliRunner(aliases={"x": []})
def test_invalid_value_type_raises(self) -> None:
"""无效类型(int)作为 alias 值应抛 TypeError."""
with pytest.raises(TypeError, match="值类型无效"):
_ = px.CliRunner(aliases={"x": 123}) # type: ignore[dict-item]
def test_invalid_list_element_type_raises(self) -> None:
"""list 中非 str/TaskSpec 元素应抛 TypeError."""
with pytest.raises(TypeError, match="列表元素类型无效"):
_ = px.CliRunner(aliases={"x": [123]}) # type: ignore[list-item]
def test_duplicate_task_name_raises(self) -> None:
"""tasks 中重名任务应抛 ValueError."""
spec = px.cmd([*ECHO_CMD, "a"], name="dup")
with pytest.raises(ValueError, match="任务名重复"):
_ = px.CliRunner(tasks=[spec, spec], aliases={"a": "dup"})
def test_commands_excludes_unreferenced_tasks(self) -> None:
"""commands 只含 aliases,不含 tasks 中未引用的任务."""
runner = px.CliRunner(
tasks=[
px.cmd([*ECHO_CMD, "a"], name="used"),
px.cmd([*ECHO_CMD, "b"], name="unused"),
],
aliases={"a": "used"},
)
assert runner.commands == ["a"]
def test_unknown_command_rejected(self) -> None:
"""未注册的 alias 名应被拒绝(不接受裸 task 名)."""
runner = px.CliRunner(
tasks=[px.cmd([*ECHO_CMD, "a"], name="task_a")],
aliases={"a": "task_a"},
)
# task_a 是任务名,不是 alias,应被拒绝
assert runner.run(["task_a"]) == CliExitCode.FAILURE.value
+144
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import json
import os
import tempfile
import time
from pathlib import Path
from typing import Any
@@ -43,6 +44,46 @@ def test_memory_backend_get_missing_raises() -> None:
b.get("nope")
def test_memory_backend_ttl_expired() -> None:
"""MemoryBackend TTL 过期后 has/get 返回 False/抛 KeyError."""
b = MemoryBackend(ttl=0.1) # 0.1 秒过期
b.save("a", 1)
assert b.has("a")
time.sleep(0.15)
assert not b.has("a")
with pytest.raises(KeyError):
b.get("a")
def test_memory_backend_ttl_load_filters_expired() -> None:
"""MemoryBackend.load() 应过滤过期的条目."""
b = MemoryBackend(ttl=0.1)
b.save("a", 1)
b.save("b", 2)
time.sleep(0.15)
# a 过期,但 b 也要过期... 需要更精确控制
# 使用 monkeypatch 更可控
b._store["expired"] = ("value", time.monotonic() - 100) # 手动设置过期时间
b._store["fresh"] = ("value2", time.monotonic())
assert "expired" not in dict(b.load())
assert "fresh" in dict(b.load())
def test_memory_backend_expired_key_not_in_store() -> None:
"""不存在的键 has 返回 False."""
b = MemoryBackend(ttl=1.0)
assert b.has("nonexistent") is False
def test_memory_backend_no_ttl_never_expired() -> None:
"""无 TTL 时永不过期."""
b = MemoryBackend()
b.save("a", 1)
b._store["a"] = (1, time.monotonic() - 1000) # 手动设置很久以前的存储
assert b.has("a") # 仍然存在
assert b.get("a") == 1
# ---------------------------------------------------------------------- #
# JSONBackend
# ---------------------------------------------------------------------- #
@@ -150,6 +191,109 @@ def test_json_backend_non_dict_content_ignored(tmp_path: Path) -> None:
assert dict(b.load()) == {}
def test_json_backend_old_format_migration(tmp_path: Path) -> None:
"""旧格式JSON(纯值)应被迁移为新格式(带ts)。"""
path = tmp_path / "state.json"
# 写入旧格式:纯值
old_data = {"a": 1, "b": "value"}
_ = path.write_text(json.dumps(old_data))
b = JSONBackend(str(path))
# 读取后应有ts字段
assert "a" in b._store
assert "value" in b._store["a"]
assert "ts" in b._store["a"]
assert b._store["a"]["value"] == 1
# ---------------------------------------------------------------------- #
# JSONBackend TTL 测试
# ---------------------------------------------------------------------- #
def test_json_backend_ttl_expired_has_returns_false() -> None:
"""JSONBackend TTL 过期后 has 返回 False."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path, ttl=0.1)
b.save("a", 1)
assert b.has("a")
time.sleep(0.15)
assert not b.has("a")
def test_json_backend_ttl_expired_get_raises_keyerror() -> None:
"""JSONBackend TTL 过期后 get 抛 KeyError."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path, ttl=0.1)
b.save("a", 1)
time.sleep(0.15)
with pytest.raises(KeyError):
b.get("a")
def test_json_backend_ttl_load_filters_expired() -> None:
"""JSONBackend.load() 应过滤过期的条目."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path, ttl=0.1)
b.save("a", 1)
b.save("b", 2)
time.sleep(0.15)
# 两个都过期了
assert dict(b.load()) == {}
def test_json_backend_expired_no_ttl() -> None:
"""无 TTL 时永不过期."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path)
b.save("a", 1)
# 手动修改 ts 为很久以前
b._store["a"]["ts"] = time.time() - 1000
assert b.has("a") is True # 无 TTL,永不过期
def test_json_backend_expired_with_ttl() -> None:
"""有 TTL 时过期键 has 返回 False."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path, ttl=1.0)
b.save("a", 1)
# 手动修改 ts 为很久以前
b._store["a"]["ts"] = time.time() - 10 # 10 秒前,超过 TTL
assert b.has("a") is False
def test_json_backend_expired_missing_ts() -> None:
"""entry 缺少 ts 时视为过期."""
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path, ttl=1.0)
b._store["a"] = {"value": 1} # 缺少 ts
# ts 默认为 0,已经过了很久
assert b.has("a") is False
def test_json_backend_save_value_error(monkeypatch: pytest.MonkeyPatch) -> None:
"""save 时 json.dumps 抛 ValueError 应转为 StorageError."""
import json as _json
with tempfile.TemporaryDirectory() as tmp:
path = str(Path(tmp) / "state.json")
b = JSONBackend(path)
original_dumps = _json.dumps
def flaky_dumps(*_args: Any, **_kwargs: Any) -> str:
raise ValueError("simulated dumps failure")
monkeypatch.setattr(_json, "dumps", flaky_dumps)
with pytest.raises(StorageError, match="not JSON-serialisable"):
b.save("a", 1)
monkeypatch.setattr(_json, "dumps", original_dumps)
# ---------------------------------------------------------------------- #
# resolve_backend
# ---------------------------------------------------------------------- #
+63
View File
@@ -0,0 +1,63 @@
"""Tests for streaming result passing (iterators between tasks)."""
from __future__ import annotations
from typing import Iterator
import pyflowx as px
def test_generator_passed_as_iterator() -> None:
"""上游返回生成器,下游应能惰性消费."""
@px.task
def source() -> Iterator[int]:
yield from range(5)
@px.task(depends_on=("source",))
def consume(source: Iterator[int]) -> int:
return sum(source)
graph = px.Graph.from_specs([source, consume])
report = px.run(graph)
assert report.success
assert report["consume"] == 10
def test_large_range_streaming() -> None:
"""大范围迭代器流式传递,避免中间列表."""
@px.task
def numbers() -> Iterator[int]:
yield from range(1000)
@px.task(depends_on=("numbers",))
def total(numbers: Iterator[int]) -> int:
return sum(numbers)
graph = px.Graph.from_specs([numbers, total])
report = px.run(graph)
assert report.success
assert report["total"] == sum(range(1000))
def test_chain_multiple_streams() -> None:
"""多个流式任务串联."""
@px.task
def gen() -> Iterator[int]:
yield from range(10)
@px.task(depends_on=("gen",))
def doubled(gen: Iterator[int]) -> Iterator[int]:
for x in gen:
yield x * 2
@px.task(depends_on=("doubled",))
def collect(doubled: Iterator[int]) -> list[int]:
return list(doubled)
graph = px.Graph.from_specs([gen, doubled, collect])
report = px.run(graph)
assert report.success
assert report["collect"] == [x * 2 for x in range(10)]
+246
View File
@@ -0,0 +1,246 @@
"""Tests for tasks/system.py."""
import os
import subprocess
from pathlib import Path
import pytest
from pyflowx.conditions import Constants
from pyflowx.tasks.system import clr, reset_icon_cache, setenv, setenv_group, which, write_file
def test_clr_creates_task_spec() -> None:
"""clr() 应创建 TaskSpec。"""
spec = clr()
assert spec.name == "clear_screen"
assert spec.fn is not None
def test_clr_executes_on_linux(monkeypatch: pytest.MonkeyPatch) -> None:
"""clr() 在 Linux 上应执行 clear 命令。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
monkeypatch.setattr(Constants, "IS_LINUX", True)
# Mock subprocess.run
ran = []
monkeypatch.setattr(
subprocess,
"run",
lambda *cmd, **__: ran.append(cmd),
)
spec = clr()
assert spec.fn is not None
spec.fn()
assert ran == [(["clear"],)]
def test_clr_executes_on_windows(monkeypatch: pytest.MonkeyPatch) -> None:
"""clr() 在 Windows 上应执行 cls 命令。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
# Mock subprocess.run
ran = []
monkeypatch.setattr(
subprocess,
"run",
lambda *cmd, **__: ran.append(cmd),
)
spec = clr()
assert spec.fn is not None
spec.fn()
assert ran == [(["cls"],)]
def test_reset_icon_cache_non_windows(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
"""reset_icon_cache() 在非 Windows 上应返回空列表并打印提示。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
specs = reset_icon_cache()
assert specs == []
captured = capsys.readouterr()
assert "仅在 Windows 上支持" in captured.out
def test_reset_icon_cache_windows(monkeypatch: pytest.MonkeyPatch) -> None:
"""reset_icon_cache() 在 Windows 上应返回任务列表。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
monkeypatch.setenv("LOCALAPPDATA", "C:\\Users\\test\\AppData\\Local")
specs = reset_icon_cache()
assert len(specs) == 4
assert specs[0].name == "kill_explorer"
assert specs[1].name == "delete_icon_cache"
assert specs[2].name == "delete_icon_cache_all"
assert specs[3].name == "restart_explorer"
def test_setenv_creates_task_spec() -> None:
"""setenv() 应创建 TaskSpec。"""
spec = setenv("TEST_VAR", "test_value")
assert spec.name == "setenv_test_var"
assert spec.verbose is True
def test_setenv_sets_environment_variable(monkeypatch: pytest.MonkeyPatch) -> None:
"""setenv() 应设置环境变量。"""
spec = setenv("PYFLOWX_TEST_VAR_1", "test_value")
assert spec.fn is not None
spec.fn()
assert os.environ["PYFLOWX_TEST_VAR_1"] == "test_value"
# Clean up
del os.environ["PYFLOWX_TEST_VAR_1"]
def test_setenv_default_not_overwrite(monkeypatch: pytest.MonkeyPatch) -> None:
"""setenv(default=True) 不应覆盖已存在的环境变量。"""
os.environ["PYFLOWX_TEST_VAR_EXISTS"] = "original"
spec = setenv("PYFLOWX_TEST_VAR_EXISTS", "new_value", default=True)
assert spec.fn is not None
spec.fn()
assert os.environ["PYFLOWX_TEST_VAR_EXISTS"] == "original"
# Clean up
del os.environ["PYFLOWX_TEST_VAR_EXISTS"]
def test_setenv_default_sets_when_missing() -> None:
"""setenv(default=True) 应在缺失时设置环境变量。"""
# Ensure variable does not exist
var_name = "PYFLOWX_TEST_VAR_MISSING"
if var_name in os.environ:
del os.environ[var_name]
spec = setenv(var_name, "default_value", default=True)
assert spec.fn is not None
spec.fn()
assert os.environ[var_name] == "default_value"
# Clean up after test
del os.environ[var_name]
def test_which_creates_task_spec() -> None:
"""which() 应创建 TaskSpec。"""
spec = which("python")
assert spec.name == "which_python"
def test_which_linux_found(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
"""which() 在 Linux 上找到命令应打印路径。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
class MockResult:
returncode = 0
stdout = "/usr/bin/python\n"
monkeypatch.setattr(
subprocess,
"run",
lambda *_, **__: MockResult(),
)
spec = which("python")
assert spec.fn is not None
spec.fn()
captured = capsys.readouterr()
assert "python ->" in captured.out
assert "/usr/bin/python" in captured.out
def test_which_windows_found(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
"""which() 在 Windows 上找到命令应打印路径。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", True)
class MockResult:
returncode = 0
stdout = "C:\\Python\\python.exe\nC:\\Python\\Scripts\\python.exe\n"
monkeypatch.setattr(
subprocess,
"run",
lambda *_, **__: MockResult(),
)
spec = which("python")
assert spec.fn is not None
spec.fn()
captured = capsys.readouterr()
assert "python ->" in captured.out
assert "C:\\Python\\python.exe" in captured.out
def test_which_not_found(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
"""which() 未找到命令应打印提示。"""
monkeypatch.setattr(Constants, "IS_WINDOWS", False)
class MockResult:
returncode = 1
stdout = ""
monkeypatch.setattr(
subprocess,
"run",
lambda *_, **__: MockResult(),
)
spec = which("nonexistent_cmd")
assert spec.fn is not None
spec.fn()
captured = capsys.readouterr()
assert "nonexistent_cmd -> 未找到" in captured.out
def test_write_file_creates_task_spec() -> None:
"""write_file() 应创建带 verbose 的 TaskSpec。"""
spec = write_file("/tmp/unused", "x")
assert spec.name == "write_file_/tmp/unused"
assert spec.verbose is True
def test_write_file_writes_content(tmp_path: Path) -> None:
"""write_file() 应将内容写入指定文件."""
f = tmp_path / "out.txt"
spec = write_file(str(f), "hello world")
assert spec.fn is not None
spec.fn()
assert f.read_text(encoding="utf-8") == "hello world"
def test_write_file_with_encoding(tmp_path: Path) -> None:
"""write_file() 应支持指定编码."""
f = tmp_path / "out.txt"
spec = write_file(str(f), "中文", encoding="utf-8")
assert spec.fn is not None
spec.fn()
assert f.read_text(encoding="utf-8") == "中文"
def test_write_file_failure_propagates(tmp_path: Path) -> None:
"""write_file() 写入失败应抛出异常(不吞异常)."""
# 父目录不存在时写入应抛 FileNotFoundError
missing = tmp_path / "no_such_dir" / "out.txt"
spec = write_file(str(missing), "x")
assert spec.fn is not None
with pytest.raises(FileNotFoundError):
spec.fn()
def test_setenv_group_creates_specs() -> None:
"""setenv_group() 应为每个环境变量创建 TaskSpec."""
envs = {"VAR_A": "1", "VAR_B": "2"}
specs = setenv_group(envs)
assert len(specs) == 2
assert specs[0].name == "setenv_var_a"
assert specs[1].name == "setenv_var_b"
def test_setenv_group_default_mode(monkeypatch: pytest.MonkeyPatch) -> None:
"""setenv_group(default=True) 不应覆盖已存在的环境变量."""
monkeypatch.setenv("PYFLOWX_GROUP_EXISTS", "original")
specs = setenv_group({"PYFLOWX_GROUP_EXISTS": "new"}, default=True)
for spec in specs:
assert spec.fn is not None
spec.fn()
assert os.environ["PYFLOWX_GROUP_EXISTS"] == "original"
+345 -4
View File
@@ -2,11 +2,21 @@
from __future__ import annotations
import os
from datetime import datetime
from pathlib import Path
import pytest
from pyflowx.task import TaskResult, TaskSpec, TaskStatus
from pyflowx.task import (
RetryPolicy,
TaskResult,
TaskSpec,
TaskStatus,
_env_and_cwd,
cmd,
task_template,
)
def _fn() -> None:
@@ -18,9 +28,9 @@ def test_spec_empty_name_rejected() -> None:
TaskSpec("", _fn)
def test_spec_negative_retries_rejected() -> None:
with pytest.raises(ValueError, match="retries"):
TaskSpec("a", _fn, retries=-1)
def test_spec_negative_max_attempts_rejected() -> None:
with pytest.raises(ValueError, match="max_attempts"):
TaskSpec("a", _fn, retry=RetryPolicy(max_attempts=0))
def test_spec_zero_timeout_rejected() -> None:
@@ -28,11 +38,318 @@ def test_spec_zero_timeout_rejected() -> None:
TaskSpec("a", _fn, timeout=0)
def test_spec_negative_timeout_rejected() -> None:
"""负数timeout应被拒绝。"""
with pytest.raises(ValueError, match="timeout"):
TaskSpec("a", _fn, timeout=-1.0)
def test_spec_self_dependency_rejected() -> None:
with pytest.raises(ValueError, match="depend on itself"):
TaskSpec("a", _fn, depends_on=("a",))
def test_spec_self_soft_dependency_rejected() -> None:
"""self dependency via soft_depends_on 也应被拒绝."""
with pytest.raises(ValueError, match="depend on itself"):
TaskSpec("a", _fn, soft_depends_on=("a",))
def test_spec_overlap_depends_rejected() -> None:
"""depends_on 和 soft_depends_on 重叠应被拒绝."""
with pytest.raises(ValueError, match="不能重叠"):
TaskSpec("a", _fn, depends_on=("b",), soft_depends_on=("b",))
# ---------------------------------------------------------------------- #
# RetryPolicy 参数验证
# ---------------------------------------------------------------------- #
def test_retry_policy_negative_delay_rejected() -> None:
with pytest.raises(ValueError, match="delay must be >= 0"):
RetryPolicy(delay=-1)
def test_retry_policy_negative_backoff_rejected() -> None:
with pytest.raises(ValueError, match="backoff must be >= 0"):
RetryPolicy(backoff=-1)
def test_retry_policy_negative_jitter_rejected() -> None:
with pytest.raises(ValueError, match="jitter must be >= 0"):
RetryPolicy(jitter=-1)
# ---------------------------------------------------------------------- #
# cmd() 工厂
# ---------------------------------------------------------------------- #
def test_cmd_factory_default_name_from_two_elements() -> None:
"""cmd() 默认 name = '_'.join(command[:2])."""
spec = cmd(["uv", "build"])
assert spec.name == "uv_build"
assert spec.cmd == ["uv", "build"]
def test_cmd_factory_default_name_single_element() -> None:
"""cmd() 单元素命令 name = command[0]."""
spec = cmd(["ls"])
assert spec.name == "ls"
def test_cmd_factory_explicit_name() -> None:
"""cmd() 显式 name 覆盖默认推导."""
spec = cmd(["ruff", "check", "--fix"], name="lint")
assert spec.name == "lint"
def test_cmd_factory_passes_depends_on() -> None:
"""cmd() depends_on 透传给 TaskSpec."""
spec = cmd(["echo", "b"], name="b", depends_on=("a",))
assert spec.depends_on == ("a",)
def test_cmd_factory_passes_extra_kwargs() -> None:
"""cmd() 其余 kwargs 透传给 TaskSpec."""
spec = cmd(["echo", "x"], name="x", timeout=10.0, tags=("t1",))
assert spec.timeout == 10.0
assert spec.tags == ("t1",)
def test_retry_policy_retries_property() -> None:
policy = RetryPolicy(max_attempts=3)
assert policy.retries == 2
def test_retry_policy_should_retry_matching() -> None:
policy = RetryPolicy(max_attempts=3, retry_on=(ValueError,))
assert policy.should_retry(ValueError("x")) is True
assert policy.should_retry(RuntimeError("x")) is False
def test_retry_policy_should_retry_empty_tuple() -> None:
"""空元组等价于不重试."""
policy = RetryPolicy(max_attempts=3, retry_on=())
assert policy.should_retry(ValueError("x")) is False
def test_retry_policy_wait_seconds_zero_attempt() -> None:
"""attempt < 1 时返回 0."""
policy = RetryPolicy(delay=1.0, backoff=2.0)
assert policy.wait_seconds(0) == 0.0
assert policy.wait_seconds(-1) == 0.0
def test_retry_policy_wait_seconds_with_backoff() -> None:
"""有 backoff 时等待时间应递增."""
policy = RetryPolicy(delay=1.0, backoff=2.0)
# attempt=1: delay * backoff^0 = 1
# attempt=2: delay * backoff^1 = 2
assert policy.wait_seconds(1) == 1.0
assert policy.wait_seconds(2) == 2.0
def test_retry_policy_wait_seconds_with_jitter() -> None:
"""有 jitter 时等待时间应增加随机量."""
policy = RetryPolicy(delay=1.0, jitter=0.5)
# 多次调用验证结果在合理范围内
for _ in range(5):
wait = policy.wait_seconds(1)
assert 1.0 <= wait <= 1.5
# ---------------------------------------------------------------------- #
# should_execute 条件异常处理
# ---------------------------------------------------------------------- #
def test_should_execute_condition_exception_returns_false() -> None:
"""条件执行抛异常时应返回 False 并记录原因."""
def bad_condition(_ctx):
raise RuntimeError("condition error")
bad_condition.__name__ = ""
spec = TaskSpec("a", _fn, conditions=(bad_condition,))
should_run, reason = spec.should_execute({})
assert should_run is False
# pyrefly: ignore [not-iterable]
assert "匿名条件(执行错误)" in reason
def test_should_execute_condition_lambda_name() -> None:
"""lambda 条件有 __name__ 为 '<lambda>'."""
spec = TaskSpec("a", _fn, conditions=(lambda _ctx: False,))
should_run, reason = spec.should_execute({})
assert should_run is False
# pyrefly: ignore [not-iterable]
assert "<lambda>" in reason
def test_should_execute_skip_if_missing_cmd_not_found() -> None:
"""skip_if_missing 且命令不存在时应跳过."""
spec = TaskSpec("a", cmd=["nonexistent_cmd_xyz"], skip_if_missing=True)
should_run, reason = spec.should_execute({})
assert should_run is False
# pyrefly: ignore [not-iterable]
assert "命令不存在" in reason
def test_should_execute_skip_if_missing_cmd_found() -> None:
"""skip_if_missing 但命令存在时应执行."""
# 使用 Python 作为已安装的命令
spec = TaskSpec("a", cmd=["echo"], skip_if_missing=True) # echo 应存在
should_run, reason = spec.should_execute({})
assert should_run is True
assert reason is None
def test_should_execute_skip_if_missing_non_list_cmd() -> None:
"""skip_if_missing 对非 list 命令不影响."""
spec = TaskSpec("a", cmd="echo hello", skip_if_missing=True)
should_run, reason = spec.should_execute({})
assert should_run is True
assert reason is None
def test_should_execute_skip_if_missing_empty_list() -> None:
"""skip_if_missing 对空列表命令返回 True."""
spec = TaskSpec("a", cmd=[], skip_if_missing=True)
# 空 list 不检查
_should_run, _reason = spec.should_execute({})
# 因为 cmd=[] 且 fn=None,这会在 __post_init__ 中抛异常
# 所以这个测试无效,我们用另一个方式测试 _is_cmd_available
def test_is_cmd_available_empty_list_returns_true() -> None:
"""_is_cmd_available 对空列表返回 True."""
spec = TaskSpec("a", cmd=[], fn=_fn) # 提供 fn 避免 __post_init__ 异常
assert spec._is_cmd_available() is True
def test_is_cmd_available_string_returns_true() -> None:
"""_is_cmd_available 对字符串命令返回 True."""
spec = TaskSpec("a", cmd="echo hello")
assert spec._is_cmd_available() is True
def test_is_cmd_available_callable_returns_true() -> None:
"""_is_cmd_available 对可调用命令返回 True."""
spec = TaskSpec("a", cmd=_fn)
assert spec._is_cmd_available() is True
# ---------------------------------------------------------------------- #
# storage_key 异常处理
# ---------------------------------------------------------------------- #
def test_storage_key_cache_key_exception_returns_name() -> None:
"""cache_key 抛预期异常(TypeError/ValueError/KeyError/AttributeError)时应返回任务名."""
def bad_cache_key(_ctx):
raise ValueError("cache key error")
spec = TaskSpec("a", _fn, cache_key=bad_cache_key)
key = spec.storage_key({})
assert key == "a"
def test_storage_key_cache_key_success() -> None:
"""cache_key 成功时应返回组合键."""
spec = TaskSpec("a", _fn, cache_key=lambda ctx: ctx.get("x", "default"))
key = spec.storage_key({"x": "value"})
assert key == "a:value"
def test_storage_key_no_cache_key() -> None:
"""无 cache_key 时返回任务名."""
spec = TaskSpec("a", _fn)
key = spec.storage_key({})
assert key == "a"
# ---------------------------------------------------------------------- #
# _env_and_cwd 上下文管理器
# ---------------------------------------------------------------------- #
def test_env_and_cwd_sets_env() -> None:
"""应临时设置环境变量。"""
var_name = "PYFLOWX_TEST_ENV_VAR_1"
with _env_and_cwd({var_name: "test_value"}, None):
assert os.environ[var_name] == "test_value"
# 退出后应恢复
assert var_name not in os.environ
def test_env_and_cwd_restores_existing_env() -> None:
"""应恢复已有的环境变量."""
os.environ["EXISTING_VAR"] = "original"
try:
with _env_and_cwd({"EXISTING_VAR": "new_value"}, None):
assert os.environ["EXISTING_VAR"] == "new_value"
# 退出后应恢复原值
assert os.environ["EXISTING_VAR"] == "original"
finally:
os.environ.pop("EXISTING_VAR", None)
def test_env_and_cwd_sets_cwd(tmp_path: Path) -> None:
"""应临时切换工作目录."""
original = Path.cwd()
with _env_and_cwd(None, tmp_path):
assert Path.cwd() == tmp_path
# 退出后应恢复
assert Path.cwd() == original
def test_env_and_cwd_no_changes() -> None:
"""无 env 和 cwd 时不应有任何变化."""
original_env = dict(os.environ)
original_cwd = Path.cwd()
with _env_and_cwd(None, None):
pass
assert dict(os.environ) == original_env
assert Path.cwd() == original_cwd
def test_spec_env_context() -> None:
"""TaskSpec.env_context 应正确工作."""
var_name = "PYFLOWX_TEST_ENV_VAR_2"
spec = TaskSpec("a", _fn, env={var_name: "value"})
with spec.env_context():
assert os.environ[var_name] == "value"
assert var_name not in os.environ
# ---------------------------------------------------------------------- #
# task_template 工厂
# ---------------------------------------------------------------------- #
def test_task_template_creates_specs() -> None:
"""task_template 应创建 TaskSpec 工厂."""
template = task_template(fn=_fn, retry=RetryPolicy(max_attempts=3))
spec = template("task1")
assert spec.name == "task1"
assert spec.retry.max_attempts == 3
def test_task_template_with_cmd() -> None:
"""task_template 可以使用 cmd."""
template = task_template(cmd=["echo", "hello"])
spec = template("task1")
assert spec.name == "task1"
assert spec.cmd == ["echo", "hello"]
def test_task_template_overrides() -> None:
"""task_template 工厂可以覆盖默认值."""
template = task_template(fn=_fn, timeout=10.0)
spec = template("task1", timeout=5.0)
assert spec.timeout == 5.0
def test_task_template_factory_name() -> None:
"""工厂函数名应为 task_template_factory."""
template = task_template(fn=_fn)
assert template.__name__ == "task_template_factory"
# ---------------------------------------------------------------------- #
# TaskResult 测试
# ---------------------------------------------------------------------- #
def test_task_result_duration_none_when_not_started() -> None:
spec: TaskSpec[None] = TaskSpec("a", _fn)
result: TaskResult[None] = TaskResult(spec=spec)
@@ -61,3 +378,27 @@ def test_task_result_default_status() -> None:
assert result.value is None
assert result.error is None
assert result.attempts == 0
# ---------------------------------------------------------------------- #
# run_command callable 命令测试
# ---------------------------------------------------------------------- #
def test_run_command_callable_verbose_with_cwd(capsys: pytest.CaptureFixture[str], tmp_path: Path) -> None:
"""callable 命令 verbose 模式应打印信息."""
from pyflowx.command import run_command
spec = TaskSpec("a", cmd=lambda: "result", verbose=True, cwd=tmp_path)
result = run_command(spec)
assert result == "result"
captured = capsys.readouterr()
assert "执行可调用命令" in captured.out
assert "工作目录" in captured.out
def test_run_command_callable_exception() -> None:
"""callable 命令抛异常应转为 RuntimeError."""
from pyflowx.command import run_command
spec = TaskSpec("a", cmd=lambda: (_ for _ in ()).throw(RuntimeError("callable error")))
with pytest.raises(RuntimeError, match="可调用命令执行异常"):
run_command(spec)
+136
View File
@@ -0,0 +1,136 @@
"""Tests for the @task decorator API."""
from __future__ import annotations
from pathlib import Path
from typing import Any, Mapping
import pyflowx as px
from pyflowx.task import RetryPolicy, TaskHooks, TaskSpec
def test_task_decorator_plain() -> None:
"""@task 无参数装饰:name 取函数名,返回 TaskSpec."""
@px.task
def extract() -> list[int]:
return [1, 2, 3]
assert isinstance(extract, TaskSpec)
assert extract.name == "extract"
assert extract.fn is not None
assert extract.depends_on == ()
def test_task_decorator_with_params() -> None:
"""@task(...) 带参数装饰:传递依赖与重试."""
@px.task(depends_on=("extract",), retry=RetryPolicy(max_attempts=3))
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
assert isinstance(double, TaskSpec)
assert double.name == "double"
assert double.depends_on == ("extract",)
assert double.retry.max_attempts == 3
def test_task_decorator_explicit_name() -> None:
"""@task(name=...) 应使用显式名称而非函数名."""
@px.task(name="custom_name")
def my_func() -> None:
return None
assert my_func.name == "custom_name"
def test_task_decorator_cmd_form() -> None:
"""@task(cmd=...) 应支持命令形式."""
spec = px.task(cmd=["ls", "-la"], name="list_files")
assert isinstance(spec, TaskSpec)
assert spec.name == "list_files"
assert spec.cmd == ["ls", "-la"]
def test_task_decorator_full_options() -> None:
"""@task 应支持全部 TaskSpec 字段."""
@px.task(
depends_on=("a",),
soft_depends_on=("b",),
defaults={"b": 0},
args=(1,),
kwargs={"x": 2},
retry=RetryPolicy(max_attempts=5),
timeout=10.0,
tags=("t1",),
conditions=(px.BuiltinConditions.IS_WINDOWS,), # type: ignore[arg-type]
cwd="/tmp",
env={"K": "v"},
verbose=True,
skip_if_missing=True,
allow_upstream_skip=True,
strategy="thread",
priority=3,
concurrency_key="db",
continue_on_error=True,
)
def f(a: int) -> int:
return a
assert f.depends_on == ("a",)
assert f.soft_depends_on == ("b",)
assert f.defaults == {"b": 0}
assert f.args == (1,)
assert f.kwargs == {"x": 2}
assert f.retry.max_attempts == 5
assert f.timeout == 10.0
assert f.tags == ("t1",)
assert len(f.conditions) == 1
assert isinstance(f.cwd, Path)
assert f.cwd == Path("/tmp")
assert f.env == {"K": "v"}
assert f.verbose is True
assert f.skip_if_missing is True
assert f.allow_upstream_skip is True
assert f.strategy == "thread"
assert f.priority == 3
assert f.concurrency_key == "db"
assert f.continue_on_error is True
def test_task_decorator_runs_in_graph() -> None:
"""装饰器生成的 TaskSpec 应能直接构建图并运行."""
@px.task
def extract() -> list[int]:
return [1, 2, 3]
@px.task(depends_on=("extract",))
def double(extract: list[int]) -> list[int]:
return [x * 2 for x in extract]
graph = px.Graph.from_specs([extract, double])
report = px.run(graph)
assert report.success
assert report["double"] == [2, 4, 6]
def test_task_decorator_hooks_passthrough() -> None:
"""@task(hooks=...) 应传递 TaskHooks 实例."""
hooks = TaskHooks(pre_run=lambda _spec: None)
spec = px.task(fn=lambda: None, hooks=hooks, name="h")
assert spec.hooks is hooks
def test_task_decorator_cache_key_passthrough() -> None:
"""@task(cache_key=...) 应传递缓存键函数."""
def ck(ctx: Mapping[str, Any]) -> str:
return "k"
spec = px.task(fn=lambda: None, cache_key=ck, name="c")
assert spec.cache_key is ck
+33 -27
View File
@@ -67,7 +67,9 @@ def test_taskspec_wrap_cmd_verbose():
def test_taskspec_wrap_cmd_error():
"""Test TaskSpec._wrap_cmd handles command error."""
spec = TaskSpec("test", cmd=["python", "-c", "import sys; sys.exit(1)"])
import sys
spec = TaskSpec("test", cmd=[sys.executable, "-c", "import sys; sys.exit(1)"])
wrapped_fn = spec.effective_fn
with pytest.raises(RuntimeError, match="命令执行失败"):
@@ -105,10 +107,10 @@ def test_taskspec_conditions_check():
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: True,),
conditions=(lambda _ctx: True,),
)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
def test_taskspec_conditions_false():
@@ -116,10 +118,10 @@ def test_taskspec_conditions_false():
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
)
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
def test_taskspec_conditions_multiple():
@@ -127,10 +129,10 @@ def test_taskspec_conditions_multiple():
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: True, lambda: True, lambda: True),
conditions=(lambda _ctx: True, lambda _ctx: True, lambda _ctx: True),
)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
def test_taskspec_conditions_multiple_one_false():
@@ -138,10 +140,10 @@ def test_taskspec_conditions_multiple_one_false():
spec = px.TaskSpec(
"test",
fn=lambda: "result",
conditions=(lambda: True, lambda: False, lambda: True),
conditions=(lambda _ctx: True, lambda _ctx: False, lambda _ctx: True),
)
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
def test_taskspec_list_cmd_timeout_mocked():
@@ -177,7 +179,7 @@ def test_taskspec_shell_cmd_file_not_found_mocked():
_ = wrapped_fn()
def test_taskspec_shell_cmd_with_cwd_verbose(capsys):
def test_taskspec_shell_cmd_with_cwd_verbose(capsys: pytest.CaptureFixture[str]):
"""Test TaskSpec._wrap_cmd with shell command, cwd and verbose=True."""
with tempfile.TemporaryDirectory() as tmpdir:
if sys.platform == "win32":
@@ -218,27 +220,28 @@ def test_taskspec_shell_cmd_os_error_mocked():
# ---------------------------------------------------------------------- #
def test_skip_if_missing_with_available_command():
"""skip_if_missing=True 时,命令存在应返回 True."""
# python 命令在测试环境中一定存在
spec = TaskSpec("test", cmd=["python", "--version"], skip_if_missing=True)
assert spec.should_execute() is True
import sys
spec = TaskSpec("test", cmd=[sys.executable, "--version"], skip_if_missing=True)
assert spec.should_execute({})[0] is True
def test_skip_if_missing_with_missing_command():
"""skip_if_missing=True 时,命令不存在应返回 False."""
spec = TaskSpec("test", cmd=["definitely_not_installed_app_xyz"], skip_if_missing=True)
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
def test_skip_if_missing_false_with_missing_command():
"""skip_if_missing=False 时,命令不存在也应返回 True(不检查)."""
spec = TaskSpec("test", cmd=["definitely_not_installed_app_xyz"], skip_if_missing=False)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
def test_skip_if_missing_with_shell_cmd_not_checked():
"""skip_if_missing=True 时,shell 命令(str)不检查,应返回 True."""
spec = TaskSpec("test", cmd="definitely_not_installed_app_xyz", skip_if_missing=True)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
def test_skip_if_missing_with_callable_cmd_not_checked():
@@ -248,7 +251,7 @@ def test_skip_if_missing_with_callable_cmd_not_checked():
return 0
spec = TaskSpec("test", cmd=custom_cmd, skip_if_missing=True)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
def test_skip_if_missing_with_fn_not_checked():
@@ -258,45 +261,48 @@ def test_skip_if_missing_with_fn_not_checked():
return 0
spec = TaskSpec("test", fn=my_fn, skip_if_missing=True)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
@pytest.mark.slow
def test_skip_if_missing_with_empty_cmd_list():
"""skip_if_missing=True 时,空命令列表应返回 True(不检查)."""
spec = TaskSpec("test", cmd=[""], skip_if_missing=True)
# 空字符串命令,shutil.which 返回 None
# 但 cmd[0] 是空字符串,shutil.which("") 返回 None
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
def test_skip_if_missing_combined_with_conditions():
"""skip_if_missing=True 与 conditions 组合使用."""
import sys
# conditions 返回 False,应跳过
spec = TaskSpec(
"test",
cmd=["python", "--version"],
cmd=[sys.executable, "--version"],
skip_if_missing=True,
conditions=(lambda: False,),
conditions=(lambda _ctx: False,),
)
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
# conditions 返回 True,命令存在,应执行
spec = TaskSpec(
"test",
cmd=["python", "--version"],
cmd=[sys.executable, "--version"],
skip_if_missing=True,
conditions=(lambda: True,),
conditions=(lambda _ctx: True,),
)
assert spec.should_execute() is True
assert spec.should_execute({})[0] is True
# conditions 返回 True,命令不存在,应跳过
spec = TaskSpec(
"test",
cmd=["definitely_not_installed_app_xyz"],
skip_if_missing=True,
conditions=(lambda: True,),
conditions=(lambda _ctx: True,),
)
assert spec.should_execute() is False
assert spec.should_execute({})[0] is False
def test_skip_if_missing_skips_task_in_run():
+153 -181
View File
@@ -8,10 +8,8 @@ import pytest
import pyflowx as px
from pyflowx.conditions import (
IS_LINUX,
IS_MACOS,
IS_WINDOWS,
BuiltinConditions,
Constants,
)
# 跨平台的 echo 命令
@@ -23,11 +21,9 @@ else:
def test_taskspec_with_cmd_list():
"""测试使用命令列表的 TaskSpec."""
graph = px.Graph.from_specs(
[
px.TaskSpec("echo_test", cmd=[*ECHO_CMD, "hello"]),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("echo_test", cmd=[*ECHO_CMD, "hello"]),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -42,11 +38,9 @@ def test_taskspec_with_cmd_string():
else:
shell_cmd = "echo 'hello from shell'"
graph = px.Graph.from_specs(
[
px.TaskSpec("shell_test", cmd=shell_cmd),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("shell_test", cmd=shell_cmd),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -58,18 +52,16 @@ def test_taskspec_with_conditions_skip():
"""测试条件不满足时任务被跳过."""
# 创建一个永远不会满足的条件
def never_true():
def never_true(_ctx):
return False
graph = px.Graph.from_specs(
[
px.TaskSpec(
"should_skip",
cmd=[*ECHO_CMD, "this should not run"],
conditions=(never_true,),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"should_skip",
cmd=[*ECHO_CMD, "this should not run"],
conditions=(never_true,),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -81,18 +73,16 @@ def test_taskspec_with_conditions_execute():
"""测试条件满足时任务正常执行."""
# 创建一个总是满足的条件
def always_true():
def always_true(_ctx):
return True
graph = px.Graph.from_specs(
[
px.TaskSpec(
"should_run",
cmd=[*ECHO_CMD, "this should run"],
conditions=(always_true,),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"should_run",
cmd=[*ECHO_CMD, "this should run"],
conditions=(always_true,),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -109,25 +99,23 @@ def test_platform_conditions():
win_cmd = ["echo", "Windows"]
posix_cmd = ["echo", "POSIX"]
graph = px.Graph.from_specs(
[
px.TaskSpec(
"win_task",
cmd=win_cmd,
conditions=(IS_WINDOWS,),
),
px.TaskSpec(
"linux_task",
cmd=posix_cmd,
conditions=(IS_LINUX,),
),
px.TaskSpec(
"macos_task",
cmd=posix_cmd,
conditions=(IS_MACOS,),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"win_task",
cmd=win_cmd,
conditions=(lambda _ctx: Constants.IS_WINDOWS,),
),
px.TaskSpec(
"linux_task",
cmd=posix_cmd,
conditions=(lambda _ctx: Constants.IS_LINUX,),
),
px.TaskSpec(
"macos_task",
cmd=posix_cmd,
conditions=(lambda _ctx: Constants.IS_MACOS,),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -149,21 +137,17 @@ def test_platform_conditions():
def test_app_installed_conditions():
"""测试应用安装条件."""
# 测试 python 应该总是安装的
if sys.platform == "win32":
python_cmd = ["python", "--version"]
else:
python_cmd = ["python3", "--version"]
# 使用 sys.executable 保证可移植
python_cmd = [sys.executable, "--version"]
py_name = "python" if sys.platform == "win32" else "python3"
graph = px.Graph.from_specs(
[
px.TaskSpec(
"python_check",
cmd=python_cmd,
conditions=(BuiltinConditions.HAS_INSTALLED("python"),),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"python_check",
cmd=python_cmd,
conditions=(BuiltinConditions.HAS_INSTALLED(py_name),),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -176,38 +160,36 @@ def test_combined_conditions():
"""测试组合条件."""
# AND 条件
and_condition = BuiltinConditions.AND(
lambda: True,
lambda: True,
lambda _ctx: True,
lambda _ctx: True,
)
# OR 条件
or_condition = BuiltinConditions.OR(
lambda: True,
lambda: False,
lambda _ctx: True,
lambda _ctx: False,
)
# NOT 条件
not_condition = BuiltinConditions.NOT(lambda: False)
not_condition = BuiltinConditions.NOT(lambda _ctx: False)
graph = px.Graph.from_specs(
[
px.TaskSpec(
"and_test",
cmd=[*ECHO_CMD, "AND"],
conditions=(and_condition,),
),
px.TaskSpec(
"or_test",
cmd=[*ECHO_CMD, "OR"],
conditions=(or_condition,),
),
px.TaskSpec(
"not_test",
cmd=[*ECHO_CMD, "NOT"],
conditions=(not_condition,),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"and_test",
cmd=[*ECHO_CMD, "AND"],
conditions=(and_condition,),
),
px.TaskSpec(
"or_test",
cmd=[*ECHO_CMD, "OR"],
conditions=(or_condition,),
),
px.TaskSpec(
"not_test",
cmd=[*ECHO_CMD, "NOT"],
conditions=(not_condition,),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -223,15 +205,13 @@ def test_taskspec_with_cwd():
else:
ls_cmd = ["ls", "-la"]
graph = px.Graph.from_specs(
[
px.TaskSpec(
"list_current",
cmd=ls_cmd,
cwd=Path.cwd(),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"list_current",
cmd=ls_cmd,
cwd=Path.cwd(),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -242,16 +222,14 @@ def test_taskspec_with_cwd():
@pytest.mark.slow
def test_taskspec_with_timeout():
"""测试超时设置."""
graph = px.Graph.from_specs(
[
# 短时间任务应该成功
px.TaskSpec(
"short_task",
cmd=["python", "-c", "import time; time.sleep(0.1)"],
timeout=1.0,
),
]
)
graph = px.Graph.from_specs([
# 短时间任务应该成功
px.TaskSpec(
"short_task",
cmd=[sys.executable, "-c", "import time; time.sleep(0.1)"],
timeout=1.0,
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -261,26 +239,24 @@ def test_taskspec_with_timeout():
def test_taskspec_dependency_with_conditions():
"""测试依赖和条件的组合."""
graph = px.Graph.from_specs(
[
px.TaskSpec(
"first",
cmd=[*ECHO_CMD, "first"],
conditions=(lambda: True,),
),
px.TaskSpec(
"second",
cmd=[*ECHO_CMD, "second"],
depends_on=("first",),
conditions=(lambda: True,),
),
px.TaskSpec(
"third",
cmd=[*ECHO_CMD, "third"],
depends_on=("second",),
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"first",
cmd=[*ECHO_CMD, "first"],
conditions=(lambda _ctx: True,),
),
px.TaskSpec(
"second",
cmd=[*ECHO_CMD, "second"],
depends_on=("first",),
conditions=(lambda _ctx: True,),
),
px.TaskSpec(
"third",
cmd=[*ECHO_CMD, "third"],
depends_on=("second",),
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -295,12 +271,10 @@ def test_taskspec_mixed_fn_and_cmd():
def my_function():
return "result from function"
graph = px.Graph.from_specs(
[
px.TaskSpec("fn_task", fn=my_function),
px.TaskSpec("cmd_task", cmd=[*ECHO_CMD, "from command"]),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("fn_task", fn=my_function),
px.TaskSpec("cmd_task", cmd=[*ECHO_CMD, "from command"]),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -315,15 +289,13 @@ def test_taskspec_cmd_overrides_fn():
def my_function():
return "should not run"
graph = px.Graph.from_specs(
[
px.TaskSpec(
"cmd_priority",
fn=my_function,
cmd=[*ECHO_CMD, "cmd takes priority"],
),
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"cmd_priority",
fn=my_function,
cmd=[*ECHO_CMD, "cmd takes priority"],
),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -338,11 +310,9 @@ def test_taskspec_callable_cmd():
def my_callable():
return "callable result"
graph = px.Graph.from_specs(
[
px.TaskSpec("callable_cmd", cmd=my_callable),
]
)
graph = px.Graph.from_specs([
px.TaskSpec("callable_cmd", cmd=my_callable),
])
report = px.run(graph, strategy="sequential")
assert report.success
@@ -403,15 +373,13 @@ class TestTaskSpecVerbose:
"""verbose=True 时失败也应打印返回码."""
from pyflowx.errors import TaskFailedError
graph = px.Graph.from_specs(
[
px.TaskSpec(
"fail",
cmd=["python", "-c", "import sys; sys.exit(1)"],
verbose=True,
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"fail",
cmd=[sys.executable, "-c", "import sys; sys.exit(1)"],
verbose=True,
)
])
with pytest.raises(TaskFailedError):
_ = px.run(graph, strategy="sequential")
captured = capsys.readouterr()
@@ -440,18 +408,16 @@ class TestTaskSpecCmdErrors:
"""命令失败时错误信息应包含 stderr."""
from pyflowx.errors import TaskFailedError
graph = px.Graph.from_specs(
[
px.TaskSpec(
"fail",
cmd=[
"python",
"-c",
"import sys; sys.stderr.write('error-msg'); sys.exit(1)",
],
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"fail",
cmd=[
sys.executable,
"-c",
"import sys; sys.stderr.write('error-msg'); sys.exit(1)",
],
)
])
with pytest.raises(TaskFailedError) as exc_info:
_ = px.run(graph, strategy="sequential")
# 非 verbose 模式下, stderr 应包含在错误信息中
@@ -469,7 +435,9 @@ class TestTaskSpecCmdErrors:
"""shell 命令失败时应抛出 RuntimeError."""
from pyflowx.errors import TaskFailedError
graph = px.Graph.from_specs([px.TaskSpec("fail", cmd='python -c "import sys; sys.exit(1)"')])
graph = px.Graph.from_specs([
px.TaskSpec("fail", cmd=f'{sys.executable} -c "import sys; sys.exit(1)"'),
])
with pytest.raises(TaskFailedError) as exc_info:
_ = px.run(graph, strategy="sequential")
assert "Shell 命令执行失败" in str(exc_info.value.cause)
@@ -479,15 +447,13 @@ class TestTaskSpecCmdErrors:
"""命令超时应抛出 RuntimeError."""
from pyflowx.errors import TaskFailedError
graph = px.Graph.from_specs(
[
px.TaskSpec(
"slow",
cmd=["python", "-c", "import time; time.sleep(5)"],
timeout=0.1,
)
]
)
graph = px.Graph.from_specs([
px.TaskSpec(
"slow",
cmd=[sys.executable, "-c", "import time; time.sleep(5)"],
timeout=0.1,
)
])
with pytest.raises(TaskFailedError) as exc_info:
_ = px.run(graph, strategy="sequential")
assert "超时" in str(exc_info.value.cause)
@@ -497,7 +463,13 @@ class TestTaskSpecCmdErrors:
"""shell 命令超时应抛出 RuntimeError."""
from pyflowx.errors import TaskFailedError
graph = px.Graph.from_specs([px.TaskSpec("slow", cmd='python -c "import time; time.sleep(5)"', timeout=0.1)])
graph = px.Graph.from_specs([
px.TaskSpec(
"slow",
cmd=f'{sys.executable} -c "import time; time.sleep(5)"',
timeout=0.1,
),
])
with pytest.raises(TaskFailedError) as exc_info:
_ = px.run(graph, strategy="sequential")
assert "超时" in str(exc_info.value.cause)
+1 -1
View File
@@ -1,6 +1,6 @@
[tox]
isolated_build = true
envlist = py38, py39, py310, py311, py312, py313
envlist = py38, py39, py310, py311, py312, py313, py314
min_version = 4.0
requires = tox-uv
skipsdist = true
+4 -4
View File
@@ -11,7 +11,7 @@ _NODE_DONE = ...
class _NodeInfo:
__slots__: list[str]
def __init__(self, node) -> None: ...
def __init__(self, node: Any) -> None: ...
class CycleError(ValueError):
"""Subclass of ValueError raised by TopologicalSorterif cycles exist in the graph
@@ -29,8 +29,8 @@ class CycleError(ValueError):
class TopologicalSorter:
"""Provides functionality to topologically sort a graph of hashable nodes"""
def __init__(self, graph=...) -> None: ...
def add(self, node, *predecessors) -> None:
def __init__(self, graph: Any) -> None: ...
def add(self, node: Any, *predecessors: Any) -> None:
"""Add a new node and its predecessors to the graph.
Both the *node* and all elements in *predecessors* must be hashable.
@@ -86,7 +86,7 @@ class TopologicalSorter:
...
def __bool__(self) -> bool: ...
def done(self, *nodes) -> None:
def done(self, *nodes: Any) -> None:
"""Marks a set of nodes returned by "get_ready" as processed.
This method unblocks any successor of each node in *nodes* for being returned
Generated
+6035 -246
View File
File diff suppressed because it is too large Load Diff