Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
- 🛡️ **健壮可靠** — PTY 色彩保留、原子写入、优雅信号处理、git 安全标签
- ⏱️ **防误标** — AI CLI 执行低于 10s 自动标记失败(防止空跑)
- 🕐 **防封号** — 任务间随机延时(默认 60-120s),降低被检测为机器人的风险
- 📢 **企业微信通知** — 批次完成、任务失败、中断时自动推送(可选)
- � **进程守护** — 支持 supervisor / systemd / nohup,自动检测非 TTY 环境或 `--daemon` 显式启用
- �📢 **企业微信通知** — 批次完成、任务失败、中断时自动推送(可选)

---

Expand Down
73 changes: 72 additions & 1 deletion docs/USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,11 @@ python run.py run FIX_CODE code-quality-fix --work-dir /other/repo
# Git 安全模式(执行前自动创建 git tag 作为回退点)
python run.py run FIX_CODE code-quality-fix --git-safety

# 单任务超时控制(默认 40 分钟 = 2400 秒)
python run.py run FIX_CODE code-quality-fix --timeout 3600 # 60 分钟
python run.py run FIX_CODE code-quality-fix --timeout 7200 # 2 小时(适合大型任务)
python run.py run FIX_CODE code-quality-fix --timeout 600 # 10 分钟(快速任务)

# 任务间延时控制(防止被检测为机器人)
python run.py run FIX_CODE code-quality-fix --delay 60-120 # 随机 60~120s(默认)
python run.py run FIX_CODE code-quality-fix --delay 30 # 固定 30s
Expand All @@ -174,6 +179,7 @@ python run.py run FIX_CODE code-quality-fix --wecom-webhook "https://..." # 命
python run.py run FIX_CODE code-quality-fix --verbose # 详细模式
python run.py run FIX_CODE code-quality-fix --quiet # 安静模式
python run.py run FIX_CODE code-quality-fix --no-color # 无颜色(CI 环境)
python run.py run FIX_CODE code-quality-fix --daemon # 进程守护模式(supervisor/systemd/nohup)

# 心跳间隔
python run.py run FIX_CODE code-quality-fix --heartbeat 30 # 每 30s 打印一次状态
Expand Down Expand Up @@ -309,6 +315,70 @@ python run.py run MY_PROJECT my-tasks
# → 自动从上次中断的位置继续
```

### 场景 5:进程守护 / 后台长时间运行

当需要在 supervisor、systemd 或 nohup 下运行时,使用 `--daemon` 模式:

```bash
# 显式指定 daemon 模式
python run.py run MY_PROJECT my-tasks --delay 111-229 --daemon

# 自动检测:当 stdout 不是 TTY 时自动启用 daemon 模式
# 所以在 supervisor / systemd / nohup 下不加 --daemon 也能正常工作
nohup python run.py run MY_PROJECT my-tasks --delay 111-229 > task.log 2>&1 &
```

**Supervisor 配置示例:**

```ini
[program:auto-task-runner]
command=/path/to/venv/bin/python /path/to/run.py run MY_PROJECT my-tasks --delay 111-229
directory=/path/to/auto-run-task
autostart=true
autorestart=false
stdout_logfile=/var/log/auto-task-runner.log
stderr_logfile=/var/log/auto-task-runner-err.log
environment=PYTHONUNBUFFERED=1
user=deploy
```

**systemd 配置示例:**

```ini
[Unit]
Description=Auto Task Runner
After=network.target

[Service]
Type=simple
User=deploy
WorkingDirectory=/path/to/auto-run-task
ExecStart=/path/to/venv/bin/python run.py run MY_PROJECT my-tasks --delay 111-229
Restart=no
Environment=PYTHONUNBUFFERED=1
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

**Daemon 模式下的行为变化:**

| 功能 | 正常模式 | Daemon 模式 |
|------|----------|-------------|
| Rich Live 面板 | ✅ 实时刷新 | ❌ 禁用(防止光标操纵乱码) |
| 终端标题 | ✅ 显示进度 | ❌ 禁用(ESC 序列污染日志) |
| 倒计时动画 | `\r` 覆盖刷新 | 单行输出(兼容日志管道) |
| 子进程模式 | PTY(保持色彩) | PIPE(兼容无终端环境) |
| 颜色输出 | ✅ Rich 彩色 | ❌ 自动禁用 |
| stdout 缓冲 | 系统默认 | 强制行缓冲(日志实时可见) |
| 断点续跑 | ✅ | ✅(状态持久化不受影响) |
| 企业微信通知 | ✅ | ✅(推荐配合使用) |

> 💡 **提示:** 在 daemon 模式下推荐配合 `--notify` 或 `--notify-each` 使用企业微信通知,
> 这样即使不盯着日志也能实时了解执行进度。

---

## 数据结构详解
Expand Down Expand Up @@ -433,10 +503,11 @@ python run.py run MY_PROJECT tasks
| 机制 | 说明 |
| --- | --- |
| **最短执行时间** | AI CLI 执行不足 10 秒自动标记为失败(防止空跑误标成功) |
| **单任务超时** | 默认 40 分钟,超时自动终止并标记失败`--timeout` 可调整 |
| **单任务超时** | 默认 40 分钟(2400 秒),超时自动终止并标记失败;`--timeout <秒数>` 可自定义,如 `--timeout 7200` 设为 2 小时 |
| **任务间延时** | 默认随机等待 60-120 秒,降低触发反爬/封号风险,`--delay 0` 可关闭 |
| **PTY 色彩保留** | 使用伪终端执行,AI CLI 的彩色输出原样呈现 |
| **自动降级** | PTY 不可用时自动切换 PIPE 模式 |
| **Daemon 兼容** | `--daemon` 或自动检测非 TTY 环境(supervisor/systemd/nohup),禁用交互特性、强制 PIPE 模式、行缓冲输出 |
| **日志全量捕获** | 终端实时输出的同时写入日志文件,同时生成去噪净化版 `.clean.log` |
| **心跳 & 标题** | 长时间运行时定期打印状态,终端标题显示任务进度 |
| **优雅中断** | 第一次 CTRL+C 优雅终止当前任务并保存状态,第二次强制退出 |
Expand Down
7 changes: 7 additions & 0 deletions task_runner/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,13 @@ def _add_execution_options(parser):
action="store_true",
help="Disable color output (useful for CI/piped output)",
)
output_group.add_argument(
"--daemon",
action="store_true",
help="Daemon/supervisor mode: disable interactive features (Live panel, "
"terminal title, \\r progress bars), force PIPE subprocess mode, "
"use line-buffered output. Auto-enabled when stdout is not a TTY.",
)


def _add_reset_subparser(subparsers):
Expand Down
16 changes: 16 additions & 0 deletions task_runner/commands/run_cmd.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,22 @@ def _execute(args, dry_run: bool = False) -> int:
verbose = getattr(args, "verbose", False)
quiet = getattr(args, "quiet", False)
no_color = getattr(args, "no_color", False)
daemon = getattr(args, "daemon", False)

# ── Daemon / supervisor mode ──
# Explicit --daemon flag OR auto-detect when stdout is not a TTY
# (e.g. supervisord, systemd, nohup, piped output).
import sys as _sys

from ..display import auto_detect_daemon_mode, enable_daemon_mode, is_daemon_mode

if daemon:
enable_daemon_mode()
elif not _sys.stdout.isatty():
auto_detect_daemon_mode()

if is_daemon_mode():
no_color = True # force no-color in daemon mode

if no_color:
from ..display import console as _console
Expand Down
6 changes: 6 additions & 0 deletions task_runner/display/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@
STATUS_ICONS,
STATUS_STYLES,
_format_elapsed,
auto_detect_daemon_mode,
console,
enable_daemon_mode,
format_elapsed,
is_daemon_mode,
reset_terminal_title,
set_terminal_title,
)
Expand Down Expand Up @@ -75,9 +78,12 @@
# Tracker
"ExecutionTracker",
"_format_elapsed",
"auto_detect_daemon_mode",
# Core
"console",
"enable_daemon_mode",
"format_elapsed",
"is_daemon_mode",
"reset_terminal_title",
"set_terminal_title",
"show_all_done",
Expand Down
65 changes: 63 additions & 2 deletions task_runner/display/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,61 @@
Core display components: console singleton, constants, and utility helpers.
"""

import os
import sys

from rich.console import Console

# ─── Daemon Mode Detection ───────────────────────────────────────

# Daemon mode is activated by:
# 1. Explicit ``--daemon`` CLI flag (sets _daemon_mode = True via enable_daemon_mode())
# 2. Auto-detection: stdout is NOT a TTY (e.g. supervisor, systemd, nohup)
#
# In daemon mode:
# - Rich Live panel is disabled (no cursor manipulation)
# - Terminal title escape sequences are suppressed
# - \r carriage-return progress is replaced by plain line output
# - PIPE mode is forced for subprocess execution (no PTY)
# - Output is line-buffered to prevent silent buffering

_daemon_mode: bool = False


def is_daemon_mode() -> bool:
"""Return True if running in daemon/supervisor mode."""
return _daemon_mode


def enable_daemon_mode() -> None:
"""Activate daemon mode and reconfigure console for non-interactive output."""
global _daemon_mode
_daemon_mode = True

# Reconfigure the shared console instance in-place so all modules that
# already imported ``console`` see the change. We disable colour and
# override ``is_terminal`` so Rich degrades gracefully (no cursor moves).
console.no_color = True
console._force_terminal = False # type: ignore[attr-defined]

# Ensure stdout/stderr are line-buffered so supervisor log capture is
# immediate. In a non-TTY environment Python may full-buffer stdout.
try:
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(line_buffering=True)
if hasattr(sys.stderr, "reconfigure"):
sys.stderr.reconfigure(line_buffering=True)
except Exception:
# Fallback: set PYTHONUNBUFFERED for child processes at least
os.environ.setdefault("PYTHONUNBUFFERED", "1")


def auto_detect_daemon_mode() -> None:
"""Auto-enable daemon mode when stdout is not a TTY."""
if not sys.stdout.isatty():
enable_daemon_mode()


# ─── Singleton Console ───────────────────────────────────────────

console = Console(highlight=False)
Expand Down Expand Up @@ -53,7 +104,12 @@


def set_terminal_title(text: str):
"""Set terminal window title via OSC escape sequence."""
"""Set terminal window title via OSC escape sequence.

Suppressed in daemon mode — escape sequences corrupt supervisor logs.
"""
if _daemon_mode:
return
try:
sys.stderr.write(f"\033]0;{text}\007")
sys.stderr.flush()
Expand All @@ -62,7 +118,12 @@ def set_terminal_title(text: str):


def reset_terminal_title():
"""Reset terminal title to default."""
"""Reset terminal title to default.

Suppressed in daemon mode.
"""
if _daemon_mode:
return
try:
sys.stderr.write("\033]0;\007")
sys.stderr.flush()
Expand Down
10 changes: 7 additions & 3 deletions task_runner/display/tracker.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from rich.live import Live
from rich.panel import Panel

from .core import SPINNER_FRAMES, _format_elapsed, console
from .core import SPINNER_FRAMES, _format_elapsed, console, is_daemon_mode


class _TrackerRenderable:
Expand Down Expand Up @@ -65,8 +65,12 @@ def __init__(self, total_all: int, total_to_execute: int, project: str, task_set
self._enabled: bool = True

def start(self):
"""Start the live display."""
if not self._enabled:
"""Start the live display.

Disabled in daemon mode — Rich Live uses cursor manipulation
that corrupts supervisor/pipe-captured logs.
"""
if not self._enabled or is_daemon_mode():
return
self._live = Live(
_TrackerRenderable(self),
Expand Down
62 changes: 50 additions & 12 deletions task_runner/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
SPINNER_FRAMES,
ExecutionTracker,
console,
is_daemon_mode,
reset_terminal_title,
set_terminal_title,
show_all_done,
Expand Down Expand Up @@ -585,19 +586,37 @@ def _inter_task_delay(self, current_idx: int, remaining_tasks, *, last_success:
import sys as _sys

label = f"next: {next_label}" if next_label else "next task"
for remaining in range(delay, 0, -1):
if self.interrupted:
_sys.stdout.write(f"\r \u23f3 Delay interrupted.{' ' * 50}\n")
_sys.stdout.flush()
return
daemon = is_daemon_mode()

# In daemon mode, print one line and sleep — \r carriage returns
# produce garbled output in supervisor log files.
if daemon:
_sys.stdout.write(
f"\r \u23f3 Waiting {remaining}s before {label} (anti-rate-limit)..."
f" \u23f3 Waiting {delay}s before {label} (anti-rate-limit)...\n"
)
_sys.stdout.flush()
time.sleep(1)
for remaining in range(delay, 0, -1):
if self.interrupted:
_sys.stdout.write(f" \u23f3 Delay interrupted.\n")
_sys.stdout.flush()
return
time.sleep(1)
_sys.stdout.write(f" \u23f3 Delay complete, resuming execution.\n")
_sys.stdout.flush()
else:
for remaining in range(delay, 0, -1):
if self.interrupted:
_sys.stdout.write(f"\r \u23f3 Delay interrupted.{' ' * 50}\n")
_sys.stdout.flush()
return
_sys.stdout.write(
f"\r \u23f3 Waiting {remaining}s before {label} (anti-rate-limit)..."
)
_sys.stdout.flush()
time.sleep(1)

_sys.stdout.write(f"\r \u23f3 Delay complete, resuming execution.{' ' * 40}\n")
_sys.stdout.flush()
_sys.stdout.write(f"\r \u23f3 Delay complete, resuming execution.{' ' * 40}\n")
_sys.stdout.flush()

# ─── Heartbeat ───────────────────────────────────────────────

Expand Down Expand Up @@ -696,7 +715,10 @@ def _execute_with_pty(self, cmd: str, log_path: Path) -> tuple[int, float]:
data = os.read(master_fd, 8192)
if not data:
break
os.write(sys.stdout.fileno(), data)
try:
os.write(sys.stdout.fileno(), data)
except (BrokenPipeError, OSError):
pass # stdout pipe closed (supervisor restart, etc.)
log_file.write(data)
log_file.flush()
except OSError as e:
Expand Down Expand Up @@ -752,7 +774,10 @@ def _execute_with_pipe(self, cmd: str, log_path: Path) -> tuple[int, float]:
self._timed_out = True
self._timeout_kill()
break
os.write(sys.stdout.fileno(), line)
try:
os.write(sys.stdout.fileno(), line)
except (BrokenPipeError, OSError):
pass # stdout pipe closed (supervisor restart, etc.)
log_file.write(line)
log_file.flush()

Expand All @@ -773,6 +798,16 @@ def _execute_with_pipe(self, cmd: str, log_path: Path) -> tuple[int, float]:

def execute_task(self, cmd: str, log_path: Path) -> tuple[int, float]:
self._timed_out = False

# In daemon mode, skip PTY entirely — PTY relies on a controlling
# terminal that doesn't exist under supervisord / systemd / nohup.
# PIPE mode is fully sufficient and avoids EIO / TIOCGWINSZ errors.
if is_daemon_mode():
try:
return self._execute_with_pipe(cmd, log_path)
finally:
self._ensure_child_cleaned_up()

try:
return self._execute_with_pty(cmd, log_path)
except Exception as e:
Expand All @@ -797,7 +832,10 @@ def _drain_fd(fd: int, log_file):
data = os.read(fd, 8192)
if not data:
break
os.write(sys.stdout.fileno(), data)
try:
os.write(sys.stdout.fileno(), data)
except (BrokenPipeError, OSError):
pass
log_file.write(data)
log_file.flush()
except OSError:
Expand Down
Loading