主要错误信息

当支持 auth.profiles 的提供程序发生请求超时时，嵌入式运行器会发出级联故障消息：

Profile openai-codex:default timed out (possible rate limit). Trying next account...
No available auth profile for openai-codex (all in cooldown or unavailable).
... provider=openai model=gpt-5.2 ...   # fallback triggered

可观察行为

• Profile 过早耗尽: 单个 profile 上的一次超时会导致立即轮换到下一个可用的 profile
• 冷却状态累积: 每次超时都会写入一个冷却条目，采用指数退避策略（~1分钟 → 5分钟 → 25分钟 → 1小时上限）
• 不必要的模型回退: 当所有 profile 都进入冷却状态时，系统会继续执行配置的模型回退，即使原始提供程序仍在正常运行
• 日志噪音: 重复的 `timed out (possible rate limit)` 消息会让人混淆实际速率限制状态

复现场景

bash

触发条件：单个请求超过 timeoutSeconds 阈值

openclaw run –agent ./my-agent.ts –timeout-seconds 30

观察结果：立即进行 auth profile 轮换，无重试

预期结果：在轮换前至少有一次带退避的重试

受影响的组件

架构分析

嵌入式运行器中的 auth-profile 故障转移循环混淆了两种不同的故障模式：

1. 强速率限制信号: HTTP 429、提供程序特定错误码（例如 error.code === "rate_limit_exceeded"）
2. 弱瞬态信号: 通用请求超时（网络抖动、流式响应慢、SDK 延迟峰值）

代码路径分解

文件: src/agents/pi-embedded-runner/run.ts

超时处理程序执行时没有重试门控：

typescript // Simplified flow (line numbers approximate) async function executeWithAuthProfile(provider, profile, request) { try { const result = await executeRequest(request, { timeout: timeoutMs }); return result; } catch (error) { if (isTimeout(error)) { // ❌ No retry gate - immediate failure marking markAuthProfileFailure(profile, { reason: “timeout” }); advanceAuthProfile(provider); // ← Triggers rotation throw new NoAvailableAuthProfileError(provider); }

if (isRateLimit(error)) {
  // ✓ Correct: strong signal warrants immediate cooldown
  markAuthProfileFailure(profile, { reason: "rate_limit" });
  advanceAuthProfile(provider);
  throw new NoAvailableAuthProfileError(provider);
}

} }

文件: src/agents/auth-profiles/usage.ts

冷却时间计算对所有故障原因应用相同的指数调度：

typescript function calculateAuthProfileCooldownMs(errorCount: number): number { // ~1m → 5m → 25m → 1h cap const baseMs = 60_000; const cooldown = baseMs * Math.pow(5, Math.min(errorCount - 1, 3)); return Math.min(cooldown, 3_600_000); // 1-hour cap }

// Called identically for “timeout” and “rate_limit” reasons

故障级联序列

1. 发生请求超时
2. markAuthProfileFailure(reason: “timeout”) 写入冷却条目
3. advanceAuthProfile() 轮换到下一个 profile
4. 如果所有 profile 都不可用： a. 抛出 NoAvailableAuthProfileError b. 检查 agents.defaults.model.fallbacks c. 执行到回退模型/提供程序 ← 过早！
5. 如果没有回退：请求完全失败

为什么这是不正确的

通用超时尚无法与以下情况区分：

• 临时网络延迟峰值
• 流式响应初始化慢
• SDK 连接开销
• 提供程序端临时负载

配置缺口

没有配置项来控制按原因区分的重试行为：

typescript // Current: No retrySameProfileOnTimeout config exists agents: { defaults: { timeoutSeconds: 30, modelFailover: { // Missing: retrySameProfileOnTimeout, retryBackoffMs } } }

推荐方案：添加最小重试门控

此修复在触发冷却和轮换之前，为超时失败添加按原因区分的重试门控。

步骤 1：扩展配置模式

文件: src/config/schema.ts

向模型故障转移配置添加新字段：

typescript // Before interface ModelFailoverConfig { fallbacks: string[]; }

// After interface ModelFailoverConfig { fallbacks: string[]; retrySameProfileOnTimeout: number; // Default: 1 retryBackoffMs: [number, number]; // Default: [300, 1200] ms (min, max jitter) }

步骤 2：在嵌入式运行器中实现重试门控

文件: src/agents/pi-embedded-runner/run.ts

修改超时处理以包含重试逻辑：

typescript // Before async function executeWithAuthProfile(provider, profile, request) { try { return await executeRequest(request, { timeout: timeoutMs }); } catch (error) { if (isTimeout(error)) { markAuthProfileFailure(profile, { reason: “timeout” }); advanceAuthProfile(provider); throw new NoAvailableAuthProfileError(provider); } // … rate limit handling } }

// After async function executeWithAuthProfile(provider, profile, request, options = {}) { const config = getConfig(); const { retrySameProfileOnTimeout = 1, retryBackoffMs = [300, 1200] } = config.agents?.defaults?.modelFailover ?? {};

// Track retries per-profile per-session const retryState = getOrCreateRetryState(profile.id);

try { return await executeRequest(request, { timeout: timeoutMs }); } catch (error) { if (isTimeout(error)) { const maxRetries = retrySameProfileOnTimeout; const currentRetries = retryState.consecutiveTimeouts;

  if (currentRetries < maxRetries) {
    // Retry same profile with jittered backoff
    const [minDelay, maxDelay] = retryBackoffMs;
    const delay = minDelay + Math.random() * (maxDelay - minDelay);
    
    console.log(
      `Profile ${profile.id} timed out. ` +
      `Retry ${currentRetries + 1}/${maxRetries} in ${Math.round(delay)}ms...`
    );
    
    retryState.consecutiveTimeouts++;
    await sleep(delay);
    
    // Re-execute on same profile (no cooldown written)
    return await executeWithAuthProfile(
      provider, profile, request, 
      { ...options, isRetry: true }
    );
  }
  
  // Retries exhausted: apply cooldown + rotate
  console.log(
    `Profile ${profile.id} timed out (${maxRetries} retries exhausted). ` +
    `Trying next account...`
  );
  
  markAuthProfileFailure(profile, { reason: "timeout" });
  clearRetryState(profile.id);  // Reset retry counter
  advanceAuthProfile(provider);
  throw new NoAvailableAuthProfileError(provider);
}

// Rate-limit handling unchanged (immediate cooldown)
if (isRateLimit(error)) {
  markAuthProfileFailure(profile, { reason: "rate_limit" });
  clearRetryState(profile.id);
  advanceAuthProfile(provider);
  throw new NoAvailableAuthProfileError(provider);
}

throw error;

} }

步骤 3：添加重试状态管理

文件: src/agents/auth-profiles/retry-state.ts (新文件)

typescript interface RetryState { consecutiveTimeouts: number; lastRetryTimestamp: number; }

const retryStateMap = new Map<string, RetryState>();

export function getOrCreateRetryState(profileId: string): RetryState { if (!retryStateMap.has(profileId)) { retryStateMap.set(profileId, { consecutiveTimeouts: 0, lastRetryTimestamp: 0 }); } return retryStateMap.get(profileId)!; }

export function clearRetryState(profileId: string): void { retryStateMap.delete(profileId); }

export function clearAllRetryStates(): void { retryStateMap.clear(); }

步骤 4：更新默认配置

文件: src/config/defaults.ts

typescript // Before export const defaultAgentsConfig = { defaults: { timeoutSeconds: 30, modelFailover: { fallbacks: [] } } };

// After export const defaultAgentsConfig = { defaults: { timeoutSeconds: 30, modelFailover: { fallbacks: [], retrySameProfileOnTimeout: 1, retryBackoffMs: [300, 1200] } } };

修复后的配置

json5 { “agents”: { “defaults”: { “timeoutSeconds”: 30, “modelFailover”: { “fallbacks”: [“gpt-4-turbo”, “claude-3-opus”], “retrySameProfileOnTimeout”: 1, // Retries before cooldown (0 = disabled) “retryBackoffMs”: [300, 1200] // [min, max] jittered delay in ms } } } }

可选方案：按原因区分的冷却调度

对于更精细的修复，按原因区分冷却调度：

文件: src/agents/auth-profiles/usage.ts

typescript const COOLDOWN_SCHEDULES = { timeout: { baseMs: 10_000, // 10 seconds (vs 60s for rate-limit) multiplier: 2, // 10s → 20s → 40s → 80s capMs: 300_000 // 5 minutes cap (vs 1 hour) }, rate_limit: { baseMs: 60_000, multiplier: 5, // 60s → 5m → 25m → 1h capMs: 3_600_000 // 1 hour cap } };

export function calculateAuthProfileCooldownMs( errorCount: number, reason: ’timeout’ | ‘rate_limit’ ): number { const schedule = COOLDOWN_SCHEDULES[reason]; const cooldown = schedule.baseMs * Math.pow(schedule.multiplier, Math.min(errorCount - 1, 3)); return Math.min(cooldown, schedule.capMs); }

单元测试：单个超时重试相同 profile

文件: src/agents/pi-embedded-runner/__tests__/timeout-retry.test.ts

typescript describe(‘Timeout retry behavior’, () => { const mockProfile = { id: ’test-profile’, provider: ‘openai-codex’ };

beforeEach(() => { clearAllRetryStates(); });

test(‘single timeout retries same profile without cooldown’, async () => { const executeRequest = jest.fn() .mockRejectedValueOnce(new TimeoutError()) .mockResolvedValueOnce({ data: ‘success’ });

const markAuthProfileFailure = jest.fn();
const advanceAuthProfile = jest.fn();

await executeWithAuthProfile('openai-codex', mockProfile, mockRequest, {
  executeRequest,
  markAuthProfileFailure,
  advanceAuthProfile,
  config: { retrySameProfileOnTimeout: 1, retryBackoffMs: [0, 10] }
});

// Verify retry occurred
expect(executeRequest).toHaveBeenCalledTimes(2);

// Verify NO cooldown was written
expect(markAuthProfileFailure).not.toHaveBeenCalled();

// Verify NO rotation occurred
expect(advanceAuthProfile).not.toHaveBeenCalled();

});

test(’exhausted retries triggers cooldown and rotation’, async () => { const executeRequest = jest.fn() .mockRejectedValue(new TimeoutError());

const markAuthProfileFailure = jest.fn();
const advanceAuthProfile = jest.fn();

await expect(
  executeWithAuthProfile('openai-codex', mockProfile, mockRequest, {
    executeRequest,
    markAuthProfileFailure,
    advanceAuthProfile,
    config: { retrySameProfileOnTimeout: 1, retryBackoffMs: [0, 10] }
  })
).rejects.toThrow(NoAvailableAuthProfileError);

// Verify retry exhausted
expect(executeRequest).toHaveBeenCalledTimes(2);

// Verify cooldown WAS written
expect(markAuthProfileFailure).toHaveBeenCalledWith(
  mockProfile, 
  { reason: 'timeout' }
);

// Verify rotation occurred
expect(advanceAuthProfile).toHaveBeenCalledWith('openai-codex');

});

test(‘rate-limit triggers immediate cooldown (no retry)’, async () => { const executeRequest = jest.fn().mockRejectedValue({ status: 429, code: ‘rate_limit_exceeded’ });

const markAuthProfileFailure = jest.fn();
const advanceAuthProfile = jest.fn();

await expect(
  executeWithAuthProfile('openai-codex', mockProfile, mockRequest, {
    executeRequest,
    markAuthProfileFailure,
    advanceAuthProfile,
    config: { retrySameProfileOnTimeout: 1, retryBackoffMs: [0, 10] }
  })
).rejects.toThrow(NoAvailableAuthProfileError);

// Verify NO retry for rate-limit
expect(executeRequest).toHaveBeenCalledTimes(1);
expect(markAuthProfileFailure).toHaveBeenCalledWith(
  mockProfile, 
  { reason: 'rate_limit' }
);

}); });

集成测试：多个 profile + 间歇性超时

typescript test(‘intermittent timeouts do not exhaust all profiles’, async () => { const profiles = [ { id: ‘profile-1’, provider: ‘openai-codex’ }, { id: ‘profile-2’, provider: ‘openai-codex’ }, { id: ‘profile-3’, provider: ‘openai-codex’ } ];

// Profile 1: timeout → retry → success // Profile 2: timeout → retry → timeout → cooldown // Profile 3: success const executeRequest = jest.fn() .mockImplementation(({ profile }) => { if (profile.id === ‘profile-1’) return Promise.resolve({ data: ‘ok’ }); if (profile.id === ‘profile-2’) return Promise.reject(new TimeoutError()); if (profile.id === ‘profile-3’) return Promise.resolve({ data: ‘ok’ }); });

const result = await runWithAuthProfiles(profiles, mockRequest, { executeRequest, config: { retrySameProfileOnTimeout: 1, retryBackoffMs: [0, 10] } });

// Should succeed using profile-1 or profile-3 expect(result).toBeDefined();

// profile-2 cooldown should be recorded expect(getProfileCooldown(‘profile-2’)).toBeDefined(); expect(getProfileCooldown(‘profile-3’)).toBeUndefined(); });

手动验证步骤

bash

1. 启用调试日志

export OPENCLAW_LOG_LEVEL=debug

2. 运行代理，使用已知的容易超时的场景

openclaw run –agent ./test-agent.ts –timeout-seconds 5

3. 修复后的预期日志输出：

[DEBUG] Profile openai-codex:default timed out. Retry 1/1 in 847ms…

[DEBUG] Request succeeded on retry

不是：首次超时时显示 “Trying next account…”

4. 修复后，重试耗尽时：

[INFO] Profile openai-codex:default timed out (1 retries exhausted). Trying next account…

[INFO] Cooldown applied: 10000ms for timeout reason

验证清单

上下文相关的错误码和历史问题

相关配置参数

历史背景

此问题根据配置表现不同：

• 高流量部署: 多个同时发生的超时可能快速耗尽所有 profile
• 低流量部署: 单次超时可能是唯一信号，但仍会导致回退
• 共享基础设施: 一个团队的超时会影响其他团队的 profile 可用性

相关 OpenClaw 组件参考

• src/agents/pi-embedded-runner/run.ts - 嵌入式运行器 auth 循环
• src/agents/auth-profiles/usage.ts - 冷却计算
• src/agents/auth-profiles/cooldown-store.ts - 持久化冷却状态
• src/config/schema.ts - 配置类型定义
• src/errors/auth-profile-errors.ts - 错误类定义