x_search Fails with Unresolved SecretRef on Canonical xAI Auth Path

Primary Error Manifestation

When invoking the built-in x_search tool with xAI credentials configured as an exec SecretRef at the canonical plugin-owned path, the operation fails immediately with the following error response:

{
  "status": "error",
  "tool": "x_search",
  "error": "plugins.entries.xai.config.webSearch.apiKey: unresolved SecretRef \"exec:onepassword_xai:value\". Resolve this command against an active gateway runtime snapshot before reading it."
}

CLI Reproduction Sequence

# Attempt to invoke x_search via gateway CLI
$ clawctl tools invoke x_search --input '{"query": "latest OpenClaw release notes"}'

# Expected: Successful search response
# Actual: Error with unresolved SecretRef

{
  "status": "error",
  "tool": "x_search",
  "error": "plugins.entries.xai.config.webSearch.apiKey: unresolved SecretRef \"exec:onepassword_xai:value\". Resolve this command against an active gateway runtime snapshot before reading it."
}

Configuration Context

The failure occurs with this configuration shape (verified working pre-April on same infrastructure):

{
  "tools": {
    "web": {
      "search": {
        "provider": "brave"
      },
      "x_search": {
        "enabled": true,
        "model": "grok-4-1-fast-non-reasoning",
        "maxTurns": 2,
        "timeoutSeconds": 30,
        "cacheTtlMinutes": 15
      }
    }
  },
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "exec",
              "provider": "onepassword_xai",
              "id": "value"
            }
          }
        }
      }
    }
  }
}

Environment Details

• OpenClaw Version: 2026.4.10 (commit 44e5b62)
• OS: macOS 25.3.0 (arm64)
• Install Method: global npm install / local gateway service
• Install Type: exec SecretRef via 1Password provider

Confounding Observation

The error persists even though:

• web_search is configured to use brave provider (not xAI)
• The xAI credentials exist at the canonical plugin-owned path
• Same configuration worked prior to April xAI refactors

Architectural Context: The April xAI Refactors

The issue stems from a sequence of breaking changes introduced in the April 2026 refactors:

1. Issue #59674 — Moved x_search configuration behind the xAI plugin boundary, making it a plugin-owned feature rather than a top-level tool.
2. Issue #59691 — Made x_search authentication plugin-owned, requiring credentials to flow through the xAI plugin's config path rather than direct tool config.

Root Cause: Runtime Snapshot Staleness with Exec Providers

The specific failure mechanism involves a race condition or snapshot staleness in how the gateway runtime resolves exec SecretRefs for plugin-owned authentication:

1. Snapshot Initialization: When the gateway starts, it creates a runtime snapshot containing resolved values for all configuration including SecretRefs.
2. Exec Provider Timing: Exec providers (like 1Password) require external command execution to retrieve secrets. These are resolved during snapshot creation.
3. Post-Initialization Access: The x_search tool path now accesses plugins.entries.xai.config.webSearch.apiKey during tool execution, not during initial snapshot creation.
4. Stale Reference: If the exec provider command output has changed since snapshot creation (or if the snapshot wasn't properly hydrated for the plugin-owned path), the SecretRef appears unresolved.

Code Path Analysis

The failure occurs in this execution sequence:

x_search tool invocation
  → xAI plugin-owned auth path
  → plugins.entries.xai.config.webSearch.apiKey
  → SecretRef resolution check
  → Runtime snapshot lookup for "exec:onepassword_xai:value"
  → Result: UNRESOLVED (snapshot doesn't contain active value)

Why This Is a Regression

Prior to the April refactors:

• x_search used a direct authentication path that didn't require plugin-owned credential resolution
• Exec SecretRefs were resolved at tool invocation time, not snapshot-dependent time
• The same 1Password exec command worked because the resolution path didn't traverse the plugin boundary

Post-April:

• x_search is now fully plugin-owned
• Auth resolution occurs against the gateway runtime snapshot
• Exec SecretRefs are evaluated as "unresolved" when the snapshot doesn't contain a current resolved value

Adjacent Issues

This regression connects to broader runtime snapshot / SecretRef resolution issues:

• #50161 — General unresolved SecretRef handling in gateway runtime
• #51263 — Snapshot staleness with dynamic providers
• #57272 — Exec provider caching vs. fresh resolution
• #54555 — Missing xAI API key detection (related auth surface)

Option A: Use Static SecretRef (Recommended for Production)

Replace the exec SecretRef with a static reference that doesn’t require runtime command execution:

{
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "static",
              "value": "${XAI_API_KEY}"
            }
          }
        }
      }
    }
  }
}

Then set the environment variable:

# Add to your shell profile or gateway environment
export XAI_API_KEY="xai-your-actual-api-key-here"

# Or inject at gateway startup
XAI_API_KEY="xai-your-actual-api-key-here" clawctl gateway start

Option B: Inline Static Value (Development Only)

For development/testing, use an inline static value directly in config:

{
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "inline",
              "value": "xai-your-actual-api-key-here"
            }
          }
        }
      }
    }
  }
}

Option C: Force Snapshot Refresh (Workaround)

If you must use exec providers, force a gateway snapshot refresh:

# Stop the gateway
clawctl gateway stop

# Clear the runtime snapshot cache
rm -rf ~/.config/openclaw/runtime/snapshots/
rm -rf ~/.local/share/openclaw/snapshots/

# Restart the gateway (this creates a fresh snapshot)
clawctl gateway start

# Verify snapshot is fresh
clawctl gateway status --verbose

Then test:

clawctl tools invoke x_search --input '{"query": "test"}'

Option D: Migrate to Native Secrets Provider

If using 1Password, migrate to the native OpenClaw 1Password integration that doesn’t require exec commands:

{
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "onepassword",
              "vault": "openclaw-secrets",
              "item": "xai-api-key",
              "field": "api_key"
            }
          }
        }
      }
    }
  }
}

Before vs. After Configuration

Before (Failing)

{
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "exec",
              "provider": "onepassword_xai",
              "id": "value"
            }
          }
        }
      }
    }
  }
}

After (Working)

{
  "plugins": {
    "entries": {
      "xai": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "static",
              "value": "${XAI_API_KEY}"
            }
          }
        }
      }
    }
  }
}

Step 1: Confirm Gateway is Running

$ clawctl gateway status

Gateway Status: RUNNING
Version: 2026.4.10 (44e5b62)
Runtime: Active
Snapshot: Valid (created: 2026-04-XX XX:XX:XX)

Step 2: Verify xAI Plugin Configuration

$ clawctl config get plugins.entries.xai.config.webSearch.apiKey

{
  "source": "static",
  "value": "${XAI_API_KEY}"
}

Step 3: Verify Environment Variable is Set

$ echo $XAI_API_KEY | head -c 10 && echo "..."

xai-sk-12...

Step 4: Test x_search Tool Invocation

$ clawctl tools invoke x_search --input '{"query": "OpenClaw troubleshooting guide"}'

{
  "status": "success",
  "tool": "x_search",
  "results": {
    "query": "OpenClaw troubleshooting guide",
    "results": [
      {
        "title": "OpenClaw Professional Troubleshooting Guide",
        "url": "https://docs.openclaw.io/guides/troubleshooting",
        "snippet": "..."
      }
    ],
    "model": "grok-4-1-fast-non-reasoning",
    "tokensUsed": 142
  }
}

Step 5: Verify Exit Code

$ echo $?
0

A successful x_search invocation should return exit code 0.

Step 6: Verify No SecretRef Errors in Logs

$ clawctl logs --tail 50 | grep -E "(x_search|SecretRef|xai)"

# Should show successful resolution, no unresolved errors:
[INFO] x_search: Tool invocation successful
[INFO] xai.plugin: Credential resolved successfully

Step 7: Verify Plugin Credential Resolution

$ clawctl debug resolve-secret --path plugins.entries.xai.config.webSearch.apiKey

Resolved: true
Value Preview: xai-sk-12****
Source: static (env: XAI_API_KEY)
TTL: Persistent

1. Gateway Service vs. User Shell Environment

Problem: Environment variables set in your shell aren’t available to the gateway service daemon.

# This sets the variable for YOUR shell:
export XAI_API_KEY="xai-..."

# But the GATEWAY SERVICE runs in a different context:
# It won't see this variable unless configured system-wide

Solution: Configure environment variables for the gateway service:

# For launchd (macOS)
# Edit /Library/LaunchDaemons/com.openclaw.gateway.plist
<key>EnvironmentVariables</key>
<dict>
    <key>XAI_API_KEY</key>
    <string>xai-...</string>
</dict>

# For systemd (Linux)
# Create /etc/systemd/system/openclaw-gateway.service.d/override.conf
[Service]
Environment="XAI_API_KEY=xai-..."

2. Snapshot Not Refreshed After Config Change

Problem: Changing from exec to static SecretRef doesn’t take effect until snapshot refresh.

Solution:

# Always restart the gateway after config changes
clawctl gateway restart

# Or force snapshot refresh
clawctl gateway snapshot --refresh

3. Plugin Not Enabled

Problem: The xAI plugin must be explicitly enabled for plugin-owned x_search to function.

Solution:

# Verify xAI plugin is enabled
clawctl plugins list --enabled

# If not enabled:
clawctl plugins enable xai

4. Version Mismatch

Problem: The April refactors (x_search plugin-owned) require OpenClaw 2026.4.0 or later.

Solution:

# Check current version
clawctl --version

# Update if needed
npm update -g @openclaw/cli
clawctl self-update

5. Exec Provider Timeout

Problem: If the 1Password CLI takes too long, exec SecretRefs timeout during snapshot creation.

Solution: Add timeout configuration to exec provider:

{
  "providers": {
    "exec": {
      "onepassword_xai": {
        "command": "op read op://.../api_key/value",
        "timeout": 10000
      }
    }
  }
}

6. macOS Gatekeeper/Notarization Issues

Problem: On macOS, the 1Password CLI (op) may be blocked by Gatekeeper if not properly signed.

Solution:

# Verify op binary is allowed
xattr -l /usr/local/bin/op
# Should not show com.apple.quarantine

# If quarantined, remove:
xattr -d com.apple.quarantine /usr/local/bin/op

7. Docker/Container Environment Variables

Problem: Environment variables must be explicitly passed to containers.

Solution:

# Pass env vars to Docker container
docker run -e XAI_API_KEY="xai-..." openclaw/gateway:latest

# Or use docker-compose.yml
environment:
  - XAI_API_KEY=xai-...

• UNRESOLVED_SECRET_REF — General error when a SecretRef cannot be resolved against the current runtime snapshot. Manifests as: unresolved SecretRef "exec:provider:id". See issues #50161, #51263.
• MISSING_XAI_API_KEY — Indicates xAI credentials are not present in configuration at all, regardless of format. May occur even when credentials exist but aren't at the canonical plugin-owned path. See issue #54555.
• SNAPSHOT_STALE — Runtime snapshot contains outdated values and requires refresh. Often manifests alongside SecretRef resolution failures.
• PLUGIN_NOT_INITIALIZED — xAI plugin hasn't been loaded or enabled. x_search requires the plugin to be active.
• EXEC_PROVIDER_TIMEOUT — External command for exec SecretRef exceeded timeout. The op (1Password CLI) command may be slow or blocked.
• INVALID_SECRET_REF_FORMAT — SecretRef doesn't conform to expected source:provider:id format.
• GATEWAY_NOT_RUNNING — Cannot invoke tools when gateway service is stopped. Ensure clawctl gateway status shows RUNNING.

Historical Issues

Documentation References

• OpenClaw Secrets Management
• xAI Plugin Configuration
• x_search Tool Reference
• Runtime Snapshots