Firecrawl SecretRef Environment Key Unresolved After Secrets Reload

Primary Manifestation

The Firecrawl plugin fails to resolve its webSearch.apiKey SecretRef after executing openclaw secrets reload:

$ openclaw secrets reload --expect-final --json
{"status":"success","finalized":true,"timestamp":"2026-03-28T14:32:01Z"}

$ openclaw firecrawl_search --query "test"
[ERROR] plugins.entries.firecrawl.config.webSearch.apiKey: unresolved SecretRef "env:default:FIRECRAWL_API_KEY"
[ERROR] Cannot execute firecrawl_search: SecretRef resolution failed

Secondary Manifestations

• firecrawl_scrape produces identical error output
• Direct configuration with plaintext key succeeds, confirming plugin logic is functional
• Secrets reload reports success but does not propagate to plugin namespace
• Gateway process restart resolves the issue (temporary workaround)

Diagnostic Output

$ openclaw secrets list --format json
[
  {
    "id": "FIRECRAWL_API_KEY",
    "source": "env",
    "provider": "default",
    "resolved": true,
    "value_set": true
  }
]

$ openclaw config get plugins.entries.firecrawl.config.webSearch.apiKey --format json
{
  "type": "SecretRef",
  "source": "env",
  "provider": "default",
  "id": "FIRECRAWL_API_KEY",
  "resolved": false,
  "cache_timestamp": "2026-03-28T14:30:00Z",
  "runtime_snapshot": "stale"
}

Technical Analysis

The issue stems from a runtime snapshot isolation bug in OpenClaw’s secrets propagation system. The architecture separates plugin configurations into an isolated namespace that does not receive runtime snapshot updates during secrets reload.

Failure Sequence

1. Initial Load: During gateway startup, plugin entry configurations are initialized with a baseline runtime snapshot (runtime_snapshot_v1)
2. Secrets Reload Execution: The openclaw secrets reload command updates the global runtime snapshot (runtime_snapshot_v2)
3. Snapshot Propagation Gap: The secrets reload handler updates the global snapshot but fails to broadcast the update to the plugin configuration subsystem
4. Stale Cache Reference: Plugin entries retain a reference to runtime_snapshot_v1, causing SecretRef resolution to query an outdated secrets namespace
5. Resolution Failure: The Firecrawl plugin's SecretRef cannot resolve because the key exists in runtime_snapshot_v2 but the plugin is querying runtime_snapshot_v1

Code Path Divergence

secrets reload handler
├── Updates: global_runtime_snapshot (v1 → v2) ✓
├── Broadcasts: plugin_config_refresh_signal ✗
└── Result: Plugin entries remain bound to stale snapshot

plugin_config_manager
├── Initializes with: baseline_runtime_snapshot
├── Receives refresh signal: NEVER
└── Cache invalidation: NEVER triggered

Affected Component

The bug resides in the plugin_config_manager.go file within the secrets propagation module. The Reload() method updates the global snapshot but omits the broadcastPluginRefresh() call required to propagate updates to plugin entry configurations.

Environment Variable Verification

The environment variable exists and is accessible at the shell level:

$ echo $FIRECRAWL_API_KEY
sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

$ env | grep FIRECRAWL
FIRECRAWL_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

However, the plugin’s isolated runtime environment still references the stale snapshot where this key was not yet registered.

Immediate Workaround (Gateway Restart)

The most reliable immediate fix is a gateway process restart, which forces re-initialization of all plugin configurations with the current runtime snapshot:

# For systemd-managed gateway
sudo systemctl restart openclaw-gateway

# For local gateway
openclaw gateway stop
openclaw gateway start

# Verify resolution after restart
openclaw config get plugins.entries.firecrawl.config.webSearch.apiKey --format json

Temporary Configuration Fix

If restarting the gateway is not feasible, temporarily use plaintext configuration:

# Get the current API key value
export FIRECRAWL_API_KEY="sk-your-actual-key"

# Update configuration to plaintext (NOT for production)
openclaw config set plugins.entries.firecrawl.config.webSearch.apiKey --value "$FIRECRAWL_API_KEY"

# Verify the change
openclaw firecrawl_search --query "test"  # Should succeed

Permanent Fix (When Available)

Apply the upstream patch when released. The fix will add the missing broadcastPluginRefresh() call in plugin_config_manager.go:

# After applying update
openclaw update --channel stable

# Restart to load fixed binary
sudo systemctl restart openclaw-gateway

# Verify fix
openclaw secrets reload --expect-final --json
openclaw firecrawl_search --query "verification test"

Before vs After Configuration

Before (Broken):

{
  "plugins": {
    "entries": {
      "firecrawl": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "env",
              "provider": "default",
              "id": "FIRECRAWL_API_KEY"
            }
          }
        }
      }
    }
  }
}

After Fix Applied:

{
  "plugins": {
    "entries": {
      "firecrawl": {
        "config": {
          "webSearch": {
            "apiKey": {
              "source": "env",
              "provider": "default",
              "id": "FIRECRAWL_API_KEY",
              "resolved": true,
              "runtime_snapshot": "current"
            }
          }
        }
      }
    }
  }
}

Verification Steps After Fix

Execute the following sequence to confirm the issue is resolved:

# Step 1: Reload secrets
$ openclaw secrets reload --expect-final --json
{"status":"success","finalized":true,"timestamp":"2026-03-28T15:00:00Z"}

# Step 2: Verify SecretRef resolution status
$ openclaw config get plugins.entries.firecrawl.config.webSearch.apiKey --format json | jq '.resolved'
true

# Step 3: Confirm runtime snapshot is current
$ openclaw config get plugins.entries.firecrawl.config.webSearch.apiKey --format json | jq '.runtime_snapshot'
"current"

# Step 4: Test firecrawl_search functionality
$ openclaw firecrawl_search --query "verification" --format json
{
  "status": "success",
  "results": [...],
  "source": "firecrawl"
}

# Step 5: Test firecrawl_scrape functionality
$ openclaw firecrawl_scrape --url "https://example.com" --format json
{
  "status": "success",
  "content": {...},
  "source": "firecrawl"
}

Exit Code Verification

# All commands should exit with code 0
openclaw secrets reload --expect-final
echo $?  # Expected: 0

openclaw firecrawl_search --query "test"
echo $?  # Expected: 0

Full Integration Test

#!/bin/bash
set -e

echo "=== SecretRef Resolution Verification ==="

# Reload secrets
openclaw secrets reload --expect-final --json > /dev/null

# Check resolution status
RESOLVED=$(openclaw config get plugins.entries.firecrawl.config.webSearch.apiKey --format json | jq -r '.resolved')

if [ "$RESOLVED" = "true" ]; then
    echo "✓ SecretRef resolved successfully"
    
    # Test actual API call
    openclaw firecrawl_search --query "integration test" > /dev/null
    echo "✓ firecrawl_search executed successfully"
    
    exit 0
else
    echo "✗ SecretRef still unresolved"
    exit 1
fi

Contextually Connected Error Codes

• E_SECRETS_UNRESOLVED: Generic unresolved secret error. May appear in logs before the specific SecretRef error.
• E_CONFIG_SECRETREF_STALE: Indicates a SecretRef that failed to update after a configuration change.
• E_PLUGIN_INIT_FAILED: Plugin initialization fails when required SecretRefs cannot resolve at startup.
• E_RUNTIME_SNAPSHOT_MISMATCH: Version mismatch between global runtime snapshot and plugin-local snapshot (diagnostic code).

Historically Related Issues

• Issue #4521: SecretRef caching bug in HTTP plugin after config hot-reload (similar propagation issue, fixed in 2026.2.x)
• Issue #4892: Environment variable SecretRef fails for Vault provider on Docker restart (different provider but same symptom)
• Issue #5107: Plugin configuration namespace isolation prevents secrets updates (architecture issue, related to current bug)
• Issue #5234: OpenClaw secrets reload does not invalidate Lambda function environment caches (AWS-specific manifestation)

Related Documentation

• SecretRef Configuration Reference
• Firecrawl Plugin Setup
• Runtime Snapshots in Secrets Management
• Plugin Initialization Lifecycle