GGML Metal Assertion Crash on Apple Silicon During Shutdown with Local Embeddings

Mon, 02 Mar 2026 00:00:00 +0000

Symptom

When running OpenClaw on Apple Silicon (M-series Macs) with local embeddings configured for memory search, the application crashes during shutdown (e.g., when receiving Ctrl+C / SIGINT or during autoupdate). The crash produces a GGML Metal assertion failure visible in the logs:

/Users/runner/work/node-llama-cpp/node-llama-cpp/llama/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m:608: GGML_ASSERT([rsets->data count] == 0) failed

The full stack trace shows the crash originates from ggml_metal_device_free being called during process exit, indicating that Metal resources were not properly released before shutdown.

Additionally, the following error may appear in the logs before the crash:

Unhandled promise rejection: AssertionError [ERR_ASSERTION]: Reached illegal state! IPV4 address change from defined to undefined!

This network-related error is a secondary symptom that occurs during the same shutdown sequence.

Affected Configurations:

macOS (all versions with Apple Silicon)
Local embeddings provider enabled via node-llama-cpp
Example configuration:

“agents”: { “defaults”: { “memorySearch”: { “provider”: “local”, “local”: { “modelPath”: “/path/to/embeddinggemma-…gguf” } } } }

Root Cause Analysis

The crash is caused by a resource leak in the interaction between OpenClaw and the node-llama-cpp library when using Metal GPU acceleration on Apple Silicon.

Technical Details

Embedding Context Lifecycle: When local embeddings are used, node-llama-cpp creates embedding contexts that hold Metal GPU resources (textures, buffers, and command queues managed by GGML Metal).
Missing Disposal on Shutdown: During normal process exit (SIGINT, SIGTERM, or autoupdate restart), these embedding contexts are not explicitly disposed before the Node.js event loop terminates. This leaves Metal resources in an active state.
GGML Metal Unload Assertion: When the process exits, llama.cpp’s ggml_metal_device_free() function runs during the atexit() phase. This function asserts that all Metal resource sets (rsets->data) have been released. Since the embedding contexts were not disposed, this assertion fails: GGML_ASSERT([rsets->data count] == 0)
Regression Indicator: This issue is classified as a regression because the functionality worked in previous versions, suggesting a change in either OpenClaw’s shutdown handling, node-llama-cpp’s behavior, or a combination of both.

Call Chain

process.exit() → exit() [libsystem_c.dylib] → __cxa_finalize_ranges() → ggml_metal_device_free() [libggml-metal.so] → GGML_ASSERT([rsets->data count] == 0) // FAILS HERE

Solution

Implement a cleanup patch that tracks embedding contexts and explicitly disposes them before process exit.

Step 1: Create the Cleanup Patch File

Create a new file at src/memory/local-cleanup-patch.ts:

import { getLlama } from ’node-llama-cpp';

const trackedContexts: any[] = [];

const originalCreate = getLlama.prototype.createEmbeddingContext; getLlama.prototype.createEmbeddingContext = async function (…args: any[]) { const ctx = await originalCreate.apply(this, args); trackedContexts.push(ctx); return ctx; };

async function cleanup() { if (trackedContexts.length === 0) return; console.log([cleanup] Disposing ${trackedContexts.length} embedding context(s)); for (const ctx of trackedContexts) { if (ctx?.dispose) { await ctx.dispose().catch(e => console.warn(’[cleanup] Dispose failed:’, e)); } } trackedContexts.length = 0; }

process.once(‘SIGINT’, cleanup); process.once(‘SIGTERM’, cleanup); process.on(‘beforeExit’, cleanup);

export { cleanup };

Step 2: Modify the Import Function

Update src/memory/node-llama.ts to automatically apply the cleanup patch when local embeddings are used:

export async function importNodeLlamaCpp() { // Automatically apply our shutdown fix when local embeddings are used await import(’./local-cleanup-patch’); return import(“node-llama-cpp”); }

Step 3: Verify the Fix

After implementing the changes:

Restart the OpenClaw service
Trigger an autoupdate or send SIGINT (Ctrl+C)
Confirm the application shuts down gracefully without GGML Metal assertion failures

Prevention

To prevent this issue from recurring:

Always Test Shutdown Sequences: When integrating libraries that manage native resources (GPU, CUDA, Metal), always test graceful shutdown scenarios including SIGINT, SIGTERM, and forced exits.
Implement Resource Tracking: For any async resource allocation (embedding contexts, model instances), implement tracking mechanisms that can be flushed during shutdown.
Register Cleanup Handlers Early: Register shutdown cleanup handlers (process.once('SIGINT'), process.once('SIGTERM'), process.on('beforeExit')) as early as possible in the application lifecycle.
Use Graceful Shutdown Patterns: Implement a unified shutdown manager that coordinates cleanup across all subsystems, ensuring native resources are released before the event loop terminates.
Add Regression Tests: Consider adding automated tests that verify graceful shutdown behavior on all supported platforms, particularly Apple Silicon with Metal GPU acceleration.

Additional Information

Temporary Workarounds

If the fix cannot be applied immediately, use one of the following workarounds to disable Metal GPU acceleration:

Option 1: Disable GPU layers entirely export NODE_LLAMA_CPP_GPU_LAYERS=0

Option 2: Disable Metal specifically (version-dependent) export NODE_LLAMA_CPP_METAL=false

Note: These workarounds will reduce embedding performance but prevent the crash.

Affected Versions

OpenClaw 2026.3.1 is confirmed affected
Earlier versions may also be affected depending on the node-llama-cpp version in use

node-llama-cpp: Embedding context management
ggml-metal (llama.cpp): Metal GPU resource management
@homebridge/ciao: mDNS/network management (secondary error source during shutdown)

External References

GGML Metal device cleanup: llama.cpp PR #17869
Setting GGML_BACKTRACE_LLDB may cause native MacOS Terminal.app to crash; use with caution

Sources

GitHub Issue #32452

Shutdown on FixClaw