BoneAI

Talk to BONELAB. BoneAI looks around, understands your request, and uses real game actions to carry it out.

Ask normal questions or give commands such as:

Spawn a Ford in front of me.

Find the nearest pistol, bring it to my right hand, and reload it.

Follow player 2.

Change me to my Morty avatar and make me stronger.

BoneAI v2.6 Compatibility Extended uses the same universal BoneAI.dll on Windows PCVR and standalone Quest/LemonLoader. The avatar catalog and all BoneAI runtime code are inside that single DLL. Codex remains the default and supports Hermes-style browser login: choose Sign In With Codex, enter the displayed one-time code in the browser, and return to BONELAB.

Start here

What you need

• BONELAB PCVR 1.744.58126
• MelonLoader 0.7.3 using the .NET 6 runtime
• BoneLib 3.2.2
• Codex CLI/App Server on the same PC, or on a securely reachable PC/server for Quest
• LabFusion 1.14.2 for multiplayer features (optional for offline play)
• SpawnLab 1.0.1 for spawning (BoneAI still loads without it, but spawn tools report that the provider is unavailable)

Only BoneAI's own DLL is included in the release. BoneLib, Fusion, SpawnLab, BONELAB, and Codex belong to their respective authors and must be installed separately.

Install BoneAI

1. Download the latest BoneAI-v*.zip from Releases.
2. Copy BoneAI.dll into BONELAB's Mods folder.
3. Use this exact same DLL for PCVR MelonLoader or Quest LemonLoader.
4. Make sure Codex is installed. You do not need to sign in before opening BONELAB.
5. Start BONELAB.
6. Open Preferences → BoneAI → Codex Sign-In → Sign In With Codex.
7. BoneAI copies the one-time code and opens the official sign-in page. Enter the code, sign in, and return to BONELAB.
8. Open Assistant, type into Prompt, and select Send.

That is the complete BoneAI install. On PCVR, BoneAI.dll directly starts the installed official Codex App Server—Python and a bridge script are not required.

Your BONELAB folder should contain:

BONELAB/
├─ Mods/
│  ├─ BoneLib.dll
│  ├─ LabFusion.dll                 (optional, for multiplayer)
│  ├─ SpawnLab.dll                  (required only for spawning)
│  └─ BoneAI.dll

AI providers

Use Preferences → BoneAI → AI Provider, choose Change Provider, edit Model if desired, and select Reconnect. Provider selection and model names are saved in UserData/MelonPreferences.cfg.

Set keys as Windows user environment variables, then restart BONELAB. BoneAI never saves or logs them. Claude uses Anthropic's native Messages/tool-use protocol. Grok, DeepSeek, OpenRouter, Ollama, and Custom use their documented OpenAI-compatible function-calling protocol. The chosen model must support tools; OpenRouter support depends on the routed model, and Ollama requires the named model to be installed locally.

Codex is the only provider that uses account login. Consumer Claude/Grok/DeepSeek logins are not reused because their official third-party API authentication uses API keys.

Codex browser sign-in

BoneAI uses Codex App Server's official chatgptDeviceCode login flow. The game receives only a verification URL and one-time code. Your password, ChatGPT session, and resulting OAuth tokens remain inside Codex App Server and are never returned to or stored by BoneAI.

1. Open Preferences → BoneAI → Codex Sign-In.
2. Choose Sign In With Codex.
3. BoneAI copies the code and opens the official Codex device sign-in page in the system browser.
4. Finish signing in and return to BONELAB. The status changes to Connected: Codex without a restart.

Use Open Sign-In Page Again if the browser was closed. Sign Out signs the connected Codex App Server out. A secure connection token entered in-game is held in memory for the current game session only and is never written to preferences or logs.

Standalone Quest setup

The mod and its game actions run on the headset under LemonLoader. Codex App Server itself is a host program and is not distributed as an Android/Quest runtime, so the headset connects to Codex App Server running on a PC or server. Browser login alone cannot replace that host process. The browser performs account authorization while the trusted backend retains the credentials.

1. Install the same universal BoneAI.dll with LemonLoader and Quest-compatible BoneLib/Fusion/SpawnLab versions matching your BONELAB patch.
2. Run Codex App Server on a PC/server reachable through a trusted HTTPS/WSS endpoint, with App Server capability-token authentication enabled.
3. In BoneAI → Codex Sign-In, set the wss:// App Server address and enter its connection token.
4. Choose Sign In With Codex. Quest opens its browser, the code is entered once, and returning to BONELAB completes the connection.

BoneAI refuses unencrypted remote ws:// by default. Allow Insecure Private LAN exists for isolated testing only and should not be used on public or shared networks. Quest never attempts to launch a local host process. The universal DLL is managed AnyCPU; because Quest game assemblies and installed mod versions must match, validate it against your headset's exact BONELAB/LemonLoader setup before public distribution.

Updating

Install new versions through Thunderstore/r2modman or download a newer GitHub release and replace BoneAI's files manually.

First conversation

BoneAI enables game actions by default. For a quick check:

1. Open Preferences → BoneAI → Assistant.
2. Confirm Connection Status says connected.
3. Enter: Tell me my current avatar and what I am holding.
4. Select Send. The prompt box clears immediately.
5. Try: Find the nearest grabbable object and bring it to my right hand.

BoneAI now has its own top-level Preferences button instead of appearing inside BoneMenu. Its interface includes Assistant, Saved Conversations, Voice AI (Beta), AI Provider, Game Permissions, and Diagnostics pages. AI requests, movement/follow state, and wake-word listening run independently of the interface and continue when Preferences is closed.

Voice AI beta

Voice is optional and off by default. It uses the game microphone through Unity, sends completed utterances to OpenAI's official transcription endpoint, and can read replies using OpenAI text-to-speech. Because Codex account authentication cannot be reused for the audio API, voice requires OPENAI_API_KEY as a Windows user environment variable even when Codex is your chat provider.

1. Set OPENAI_API_KEY, then restart BONELAB.
2. Open Preferences → BoneAI → Voice AI (Beta).
3. Enable Voice AI Beta. Enable Require Wake Word for hands-free mode.
4. Edit Wake Word (default: Hey BoneAI) or use Listen Once to accept one utterance without the wake word.
5. Optionally enable Speak AI Replies.

Wake-word detection is transcript-gated: short microphone recordings are transcribed, then BoneAI checks for the configured phrase before authorizing a command. It is not offline keyword spotting. Audio replies are AI-generated. Voice is labelled beta because microphone routing and permissions vary between Windows VR setups; a voice failure does not stop text chat or gameplay tools.

What BoneAI can do

BoneAI exposes exactly 350 structured game tools. The catalog combines the original high-level actions with component-specific world queries, SpawnLab presets, verified interaction invocations, directional physics, combat strengths, player presets, turn presets, and radius scans. Codex receives the full namespaced catalog. Providers with documented tool-count limits receive a prompt-relevant subset plus tools.search, so they can discover and call anything in the full catalog without exceeding their API limit. The DLL never treats free-form text as a completed game action.

Every action has an ID and returns success, failed, or cancelled with real result data. If an object disappears or an API cannot perform the requested interaction, BoneAI reports the failure instead of claiming success.

Avatar discovery

The catalog provider built into BoneAI.dll builds a unified avatar list from the live Marrow warehouse, installed pallet manifests, and WristHub's generated avatar index when WristHub is present. WristHub is not required. Results are deduplicated and searchable by display name or barcode.

If the list is empty, ask BoneAI to run avatar.catalog_status and avatar.refresh and verify that installed content is visible to the game.

Fusion multiplayer

BoneAI uses normal, verified BONELAB/Fusion paths. It does not label a local-only transform as synchronized and does not bypass host or object authority.

BoneAI does not accept remote AI commands. Other players, player names, chat, map text, object names, and mod descriptions are context—not trusted instructions.

Connection and privacy

On PCVR, BoneAI locates the installed Codex executable and directly starts:

codex app-server --listen ws://127.0.0.1:4500

and stops it when BONELAB exits. The default PC connection is localhost-only. Account authentication remains inside Codex. BoneAI sees device-login status but never reads, stores, or logs passwords or OAuth tokens. Remote transport bearer tokens are session-only.

BoneAI's tool surface is limited to its registered BONELAB actions. It does not provide prompts with shell access, arbitrary process execution, arbitrary filesystem access, credential access, DLL loading, or unrestricted web requests.

Troubleshooting

Connection says unavailable

• Open BoneAI → Codex Sign-In and choose Sign In With Codex.
• Confirm the Codex desktop app or CLI is installed.
• Select Reconnect in Preferences → BoneAI → Codex Sign-In.
• Search MelonLoader/Latest.log for [BoneAI].

The Codex App Server exits

Install/open Codex, then restart BONELAB. BoneAI searches the system PATH and standard Codex desktop installation folders under %LOCALAPPDATA%. Account sign-in can be completed afterward from BoneAI's Codex Sign-In page.

Spawning fails

Install SpawnLab 1.0.0, restart BONELAB, and refresh SpawnLab's catalog once. Then ask BoneAI to use spawn.refresh or spawn.list before spawning.

An object command fails

Ask BoneAI to look at the object or search nearby again. Object handles are deliberately invalidated when an object is destroyed or a level changes. Some modded doors/buttons expose no safe public interaction method; BoneAI reports that limitation and can often manipulate the object's rigidbody instead.

Fusion action appears local

Ask BoneAI for fusion.get_sync_report. Local player stat changes, UI, conversation state, and movement assistance are local by design.

Known limitations

• Voice is beta, requires an OpenAI API key, and may fail on unsupported Windows/VR microphone routes; text remains fully supported.
• Navigation is collision-unaware incremental movement, not full navmesh pathfinding.
• Climbing, crouching, and generalized vehicle steering are not automated.
• Modded interaction components vary widely; unsupported controls fail cleanly.
• A spawn callback may finish after the initial tool response; the object will appear in the next nearby query.
• The exact supported game stack matters because BONELAB uses generated IL2CPP assemblies.

Build from source

Install the .NET 6 SDK and point the build at a compatible BONELAB installation:

dotnet build BoneAI.csproj -c Release -p:BONELAB_DIR="D:\Games\BONELAB"

Output: bin/Release/net6.0/BoneAI.dll. This one managed DLL contains the catalog provider and is packaged unchanged for PCVR and Quest.

The code is separated into Codex communication, tool dispatch, game controllers, Fusion compatibility, BoneMenu UI, avatar catalogs, and infrastructure. Version-sensitive reflection is isolated in the relevant compatibility adapters.

Provider transports follow the official Codex App Server, Anthropic tool use, xAI function calling, DeepSeek function calling, OpenRouter tool calling, and Ollama tool calling documentation.

License

BoneAI source code is available under the MIT License. BONELAB, BoneLib, LabFusion, SpawnLab, and Codex are separate projects and are not redistributed here.

Changelog

3.3.0 — More voice paths

• Browser Voice Automatic mode now tries on-device recognition, the browser speech service, Groq Whisper, then Cloudflare Whisper where configured. Typed commands remain available after all voice paths fail.
• Added separate Groq and Cloudflare STT credentials in the Voice menu, stored through BoneAI's protected key store. Cloudflare also requires an Account ID. Credentials are not exposed to the browser page.
• Added bounded WAV microphone capture in the browser, server-side transcription over documented HTTPS endpoints, status/error reporting, and tests for the fallback sequence.
• Groq and Cloudflare offer limited free tiers, not unlimited or anonymous usage. The web fallbacks need user accounts/keys and send recorded audio to those providers. No autoupdater; the Quest native App Server library is unchanged.

3.2.0 — Model choice, voice fallback, and smoother game actions

• Added an in-game Codex model browser backed by the App Server's paginated model/list response, including optional older/hidden entries. A selected model and its supported default effort are saved; an explicit Codex-default option clears the override.
• Browser voice now defaults to automatic on-device-first recognition when supported, with browser-service and typed-command fallbacks. Added a direct Open On-Device Voice control. An offline STT engine is not bundled; on-device availability depends on the browser.
• Limited queued main-thread work per frame, reduced automatic nearby-world context, reused a bounded physics-overlap buffer, and avoided child-component scans for routine object descriptions. The live spawn catalog now refreshes in small batches instead of rescanning the entire warehouse during an action. Slow tools now log their execution time.
• Moved avatar manifest refresh and WAV encoding off the game thread, bounded recent-log reads, and throttled repeated movement teleports. Jump targets the verified PhysicsRig.rbFeet body instead of every rig rigidbody. Rapid gun calls and extreme physics/avatar values are bounded.
• Added model-catalog, dispatcher-budget, and browser-voice fallback tests. No autoupdater.

3.1.1 — Browser voice recovery

• Stopped the browser voice page from endlessly restarting after a fatal speech-service network error. It now reports the browser-side failure and lets the player retry deliberately.
• Added feature-detected on-device speech recognition with browser-managed language-model installation, where supported. Browsers without the experimental API keep the browser-service option.
• Added a typed-command fallback on the same localhost page so a speech outage does not block commands; the field clears after submission.
• Embedded the page as a separate, testable asset and added browser-behavior tests and safer JSON escaping for the wake word.
• No autoupdater and no change to the pinned Quest Codex native library.

3.1.0 — Built-in spawning

• Removed the SpawnLab runtime and Thunderstore dependency. BoneAI now indexes the loaded Marrow warehouse itself, including base-game and installed-mod spawnables.
• Added exact-barcode and unique-name resolution, ambiguity errors, a selectable spawn position/rotation, spawn.status for local callback results, and a manual Spawn Catalog Preferences page.
• Offline spawning uses BONELAB's AssetSpawner. Fusion sessions use the verified NetworkAssetSpawner server request, with a local permission/gamemode check; requests are blocked rather than silently spawning only locally when network synchronization is unavailable.
• Time-bounded unanswered Fusion callback registrations and kept peer visibility explicitly unconfirmed without a second-client observation.
• Added spawn-catalog matching tests. No third-party SpawnLab code or DLL is redistributed, and there is no autoupdater.
• The Android native Codex library is unchanged from v3; physical Quest and two-client Fusion testing are still required.

3.0.1 — Reliability and truthful action results

• Fixed cancellation of queued game actions so they cannot run after cancellation; actions already executing now return their real result rather than a misleading cancelled result.
• Added pending results for fire-and-forget SpawnLab, input, combat, and network requests. The assistant is instructed to verify the game world instead of claiming completion prematurely.
• Added local Fusion entity ownership checks before direct manipulation of registered network objects and made synchronization reports distinguish local acceptance from peer confirmation.
• Replaced wrong-best-match spawning with exact/unique catalog resolution and an ambiguity error when multiple spawnables match.
• Added typed argument schemas for core actions, improved tool selection, and bounded recent action history.
• Added automated dispatcher/schema smoke tests, source CI, a physical-device checklist, and release checksums. No autoupdater.
• Reuses the same pinned Android native Codex App Server library as v3.0.0. Physical Quest and two-client Fusion behavior still require user testing.

3.0.0 — Native Quest Codex

• Integrated an Android ARM64 Codex App Server from the BoneAI branch of the user's Codex fork. Quest can start it inside BONELAB and use persistent ChatGPT/Codex device-code sign-in without a PC or API key.
• Kept one managed BoneAI.dll for PCVR and Quest. The Quest package additionally contains the native .so and checksum under UserLibs; the native source remains in the pinned Codex fork.
• Restricted the embedded App Server to BoneAI's dynamic game tools and added a random per-session bearer token to its localhost WebSocket.
• Added checksum validation, native ABI version checking, private Android storage for Codex state, bounded startup readiness checks, and clean failure status.
• Reorganized the in-game BoneAI pages with clearer assistant, provider, voice, account, and runtime status controls.
• Kept OpenRouter Free, other API providers, voice beta, SpawnLab, Fusion, saved conversations, and the existing tool catalog.
• Android ARM64 build validation is available; physical Quest/LemonLoader gameplay testing remains necessary.
• No autoupdater.

2.7.0 — Free Voice

• Added OpenRouter Free, backed by the official openrouter/free router. Users bring their own OpenRouter key, while the router selects a currently available free model that supports the requested features, including tool calling when available.
• Made OpenRouter Free the default provider for new Quest installations. Existing provider selections are preserved.
• Added Free Browser Voice (Beta). BoneAI hosts a localhost-only speech page, opens it in Edge/Chrome/Quest Browser, waits for the configurable wake word, and sends recognized commands back through a random per-session capability token.
• Browser voice requires no speech API key and BoneAI imposes no usage quota. Recognition availability, privacy, networking, and limits are controlled by the selected browser and its speech service.
• Added bounded HTTP parsing, body limits, CSP, no-store/no-referrer headers, constant-time token checks, and loopback-only binding to the browser voice bridge.
• Added bounded retry with jitter for temporary provider rate-limit and availability errors, plus official OpenRouter attribution headers.
• Removed the obsolete insecure-remote-Codex preference and improved Quest Codex errors so the mod no longer tries a nonexistent local App Server.
• Revalidated the official Codex boundary: browser/device-code login still requires a running Codex App Server. OpenAI does not publish an Android App Server target, so v2.7.0 does not copy private OAuth behavior or falsely claim standalone Quest Codex login.
• Contains no updater and does not modify an existing installed DLL during release creation.

2.6.2 — Secure Persistence

• Codex account sign-in now explicitly reuses and refreshes the official Codex App Server login across game restarts; Sign Out clears that Codex-owned login.
• Provider API keys entered in-game are now saved until cleared. Windows uses current-user DPAPI; Quest uses authenticated device-bound encryption. Plaintext keys are never written to preferences or logs.
• Added an explicit Clear Current API Key action that removes the selected provider key from memory and protected storage.
• Restricted Codex App Server connections to loopback and launch an owned server on a random local port, removing remote transport-token and insecure-LAN controls.
• Restricted built-in provider keys to each provider's official HTTPS host. Custom endpoints require HTTPS except for explicit localhost development; Ollama is localhost-only.
• Validates Codex browser-login URLs, caps WebSocket/API/voice response sizes, bounds pending requests and timeouts, and cleans up cancelled/disconnected requests.
• Added automatic secret redaction to logs and log-query tool results.
• Contains no updater.

2.6.1 — Quest Standalone Fix

• Added true no-PC Quest operation through direct HTTPS provider connections from the headset.
• Added OpenAI Responses API support with structured BoneAI tool calling and gpt-5.3-codex as the default standalone OpenAI model.
• Added in-game provider API-key entry. Keys are session-only, cleared from the visible field, never written to disk, and never logged.
• Added a one-click Quest Standalone OpenAI setup option and made OpenAI direct mode the default for new Quest installations.
• Made the same in-game key path available to Claude, Grok, DeepSeek, OpenRouter, Custom, and Voice AI.
• Corrected the documentation: Codex account/browser login remains PCVR-only because the official App Server has no supported Android/Quest runtime.
• Still uses one universal BoneAI.dll, contains no Python bridge, and contains no updater.

2.6.0 — Compatibility Extended

• Replaced separate PCVR and Quest mod binaries with one AnyCPU BoneAI.dll used unchanged on both platforms.
• Folded the avatar/pallet catalog provider into BoneAI.dll; BoneAI.Catalogs.dll and the UserLibs install step are no longer required.
• Removed the Python bridge requirement. On PCVR, BoneAI directly launches the installed official Codex App Server; on Quest it connects to the configured remote App Server.
• Made installed-content discovery use Unity's platform-correct persistent data directory on both Windows and Android.
• Kept official browser/device-code Codex sign-in, all 350 tools, Fusion, SpawnLab, saved conversations, providers, performance work, and Voice AI beta.
• Contains no updater.

2.5.0

• Added official Codex device-code authentication from inside BONELAB: BoneAI copies the one-time code, opens the system browser, and detects completion automatically.
• Added a dedicated Codex Sign-In page with reconnect, sign-out, sign-in-page reopening, remote App Server address, and session-only transport-token entry.
• Added standalone Quest runtime detection. Quest never attempts to launch Python and instead connects to a securely hosted Codex App Server.
• Added authenticated remote App Server support using bearer transport tokens and secure wss:// enforcement by default.
• Changed the mod assembly to AnyCPU for PCVR/Quest managed-runtime compatibility; no auto-updater is included.
• Retains the complete v2.4 performance, 350-tool, Fusion, SpawnLab, saved-conversation, provider, and Voice AI beta feature set.

2.4.0

• Removed the updater and all updater packaging; BoneAI updates are installed manually or through Thunderstore/r2modman.
• Reworked nearby-world queries with a short-lived, movement-aware cache, squared-distance filtering, and scene-change invalidation.
• Moved stale object-handle pruning off the per-frame path and removed its normal-case temporary allocations.
• Reduced repeated Unity/IL2CPP component traversal by describing NPCs, interactables, and component types from one cached scan per object.
• Cached Fusion player snapshots and SpawnLab catalog reflection results to avoid repeating expensive reflection work during multi-step actions.
• Precomputed tool-search text and cached the legacy catalog JSON, eliminating repeated concatenation and serialization across the 350-tool registry.
• Bounded provider tool-selection context to the current prompt and recent history instead of serializing the entire conversation every round.
• Cached saved-conversation menu snapshots so the closed/open Preferences refresh loop no longer continually sorts and clones history.
• Reused the voice activity polling buffer and throttled missing-key retries, reducing allocations in the beta microphone loop.
• Fixed cancellation-token lifetime races during overlapping cancel/send operations and serialized concurrent connection attempts.
• Added defensive reporting for providers that return HTML or other non-JSON errors.
• Added diagnostics.performance for physical world-scan counts, cache hit rate, average scan time, and cached-object count.
• Kept the complete v2.3 feature set: exactly 350 tools, standalone Preferences UI, saved conversations, voice beta, wake word, multi-provider support, SpawnLab, and Fusion integration.

2.3.0

• Includes the v2.2 AI providers: Claude, Grok, DeepSeek, OpenRouter, Ollama, and Custom.
• Expanded BoneAI to exactly 350 internal structured gameplay tools, with searchable discovery and provider-aware subsets for APIs with lower tool limits.
• Added a standalone BoneAI button to BONELAB Preferences, based on the verified Fusion 1.14.2 Preferences injection lifecycle.
• Redesigned the in-game interface into Assistant, Saved Conversations, Voice AI (Beta), AI Provider, Game Permissions, and Diagnostics pages.
• Added persistent saved-conversation metadata and one-click resume for Codex and direct API providers.
• Added opt-in microphone transcription, configurable transcript-gated wake word, listen-once mode, and optional AI speech replies.
• Kept AI work, follow/movement state, and voice polling independent of the open/closed Preferences menu.
• Simplified the release package to the main mod, bridge, and catalog library; new versions are installed manually or through Thunderstore/r2modman.

2.2.0

• Added official tool-calling adapters for Anthropic Claude, xAI Grok, DeepSeek, OpenRouter, and local Ollama.
• Added a configurable OpenAI-compatible provider for other services.
• Kept Codex App Server as the default and only login-based provider; API-provider secrets are read exclusively from environment variables.
• Added provider, model, and custom endpoint controls to BoneMenu.

2.1.1

• Added a valid Thunderstore manifest.json with current BONELAB dependencies.
• Added an exact 256×256 icon.png based on the official BoneAI logo.
• Added the full-resolution logo to the repository and README.
• Rebuilt the release archive with all required Thunderstore files at its root.

2.1.0

• Renamed the project, assemblies, menu, settings, bridge, and logs to BoneAI.
• Added the first-time-player installation package and redesigned README.
• Published the complete source and compiled release under the MIT License.