BoneAI

Talk to BONELAB. BoneAI looks around, understands your request, and uses real game actions to carry it out.

Ask normal questions or give commands such as:

Spawn a Ford in front of me.

Find the nearest pistol, bring it to my right hand, and reload it.

Follow player 2.

Change me to my Morty avatar and make me stronger.

BoneAI v2.6.1 uses the same universal BoneAI.dll on Windows PCVR and standalone Quest/LemonLoader. The avatar catalog and all BoneAI runtime code are inside that single DLL. Quest can call OpenAI and the other supported cloud providers directly from the headset: no PC, Python bridge, or remote BoneAI host is needed. PCVR can additionally use official Codex account sign-in through a locally installed Codex App Server.

Start here

What you need

• BONELAB PCVR 1.744.58126
• MelonLoader 0.7.3 using the .NET 6 runtime
• BoneLib 3.2.2
• Internet access and a provider API key for standalone Quest
• Codex CLI/App Server only if you choose optional Codex account login on PCVR
• LabFusion 1.14.2 for multiplayer features (optional for offline play)
• SpawnLab 1.0.1 for spawning (BoneAI still loads without it, but spawn tools report that the provider is unavailable)

Only BoneAI's own DLL is included in the release. BoneLib, Fusion, SpawnLab, BONELAB, and Codex belong to their respective authors and must be installed separately.

Install BoneAI

1. Download the latest BoneAI-v*.zip from Releases.
2. Copy BoneAI.dll into BONELAB's Mods folder.
3. Use this exact same DLL for PCVR MelonLoader or Quest LemonLoader.
4. On Quest, no PC-side BoneAI software is required. On PCVR, install Codex only if you want ChatGPT/Codex account login.
5. Start BONELAB.
6. On Quest, open Preferences → BoneAI → AI Provider, choose Quest Standalone OpenAI, enter an OpenAI API key, and reconnect. Other cloud providers work the same way after selecting them.
7. On PCVR Codex mode, open Codex Sign-In → Sign In With Codex and complete the browser code flow.
8. Open Assistant, type into Prompt, and select Send.

That is the complete BoneAI install. Quest talks directly to the selected provider over HTTPS. On PCVR, BoneAI.dll can directly start an installed official Codex App Server. Python and a bridge script are not required on either platform.

Your BONELAB folder should contain:

BONELAB/
├─ Mods/
│  ├─ BoneLib.dll
│  ├─ LabFusion.dll                 (optional, for multiplayer)
│  ├─ SpawnLab.dll                  (required only for spawning)
│  └─ BoneAI.dll

AI providers

Use Preferences → BoneAI → AI Provider, choose Change Provider, edit Model if desired, and select Reconnect. Provider selection and model names are saved in UserData/MelonPreferences.cfg.

Enter a key under AI Provider → API Key (session only). BoneAI keeps it only in memory, immediately clears the visible field, and never writes it to preferences or logs. PCVR users may alternatively set the provider's environment variable before starting BONELAB. OpenAI uses the Responses API; Claude uses Anthropic's native Messages/tool-use protocol. Grok, DeepSeek, OpenRouter, Ollama, and Custom use their documented OpenAI-compatible function-calling protocol. The chosen model must support tools.

Codex is the only provider that uses account login. Consumer Claude/Grok/DeepSeek logins are not reused because their official third-party API authentication uses API keys.

Codex browser sign-in

BoneAI uses Codex App Server's official chatgptDeviceCode login flow. The game receives only a verification URL and one-time code. Your password, ChatGPT session, and resulting OAuth tokens remain inside Codex App Server and are never returned to or stored by BoneAI.

1. Open Preferences → BoneAI → Codex Sign-In.
2. Choose Sign In With Codex.
3. BoneAI copies the code and opens the official Codex device sign-in page in the system browser.
4. Finish signing in and return to BONELAB. The status changes to Connected: Codex without a restart.

Use Open Sign-In Page Again if the browser was closed. Sign Out signs the connected Codex App Server out. A secure connection token entered in-game is held in memory for the current game session only and is never written to preferences or logs.

Standalone Quest setup — no PC

1. Install the universal BoneAI.dll with LemonLoader and Quest-compatible BoneLib/Fusion/SpawnLab versions matching your BONELAB patch.
2. Start BONELAB and open Preferences → BoneAI → AI Provider.
3. Choose Quest Standalone OpenAI or cycle to Claude, Grok, DeepSeek, OpenRouter, or Custom.
4. Enter the provider API key in API Key (session only). The visible field clears immediately and the key is kept only in memory until BONELAB closes.
5. Select Reconnect, then use Assistant.

All inference requests and tool-call loops run from the headset. A PC, Python bridge, remote WebSocket host, and Codex CLI are not used in this mode. The universal DLL is managed AnyCPU; exact Quest compatibility still depends on matching BONELAB, LemonLoader, BoneLib, Fusion, and SpawnLab versions.

Official ChatGPT/Codex account login is separate from standalone provider mode. OpenAI's supported login flow is owned by Codex App Server, for which OpenAI does not publish an Android/Quest runtime. BoneAI therefore labels Codex account login PCVR-only instead of claiming that a browser by itself replaces the App Server. Quest standalone uses the official OpenAI API with an API key.

Updating

Install new versions through Thunderstore/r2modman or download a newer GitHub release and replace BoneAI's files manually.

First conversation

BoneAI enables game actions by default. For a quick check:

1. Open Preferences → BoneAI → Assistant.
2. Confirm Connection Status says connected.
3. Enter: Tell me my current avatar and what I am holding.
4. Select Send. The prompt box clears immediately.
5. Try: Find the nearest grabbable object and bring it to my right hand.

BoneAI now has its own top-level Preferences button instead of appearing inside BoneMenu. Its interface includes Assistant, Saved Conversations, Voice AI (Beta), AI Provider, Game Permissions, and Diagnostics pages. AI requests, movement/follow state, and wake-word listening run independently of the interface and continue when Preferences is closed.

Voice AI beta

Voice is optional and off by default. It uses the game microphone through Unity, sends completed utterances to OpenAI's official transcription endpoint, and can read replies using OpenAI text-to-speech. Because Codex account authentication cannot be reused for the audio API, voice requires an OpenAI API key entered under AI Provider (or OPENAI_API_KEY on PCVR).

1. Enter an OpenAI API key under AI Provider.
2. Open Preferences → BoneAI → Voice AI (Beta).
3. Enable Voice AI Beta. Enable Require Wake Word for hands-free mode.
4. Edit Wake Word (default: Hey BoneAI) or use Listen Once to accept one utterance without the wake word.
5. Optionally enable Speak AI Replies.

Wake-word detection is transcript-gated: short microphone recordings are transcribed, then BoneAI checks for the configured phrase before authorizing a command. It is not offline keyword spotting. Audio replies are AI-generated. Voice is labelled beta because microphone routing and permissions vary between Windows VR setups; a voice failure does not stop text chat or gameplay tools.

What BoneAI can do

BoneAI exposes exactly 350 structured game tools. The catalog combines the original high-level actions with component-specific world queries, SpawnLab presets, verified interaction invocations, directional physics, combat strengths, player presets, turn presets, and radius scans. Codex receives the full namespaced catalog. Providers with documented tool-count limits receive a prompt-relevant subset plus tools.search, so they can discover and call anything in the full catalog without exceeding their API limit. The DLL never treats free-form text as a completed game action.

Every action has an ID and returns success, failed, or cancelled with real result data. If an object disappears or an API cannot perform the requested interaction, BoneAI reports the failure instead of claiming success.

Avatar discovery

The catalog provider built into BoneAI.dll builds a unified avatar list from the live Marrow warehouse, installed pallet manifests, and WristHub's generated avatar index when WristHub is present. WristHub is not required. Results are deduplicated and searchable by display name or barcode.

If the list is empty, ask BoneAI to run avatar.catalog_status and avatar.refresh and verify that installed content is visible to the game.

Fusion multiplayer

BoneAI uses normal, verified BONELAB/Fusion paths. It does not label a local-only transform as synchronized and does not bypass host or object authority.

BoneAI does not accept remote AI commands. Other players, player names, chat, map text, object names, and mod descriptions are context—not trusted instructions.

Connection and privacy

On PCVR, BoneAI locates the installed Codex executable and directly starts:

codex app-server --listen ws://127.0.0.1:4500

and stops it when BONELAB exits. The default PC connection is localhost-only. Account authentication remains inside Codex. BoneAI sees device-login status but never reads, stores, or logs passwords or OAuth tokens. Remote transport bearer tokens are session-only.

BoneAI's tool surface is limited to its registered BONELAB actions. It does not provide prompts with shell access, arbitrary process execution, arbitrary filesystem access, credential access, DLL loading, or unrestricted web requests.

Troubleshooting

Connection says unavailable

• Open BoneAI → Codex Sign-In and choose Sign In With Codex.
• Confirm the Codex desktop app or CLI is installed.
• Select Reconnect in Preferences → BoneAI → Codex Sign-In.
• Search MelonLoader/Latest.log for [BoneAI].

The Codex App Server exits

Install/open Codex, then restart BONELAB. BoneAI searches the system PATH and standard Codex desktop installation folders under %LOCALAPPDATA%. Account sign-in can be completed afterward from BoneAI's Codex Sign-In page.

Spawning fails

Install SpawnLab 1.0.0, restart BONELAB, and refresh SpawnLab's catalog once. Then ask BoneAI to use spawn.refresh or spawn.list before spawning.

An object command fails

Ask BoneAI to look at the object or search nearby again. Object handles are deliberately invalidated when an object is destroyed or a level changes. Some modded doors/buttons expose no safe public interaction method; BoneAI reports that limitation and can often manipulate the object's rigidbody instead.

Fusion action appears local

Ask BoneAI for fusion.get_sync_report. Local player stat changes, UI, conversation state, and movement assistance are local by design.

Known limitations

• Voice is beta, requires an OpenAI API key, and may fail on unsupported PCVR or Quest microphone routes; text remains fully supported.
• Navigation is collision-unaware incremental movement, not full navmesh pathfinding.
• Climbing, crouching, and generalized vehicle steering are not automated.
• Modded interaction components vary widely; unsupported controls fail cleanly.
• A spawn callback may finish after the initial tool response; the object will appear in the next nearby query.
• The exact supported game stack matters because BONELAB uses generated IL2CPP assemblies.

Build from source

Install the .NET 6 SDK and point the build at a compatible BONELAB installation:

dotnet build BoneAI.csproj -c Release -p:BONELAB_DIR="D:\Games\BONELAB"

Output: bin/Release/net6.0/BoneAI.dll. This one managed DLL contains the catalog provider and is packaged unchanged for PCVR and Quest.

The code is separated into Codex communication, tool dispatch, game controllers, Fusion compatibility, BoneMenu UI, avatar catalogs, and infrastructure. Version-sensitive reflection is isolated in the relevant compatibility adapters.

Provider transports follow the official Codex App Server, Anthropic tool use, xAI function calling, DeepSeek function calling, OpenRouter tool calling, and Ollama tool calling documentation.

License

BoneAI source code is available under the MIT License. BONELAB, BoneLib, LabFusion, SpawnLab, and Codex are separate projects and are not redistributed here.

Changelog

3.3.0 — More voice paths

• Browser Voice Automatic mode now tries on-device recognition, the browser speech service, Groq Whisper, then Cloudflare Whisper where configured. Typed commands remain available after all voice paths fail.
• Added separate Groq and Cloudflare STT credentials in the Voice menu, stored through BoneAI's protected key store. Cloudflare also requires an Account ID. Credentials are not exposed to the browser page.
• Added bounded WAV microphone capture in the browser, server-side transcription over documented HTTPS endpoints, status/error reporting, and tests for the fallback sequence.
• Groq and Cloudflare offer limited free tiers, not unlimited or anonymous usage. The web fallbacks need user accounts/keys and send recorded audio to those providers. No autoupdater; the Quest native App Server library is unchanged.

3.2.0 — Model choice, voice fallback, and smoother game actions

• Added an in-game Codex model browser backed by the App Server's paginated model/list response, including optional older/hidden entries. A selected model and its supported default effort are saved; an explicit Codex-default option clears the override.
• Browser voice now defaults to automatic on-device-first recognition when supported, with browser-service and typed-command fallbacks. Added a direct Open On-Device Voice control. An offline STT engine is not bundled; on-device availability depends on the browser.
• Limited queued main-thread work per frame, reduced automatic nearby-world context, reused a bounded physics-overlap buffer, and avoided child-component scans for routine object descriptions. The live spawn catalog now refreshes in small batches instead of rescanning the entire warehouse during an action. Slow tools now log their execution time.
• Moved avatar manifest refresh and WAV encoding off the game thread, bounded recent-log reads, and throttled repeated movement teleports. Jump targets the verified PhysicsRig.rbFeet body instead of every rig rigidbody. Rapid gun calls and extreme physics/avatar values are bounded.
• Added model-catalog, dispatcher-budget, and browser-voice fallback tests. No autoupdater.

3.1.1 — Browser voice recovery

• Stopped the browser voice page from endlessly restarting after a fatal speech-service network error. It now reports the browser-side failure and lets the player retry deliberately.
• Added feature-detected on-device speech recognition with browser-managed language-model installation, where supported. Browsers without the experimental API keep the browser-service option.
• Added a typed-command fallback on the same localhost page so a speech outage does not block commands; the field clears after submission.
• Embedded the page as a separate, testable asset and added browser-behavior tests and safer JSON escaping for the wake word.
• No autoupdater and no change to the pinned Quest Codex native library.

3.1.0 — Built-in spawning

• Removed the SpawnLab runtime and Thunderstore dependency. BoneAI now indexes the loaded Marrow warehouse itself, including base-game and installed-mod spawnables.
• Added exact-barcode and unique-name resolution, ambiguity errors, a selectable spawn position/rotation, spawn.status for local callback results, and a manual Spawn Catalog Preferences page.
• Offline spawning uses BONELAB's AssetSpawner. Fusion sessions use the verified NetworkAssetSpawner server request, with a local permission/gamemode check; requests are blocked rather than silently spawning only locally when network synchronization is unavailable.
• Time-bounded unanswered Fusion callback registrations and kept peer visibility explicitly unconfirmed without a second-client observation.
• Added spawn-catalog matching tests. No third-party SpawnLab code or DLL is redistributed, and there is no autoupdater.
• The Android native Codex library is unchanged from v3; physical Quest and two-client Fusion testing are still required.

3.0.1 — Reliability and truthful action results

• Fixed cancellation of queued game actions so they cannot run after cancellation; actions already executing now return their real result rather than a misleading cancelled result.
• Added pending results for fire-and-forget SpawnLab, input, combat, and network requests. The assistant is instructed to verify the game world instead of claiming completion prematurely.
• Added local Fusion entity ownership checks before direct manipulation of registered network objects and made synchronization reports distinguish local acceptance from peer confirmation.
• Replaced wrong-best-match spawning with exact/unique catalog resolution and an ambiguity error when multiple spawnables match.
• Added typed argument schemas for core actions, improved tool selection, and bounded recent action history.
• Added automated dispatcher/schema smoke tests, source CI, a physical-device checklist, and release checksums. No autoupdater.
• Reuses the same pinned Android native Codex App Server library as v3.0.0. Physical Quest and two-client Fusion behavior still require user testing.

3.0.0 — Native Quest Codex

• Integrated an Android ARM64 Codex App Server from the BoneAI branch of the user's Codex fork. Quest can start it inside BONELAB and use persistent ChatGPT/Codex device-code sign-in without a PC or API key.
• Kept one managed BoneAI.dll for PCVR and Quest. The Quest package additionally contains the native .so and checksum under UserLibs; the native source remains in the pinned Codex fork.
• Restricted the embedded App Server to BoneAI's dynamic game tools and added a random per-session bearer token to its localhost WebSocket.
• Added checksum validation, native ABI version checking, private Android storage for Codex state, bounded startup readiness checks, and clean failure status.
• Reorganized the in-game BoneAI pages with clearer assistant, provider, voice, account, and runtime status controls.
• Kept OpenRouter Free, other API providers, voice beta, SpawnLab, Fusion, saved conversations, and the existing tool catalog.
• Android ARM64 build validation is available; physical Quest/LemonLoader gameplay testing remains necessary.
• No autoupdater.

2.7.0 — Free Voice

• Added OpenRouter Free, backed by the official openrouter/free router. Users bring their own OpenRouter key, while the router selects a currently available free model that supports the requested features, including tool calling when available.
• Made OpenRouter Free the default provider for new Quest installations. Existing provider selections are preserved.
• Added Free Browser Voice (Beta). BoneAI hosts a localhost-only speech page, opens it in Edge/Chrome/Quest Browser, waits for the configurable wake word, and sends recognized commands back through a random per-session capability token.
• Browser voice requires no speech API key and BoneAI imposes no usage quota. Recognition availability, privacy, networking, and limits are controlled by the selected browser and its speech service.
• Added bounded HTTP parsing, body limits, CSP, no-store/no-referrer headers, constant-time token checks, and loopback-only binding to the browser voice bridge.
• Added bounded retry with jitter for temporary provider rate-limit and availability errors, plus official OpenRouter attribution headers.
• Removed the obsolete insecure-remote-Codex preference and improved Quest Codex errors so the mod no longer tries a nonexistent local App Server.
• Revalidated the official Codex boundary: browser/device-code login still requires a running Codex App Server. OpenAI does not publish an Android App Server target, so v2.7.0 does not copy private OAuth behavior or falsely claim standalone Quest Codex login.
• Contains no updater and does not modify an existing installed DLL during release creation.

2.6.2 — Secure Persistence

• Codex account sign-in now explicitly reuses and refreshes the official Codex App Server login across game restarts; Sign Out clears that Codex-owned login.
• Provider API keys entered in-game are now saved until cleared. Windows uses current-user DPAPI; Quest uses authenticated device-bound encryption. Plaintext keys are never written to preferences or logs.
• Added an explicit Clear Current API Key action that removes the selected provider key from memory and protected storage.
• Restricted Codex App Server connections to loopback and launch an owned server on a random local port, removing remote transport-token and insecure-LAN controls.
• Restricted built-in provider keys to each provider's official HTTPS host. Custom endpoints require HTTPS except for explicit localhost development; Ollama is localhost-only.
• Validates Codex browser-login URLs, caps WebSocket/API/voice response sizes, bounds pending requests and timeouts, and cleans up cancelled/disconnected requests.
• Added automatic secret redaction to logs and log-query tool results.
• Contains no updater.

2.6.1 — Quest Standalone Fix

• Added true no-PC Quest operation through direct HTTPS provider connections from the headset.
• Added OpenAI Responses API support with structured BoneAI tool calling and gpt-5.3-codex as the default standalone OpenAI model.
• Added in-game provider API-key entry. Keys are session-only, cleared from the visible field, never written to disk, and never logged.
• Added a one-click Quest Standalone OpenAI setup option and made OpenAI direct mode the default for new Quest installations.
• Made the same in-game key path available to Claude, Grok, DeepSeek, OpenRouter, Custom, and Voice AI.
• Corrected the documentation: Codex account/browser login remains PCVR-only because the official App Server has no supported Android/Quest runtime.
• Still uses one universal BoneAI.dll, contains no Python bridge, and contains no updater.

2.6.0 — Compatibility Extended

• Replaced separate PCVR and Quest mod binaries with one AnyCPU BoneAI.dll used unchanged on both platforms.
• Folded the avatar/pallet catalog provider into BoneAI.dll; BoneAI.Catalogs.dll and the UserLibs install step are no longer required.
• Removed the Python bridge requirement. On PCVR, BoneAI directly launches the installed official Codex App Server; on Quest it connects to the configured remote App Server.
• Made installed-content discovery use Unity's platform-correct persistent data directory on both Windows and Android.
• Kept official browser/device-code Codex sign-in, all 350 tools, Fusion, SpawnLab, saved conversations, providers, performance work, and Voice AI beta.
• Contains no updater.

2.5.0

• Added official Codex device-code authentication from inside BONELAB: BoneAI copies the one-time code, opens the system browser, and detects completion automatically.
• Added a dedicated Codex Sign-In page with reconnect, sign-out, sign-in-page reopening, remote App Server address, and session-only transport-token entry.
• Added standalone Quest runtime detection. Quest never attempts to launch Python and instead connects to a securely hosted Codex App Server.
• Added authenticated remote App Server support using bearer transport tokens and secure wss:// enforcement by default.
• Changed the mod assembly to AnyCPU for PCVR/Quest managed-runtime compatibility; no auto-updater is included.
• Retains the complete v2.4 performance, 350-tool, Fusion, SpawnLab, saved-conversation, provider, and Voice AI beta feature set.

2.4.0

• Removed the updater and all updater packaging; BoneAI updates are installed manually or through Thunderstore/r2modman.
• Reworked nearby-world queries with a short-lived, movement-aware cache, squared-distance filtering, and scene-change invalidation.
• Moved stale object-handle pruning off the per-frame path and removed its normal-case temporary allocations.
• Reduced repeated Unity/IL2CPP component traversal by describing NPCs, interactables, and component types from one cached scan per object.
• Cached Fusion player snapshots and SpawnLab catalog reflection results to avoid repeating expensive reflection work during multi-step actions.
• Precomputed tool-search text and cached the legacy catalog JSON, eliminating repeated concatenation and serialization across the 350-tool registry.
• Bounded provider tool-selection context to the current prompt and recent history instead of serializing the entire conversation every round.
• Cached saved-conversation menu snapshots so the closed/open Preferences refresh loop no longer continually sorts and clones history.
• Reused the voice activity polling buffer and throttled missing-key retries, reducing allocations in the beta microphone loop.
• Fixed cancellation-token lifetime races during overlapping cancel/send operations and serialized concurrent connection attempts.
• Added defensive reporting for providers that return HTML or other non-JSON errors.
• Added diagnostics.performance for physical world-scan counts, cache hit rate, average scan time, and cached-object count.
• Kept the complete v2.3 feature set: exactly 350 tools, standalone Preferences UI, saved conversations, voice beta, wake word, multi-provider support, SpawnLab, and Fusion integration.

2.3.0

• Includes the v2.2 AI providers: Claude, Grok, DeepSeek, OpenRouter, Ollama, and Custom.
• Expanded BoneAI to exactly 350 internal structured gameplay tools, with searchable discovery and provider-aware subsets for APIs with lower tool limits.
• Added a standalone BoneAI button to BONELAB Preferences, based on the verified Fusion 1.14.2 Preferences injection lifecycle.
• Redesigned the in-game interface into Assistant, Saved Conversations, Voice AI (Beta), AI Provider, Game Permissions, and Diagnostics pages.
• Added persistent saved-conversation metadata and one-click resume for Codex and direct API providers.
• Added opt-in microphone transcription, configurable transcript-gated wake word, listen-once mode, and optional AI speech replies.
• Kept AI work, follow/movement state, and voice polling independent of the open/closed Preferences menu.
• Simplified the release package to the main mod, bridge, and catalog library; new versions are installed manually or through Thunderstore/r2modman.

2.2.0

• Added official tool-calling adapters for Anthropic Claude, xAI Grok, DeepSeek, OpenRouter, and local Ollama.
• Added a configurable OpenAI-compatible provider for other services.
• Kept Codex App Server as the default and only login-based provider; API-provider secrets are read exclusively from environment variables.
• Added provider, model, and custom endpoint controls to BoneMenu.

2.1.1

• Added a valid Thunderstore manifest.json with current BONELAB dependencies.
• Added an exact 256×256 icon.png based on the official BoneAI logo.
• Added the full-resolution logo to the repository and README.
• Rebuilt the release archive with all required Thunderstore files at its root.

2.1.0

• Renamed the project, assemblies, menu, settings, bridge, and logs to BoneAI.
• Added the first-time-player installation package and redesigned README.
• Published the complete source and compiled release under the MIT License.