Muse Gadget SDK: ESP32 & Linux Device SDK

1. Project Overview

Muse Gadget SDK is an open-source SDK that turns off-the-shelf hardware, such as ESP32 boards and Raspberry Pi or other Linux machines, into "gadgets" that pair with the Muse mobile app on iOS and Android, so developers can build custom connected devices without designing hardware from scratch.

2. Background & Positioning

Why it exists

Muse Gadgets lets makers extend the Muse app with their own devices: displays, sensors, buttons, microphones, speakers and actuators. The SDK supplies the firmware and client code that handle pairing and communication, so you can focus on what your gadget does.

How it differs from similar projects

Companion-app oriented: devices pair with the Muse mobile app through a Developer Mode flow rather than a generic cloud dashboard.
Two device families in one repository: C-based ESP32 firmware for microcontrollers and a Python-based SDK for full Linux systems.
Agent-friendly: each SDK ships an AGENTS.md so AI coding agents (including Muse Code) can help build, flash and extend gadgets.
Hacker-friendly by design: the project describes itself as "built by hackers, for hackers, just for fun", so custom firmware carries risk and is used at your own responsibility.

3. Feature Categories

🔌 ESP32 Device SDK

Firmware and components for ESP32 boards, with five board families documented in the README. Examples:

ESP32-C5 DevKitC-1 (status light and button)
ideaspark ESP32 (1.9" display)
Seeed SenseCAP Indicator (4" screen)
Waveshare AMOLED boards (full UI, push-to-talk)
Home Assistant Voice Preview (LED ring, voice)

Purpose: build screen, audio and sensor gadgets on low-cost microcontrollers.

🐧 Linux Device SDK

A Python package (musegadget) that makes a Raspberry Pi or other Bluetooth LE-capable Linux computer appear as a Muse gadget. Examples:

system.run to execute shell commands
file.read / file.write for file access in 64 KB chunks
device.health for uptime, load, memory, disk and temperature
Custom commands for Home Assistant or system administration

Purpose: control and monitor Linux machines from your phone.

🖥️ Simulator & Tooling

Located in esp32/simulator, esp32/tools and esp32/tests. Examples:

Desktop UI simulator (SDL-based) to preview screens
tools/board.sh for board-specific build, flash and monitor
Unit tests for the ESP32 code
Per-board configurations under esp32/devices

Purpose: shorten the edit-build-flash loop.

🧩 Skills

The skills/ directory holds additional functionality modules that extend what gadgets and coding agents can do.

Purpose: reusable add-ons for agent-assisted development.

4. Key Highlights

Off-the-shelf hardware: no custom PCB needed; common ESP32 boards and Raspberry Pi models work.
Phone pairing: Developer Mode in the Muse app connects your device using an SDK token.
Rich device UI: ESP32 builds support displays (LVGL-based UI), avatars, audio input/output and push-to-talk on supported boards.
Remote control for Linux: run commands, transfer files and read health metrics from the app.
AI-assisted workflow: AGENTS.md files guide coding agents through setup and builds.
Permissive license: Apache-2.0 for the main code, with clearly listed third-party exceptions.

5. Use Cases by Role

General developers / makers: prototype a handheld display, voice device or sensor gadget on an ESP32 board.
DevOps / SRE: expose a home-lab or edge Linux box to your phone for quick commands and health checks (mind the permission notes below).
Embedded engineers: use the simulator and board scripts to iterate on firmware UI without constant flashing.
Smart-home enthusiasts: build Home Assistant-oriented gadgets such as voice or LED-ring devices.

6. Getting Started

Find what you need

Browse the repository and read the README in each SDK directory:

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
cd muse-gadget-sdk

Documentation: https://gadgets.muse.ai

Install / integrate

You first need an SDK token from gadgets.muse.ai/settings/sdk-tokens and should review the Gadget SDK Terms at gadgets.muse.ai/sdk-terms.

ESP32 (requires ESP-IDF v6.0.1):

cd esp32
idf.py menuconfig
idf.py build
idf.py -p /dev/cu.usbmodem1101 flash monitor

# Board-specific builds
tools/board.sh ideaspark build

Linux / Raspberry Pi:

curl -fsSL https://raw.githubusercontent.com/facebookincubator/muse-gadget-sdk/main/linux/install.sh -o install.sh
bash install.sh --sdk-token mgst_…

Contribute

Read CONTRIBUTING.md in the repository, then open an issue or pull request: https://github.com/facebookincubator/muse-gadget-sdk

7. Project Structure

muse-gadget-sdk/
├── esp32/
│   ├── avatar/       # Avatar graphics and rendering
│   ├── cmake/        # Build configuration
│   ├── components/   # Firmware components
│   ├── devices/      # Board-specific configurations
│   ├── main/         # Core firmware code
│   ├── simulator/    # Desktop UI preview
│   ├── tests/        # Unit tests
│   └── tools/        # Build and utility scripts (board.sh)
├── linux/
│   ├── examples/     # Sample implementations
│   ├── src/musegadget/  # Core Python SDK
│   ├── tests/        # Test suite
│   └── install.sh    # Installer
├── skills/           # Additional functionality modules
└── .github/          # Workflows and documentation assets

8. Related Ecosystem

Muse app (iOS and Android): the companion app that pairs with gadgets.
ESP-IDF v6.0.1: Espressif's development framework for ESP32 firmware.
LVGL: embedded graphics library fetched at build time.
SDL: used by the desktop simulator.
Home Assistant: a target ecosystem for voice and custom-command gadgets.
uv and pytest: used for Linux SDK development and testing.
Muse Code: a coding agent supported through AGENTS.md.

9. License

✅ Use, modify and distribute the code under Apache License 2.0, including commercially
✅ Keep the license and copyright notices when redistributing
❌ Do not assume the Jollybot avatar is covered by Apache-2.0; it is excluded
ℹ️ Third-party components keep their own licenses: minimp3.h is CC0-1.0 and pixel_font.c is BSD-2-Clause
ℹ️ Use of the Gadget SDK is also subject to the Gadget SDK Terms at gadgets.muse.ai/sdk-terms
ℹ️ Custom firmware and the Linux SDK run with the installing account's permissions (including sudo, if available), so use at your own risk

10. FAQ

Q: Do I need a token to use the SDK?
A: Yes. Get an SDK token at gadgets.muse.ai/settings/sdk-tokens before flashing or pairing a gadget.

Q: Which ESP-IDF version should I use?
A: The ESP32 README specifies ESP-IDF v6.0.1.

Q: Which Linux systems are supported?
A: Raspberry Pi 3B+, 4, 5 and Zero 2 W, or any Bluetooth LE-capable Linux computer running Raspberry Pi OS Bullseye+, Debian 11+ or Ubuntu 22.04+.

Q: Can I preview the UI without hardware?
A: Yes, use the desktop simulator in esp32/simulator.

Q: Is it safe to run on my main machine?
A: The Linux SDK can run shell commands and access files with the installing user's rights. Prefer a dedicated device and read the README first.

11. Quick Links

Repository: https://github.com/facebookincubator/muse-gadget-sdk
Documentation: https://gadgets.muse.ai
SDK terms: https://gadgets.muse.ai/sdk-terms
SDK tokens: https://gadgets.muse.ai/settings/sdk-tokens
Contributing guide: CONTRIBUTING.md in the repository
Community: Discord (linked from the repository README)

12. Summary

Muse Gadget SDK gives makers and developers a practical path to turn inexpensive ESP32 boards and Linux machines into phone-connected gadgets. It suits hardware tinkerers, smart-home builders and anyone who wants AI-assisted firmware development with a permissive Apache-2.0 license.