TL;DR
- A developer used a Google Nest Mini, replaced its PCB and MCU, and hacked together a solution that employs AI for a powerful AI voice assistant.
- The project gives us a glimpse into the future of how AI-powered smart speakers could turn out to be.
The Google Nest Mini is a good smart speaker that can do much despite its tiny package. But in this day and age of AI, a simple Google Assistant-powered smart speaker may feel somewhat underwhelming. While we cross our fingers for AI reaching smart speakers soon, you can take matters into your own hands and create your own AI-powered Google Nest Mini.
Twitter user Justin Alvey modified their Google Nest Mini to run their own LLM, agent, and voice model.
I “jailbroke” a Google Nest Mini so that you can run your own LLM’s, agents and voice models.
Here’s a demo using it to manage all my messages (with help from @onbeeper)
🔊 on, and wait for surprise guest!
The hacking process starts with a PCB replacement, which is already beyond what most ordinary users can do by themselves. But if you do opt for this process, you can use a cheap and developer-friendly Wi-Fi chip that works well with various audio frameworks. This, in turn, opens up the ability to use the speaker for audio and voice detection while offloading the AI processing to a more powerful local machine.
Further, the solution relies on Beeper, a universal chat app. This is where the messages arrive and get parsed through an API. The developer then uses GPT 3.5 for summarizing the messages and other AI functions. The result is passed on to ElevenLabs’ text-to-speech engine.
The result of this experimental project is a fun imagination of what smart speakers could sound like if AI powered them.
I’ve been experimenting with multiple of these, announcing important messages as they come in, morning briefings, noting down ideas and memos, and browsing agents.
I couldn’t resist – here’s a playful (unscripted!) video of two talking to each other prompted to be AI’s from “Her pic.twitter.com/f9lOUciO3Y
The developer said they are working on open-sourcing the PCB design, build instructions, firmware, bot, and server code. We should have a better idea of what is happening under the hood when these are open-sourced.
Nonetheless, this was an exciting project to witness. It gives us a glimpse of what the AI-powered future could look like in our everyday lives. Instead of having a smart speaker read things out, AI could make it all the more valuable by providing a summary of our notifications.
Fitting all of that software magic within the footprint of a tiny Google Nest Mini would also be fantastic. Granted, the processing in this project was happening on a different machine, but that’s not too different from how the Nest Mini needs a constant internet connection to work with Assistant.