Silence is one screen that sits between two people. One draws in the air and is heard out loud. The other speaks and is read as text. Neither waits for the other to type.
Blue is the person who can hear. Pink is the person who cannot speak. Each step names what does the work.
The band's microphone hears the other person and streams the sound to Deepgram, which sends the words back while they are still talking. They appear in large type on the blue side of the screen.
Vexo band microphone, Deepgram live transcription
You make a fist in front of the laptop camera and draw in the air. Opening your hand ends the drawing. Google's MediaPipe hand model follows your hand inside the page. The camera picture is never shown and never leaves the laptop: only your lines are kept.
Laptop camera, Google MediaPipe, running in the browser
The line drawing and the last few lines of the conversation go to OpenAI. It answers with the one sentence you most likely mean as a reply. A cup, after "What can I get you?", becomes "I would like a coffee, please." If the drawing is unclear it answers nothing, and nothing is said.
OpenAI, looking at the lines and the conversation
ElevenLabs turns the sentence into a natural voice and the laptop plays it. The sentence also joins the conversation on screen, so both people can read back what was said.
ElevenLabs text to speech
While the page is speaking it stops listening, so it does not caption its own voice. Then the other person's reply appears, and it becomes part of what the next drawing is understood against.
The conversation so far is the context for every reply
The wearable
Worn on the wrist of the person who cannot speak, and connected straight from the browser over Bluetooth. Its microphone is the ear of the conversation. Its motion sensor feels the wrist, which keeps a line unbroken when a fast hand blurs on camera, and reads a slow or fast shake as a quick yes or no. Its light shows what the page is doing.
Hearing
Live speech to text. The band sends 16 kHz Opus sound in 20 millisecond packets, and they are passed to Deepgram's streaming model as they are, with nothing decoded on the way. Words come back as first guesses and then as finished sentences.
Meaning
Reads a rough air drawing together with the conversation and writes the sentence the person means, in the first person. It is told to stay with what was drawn and what was said, and to stay silent when unsure.
Voice
Speaks every sentence. A phrase is fetched once and kept, so saying it again is instant.
Seeing the hand
Finds 21 points on the hand in every camera frame, inside the browser. We measure the finger joints to tell a closed fist, which draws, from an open hand, which stops.