For decades, voice-to-text has been a standard feature on every smartphone. Deaf and hard of hearing users have had no equivalent. That changes today: Google DeepMind has announced SL2T (Sign Language to Text), a massively multilingual sign language translation model that powers new sign-to-text dictation features in Gboard and Live Transcribe, launching first on the Pixel 11 family with American Sign Language (ASL) to English support.

What it does and how to use it

Google has embedded the model inside Gboard and Live Transcribe, where it allows Deaf and hard of hearing users to sign to their phone anywhere they would have normally typed. That means signing to search the web, draft messages, ask Gemini a question, or respond in a Live Transcribe conversation, all without touching the keyboard. In practice, this gives users a faster, more natural way to search the web, write messages, and converse with Google's Gemini chatbot.

Sign-to-Text lets you sign into the front camera while the viewer sees the transcription on the outer display -- a particularly useful setup for the Pixel 11 Pro Fold. The feature is available at no additional cost, and Google plans to expand it to more Android devices beyond the Pixel 11 line.

The technical challenge nobody solved until now

Sign language AI has been a notoriously hard problem, and not just for the obvious reasons. AI's ability to process spoken languages has advanced rapidly over recent decades, enabling automatic translation, dictation, and conversational interfaces that feel effortless to hearing users -- yet this technological revolution has not reached the world's more than 200 sign languages and the estimated 70 million Deaf and hard of hearing people who use them.

Two core problems have blocked progress. First, sign languages are not simply encoded versions of spoken languages. Transcribing speech is a matter of performing a sequential mapping from sound to text in the same language, whereas sign languages are independent, natural languages with their own distinct grammars and lexicons -- requiring true machine translation rather than a sequential process of sign-to-word transformations. Second, the model must learn to see and understand physical movement: sign languages convey meaning through simultaneous movements of the hands, arms, torso, head, and face, and accurately tracking these at high frame rates is a difficult and computationally demanding computer vision task.