Nsila
Recognising Nigerian Sign Language on the phone, with no server.
- Role
- Product Engineer
- Context
- Personal product, free and non-commercial
- Timeline
- October 2026 to present
- Platforms
- Android, iOS
- Status
- Primary metric
- 82.5%letter accuracy on capture sessions the model never saw
Overview
Nsila recognises Nigerian Sign Language (NSL) fingerspelling from the phone camera in real time. You sign a letter with one hand; the app shows it, builds letters into words and can speak them aloud. A daily practice mode with streaks teaches the alphabet to hearing people such as family members, teachers and colleagues.
Most sign recognition apps target American or British Sign Language. NSL is related to ASL but is its own language, with very few learning tools. Beyond letters, you can record the word signs you use and the app recognises them from your recordings straight away. Those recordings can be shared, as hand landmarks only, to build words that work for everyone.
It is one product built twice, natively: Kotlin and Jetpack Compose on Android, Swift and SwiftUI on iOS. Video and images never leave the phone. It is a fingerspelling and learning tool, not a sign language translator or a replacement for interpreters, and the app says so. The letter model is trained on the dataset by Kolawole, Osakuade, Saxena and Olorisade (IJCAI-22), whose CC BY-NC licence keeps Nsila free.
See it work
My role
I own the whole product: the requirements, the interface, the Python pipeline that extracts landmarks and trains the model, and both native apps. I also wrote the tooling that merges contributed recordings back into training.
The challenge
A letter has to appear about half a second after the hand settles, on the mid-range and budget Android phones common in Nigeria, with no server to lean on. The training data is small, about 1,200 hands across 25 letters, and several NSL letters look almost the same to a camera.
- Recognition runs on the phone: both hand tracking and the letter model ship inside the app.
- No video or images leave the device. Sharing recordings is opt-in and landmarks only.
- The dataset licence rules out ads, paid features or a paid version.
- Deaf NSL users are the experts on correctness, so the app must never overstate what it can do.
Architecture
Camera
Built by me
Hand Landmarker
Third party
Letter classifier
Built by me
Smoother and text
Built by me
More screens
Engineering challenges
One model, two native apps
- problem
- The model is trained in Python but has to behave the same in Kotlin and in Swift.
- constraint
- If the app computes one hand feature differently from training, nothing crashes. Accuracy just drops, silently.
- approach
- I export the model as one JSON file and run it as plain code in both apps, with no ML runtime or conversion step. On every retrain, Python writes golden test vectors from real hands, and an Android unit test fails if any of the 154 features differs by more than 1e-4 or any probability by more than 1e-3.
- result
- Both apps ship the same model file, and a retrain cannot reach Android with features that have drifted from training.
An accuracy number worth trusting
- problem
- R and U, or T, S, M and N, look nearly identical to the model, and a still photo of Z is just a pointing finger: it scored 0 of 24.
- constraint
- A random train and test split flatters the model, because photos from one capture session land on both sides.
- approach
- I score only on capture sessions the model never saw. Z came out of the model and waits for motion detection, marked as coming soon in the app. Practice accepts a letter as the first or second guess so known confusions do not fail a beginner, and the Contribute screen asks volunteers for the letters the model confuses most.
- result
- 82.5% across 25 letters on unseen sessions, with a per-letter report that says which recordings to collect next.
Real time on a budget phone
- problem
- Hand tracking and classification have to keep up with the camera on low-end Android hardware.
- constraint
- MediaPipe’s GPU delegate can crash natively on budget GPUs, and a native crash cannot be caught. Newer MediaPipe versions also pull in a runtime that breaks the app’s 30MB download target.
- approach
- Tracking runs on the CPU at about 640x480, and the camera keeps only the latest frame. MediaPipe stays pinned to 0.10.x, and release builds ship only the two ARM ABIs, published as an App Bundle so each phone downloads one.
- result
- A slow phone skips frames instead of falling behind the camera, and tracking avoids the GPU crash path entirely.
Word signs with no retraining
- problem
- People want to sign whole words, but the dataset’s word signs are single photos of signs that mostly involve movement.
- constraint
- There is no server, and a model cannot be retrained on the phone every time someone adds a word.
- approach
- A word is recorded as a few short takes of hand and body landmarks, then a live sign is compared against them with dynamic time warping, which tolerates signing faster or slower. A match counts only if it beats the word’s own threshold and is 15% closer than the runner-up. Otherwise the app says it is not sure and offers the top two.
- result
- A new word works as soon as its takes are saved, tuned to the person who recorded it. A shared word ships only after approval, with 5 contributors and 80% accuracy on people it has not seen.
Outcome
82.5%
letter accuracy on unseen capture sessions, A to Y
Kotlin + Swift
two native apps, one product
0
video frames sent off the phone
Tuning the model gained under two points. The next real improvement has to come from more signers, Deaf NSL users above all, which is why contributing is built into the app.
Technologies
Links
Nsila is awaiting approval and is not on the App Store or Google Play yet, and it still confuses some look-alike letters. NSL signers who would like to test it are welcome to get in touch. These posts show it running.