Keep only the latest frame

This note is about a concept used in Nsila, rather than a bug I had to chase.

A camera at 30 frames per second delivers a new frame about every 33 milliseconds. If processing one takes longer than that, frames arrive faster than they can be handled. Put them in a queue and the queue keeps growing, so the app shows what your hand was doing a moment ago, and that delay gets worse the longer it runs.

For live recognition, an old frame has no value. The only frame worth processing is the newest one.

Both platforms have a setting for exactly this:

// Android, CameraX
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)

// iOS, AVFoundation
videoOutput.alwaysDiscardsLateVideoFrames = true

A slower phone then processes fewer frames, but each one is current. It gets less smooth instead of falling behind.

Keep the work off the main thread

The main thread draws the interface, so frame processing cannot run there. Nsila gives it a dedicated executor on Android and a dedicated dispatch queue on iOS, and results move back to the main thread only to update the screen.

On Android there is one more rule. Each frame must be closed when you are done with it, or the analyzer stops receiving new ones. That is why the close call sits in a finally block, so it runs even if something fails.

Why dropping frames is safe here

Dropping frames only works if the code after it does not depend on receiving every one. In Nsila, the logic that decides on a letter works over a window of time instead of a count of frames, so a slower phone gets fewer samples in that window and the same half-second behaviour.