Why my iPhone camera went black but the hand skeleton kept working

Nsila has a main camera screen that recognises letters, and separate screens that use the camera to record letters and words.

On my Android phone, all of this worked fine. The bug only showed up when I tested on my iPhone. I would open a recording screen, close it, and go back to the first screen. The camera view was now blank. Everything was black, except that I could still see the hand skeleton showing.

I checked the code but could not find where the issue was coming from, because I still have little knowledge of Swift. So I asked Claude, and that is how I found out what the issue was.

What was actually happening

On iOS, an AVCaptureSession is the object that runs the camera, and it can send what it captures to more than one place. In Nsila it sends frames to a video output for hand tracking, and to a preview layer that shows the picture on screen.

That explains why the skeleton survived. The session was still running, and frames were still reaching hand tracking, so the skeleton kept being drawn. Only the picture was gone.

The cause is that a session feeds one preview layer at a time. Each screen was creating its own. When the recording screen attached its preview layer, it took the connection away from the one on the first screen. Closing the recording screen did not give it back, so the first screen stayed black. Attaching a second layer is also slow on the main thread, which is where the hanging came from.

The fix

Instead of each screen creating a preview layer, there is now one shared preview. Each screen gets a small host view, and the shared preview moves into whichever host was added most recently. When that host leaves the screen, the preview goes back to the one before it.

override func didMoveToWindow() {
    super.didMoveToWindow()
    if window != nil { CameraPreview.push(self) } else { CameraPreview.remove(self) }
}

When the recording screen closes, its host is removed, and the first screen gets the preview back.

What I took from it

Working on Android told me nothing about iOS here. The feature was the same, but the camera APIs underneath are different and have different rules. A native app has to be tested on each platform, because passing on one is not evidence for the other.

It also taught me to read a symptom more carefully. A black screen sounds like the camera has stopped, but the skeleton still moving showed that it had not. Once you know that, the problem is much smaller.