How Face Tracking Actually Works (And Why Most Tools Get It Wrong)

I spent three months building face tracking into ApexClip, and I can tell you — it's way harder than it looks. Most tutorials make it sound simple: "Just detect the face and center it in the frame!" But in practice, there are about a dozen things that can go wrong.

Let me walk you through what actually happens.

Step 1: Face Detection (The Easy Part)

Modern face detection is remarkably good. MediaPipe's Face Landmarker can detect faces in real-time with sub-millisecond latency. It gives you not just the bounding box, but 478 individual landmark points on the face — eyes, nose, mouth, jawline, everything.

This part is genuinely easy. The hard part comes next.

Step 2: Choosing Which Face to Track

What happens when there are two people in the frame? Or three? Or someone walks in and out of the shot?

Most tools just pick the "most prominent" face — usually the one closest to the center or the largest one. But this is wrong. You want to track the person who's speaking, not just the person who's closest to the camera.

In ApexClip, we solve this by correlating face detection with the audio. We know who's speaking at each moment (from the transcript), and we track that person's face specifically. It's not perfect, but it's way better than just picking the biggest face.

Step 3: Smoothing the Movement

Raw face positions are jittery. Even a professional speaker moves their head constantly — nodding, turning, leaning forward. If you follow the face exactly, the camera will shake constantly.

The solution is spring-based smoothing. Think of the camera as being attached to the face by a spring. When the face moves, the spring pulls the camera, but with some delay and damping. This creates smooth, natural-looking camera movement.

The key parameters are:

Finding the right balance is an art. Too responsive and the camera shakes. Too smooth and the camera lags behind the speaker.

Step 4: Keeping the Face in Frame

Here's where most tools fail. If the speaker moves to the edge of the frame, naive tracking will try to follow them — but then the face goes off-screen on the other side.

The solution is boundary clamping. You calculate the ideal camera position, then check if the face would go off-screen. If it would, you adjust the camera position to keep the face at the edge of the frame rather than letting it disappear.

But this creates another problem: if the face is near the edge, the camera might oscillate between two positions. This is called "edge bouncing" and it looks terrible.

We solve this with hysteresis — the threshold for moving the camera toward the edge is different from the threshold for moving it back. This prevents the camera from getting stuck in a loop.

Step 5: Handling No Face

What happens when no face is detected? Maybe the person turned away, or there's a B-roll segment, or the lighting is bad.

Most tools just freeze the camera in place, which looks awkward. In ApexClip, we slowly drift the camera back to center during no-face segments. This creates a natural "reset" that looks intentional rather than like a glitch.

The Results

After three months of tuning, our face tracking is... decent. Not perfect — nothing is. But it handles most scenarios well:

Is it as good as a professional camera operator? No. But it's way better than no tracking, and for most short-form content, it's more than good enough.