How to measure accuracy when there is no ground truth

Our technology reconstructs an athlete’s body in 3D from ordinary video, estimating the positions of individual joints from a single camera. The challenge isn’t producing a convincing-looking 3D avatar. It’s producing measurements accurate and repeatable enough for a strength-and-conditioning coach, physiotherapist or sports physician to make decisions based on them.

To understand how reliable these measurements are, we are building a validation programme: comparing independent measurements of the same movement, understanding where they disagree, and repeating the experiment under progressively more realistic conditions. We started in a motion-capture lab in Nicosia.

The full whitepaper covers the Nicosia experiment end to end: the setup, Xsens reference system, synchronization and 3D alignment, joint-by-joint methodology, and results across nine athletes performing treadmill running, shuttle runs, and occlusion exercises. It also examines the limitations of using Xsens as a reference system. Read the Whitepaper →

Looking for ground truth

For the first experiment, we partnered with TheStudio CYENS, a motion-capture laboratory equipped with an Xsens inertial motion-capture system.

Tracking technologies commonly used in professional sports can tell you where an athlete is on the pitch and provide metrics such as speed and acceleration. For biomechanics, we also need a reference for finer measurements.

Xsens provides one. Inertial sensors worn on different body segments are combined with the athlete’s body measurements and a biomechanical model to reconstruct a full 3D skeleton that we can compare joint by joint with FitWise. FitWise starts with pixels and reconstructs the body through a different model.

Xsens is a reference system, not ground truth. Its sensors measure the motion and orientation of body segments, while the positions of individual joints are inferred through a biomechanical model. This means that when FitWise and Xsens disagree, the difference cannot automatically be treated as an error in the FitWise reconstruction. It is a discrepancy between two independently reconstructed versions of the same movement.

Building the first experiment

The lab session was designed as a controlled test of the part of our pipeline that reconstructs the athlete in 3D. We invited nine athletes (five men and four women), all marathon runners. Each was recorded simultaneously by a calibrated camera and the Xsens system.

Before recording, we measured their body dimensions, attached the sensors, and calibrated the Xsens suit for each athlete. They then performed three exercises.

• The first was treadmill running. This allowed us to capture high-speed running while keeping the athlete inside the relatively small indoor recording area.
• The second was a shuttle run: repeated acceleration, deceleration and changes of direction.
• The third deliberately made things harder for our own system. Another person moved in front of the athlete, partially blocking them from the camera.

Together, these cover high-speed movement, changes of direction and occlusion — a common problem in team-sport footage. Each exercise now had two independent recordings of the same movement.

The next problem was making them comparable.

Two systems, two clocks, two skeletons

You cannot simply take frame 1,000 from FitWise and compare it with data point 1,000 from Xsens. The two systems record independently. The FitWise pipeline produced outputs at 119.88 fps, while Xsens recorded at 60 Hz. Their clocks were not synchronized, their coordinate systems were different, and the FitWise skeleton did not initially have an absolute physical scale.

So the comparison had three basic stages:

1. Align them in time.
2. Align them in space.
3. Measure what is left.

First, we synchronized the streams using the movement itself. Then we brought the skeletons into the same 3D space, giving the FitWise reconstruction a single scale factor for each person and rotating and translating the skeletons so they could be overlaid.

We could then measure the remaining 3D distance between corresponding joints — knee to knee, ankle to ankle, shoulder to shoulder.

The spatial alignment follows PA-MPJPE, a standard protocol in 3D human-pose research. It removes differences in overall position, orientation and scale, while differences in the relative positions of the joints remain. This comparison also showed that the two systems can disagree for reasons that have little to do with the accuracy of the system being tested.

Understanding disagreement

Take the treadmill. As running speed increased, the discrepancy between the two reconstructions tended to grow at the fastest-moving joints, particularly the ankles. At first glance, this could suggest that FitWise becomes less accurate at higher speeds. But the Xsens data pointed to another explanation.

Its offline processing fits the athlete’s trajectory into detected foot strikes using a static-ground assumption. On a treadmill, Xsens therefore reports the foot becoming nearly stationary at ground contact even though the belt is carrying it backwards. Reported foot speed dropped from roughly 5 m/s in mid-swing to around 0.3 m/s during contact. The Xsens ankle trajectory also suggested that post-processing and smoothing had been applied to the raw data.

Some body-worn sensors also became loose during fast movement, making the corresponding Xsens segments unreliable. Because Xsens fits all of its sensors into one connected biomechanical model, an unreliable sensor can also affect neighboring joints.

Sometimes a “joint” isn’t the same joint

The hip measurements exposed a different problem. For some athletes, the FitWise and Xsens hip positions were noticeably farther apart, but the discrepancy stayed similar across different exercises while their knees behaved normally.

Xsens and FitWise use different definitions for the hip point. The point reported by Xsens corresponds to an anatomical hip-joint center around the center of rotation of the femoral head, while the corresponding point in the FitWise body model represents a slightly different anatomical location. As a result, the two points are not directly equivalent, even when both systems consistently track the athlete’s movement.

The difference travels with the athlete instead of changing with the movement. We therefore did not treat the raw hip distance as tracking discrepancy in the main comparison. The ankles, knees, shoulders and neck/torso center have sufficiently compatible definitions; the hips require bias correction first.

Before interpreting a distance, we need to understand what produced it.

What comes next

The Nicosia study gives us a baseline for testing future versions of FitWise. We can repeat the same measurements as the pipeline develops and see how the results change.

But the lab is only the first part of our validation programme. FitWise is built for sport, so the testing also has to move to where sport actually happens. Next, we are taking FitWise onto the football pitch. Five professional players, three independent tracking systems, and a very different experiment. That’s our next story.