Google's PhotoScan: Smartphone Photos Estimate Body Composition and Insulin Resistance

Google Research has published a paper demonstrating that a deep learning model called PhotoScan can estimate body composition metrics—body fat percentage, android-to-gynoid fat ratio, and visceral-to-subcutaneous fat ratio—directly from standard 2D smartphone photos. In a clinical validation, the model predicted insulin resistance with accuracy comparable to DXA scans, the current gold standard for body composition measurement.

Insulin resistance is a key driver of metabolic disease, often going undiagnosed for years before type 2 diabetes develops. Current screening relies on blood tests like HOMA-IR, but body composition metrics offer a complementary structural view. DXA scans are precise but expensive, require specialized equipment, and expose patients to low radiation doses—making them impractical for routine screening.

How PhotoScan works

PhotoScan bypasses clinical measurements by extracting geometric body information from smartphone images. The model was built in three phases:

• Pre-training: A ResNet-50 backbone, initialized with ImageNet weights, was trained on 35,323 UK Biobank records, using 2D frontal and lateral projections generated from 3D MRI scans, with DXA as ground truth.
• Fine-tuning: The model was fine-tuned on a new cohort of 677 adults, using real smartphone photos paired with DXA ground truth. An automated landmark detection pipeline selected optimal pose frames from 360-degree videos.
• Validation: On an independent cohort of 132 participants from a 30-week longitudinal trial, PhotoScan achieved strong agreement with DXA across all three metrics.

Key results

In the fine-tuning cohort, PhotoScan achieved a mean absolute error (MAE) of 2.15 for body fat percentage, compared to 2.91 for smartwatch-based bioelectrical impedance analysis (BIA) sensors. The MAE for A/G ratio was 0.107 and for V/S ratio 0.094. On the independent validation cohort, errors were comparable: 2.13 for BF%, 0.085 for A/G, and 0.085 for V/S.

PhotoScan also unlocks A/G and V/S ratios, which BIA cannot measure. These metrics are clinically significant: elevated A/G ratios and higher visceral fat correlate strongly with insulin resistance.

The researchers note that the model's accuracy on A/G and V/S was slightly better in the validation cohort, likely due to a higher proportion of female participants, who typically have lower absolute ratios.

This work builds on Google's broader push into passive health monitoring via smartphones, including continuous heart-rate monitoring. PhotoScan is still investigational and not yet a consumer product, but the results suggest a future where a simple photo could flag metabolic risk without a clinic visit.