← Back to feed

Google Research has published PhotoScan, a deep learning framework that estimates body composition directly from standard 2D smartphone photos. In validation testing, it approached the performance of Dual-Energy X-ray Absorptiometry, the clinical gold standard for measuring body fat distribution, which requires specialist equipment, costs several hundred dollars per scan, and involves low-dose radiation.

The paper, posted to arxiv as 2603.27017 and described in a Google Research blog post on August 19, was authored by a 22-person team including Shwetak Patel, who has spent years building smartphone-based health sensing systems, and Aravind Natarajan, who leads health AI work at Google.

What BMI Misses

BMI divides weight by height squared. It does not distinguish fat from muscle, does not locate fat within the body, and does not differentiate between metabolically active visceral fat (surrounding internal organs) and subcutaneous fat (beneath the skin). Two people with identical BMI readings can have meaningfully different cardiometabolic risk profiles depending on where their body fat sits and what ratio it occupies relative to lean mass.

The metrics that matter clinically are harder to measure. The Android-to-Gynoid fat ratio compares trunk fat to hip-and-thigh fat. The Visceral-to-Subcutaneous fat area ratio distinguishes the metabolically active fat wrapped around organs from the fat just beneath the skin. Elevated A/G ratios and high visceral fat burden correlate strongly with insulin resistance, which typically precedes type 2 diabetes by years and impairs vascular and liver function long before blood sugar rises into diagnostic range.

DXA measures all of this with precision. A scan generates a full body composition profile: bone density, lean mass, fat mass by region, and derived ratios including A/G and V/S. It does so by passing two X-ray beams of different energy through the body and measuring differential absorption. The results are reliable. The infrastructure is not: DXA machines require clinical settings, trained operators, and a scheduling appointment. Availability outside high-income healthcare systems is limited.

How PhotoScan Works

Google’s approach is to learn the mapping from 2D appearance to underlying body composition by training on paired data: photographs and DXA measurements from the same individuals.

The training corpus is the UK Biobank, a long-running population health study that has collected DXA scans alongside lifestyle, genomic, and imaging data from hundreds of thousands of participants. PhotoScan was pre-trained on over 35,000 participant records from that dataset and then fine-tuned on a separate validation cohort.

The model takes standard 2D smartphone photos as input and outputs estimates of three body composition metrics: body fat percentage (BF%), the Android-to-Gynoid fat ratio (A/G), and the Visceral-to-Subcutaneous fat area ratio (V/S). These are the same metrics a DXA scan provides.

The paper does not describe the precise architecture, but the approach is a form of digital phenotyping: using visual signals that correlate with internal physiology to infer physiological state without direct measurement. The correlation works because fat distribution has visible surface expression, particularly in the ratios that distinguish visceral from subcutaneous burden.

The Cardiometabolic Risk Connection

The output feeds into HOMA-IR estimation. HOMA-IR, the Homeostasis Model Assessment for Insulin Resistance, models the feedback loop between liver glucose production and insulin secretion under fasting conditions. A HOMA-IR score above 2.9 indicates insulin resistance.

The connection from PhotoScan outputs to HOMA-IR is not direct prediction, but it is mechanistic. A/G and V/S ratios that track visceral adiposity are established predictors of insulin resistance. A model that estimates those ratios from a photo inherits the predictive relationship.

The practical implication: a smartphone-accessible screening tool could flag elevated metabolic risk in populations that have no routine access to DXA, metabolic panels, or specialist care. It would not replace clinical diagnosis but could prioritise who needs it.

Where This Fits

Google Research has been building a body of health sensing work that treats the smartphone camera as a diagnostic instrument. Earlier published work included heart-rate monitoring from camera video. PhotoScan extends that direction into structural body composition, which is a more complex inference problem because it requires estimating three-dimensional internal distribution from a two-dimensional surface view.

The validation comparison to DXA is the key claim in the paper. The phrase “approached DXA performance” appears in the Google Research blog post summary and in corroborating coverage from other sources, but the paper itself is where the specific correlation coefficients and error bounds will sit. The arxiv preprint is available at 2603.27017.

Google has not announced any product integration. PhotoScan is described as an investigational framework. The regulatory path for a cardiometabolic risk tool built on smartphone photos would require clinical validation at a scale and rigour that a UK Biobank pre-training run does not yet satisfy. That process will take time. The research establishes feasibility; the deployment question is separate.