The itch
Nutrition tracking is supposed to be effortless: point your phone at your plate, get a calorie count. In practice, the moment your plate has dal, sambar, or a thali with six things touching each other on one steel plate, most food-recognition models fall apart. I went looking for why, and the answer was unglamorous: the benchmark datasets these models are trained on — the ones everyone fine-tunes against — skew heavily toward Western and generically "global" cuisine categories. Indian food, with its regional variety and visually similar dishes, is underrepresented almost everywhere.
What already existed
The mainstream calorie-tracking apps mostly lean on manual entry or barcode scanning for packaged food, and where they do offer image recognition, it's usually a thin wrapper over a general-purpose food classifier that was never shown enough Indian dishes to tell a paratha from a naan, let alone separate items on a mixed thali. On the research side, I found plenty of food-image datasets, but very few sized and labeled specifically for the Indian food space at a resolution useful for a real application.
How I actually built it
I trained on 4,000 images across 80 Indian food categories, and I fine-tuned a Vision Transformer (ViT-Base) rather than a conventional CNN. With a dataset this size, ViT's pretrained attention layers transfer surprisingly well — the model isn't learning to see edges and textures from scratch, it's learning to redirect attention it already has toward the specific visual differences that separate one lentil curry from another.
Because the real-world class distribution isn't even — some dishes just have far more photos available than others — I used a stratified 60/40 train/test split to keep every category fairly represented in both sets, and I evaluated with macro-F1 instead of leaning on raw accuracy. Accuracy alone would have let the model coast on the well-represented categories while quietly failing the rarer ones; macro-F1 forces every class to actually earn its score.
From prediction to plate
A dish label alone isn't useful — you need calories. I built a FastAPI inference pipeline that takes the ViT's prediction, maps it into a calorie database, and adjusts the estimate for serving size instead of assuming every portion is identical, since a home-cooked serving and a restaurant serving of the same dish can differ by a lot.
The numbers
- 75.19% test accuracy across 80 Indian food categories
- 0.735 macro-F1, chosen specifically to expose weak performance on underrepresented dishes instead of hiding it
- Stratified 60/40 split to keep class balance honest
- End-to-end FastAPI pipeline returning dish label, confidence, and serving-size-adjusted calorie estimate
Why it matters
Nutrition tools that don't recognize what a huge share of the world actually eats aren't neutral — they quietly push people toward logging food as the nearest Western analogue, which distorts the data before it even reaches the user. SmartPlate is a small, direct fix aimed at exactly that gap: food recognition that was actually trained on the food it's supposed to recognize.