Airfryer: computer vision for intelligent cooking

The process starts with an image. The orchestrator corrects its orientation, coordinates the models through adapters, and saves intermediate results, logs and a report for each run. The modules exchange masks and structured information linked by a stable identifier for each piece.

The run generates masks, per-identifier predictions, measurements in pixels, a proposed time and temperature, and relevant warnings. pipeline.json defines the stages, adapters and environments. Development retains the training, selection and history. Experiments and alternatives outside the active selection provide technical evidence but do not run in the runtime.

The pipeline assigns a specific task to each model instead of asking one network to solve different problems.

Development data combines photographs of real baskets with images gathered to increase variety. Each task has its own labels and splits. The instance study brought together 1,255 photos: 1,023 for training, 114 for validation, 115 in a historical test and three continuous captures outside the count. Acquisition, annotation and cleaning tools support experimentation but do not run during basket inference.

• Food and background — SegFormer-B0. It produces a global semantic mask and estimates occupancy. In a 114-image comparison it achieved 92.82% IoU and 1.01 percentage points of mean coverage error. It outperformed semantic YOLO26n on those metrics (90.63% IoU and 1.55 error points), so it remained the selected model.
• Piece separation — YOLO11n 512 + SAM 2.1 Hiera Tiny. YOLO proposes a mask for each piece and SAM refines its contours. Selection used a validation criterion combining count error with minimum quality thresholds. On 114 validation photos (2,036 pieces), it achieved a count MAE of 0.956, 61.4% exact counts and mask AP of 0.825. On the historical test of 115 photos, MAE was 1.009 and AP was 0.816. The test was not used to change the selection.
• Cut classification — fine-tuned DINOv3 ConvNeXt-B. It processes each crop at 256 × 256 and a linear head predicts one of seven classes. Recorded validation achieved 92.40% accuracy and 92.17% macro-F1; the documented test evaluation achieved 91.71% accuracy. The system assigns an operational class from 0.90 confidence. Because the statistical rejection mechanism did not certify the 95% precision target, some pieces may be left for review.
• Breading attribute — DINOv3 ViT-B/16 + RBF SVM. The encoder generates features classified by an SVM calibrated with logistic regression. On a grouped test of 1,449 images it achieved 96.20% balanced accuracy, 89.35% precision and 93.79% recall for “Breaded”. The measured precision and the difference between training images and camera crops mean the result requires human confirmation before changing the plan.
• Geometry and cooking. The summary calculates area, bounding box, centroid, elongation and solidity in pixels; it can also produce provisional shape consensus. The planner is a deterministic Python component with JSON profiles, not a trained model. It adjusts time ranges from available parameters and returns actions and temperature stages when configured.

Other model families and configurations were also explored, including semantic YOLO, instance segmentation variants and ViT classifiers with linear heads or classical methods. The comparisons helped select a suitable component for each task. For classification, ConvNeXt achieved the best single validation result against the ViT variants tested. For instances, the selected combination met the pre-set count and mask thresholds. Full results and recipes remain in the Development modules.

One of the main challenges was preventing split quality from inflating the metrics. Exact duplicates or near-identical images spread across sets can make performance look too optimistic. Preparation uses exact and perceptual hashes, grouping by provenance and label-conflict review to reduce that risk. Splits and metrics remain specific to each task.

In some images, pieces are very close together or overlap. Keeping the global food mask separate from instance segmentation makes it possible to evaluate each problem independently. SAM refines YOLO proposals and exported masks retain unique identifiers. The semantic mask serves as a control; using it to crop instances automatically has not yet been validated.

The integrity of checkpoints, dependencies and outputs also had to be preserved. V2 separates inference environments, configures every stage explicitly and checks hashes and data contracts. If GPU memory is unavailable, automatic mode can fall back to CPU. For instance analysis, CPU execution omits SAM and is recorded as a different variant from the selected method.

Evaluation combines task-specific metrics with integration checks. IoU and Dice measure mask overlap; AP and F1 evaluate instance detection; MAE and exact-count rate describe counting; accuracy, macro-F1 and balanced accuracy summarise classification. A smoke test checks artifact loading, execution and identifier correspondence throughout the pipeline, but does not measure predictive quality.

V2 can complete a run from the food mask to a JSON plan: it produces instances, classification, breading analysis and geometry. In a saved example, the run finished with seven pieces and recorded three warnings: classes did not exceed the operational threshold, the frame was not calibrated to cm², and the cooking proposal needed review. The example shows why uncertainty and provenance should be recorded alongside each prediction.

Model figures come from different datasets and splits; they are not a single overall score and do not guarantee performance with a commercial camera or airfryer. In particular, the breading result remains under review and the instance test is historical, not an independent per-session evaluation.

Current status: V2 runs the workflow from an image to a provisional plan. It has central configuration, selected models, traceable outputs, and documented structural and smoke tests. The main classifier covers seven chicken cuts; physical measurements need calibration and the planner requires human review.

Next steps: validate the system on crops from a real camera and independent sessions; expand and balance foods and preparations; calibrate the camera to convert pixels into physical measurements; collect trials for time, weight, thickness and internal temperature on a specific appliance; recalibrate profiles and thresholds with those data; and measure performance and latency on the target hardware. The project’s expansion and professionalisation are ongoing.