Skip to content

ML & Chemometrics: Model Evaluation and Validation

Robust evaluation is essential for trustworthy food spectroscopy models. This page follows the WHAT/WHY/WHEN/WHERE template and adds concrete guidance for visualizing cross-validation (CV) results.

For notation see the Glossary. Metrics: Metrics & Evaluation.

What?

Defines validation schemes (train/test, stratified CV, group-aware CV, permutation tests), the metrics to report, and how to visualize per-fold outcomes (confusion matrices, residuals, calibration).

Why?

Spectral datasets are often small, imbalanced, or batch-structured. Validation guards against overfitting/leakage, provides uncertainty via fold variability, and underpins protocol-grade reporting.

When?

Use: stratified k-fold for classification; group-aware CV when batches/instruments matter; permutation tests when checking above-chance performance.
Limitations: tiny n inflates variance; imbalance makes accuracy unreliable; always scale/normalize within folds to avoid leakage.

Where? (pipeline)

Upstream: fixed preprocessing/feature steps.
Validation: CV/permutation.
Downstream: metrics + plots + stats on key ratios.

flowchart LR
  A[Preprocess + features] --> B[CV / permutation]
  B --> C[Metrics + per-fold plots]
  C --> D[Reporting + stats tables]

Validation designs

  • Stratified k-fold (classification): preserve class proportions.
  • Group-aware CV: avoid leakage across batches/instruments.
  • Train/test split: simple, less stable on small n.
  • Permutation tests: label-shuffle to test above-chance performance.
  • Pitfalls: normalize within folds; do not tune on test; document seeds/splits.

Metrics (by task)

  • Classification: F1_macro/balanced accuracy + confusion matrix; ROC/PR for imbalance.
  • Regression/calibration: RMSE/MAE/R²/Adjusted R² + predicted vs true + residuals; calibration with CI bands; Bland–Altman for agreement.
  • Embeddings: silhouette, between/within F-like stats with permutation p_perm (see metrics chapter).

Visualizing CV folds (guidance replacing TODO)

Pattern: collect per-fold predictions and metrics, then plot distributions: 

from foodspec.chemometrics.validation import cross_validate_pipeline
from foodspec.viz import plot_confusion_matrix, plot_regression_calibration, plot_residuals

cv = cross_validate_pipeline(pipeline, X_feat, y_labels, cv_splits=5, scoring="f1_macro")
# Per-fold metrics
print(cv["metrics_per_fold"])  # e.g., list of F1s
# Example per-fold confusion matrix (if returned/recomputed)
plot_confusion_matrix(cv["confusion_matrices"][0], labels=class_labels)
- For regression folds: loop over folds, plot residuals or predicted vs true per fold, or aggregate predicted/true across folds and plot once.
- For a quick visual summary of fold metrics: make a boxplot/violin of the per-fold metric list.

Examples

Classification (stratified CV)

cv = cross_validate_pipeline(clf, X_feat, y_labels, cv_splits=5, scoring="f1_macro")
f1s = cv["metrics_per_fold"]
# visualize distribution of f1s with a simple boxplot (matplotlib/seaborn)

Regression

cv = cross_validate_pipeline(pls_reg, X_feat, y_cont, cv_splits=5, scoring="neg_root_mean_squared_error")
# After CV, refit on full data if appropriate; visualize calibration/residuals on a held-out set or via CV predictions.

Sanity checks and pitfalls

  • Very high scores on tiny n → suspect overfitting/leakage.
  • Imbalance → use macro metrics; inspect per-class supports.
  • Re-run with different seeds/folds to test stability; report mean ± std/CI across folds.
  • Keep preprocessing identical across folds; document seeds, splits, hyperparameters.

Typical plots (with metrics)

  • Confusion matrix (per fold or aggregate) + F1/accuracy/supports.
  • ROC/PR for rare-event tasks.
  • Predicted vs true + residuals for regression; calibration with CI (plot_calibration_with_ci).
  • Fold-metric distribution plot (box/violin of per-fold F1 or RMSE).

Summary

  • Choose validation design aligned with data structure (stratified, group-aware).
  • Pair metrics with uncertainty (fold variability, bootstrap CIs).
  • Avoid leakage; report seeds/splits/preprocessing.
  • Visualize per-fold behavior to reveal instability or class-specific failures.

When Results Cannot Be Trusted

⚠️ Red flags for validation design and model evaluation:

  1. Data leakage in preprocessing (mean/std computed on entire dataset before train/test split)
  2. Information from test set influences training, inflating metrics
  3. Leakage can be subtle; preprocessing should be inside CV loop

  4. Fix: Use sklearn Pipeline to chain preprocessing + model; fit only on training folds; compute statistics on training data only

  5. Same data used for hyperparameter tuning and final evaluation

  6. Hyperparameters optimized on test set produce inflated performance estimates
  7. Proper workflow: use training set for tuning, held-out test set for final evaluation

  8. Fix: Use nested CV (outer folds for evaluation, inner folds for tuning) or separate tune/test sets

  9. Stratification not applied to small, imbalanced datasets

  10. Random train/test splits of imbalanced data can yield train set with even worse imbalance
  11. Causes high fold-to-fold variability

  12. Fix: Use stratified CV (StratifiedKFold); ensures all folds have similar class distribution

  13. Metrics reported without uncertainty (accuracy = 0.92, no confidence interval or SD across folds)

  14. Point estimates hide fold-to-fold variability; single fold may differ significantly
  15. No uncertainty makes it impossible to assess significance of differences between models

  16. Fix: Report mean ± SD across folds; compute bootstrap CI; show per-fold metrics in plot

  17. Batch structure ignored in CV (all samples from Device A in train, all from Device B in test)

  18. Temporal or instrument drift confounds model learning
  19. Model may learn device artifacts, not generalizable patterns

  20. Fix: Use GroupKFold to keep batches together in splits; validate across batch/device boundaries

  21. Perfect metrics on test set (accuracy 1.0, AUC 1.0) without investigation

  22. Too-perfect results suggest overfitting, data leakage, or class separation artifacts
  23. Real food data rarely separates perfectly

  24. Fix: Check for leakage; visualize test set; validate on completely independent, external data

  25. Class-specific metrics not reported (overall accuracy = 0.90, but minority class recall = 0.10)

  26. Aggregate metrics mask poor performance on minority class
  27. Misleading for imbalanced tasks

  28. Fix: Report per-class precision/recall/F1; use confusion matrix; consider weighted F1 or balanced accuracy

  29. Cross-validation with single fold or unrepeated splits

  30. 1-fold CV gives no sense of variability; non-repeated splits depend on random seed
  31. Small datasets need more folds (5–10) for stable estimates

  32. Fix: Use at least 5-fold CV; consider RepeatedStratifiedKFold for small datasets; report mean across repeats

  33. Temporal structure ignored (time-series data: train on future, test on past)

  34. Leakage through time leads to optimistic metrics
  35. Temporal CV (train on earlier times, test on later) is more realistic
  36. Fix: Use time-aware CV (TimeSeriesSplit) for sequential data; no forward-looking information in training
  37. Workflows