Uncertainty Evaluation in Engineering Measurements and Machine Learning

---

Instructor: Miodrag Bolic, University of Ottawa, Ottawa, Canada
www.site.uottawa.ca/~mbolic
Time and place: Monday 8:30 - 11:30, MNO E218
Course code: ELG 5218 (EACJ 5600)

Course information

Calendar Style Description: Uncertainty, Uncertainty propagation, Bayesian Inference, Bayesian Filtering, Data fusion, Metrology, Measurement Science, Error Analysis, Measures of Agreement, Data Quality, Data quality index. Case studies will be drawn from various fields including biomedical instrumentation, sensors and signal processing.

Prerequisites: We expect participating students to bring basic knowledge and experience in

• Elementary Probability
• Elementary Statistics
• Signal processing
• Machine learning

Grading: For collecting the credits the student are expected to

• Assignments or projects (30% of the grade)
• Midterm (15% of the grade)
• Scribing (20% of the grade)
• Final exam (35% of the grade)

All lectures are given online using Zoom this year.

About the course Over the last several years, deep neural networks advanced many applications including vision, language understanding, speech understanding, robotics and so on. But a major challenge still remains and that is: how to modeling uncertainty. Good models of uncertainty are crucial whenever decision needs to be made or an algorithm needs to decide how and when to acquire new information. Uncertainty quantification is related to combining computational models, physical observations, and possibly expert judgment to make inferences about a physical system. Types of uncertainties include:

• Experimental uncertainty (measurement errors)
• Model uncertainty/discrepancy
• Input/parameter uncertainty
• Prediction uncertainty.

Why uncertainty:

• Uncertainty quantification is a fundamental component of model validation
• The objective is to replace the subjective notion of confidence with a mathematical rigorous measure
• Uncertainties relate to the physics of the problem of interest and not to the errors in the mathematical description/solution.

---

Introduction

Lecture
Reading: Z. Ghahramani, “Probabilistic machine learning and artificial intelligence,” Nature, 2015.

Intro slides

---

Topic 1 Probabilistic reasoning

Reading: Appendix D Probability by Kevin Murphy
A Comprehensive Tutorial to Learn Data Science with Julia from Scratch

Lecture
Prof. Rai's slides Lec1 9 t0 26
Lec 1 Notebook

Additional Material
Prof. Rai's slides Lec2

---

Topic 2 Bayesian Inference for Gaussians, Bayesian linear and logistic regression

Lecture
Lec 2 Notebook
Prof. Rai's slides Lec5
Prof. Rai's slides Lec6

Mandatory Exercise problems
Rehearsal Exercises->Probability Theory Review, Bayesian Machine Learning, Continuous Data and the Gaussian Distribution, Regression from Exercise: Bayesian Machine Learning and Information Processing and Solutions

Additional Material
Reading:
Chapters 3.6 and 11 by Kevin Murphy
Prof. Rai's slides Lec3
Prof. Rai's slides Lec4
Code:
Continuous Data and the Gaussian Distribution
Bayesian linear regression
Bayesian logistic regression

---

Topic 3.1 Monte Carlo, MCMC

Lecture
Prof. Rai's slides Lec15, Lec16, Lec17
Lec 3 Notebook
Bayesian inference with Stochastic Gradient Langevin Dynamics

Exercise problems
Additional Material

---

Topic 3.2 - Variational Inference

Prof. Rai's slides Lec13, Lec14
Variational Inference: Foundations and Modern Methods by David Blei, Rajesh Ranganath, Shakir Mohamed, slides 55-81
Tutorials by Chantriolnt-Andreas Kapourani

Code: Turing Variational Inference

Additional Material Reading: Blei et al JASA | Tran's VI Notes
Other material: Natural gradient notes | autograd in python | ForwardDiff in Julia
Code: SVI in Pyro

Assignment 1

---

Topic 3.3 Probabilistic programming

Slides: Probabilistic Machine Learning with PyMC3: Statistical Modeling for Engineers by Dr. Thomas Wiecki

Reading: Probabilistic programming by Dan MacKinlay
Christopher Krapu, Mark Borsuk, "Probabilistic programming: A review for environmental modellers," Environmental Modelling & Software, Volume 114, 2019, Pages 40-48. https://doi.org/10.1016/j.envsoft.2019.01.014.

---

Topic 4 Bayesian models: hierarchical and mixture models

Reading: Prior Choice Recommendations

Lecture
Hierarchical models slides by Taylor R. Brown
Mixture Models slides by Russ Salakhutdinov

Code: Turing Mixture Models
PyMC3: A Primer on Bayesian Methods for Multilevel Modeling

---

Topic 5.1 - Bayesian Model Checking and Selection

Model checking by Taylor
[Evaluating, comparing and expanding models by Taylor]https://github.com/tbrown122387/stat_6440_slides/blob/master/7/7.pdf

Reading (ordered by priority): Bayesian Data Analysis - Chapters 6 and 7 | Bayesian Model Selection, Model Comparison, and Model Averaging article

Lecture

Topic 5.2 - Expanding traditional models

Expending regression models with measurement errors

Slides: Errors and uncertainty in variables by Stefanie Muff
Reading: Bayesian analysis of measurement errors
Example of measurement error effects

Expanding neural networks - Bayesian neural networks

Slides: Probabilistic Modeling meets Deep Learning by Prof. Rai slides 1-9
Stochastic variational inference and Bayesian neural networks by Nadezhda Chirkova

Reading: An introduction to Bayesian neural networks
Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users and video

Code: Turing Julia Bayesian Neural Networks

Assignment 2, Dataset
Midterm Review

Topic 6 - Gaussian Processes Regression and Classification

Lecture
Slides by Andreas Lindholm 1 and 2

Reading: Gaussian Processes for Machine Learning - Chapters 1, 2.1-2.5, 3.1-3.4, 3.7, 4.1-4.3.
Code: GP Stheno Python and Julia | GPy for Python | GP summer school - Labs in Python | GP in Turing Regression | GP in Turing Classification Other material: Visualize GP kernels
More slides

Topic 7 Clustering with uncertainties: probabilistic PCA, VAR, normalizing flows

Reading:
Code: Vae with MNIST | Normalizing flow

Lecture
slides

Exercise problems

---

Topic 8 Time series models, forecasting and classification

Reading:
Code: ForneyLab, ForneyLab Documentation

Lecture
slides

Exercise problems

---

Topic 9 Sensor fusion

Reading:

Lecture
slides

Code BayesianDataFusion.jl

---

Topic 10 Integration of physical and machine learning models

Code: SciMLTutorials.jl: Tutorials for Scientific Machine Learning and Differential Equations
ADCME Reading:

Lecture
slides

Exercise problems

---

Topic 11 Sequential decision making

Reading: Algorithms for Decision Making

Lecture
slides

Exercise problems

---

Links

• This course at Github
  
• Bright Space at the University of Ottawa

Relevant courses

• Bayesian Learning by deep.TEACHING project
• Bayesian Machine Learning and Information Processing (5SSD0) by Prof.dr.ir. Bert de Vries
• Advanced Bayesian Learning by Mattias Villani
• Probabilistic Machine Learning (Summer 2020)

Books

• Model-Based Machine Learning by John Winn

Support or Contact

Miodrag Bolic email