Title: Mathematical foundations of deep learning
Instructor:Mohammad Motamed
General description
We will study several concepts related to deep learning from an applied mathematics perspective. We will review recent articles on the subject. We will also be working with Matlab and Python (Keras) for implementation. We will be covering the following topics.
- Deep Neural Networks: formalization and key concepts
- What is a neural network? A parametric map with a compositional structure.
- What is the use of a neural network? It solves regression and classification problems.
- What is network training? It is an optimization problem.
- How to solve the optimization problem? By stochastic gradient descent & back propagation.
- Choices of loss functions and activation functions
- Regularization
- Good practices for training neural networks.
- Approximation theory for neural networks
- Density: the theoretical ability to approximate well (when the number of parameters is very large)
- Convergence rates: how well is the approximation for a fixed number of parameters?
- Complexity: how many parameters needed to achieve a desired accuracy?
- Optimization methods to train dee networks
- The fundamentals of optimization for machine learning
- When does gradient descent fail?
- Deep networks for solving high-dimensional PDEs
- How to solve a PDE by neurla networks?
- Dimension independent approximability of neural networks
Reading list
1) HH: C F Higham and D J Higham. Deep learning: An introduction for applied mathematicians. SIAM Review 61 (2019), pp. 860-891. 2) P: A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica 8 (1999), pp. 143-195. 3) Y: D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks 94 (2017), pp. 103-114. 4) BCN: L Bottou, F E Curtis, and J Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review 60.2 (2018), pp. 223-311. To be completed ... Grading: In-class exercises and a final project motamed@unm.edu
Last updated: Fall 2021