Tutorials
The final day of the conference features tutorial presentations, which are open to the public. These tutorials present an up-to-date account of the intersection between low-dimensional modeling and deep learning in an accessible format.
The tutorials consist of two parallel tracks, respectively titled Learning Deep Low-dimensional Models from High-Dimensional Data: From Theory to Practice, and Advances in Machine Learning for Image Reconstruction: Sparse Models to Deep Networks.
Each track consists of four lectures. The planned content of the two tracks is summarized below. See the schedule for the precise times of each tutorial.
Track: Learning Deep Low-dimensional Models from High-Dimensional Data: From Theory to Practice
Lecture 1 – ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
Yi Ma
UC Berkeley / HKU IDS
Abstract
To begin, we will focus on the special yet highly useful case of learning the data distribution and transforming it to an linear discriminative representation (LDR). We will discuss the information theoretic and statistical principles behind such a representation, and design a loss function, called the coding rate reduction, which is optimized at such a representation. By unrolling the gradient ascent on the coding rate reduction, we will construct a deep network architecture, called the ReduNet, where each operator in the network has a mathematically precise (hence white-box and interpretable) function in the transformation of the data distribution towards an LDR. Also, the ReduNet may be constructed layer-wise in a forward-propagation manner, that is, without any back-propagation required.
Lecture 2 – White-Box Transformers via Sparse Rate Reduction
Sam Buchanan
TTIC
Abstract
We demonstrate how combining sparse coding and rate reduction yields sparse linear discriminative representations using an objective named “sparse rate reduction”. We develop CRATE, a deep network architecture, by unrolling the optimization of this objective and parameterizing feature distribution in each layer. CRATE’s operators are mathematically interpretable, with each layer representing an optimization step, making the network a transparent “white box”. Although CRATE’s design significantly differs from ReduNet, both aim for a similar goal, showcasing the versatility of the unrolled optimization approach. Remarkably, CRATE closely resembles the transformer architecture, suggesting that the interpretability gains from such networks might also improve our understanding of current, practical deep architectures.
Lecture 3 – Understanding Deep Representation Learning via Neural Collapse
Zhihui Zhu
Ohio State University
Abstract
The session focuses on the strong conceptual connections between low-dimensional structures and deep models in terms of learned representation. We start with the introduction of an intriguing Neural Collapse phenomenon in the last-layer representation and its universality in deep network, and lays out the mathematical foundations of understanding its cause by studying its optimization landscapes. We then generalize and explain this phenomenon and its implications under data imbalanceness. Furthermore, we demonstrate the practical algorithmic implications of Neural Collapse on training deep neural networks.
Lecture 4 – Invariant Low-Dimensional Subspaces in Gradient Descent for Learning Deep Networks
Qing Qu
University of Michigan
Abstract
To conclude, we show that low-dimensional structures also emerge in training dynamics of deep networks. Specifically, we show that the evolution of gradient descent only affects a minimal portion of singular vector spaces across all weight matrices. The analysis enables us to considerably improve training efficiency by taking advantage of the low-dimensional structure in learning dynamics. We can construct smaller, equivalent deep linear networks without sacrificing the benefits associated with the wider counterparts. Moreover, it allows us to better understand deep representation learning by elucidating the progressive feature compression and discrimination from shallow to deep layers.
Track: Advances in Machine Learning for Image Reconstruction: Sparse Models to Deep Networks
Lectures 1-3 will cover a diverse spectrum of topics across sparse modeling and deep learning and theory with applications in medical imaging and image restoration/computer vision. A subset of works across topics will be discussed including works from tutorial presenters.
Lecture 1 – Sparse Modeling for Image Reconstruction
Saiprasad Ravishankar
Michigan State University
Abstract
Data-driven and machine learning techniques have received increasing attention in recent years for solving various problems in computational imaging. First, we will introduce basics of image reconstruction and sparse modeling before discussing the learning of various classical sparse signal models, particularly synthesis dictionaries and sparsifying transforms. The application of dictionary and transform learning to primarily medical image reconstruction will be discussed. We will also briefly discuss the combination of sparsity with other priors (e.g., low-rank) and variants such as convolutional dictionary learning and multi-layer sparse models used in image reconstruction.
Time: 50 minutes
Lecture 2 – Sparse Modeling to Deep Learning for Image Restoration
Bihan Wen
Nanyang Technological University
Abstract
The second talk will focus more on integration of learned sparsity and nonlocal image modeling via group sparsity, low-rank structures, etc. Applications will be shown for image restoration and medical imaging. This part will also transition to aspects of deep learning for image restoration and offer several examples. How deep learning can be combined with conventional model-based approaches will also be touched upon.
Time: 50 minutes
Lecture 3 – Deep Learning for Imaging
Saiprasad Ravishankar
Michigan State University
Ismail Alkhouri
Michigan State University / University of Michigan, Ann Arbor
Avrajit Ghosh
Michigan State University
Gabriel Maliakal
Michigan State University
Abstract
We will introduce modern deep learning-based methods for image reconstruction, particularly in medical imaging, including image-domain or sensor-domain neural network denoisers, and hybrid-domain deep learning schemes that combine physics-based forward models together with neural networks. Hybrid methods, both supervised and unsupervised or self-supervised, will be a key focus and amongst them, we will review plug and play (PnP) priors, consensus equilibrium, regularization by denoising (RED), deep unrolling methods, deep equilibrium models, and bilevel optimization based methods. Then, generative models for image reconstruction will be presented including generative adversarial networks (GANs), deep image prior, and diffusion models (DMs). We will provide a brief introduction to score-based DMs and introduce Diffusion Posterior Sampling ReSample algorithms. Other recent trends in image reconstruction will also be covered including exploiting deep reinforcement learning, unifying deep learning and sparse modeling, and methods focused on improving robustness of deep learning based image reconstruction (to various perturbations, train-test disparities, etc.) via randomized smoothing, diffusion models, etc. Other topics will also be covered briefly including learning sparse neural networks and joint or end-to-end training of sensing and image reconstruction setups. The session will conclude with brief discussion of future directions for the field.
Lecture 3 will involve multiple speakers covering different themes. Avrajit Ghosh will present many of the hybrid or physics-based deep learning methods. Dr. Ismail Alkhouri will speak on key ideas involving diffusion models. Gabriel Maliakal will discuss GANs and deep reinforcement learning and Dr. Saiprasad Ravishankar will speak on the key other topics.
Time: 140–150 minutes (coffee break in between)