Supervision : Ludovic DENOYER

Co-supervision : RANZATO Marc'Aurelio

Neural Architecture Search under Budget Constraints

The recent increase in computing power and amount of data available has catalyzed the rise in popularity of deep learning algorithms. However, these two factors combined with the memory and energy footprint of these algorithms, their latency as well as the expertise required to build them are all obstacles preventing their use in a larger number of applications. In this thesis, we propose several methods to build the architecture of deep learning models in a more efficient and automated way.
First, we focus on learning efficient architectures for image processing. We propose a new method in which the user can guide the learning procedure by specifying a cost function and a maximum budget to be respected during inference. This budget can take various forms, "less than 200ms per image on this mobile device" for example. Our method then automatically learns a model and its architecture by jointly optimizing the predictive performance and the cost function specified by the user. Next, we consider the problem of sequence classification, where a model can be made even more efficient by dynamically adapting its size to the complexity of the signal to be processed. We show that both approaches result in significant budget savings over a range of cost functions and classes of model. Finally, we address the efficiency problem through the lens of transfer learning. A learning procedure can be made even more efficient if, instead of starting tabula rasa, it builds on the knowledge gained from previous experiments. We present a new evaluation protocol that allows a fine-grained analysis of the different types of transfer and show that our neural architecture search approach is able to beat existing methods on most of these dimensions.

2018-2021 Publications

• 2021
  • T. Veniat : “distributed machine learning”, thesis, phd defence 07/01/2021, supervision Denoyer, Ludovic, co-supervision : Ranzato, Marc'Aurelio (2021)
  • T. Véniat, L. Denoyer, M. Ranzato : “Efficient Continual Learning with Modular Networks and Task-Driven Priors”, 9^th International Conference on Learning Representations, ICLR 2021, Vienna, Austria (2021)
• 2019
  • T. Véniat, O. Schwander, L. Denoyer : “STOCHASTIC ADAPTIVE NEURAL ARCHITECTURE SEARCH FOR KEYWORD SPOTTING”, ICASSP 2019 - International Conference on Acoustics, Speech, and Signal Processing, Brighton, United Kingdom (2019)
• 2018
  • T. Véniat, L. Denoyer : “Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks”, 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, United States, pp. 3492-3500, (IEEE) (2018)