There Will Be a Scientific Theory of Deep Learning

The authors argue that there will be a scientific theory of deep learning; that we can see pieces of this theory starting to emerge; and that this theory will take the form of a mechanics of the learning process. They suggest, The measurability of deep learning makes observation and empiricism a particularly fruitful approach, since experimentation can be iterated on quickly, while revealing mathematically simple relations and structure in trained models. This is based on the manipulation of what they call numerical knobs, termed hyperparameters, which include optimization hyperparameters such as the learning rate, batch size, momentum, and initialization variance, as well as architecture hyperparameters such as width, and depth. This opens the possibility of universality in representations: It has been shown that networks trained to solve different tasks learn similar representations across training datasets. A combination of top-down hypotheses and empirical observation may well yield the theory of deep learning the authors are looking for. Web: [Direct Link] [This Post]