Curriculum
  /  
Descriptions
ELEC_ENG 335, 435: Deep Learning Foundations from Scratch


VIEW ALL COURSE TIMES AND SESSIONS

Prerequisites

Prior machine learning experience (e.g., an introductory machine learning course ELEC_ENG 375/475 or COMP_SCI 349 or a similar course), a thorough understanding of Linear Algebra and Vector Calculus, and strong familiarity with the Python programming language (e.g., basic data manipulation libraries, how to construct functions and classes, etc.). Python will be used for all coding assignments. No other language can be used to complete programming assignments.

Description

The​ ​last​ ​decade​ ​has​ ​seen​ ​a​ ​resurgence​ ​of​ ​deep​ ​neural​ ​networks​ ​as​ ​the​ ​machine​ ​learning  tool​ ​of​ ​choice​ ​for​ ​many​ ​application​ ​areas​ ​including​ ​computer​ ​vision,​ ​reinforcement​ ​learning,​ ​and​ ​natural  language​ ​processing.​ ​This​ ​course​ ​offers​ ​a​ ​holistic​ ​and​ ​hands-on​ ​introduction​ ​to​ ​deep​ ​networks,​ ​their​ ​many  varieties​ ​and​ ​applications,​ ​as​ ​well​ ​as​ ​the​ ​algorithms​ ​used​ ​to​ ​train​ ​them.​ ​In​ ​this​ ​course​ ​we​ ​discuss​ ​the  foundations​ ​of​ ​Feedforward​ ​Networks,​ ​Convolutional​ ​Networks,​ ​and​ ​Recurrent​ ​Networks,​ ​as​ ​well​ ​as​ ​their  usage​ ​within​ ​popular​ ​Reinforcement​ ​Learning​ ​frameworks.​ ​​Using​ ​real​ ​datasets​ ​and​ ​basic​ ​Python​ ​libraries​ ​for  data​ ​manipulation,​ ​vector/matrix​ ​algebra,​ ​and​ ​automatic​ ​differentiation​ ​students​ ​will​ ​code​ ​up​ ​-​ ​from​ ​scratch​ ​-  popular​ ​examples​ ​of​ ​each​ ​such​ ​network​ ​architecture​ ​and​ ​apply​ ​them​ ​to​ ​applications​ ​in​ ​natural​ ​language  processing,​ ​computer​ ​vision,​ ​speech​ ​recognition,​ ​and​ ​automatic​ ​control.

  • This course fulfills the Technical Elective requirement.

INSTRUCTOR: Prof. Aggelos Katsaggelos

COURSE OUTLINE:

1. Deep Feedforward Networks

  • a. Supervised learning recap
  • b. Feedforward networks in the context of function approximation
  • c. Simple recipes for building deep networks
  • d. Deep nets vs Kernel methods
  • e. How deep networks came back

2. Technical Issues with Deep Networks

  • a. Mathematical optimization recap
  • b. Automatic differentiation, implementations in Python
  • c. First order stochastic gradient methods
  • d. Regularization techniques
  • e. Technical tricks

3. Convolutional Networks

  • a. From fixed feature extractors to convolutional networks
  • b. Composing filters and functions
  • c. Applications in computer vision and speech processing
  • d. Transfer Learning

4. Recurrent Networks

  • a. Recurrence relations
  • b. Parameterized recurrence relations and feedforward networks
  • c. Deriving vanilla recurrent networks
  • d. Popular architectures, Long-Short-Term-Memory
  • e. Applications in NLP and speech recognition

5. Deep Representation Learning

  • a. Unsupervised learning recap
  • b. Nonlinear extensions of PCA, autoencoders

COURSE HAND-OUTS: This course has no required textbook. Handouts authored by the instructors will be made freely available to students for notes. In addition a small number of relevant sections from the author’s textbook Machine Learning Refined (Cambridge University Press) will be made freely available to students.

PROBLEM SETS: 4-5 problem sets will be assigned and graded.  

COURSE PROJECT: There will be a project for the course

COURSE GRADE: Final grades for the course are based on homework (65%) and project (35%)