Conditional Probability Estimation

Summary: This will be the beginning of a new topic: Graph Neural Networks. In this new series, we will use the textbook by Hamilton1. For the first episode, we will discuss some basics about graphs to make sure we are all on the same page. @Steven will lead the discussion. Hamilton2020 Hamilton WL. Graph representation learning. Synth lect artif intell mach learn. 2020;14: 1–159. doi:10.2200/s01045ed1v01y202009aim046 ↩︎

Pages: 22

²⁰ Self-supervised Learning: Theories (Part 2)

Published: 2021-08-26

Tags:

#Self-supervised Learning

References: - Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.08218 - Wang T, Isola P. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2005.10242 - Newell A, Deng J. How Useful is Self-Supervised Pretraining for Visual Tasks? arXiv [cs.CV]. 2020. Available: http://arxiv.org/abs/2003.14323 - Tschannen M, Djolonga J, Rubenstein PK, Gelly S, Lucic M. On Mutual Information Maximization for Representation Learning. arXiv [cs.LG]. 2019. Available: http://arxiv.org/abs/1907.13625 - van den Oord A, Li Y, Vinyals O. Representation learning with Contrastive Predictive Coding. arXiv [cs.LG]. 2018. Available: http://arxiv.org/abs/1807.03748 - Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. arXiv [cs.LG]. 2016. Available: http://arxiv.org/abs/1606.03657 - Nowozin S, Cseke B, Tomioka R. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. arXiv [stat.ML]. 2016. Available: http://arxiv.org/abs/1606.00709

Summary: Theories of self-supervised learning

Pages: 22

¹⁹ Self-supervised Learning: Theories (Part 1)

Published: 2021-08-26

Tags:

#Self-supervised Learning

References: - Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.08218

Summary: Theories of self-supervised learning

Pages: 22

¹⁸ Self-supervised Learning: GAN

Published: 2021-08-01

Tags:

#Self-supervised Learning

References: - Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative Adversarial Networks. arXiv [stat.ML]. 2014. Available: http://arxiv.org/abs/1406.2661 - Mirza M, Osindero S. Conditional Generative Adversarial Nets. arXiv [cs.LG]. 2014. Available: http://arxiv.org/abs/1411.1784 - Goodfellow I. NIPS 2016 Tutorial: Generative Adversarial Networks. arXiv [cs.LG]. 2016. Available: http://arxiv.org/abs/1701.00160 - GAN Course Introduction - Intuitive Intro To Generative Adversarial Networks. [cited 1 Aug 2021]. Available: https://deeplizard.com/lesson/gaa1ilrazd - Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.08218

Summary: Let's talk about GAN this time.

Pages: 22

¹⁷ Self-supervised Learning: Generative or Contrastive

Published: 2021-07-03

Tags:

#Self-supervised Learning

References: - Liu X, Zhang F, Hou Z, Wang Z, Mian L, Zhang J, et al. Self-supervised Learning: Generative or Contrastive. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.08218

Summary: Some biological neural network knowledge

Pages: 22

¹⁶ LTD/LTP

Published: 2021-06-21

Tags:

#Biological Neural Network

References: - Abraham WC, Jones OD, Glanzman DL. Is plasticity of synapses the mechanism of long-term memory storage? NPJ Sci Learn. 2019;4: 9. doi:10.1038/s41539-019-0048-y

Summary: Some biological neural network knowledge

Pages: 22

¹⁵ Predictive Coding Approximates Backprop along Arbitrary Computation Graphs

Published: 2021-06-21

Tags:

#Biological Neural Network

References: - Millidge B, Tschantz A, Buckley CL. Predictive Coding Approximates Backprop along Arbitrary Computation Graphs. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2006.04182

Summary: In this meetup, we will discuss this paper: https://arxiv.org/abs/2006.04182 Why? Feedforward-backprop usually has a loss function that involves all the parameters. Backprop means we need this huge global loss $\mathcal L({w_{ij}})$. However, it is hard to imaging such global loss calculations in our brain. One of the alternatives is predictive coding, which only utilizes local connection information. In this paper (2006.04182), the author proves the equivalence of backprop and predictive coding on arbitary graph.

Pages: 22