# 30. Learning Equations

Unsupervised learning

## Outlines

### Learning in Rate Models

- PCA
- Correlation matrix $C_{ij} = \langle \xi^\mu_i \xi^\mu_j\rangle_\mu$, where $\xi^\mu_i$ is the $i$th component of a vector $\boldsymbol{\xi}^\mu$.
- Covariance matrix $V_{ij} = \left\langle (\xi^\mu_i - \langle \xi^\mu_i\rangle)( \xi^\mu_j - \langle \xi^\mu_j\rangle ) \right\rangle_\mu$.
- Principal components of the vectors are the eigenvectors of the covariance matrix $V$.
- The first principal component is the direction where the variance is maximal.

- Evolution of synaptic weights
- A neuron takes $\mu$ inputs at each time step, which are either 0’s or 1’s.
- At each time step, the input forms a $N$ dimensional vector ($N$ input neurons).
- For a total time step of $p$, we have $p$ $N$ dimensional vectors.
- At each time step, the weight change according to Hebbian learning rule $\Delta w = \gamma \nu^{\text{p}} \nu^{\text{pre}}_i$, where $\gamma$ is the learning rate.
- Using linear model of post synaptic rate, $\nu^{\text{post}} = \sum_i w_i \nu_i^{\text{pre}}$.
- The author derived the relation between weight and correlation matrix, as well as the eigenvalues and eigenvectors of it.
- The growth of the
**expectation value of weight**will be dominated by the first principal component.

- Exponential growth of weight mean blowing up in biological systems, which should not happen for a working brain. Thus modified Hebbian learning rule should be used and tested. Here we talk about normalization of weight.
- Three key ideas:
- Normalize sum of weights or quadratic norm of weights;
- Multiplicative normalization or subtractive normalization (to make sure that $\sum_i w_i$ is a constant).
- Might not be just local learning rules.

- Subtractive normalization: $\Delta w_i = \Delta \tilde w_i - \sum_j \Delta w_j /N$ so that $\sum_i \Delta w_i=0$.
- Multiplicative normalization: Oja’s learning rule as an example. The natural choice of normalization is to divide the weight at each step by its norm.

- Three key ideas:
- Neurons of visual system have their receptive fields.

Planted:
by OctoMiao;

## Table of Contents

**References:**

**Current Ref:**

- snm/30.learning-equations.md