Intro to Neural Networks

You've probably seen images like the ones below that represent machine learning models, but what do they actually mean? The goal of this lesson is to begin to build some intuition about these models.

Neural networks are a sub-category of machine learning models.

The diagrams below all represent the simplest category of neural networks called a Feed Forward Networks (FFNN). You will also encounter the related names Fully Connected Neural Network (FCNN) and Multilayer Perceptron (MLP).

Neural Networks come in many shapes and sizes.

In practice, networks can have many millions of nodes.

"Feed Forward" and "Fully Connected"

Can you explain what makes these networks fully connected? What makes them feed forward?

Inputs and Outputs

In machine learning, we need to represent our input and output data with numbers. There are lots of different ways to do that. Any of the methods below are valid targets for the inputs and outputs of a neural network.

Some formats work better than others!

24-16661424-8-642145

Weights

How did we calculate the 24 in the first hidden layer?

Where do these orange values come from?

24-166614241.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0-8-642145

A more practical example

Let's discuss how we could create a neural network that generates chord sequences.

-3.644.25-4.163.95-0.84-4.23-2.531.94-0.10.6-0.5-0.1-0.0-0.60.8-0.30.0-0.4-0.3-0.0-0.50.70.30.30.50.20.3-0.4-0.80.3-0.6-0.40.5-0.50.70.8-0.6-0.40.1-0.20.70.5-0.30.4-0.50.0-0.40.1-0.20.00.50.60.20.70.20.00.50.70.0-0.6-0.6-0.6-0.6-0.7-1.460.63-1.64-1.04-2.06-0.51-2.781451

Points so far

Neurons with bias

Remember I said that we skipped over some of the details of the neurons? Let us add it for completeness.

+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021-1676111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0-14-635153

Neurons with bias and "activation"

The activation function is applied to the output of hidden layers.

Activation:
+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021076111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.01817-11153

Softmax

Take a look at the output of the matrix above. Data flowing through a neural network is typically represented as a column vector.

z=[18170]z = {\def\arraystretch{1}\begin{bmatrix} 18 \\ 17 \\ 0 \end{bmatrix}}

What we want is an array of probabilities for each possible output. That is, we want a vector that sums to 11. A naive way to do this would be to divide each value by the sum of the vector.

[1818+17+01718+17+0018+17+0]≈[0.510.490]{\def\arraystretch{1.8}\begin{bmatrix} \frac{18}{18 + 17 + 0} \\ \frac{17}{18 + 17 + 0} \\ \frac{0}{18 + 17 + 0} \end{bmatrix}} \approx {\def\arraystretch{1.8}\begin{bmatrix} 0.51 \\ 0.49 \\ 0 \end{bmatrix}}

Here is how this would be written in machine learning resources:

y^i=zi∑j=1Kzj\hat{y}_i = \frac{z_i}{\sum_{j=1}^{K} z_j}

In practice, we use a slightly different approach called the softmax function. For an input vector, z=[z1,z2,…,zK]\mathbf{z}=[z_1,z_2,\dots,z_K], the softmax function is defined as:

y^i=softmax(zi)=ezi∑j=1Kezj\hat{y}_i = \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

This evaluates to a probability distribution, which is a vector of probabilities that sum to 1.

y^=[e18e18+e17+e0e17e18+e17+e0e0e18+e17+e0]≈[0.730.270.00]\hat{y} = {\def\arraystretch{2}\begin{bmatrix} \frac{e^{18}}{e^{18} + e^{17} + e^{0}} \\ \frac{e^{17}}{e^{18} + e^{17} + e^{0}} \\ \frac{e^{0}}{e^{18} + e^{17} + e^{0}} \end{bmatrix}} \approx {\def\arraystretch{2}\begin{bmatrix} 0.73 \\ 0.27 \\ 0.00 \end{bmatrix}}

Notice how the softmax exaggerates the difference between the values? This is one of the reasons it is preferred over the naive approach.

Loss

Now that we have a probability distribution, there is one more function we need to complete our model. The loss function measures how "wrong" our model weights are. Classification models and generative models like the Transformer often use the cross-entropy loss function.

For a single example with a true one-hot label vector y=[y1,…,yK]y = [y_1, \dots, y_K] and predicted probabilities y^=[y^1,…,y^K]\hat{y} = [\hat{y}_1, \dots, \hat{y}_K], the cross-entropy loss is:

L(y,y^)=−∑i=1Kyilog⁡(y^i)L(y, \hat{y}) = - \sum_{i=1}^{K} y_i \log(\hat{y}_i)

With a one-hot label, only the correct class cc has yc=1y_c = 1 (all others are 0), so this reduces to the negative log-likelihood of the model’s probability for the correct class:

y=[100]y = {\def\arraystretch{1}\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}}
L(y,y^)=−log⁡(y^c)L(y, \hat{y}) = - \log(\hat{y}_c)
Example: L=−log⁡(0.73)≈0.313\text{Example: } L = -\log(0.73) \approx 0.313

Softmax Output

Activation:
+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021076111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0180.731170.269-110.000153

Cross-Entropy Loss

True class:
Activation:
+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021076111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0180.7311170.2690-110.0000L ≈ 0.313153

Resources