From neurons to networks

Neural networks sound mysterious, but they're built from a very simple part repeated many times. Let's start with that part.

One artificial neuron

A neuron takes several numbers as input, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function.

output = activation(w1×x1 + w2×x2 + w3×x3 + b)

In a real network, nobody sets the weights by hand. They are learned from data. That's the whole point.

Activation functions

Without activation functions, stacking neurons would only ever produce straight-line relationships. Activations add the "bends" that let networks learn complex patterns.

Activation	What it does	Typical use
ReLU	Outputs the input if positive, otherwise 0	Hidden layers. The default choice.
Sigmoid	Squashes any number into 0 to 1	Output for yes/no probability
Softmax	Turns a list of numbers into probabilities that add up to 1	Output for choosing among several classes

Layers make a network

• Input layer: the raw data, such as the 784 pixel values of a 28×28 image.
• Hidden layers: neurons that transform the data step by step. "Deep" learning means many hidden layers.
• Output layer: the prediction, such as 10 probabilities for the digits 0 to 9.

In a dense (fully connected) layer, every neuron receives every output from the previous layer. A network with 784 inputs, a hidden layer of 128 neurons and 10 outputs already has about 100,000 weights. Large language models have billions.

Why depth helps

Each layer builds on the one before. In an image network, early layers learn edges and colours, middle layers combine them into textures and shapes, and later layers recognise objects. The network discovers these features itself, which is why deep learning replaced hand-written feature engineering for images, audio and text.

A neuron in code

import numpy as np

def relu(z):
    return np.maximum(0, z)

x = np.array([1, 0, 2])          # inputs
w = np.array([3, 2, -1.5])       # weights
b = -1                           # bias
print(relu(np.dot(w, x) + b))    # 0.0 — the neuron stays "off"

# a whole layer is just a matrix multiplication
W = np.random.randn(4, 3)        # 4 neurons, 3 inputs each
B = np.zeros(4)
print(relu(W @ x + B))           # 4 outputs

That last line is the heart of deep learning: multiply by a weight matrix, add a bias, apply an activation. GPUs are so important because they do huge matrix multiplications very fast.

Key takeaways

• A neuron computes a weighted sum plus bias, then applies an activation.
• ReLU for hidden layers, sigmoid or softmax for outputs.
• Layers stack to learn features from simple to complex. Weights are learned, not set.