Build a neural network
Follow numbers through a network, learn with gradients, and check backpropagation. This module has 3 lessons and takes about 1h 30m. Each lesson has a practice exercise that runs real Python in your browser and a short quiz.
Start module 16 in the Learning LabFollow a forward pass
Compute the hidden activations and output of a small neural network.
A neural network repeatedly combines inputs with weights, adds a bias, and applies a function. A hidden layer can build new features before an output layer makes a prediction. Here the inputs are two fictional machine signals already scaled to similar ranges.
The @ operator multiplies matrices. For three machines with two features and three hidden units, X @ W1 has shape (3, 3). ReLU keeps positive values and replaces negative values with zero, which introduces a non-linear step.
The weights below are chosen by hand so you can inspect the arithmetic. These outputs are untrained scores, not calibrated probabilities or evidence of a fault. Training will change the weights using examples and a loss function.
See it in action
import numpy as np
X = np.array([[0.2, 0.4], [0.8, 0.1], [0.9, 0.9]])
W1 = np.array([[1.0, -0.5, 0.3], [0.2, 0.7, -0.8]])
b1 = np.array([0.0, 0.1, 0.2])
W2 = np.array([[0.6], [-0.4], [0.9]])
b2 = np.array([0.1])
hidden_raw = X @ W1 + b1
hidden = np.maximum(0, hidden_raw)
output = hidden @ W2 + b2
print("Hidden activations:", hidden.round(3))
print("Untrained scores:", output.ravel().round(3))
print("Shapes:", X.shape, hidden.shape, output.shape)
Track matrix shapes to understand how information flows through a network.
Learn weights with gradient descent
Train a simple demand relationship by repeatedly moving weights downhill on loss.
Suppose a fictional building's scaled heating demand grows with a temperature-gap feature. The model predicts w * x + b. Mean squared error measures how far those predictions are from the observations.
A gradient tells you how loss changes when a weight changes slightly. Gradient descent subtracts a small learning rate times the gradient. For mean squared error, the weight gradient is 2 * mean(error * x), and the bias gradient is 2 * mean(error).
Print the loss before and after training. Try a much larger learning rate and watch how unstable updates can overshoot. A falling training loss checks optimisation, while a separate test set checks whether the fitted relationship generalises.
See it in action
import numpy as np
x = np.linspace(0, 1, 20)
y = 2.5 * x + 0.7 # Synthetic relationship, not field measurements.
w, b, learning_rate = 0.0, 0.0, 0.2
for step in range(401):
error = w * x + b - y
loss = np.mean(error ** 2)
if step % 100 == 0:
print(step, "MSE:", round(float(loss), 6))
dw = 2 * np.mean(error * x)
db = 2 * np.mean(error)
w -= learning_rate * dw
b -= learning_rate * db
print("Learned slope and bias:", round(float(w), 3), round(float(b), 3))
Loss measures the error, gradients guide the update, and validation measures usefulness.
Backpropagate and check a gradient
Train a hidden-layer network and verify one derivative with a numerical check.
A synthetic calibration curve is y = x squared. A hidden layer with tanh activations can bend its response to model this non-linear relationship. Backpropagation applies the chain rule from the output error back through each layer.
Read the shapes while following the code. dy is the loss derivative at the output. Multiplying by W2.T sends it into the hidden layer, and multiplying by 1 - h squared applies the derivative of tanh. Only update weights after all gradients have been calculated.
The final check perturbs one weight in each direction and estimates its derivative from the loss difference. Close analytical and numerical gradients help catch sign and shape mistakes. This is a tiny network demonstration, not a substitute for a deep-learning framework.
See it in action
import numpy as np
rng = np.random.default_rng(16)
X = np.linspace(-1, 1, 25).reshape(-1, 1)
y = X ** 2
W1 = rng.normal(0, 0.7, (1, 8))
b1 = np.zeros((1, 8))
W2 = rng.normal(0, 0.3, (8, 1))
b2 = np.zeros((1, 1))
def loss():
prediction = np.tanh(X @ W1 + b1) @ W2 + b2
return np.mean((prediction - y) ** 2)
print("Initial MSE:", round(float(loss()), 5))
for step in range(2000):
h = np.tanh(X @ W1 + b1)
prediction = h @ W2 + b2
dy = 2 * (prediction - y) / len(X)
dW2, db2 = h.T @ dy, dy.sum(axis=0, keepdims=True)
dz = (dy @ W2.T) * (1 - h ** 2)
dW1, db1 = X.T @ dz, dz.sum(axis=0, keepdims=True)
W1 -= 0.05 * dW1
b1 -= 0.05 * db1
W2 -= 0.05 * dW2
b2 -= 0.05 * db2
print("Final MSE:", round(float(loss()), 5))
h = np.tanh(X @ W1 + b1)
dy = 2 * (h @ W2 + b2 - y) / len(X)
analytical = (X.T @ ((dy @ W2.T) * (1 - h ** 2)))[0, 0]
old, epsilon = W1[0, 0], 1e-5
W1[0, 0] = old + epsilon
plus = loss()
W1[0, 0] = old - epsilon
minus = loss()
W1[0, 0] = old
numerical = (plus - minus) / (2 * epsilon)
print("Gradient difference:", abs(float(analytical - numerical)))
Check gradients on a tiny example before trusting a larger training run.
Free, open and run by a nonprofit.
This page is part of Python to AI, a free course of 20 modules and 60 lessons in the engineering.fit( ) Learning Lab. The Learning Lab is built and run by engineering.fit( ), a nonprofit founded by Abdullateef Opeyemi Bakare that helps African engineering students gain research experience. You do not need an account or pay a fee to learn.