The Linear Model: Features, Weights and Bias

1 Learning objectives

2 Motivating example

Tutorial 1 left us with a well-posed problem: predict the number of bike rentals at a station in the next hour, from features that will be known at the time the prediction is made.

We have a target, we have features, and we have a baseline. What we do not yet have is a model — a specific mathematical form that takes a feature vector and produces a prediction. There are infinitely many such forms. We could predict that rentals equal the square root of temperature, or that they oscillate sinusoidally with the hour of the day, or that they depend on some complicated interaction between the two.

The linear model is the simplest of these forms, and it is the one we will start with. It says: the prediction is a weighted sum of the features, plus a constant.

That is all it says. There is no hidden structure, no clever transformation, no interaction between features. Each feature contributes independently, in proportion to its value, scaled by its own weight. The model then adds everything up.

ŷ = w₁x₁ + w₂x₂ + … + w_dx_d + b

For the bike-share problem with a single temperature feature, this reduces to the equation of a straight line: ŷ = wx + b. With two features it becomes a flat plane in three dimensions. With many features it is a flat hyperplane in a space you cannot draw — but the arithmetic is unchanged.

You have seen this equation before, in secondary school algebra. What is new is not the mathematics. What is new is the interpretation: the weights are not arbitrary numbers to be found, they are statements about the data, and reading them correctly is most of what makes a linear model useful in practice.

3 Concept explanation

3.1 The hypothesis function

A supervised model is a function that maps a feature vector to a prediction. That function is called the hypothesis, written h(x), and it is parameterised by the weights and the bias.

The subscript on h records the fact that the function depends on the parameters. Change the weights and you get a different hypothesis from the same family. Fitting or training the model means choosing the specific w and b that make the hypothesis fit the training data well. That is the subject of Tutorials 3, 4 and 5.

This tutorial is about the family itself — what it can represent, what the parameters mean, and how to write it down efficiently.

3.2 What a weight means

Take the weight w_j. It multiplies the feature x_j. To understand what it does, ask what happens when x_j increases by one unit while every other feature is held constant. The prediction changes by exactly w_j, because w_j is the only term in the sum that involves x_j.

Three things are worth noticing about that sentence:

• "holding all other features fixed" is essential. With multiple features the coefficient is a partial effect. If two features move together in the data, each coefficient describes what happens when only one of them changes — a situation that may never occur in reality. This is why interpreting coefficients from observational data as causal effects is dangerous.
• The units matter. w_j carries the units of the target divided by the units of the feature. A coefficient of 20.65 in a model predicting rentals from temperature is measured in "rentals per degree Celsius". Change the feature to Fahrenheit and the coefficient changes; the predictions do not.
• The sign is meaningful. A positive coefficient means the feature pushes the prediction up; a negative coefficient pushes it down. Both are informative, and a sign that contradicts domain knowledge is usually a signal that something is wrong with the data or the framing.

3.3 The bias term

The parameter b is called the bias, the intercept, or the offset. It is the value the model predicts when every feature is exactly zero:

h(0) = b

Its role is to let the model shift up or down without changing any of the slopes. Without it, the model would be forced to pass through the origin: if all features are zero, the prediction must be zero. That is occasionally the right constraint — if you are predicting a distance from a known starting point, for example — but it is wrong far more often than it is right.

Mechanically, the bias is easy to overlook because it is not attached to any feature. The cleanest way to handle it is to pretend it is attached to a feature that is always equal to 1. That is exactly what the augmented design matrix does, in Section 3.6.

3.4 From one feature to many

With a single feature, the model is a line: ŷ = wx + b. With two features it is a plane. With three or more it is a hyperplane — a flat surface in a space with as many dimensions as there are features.

The arithmetic does not change. You multiply each feature by its weight, add the results, and add the bias. What changes is only how many terms there are.

The word hyperplane sounds intimidating, but it just means "flat". A hyperplane in four dimensions is no more conceptually difficult than a plane in three; you simply cannot picture it. The algebra is identical, and the vector notation of the next section makes that explicit.

3.5 Vector notation

Writing out a sum with a dozen terms is tedious and error-prone. The vector form is compact and generalises to any number of features:

h_w,b(x) = wᵀx + b

Here w = [w₁, w₂, …, w_d]ᵀ is the weight vector and x = [x₁, x₂, …, x_d]ᵀ is the feature vector for one example. wᵀx is the dot product:

wᵀx  =  w₁x₁ + w₂x₂ + … + w_d x_d

      ⎡ x₁ ⎤
   =  [w₁ w₂ … w_d] ⎢ x₂ ⎥  =  a single number
      ⎢ ⋮  ⎥
      ⎣ x_d⎦

The dot product has a useful geometric interpretation: it measures how much the feature vector aligns with the weight vector. Features whose values point in the same direction as their weights contribute positively to the prediction; features pointing against their weights contribute negatively. A large weight on a feature that is irrelevant to the target will do nothing if that feature's values are consistently near zero — which is why feature selection matters.

3.6 The augmented design matrix

Carrying the bias term separately through every derivation is awkward. The standard trick is to absorb it into the weight vector by introducing an extra feature that is always equal to 1.

Define:

x₀ = 1,    w₀ = b

Then the model becomes a pure dot product with no separate constant:

h(x) = w₀x₀ + w₁x₁ + … + w_dx_d = wᵀx

For the whole dataset at once, stack the feature vectors into the design matrix X:

      ⎡ 1  x₁₁  x₁₂  ⋯  x₁d ⎤         ⎡ w₀ ⎤          ⎡ y₁ ⎤
X  =  ⎢ 1  x₂₁  x₂₂  ⋯  x₂d ⎥ ,   w = ⎢ w₁ ⎥ ,   y =  ⎢ y₂ ⎥
      ⎢ ⋮   ⋮    ⋮   ⋱   ⋮  ⎥         ⎢ ⋮  ⎥          ⎢ ⋮  ⎥
      ⎣ 1  xₙ₁  xₙ₂  ⋯  xₙd ⎦         ⎣ wd ⎦          ⎣ yₙ ⎦

                    ŷ  =  X w          (n × 1)

The design matrix has n rows — one per training example — and d+1 columns: a leading column of ones for the bias, plus one column per feature. The predictions for all n examples come from a single matrix–vector product.

3.7 What "linear" actually means

This is the point in the course where the word linear most often causes confusion, so it is worth being precise.

A linear model is linear in its parameters. It is not necessarily linear in the features.

Consider the model:

ŷ = w₀ + w₁x + w₂x²

Plotted against x, this is a parabola — clearly curved. But as a function of the parameters w₀, w₁ and w₂, it is a weighted sum. Define x₁ = x and x₂ = x², and it becomes exactly the linear model from Section 3.1. The weights enter linearly, so every technique in this unit applies without modification.

The practical consequence is large. By constructing new features — squares, logs, products, indicator variables — you can fit a rich family of curved and interacting relationships using nothing more than the linear machinery of this unit. The engineering effort moves from the optimisation to the feature construction, which is where a great deal of applied machine learning actually happens.

3.8 Scaling changes the weights, not the model

Suppose you measure temperature in Celsius and fit the bike-share model, obtaining a slope of about 20.65 rentals per degree. Now convert the feature to Fahrenheit and refit. The new slope will be approximately 20.65 / 1.8 ≈ 11.47, because a one-degree change in Fahrenheit is a smaller physical change than one degree Celsius.

The predictions are identical. The model has not changed; only its parameterisation has. This has two consequences worth internalising:

• You cannot compare coefficient magnitudes across features measured in different units. A coefficient of 1,000 on a feature measured in millimetres and a coefficient of 1 on a feature measured in metres may represent the same physical effect. If you want to compare the relative importance of features, standardise them first — then all the coefficients are on the same scale.
• Feature scaling changes the loss surface. Tutorial 4 will show that the conditioning of the surface depends on the scale of the features, and Tutorial 5 will demonstrate that gradient descent can need tens of thousands of times more iterations on unscaled features than on scaled ones. The model is unchanged; the difficulty of finding it is not.

3.9 Limits of the linear model

A linear model assumes that the effect of each feature on the target is additive and constant. Both assumptions are often wrong.

The remedies in the third column all keep the model linear in its parameters, which means everything you learn in the rest of this unit still applies. But each one adds parameters, and each one therefore moves you closer to the overfitting problem. The discipline is to add complexity only when the residuals demand it — which is the subject of Tutorial 6.

4 Visual illustration

5 Worked example

We will build and interpret a two-feature linear model for the bike-share problem, then compute predictions by hand and reason about what the coefficients mean.

w₁  =  18.40      rentals per °C
w₂  =  62.50      rentals on weekends vs weekdays
b   = −40.20      rentals

Model:  ŷ = 18.40 · temperature + 62.50 · is_weekend − 40.20

x₁ = 20 ,  x₂ = 0

ŷ = 18.40(20) + 62.50(0) − 40.20
  = 368.00 + 0 − 40.20
  = 327.80 rentals

x₁ = 20 ,  x₂ = 1

ŷ = 18.40(20) + 62.50(1) − 40.20
  = 368.00 + 62.50 − 40.20
  = 390.30 rentals

x₁ = 12 ,  x₂ = 1

ŷ = 18.40(12) + 62.50(1) − 40.20
  = 220.80 + 62.50 − 40.20
  = 243.10 rentals

      ⎡ 1  20  0 ⎤            ⎡ −40.20 ⎤
X  =  ⎢ 1  20  1 ⎥ ,   w  =   ⎢  18.40 ⎥
      ⎢ 1  12  1 ⎥            ⎣  62.50 ⎦
      ⎣ 1  12  0 ⎦

ŷ = Xw

  ⎡ 1  20  0 ⎤ ⎡ −40.20 ⎤     ⎡ 1(−40.20) + 20(18.40) + 0(62.50)  ⎤     ⎡ 327.80 ⎤
  ⎢ 1  20  1 ⎥ ⎢  18.40 ⎥  =  ⎢ 1(−40.20) + 20(18.40) + 1(62.50)  ⎥  =  ⎢ 390.30 ⎥
  ⎢ 1  12  1 ⎥ ⎢  62.50 ⎥     ⎢ 1(−40.20) + 12(18.40) + 1(62.50)  ⎥     ⎢ 243.10 ⎥
  ⎣ 1  12  0 ⎦ ⎣         ⎦    ⎣ 1(−40.20) + 12(18.40) + 0(62.50)  ⎦     ⎣ 180.60 ⎦

The same computation in code

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    "temp_c":     [20, 20, 12, 12],
    "is_weekend": [ 0,  1,  1,  0],
    "rentals":    [330, 395, 248, 186],
})

X = df[["temp_c", "is_weekend"]].to_numpy()
y = df["rentals"].to_numpy()

# --- Route 1: the augmented design matrix, by hand ---
n = len(X)
X_aug = np.column_stack([np.ones(n), X])          # prepend a column of ones

w_aug = np.linalg.solve(X_aug.T @ X_aug, X_aug.T @ y)
b, w1, w2 = w_aug

print(f"by hand  : ŷ = {w1:.2f}·temp + {w2:.2f}·weekend + ({b:.2f})")

# --- Route 2: scikit-learn ---
model = LinearRegression().fit(X, y)
print(f"sklearn  : ŷ = {model.coef_[0]:.2f}·temp "
      f"+ {model.coef_[1]:.2f}·weekend + ({model.intercept_:.2f})")

# --- Predict three cases ---
cases = pd.DataFrame({
    "case":       ["Wed 20°C", "Sat 20°C", "Sun 12°C"],
    "temp_c":     [20, 20, 12],
    "is_weekend": [ 0,  1,  1],
})
cases["predicted"] = model.predict(cases[["temp_c", "is_weekend"]])
print()
print(cases.to_string(index=False))

# --- Fitted values and residuals on the training data ---
df["y_hat"]    = model.predict(X)
df["residual"] = df["rentals"] - df["y_hat"]
print()
print(df.to_string(index=False))
print(f"\nRMSE = {np.sqrt((df['residual'] ** 2).mean()):.2f} rentals")

by hand  : ŷ = 18.40·temp + 62.50·weekend + (-40.20)
sklearn  : ŷ = 18.40·temp + 62.50·weekend + (-40.20)

      case  temp_c  is_weekend  predicted
  Wed 20°C      20           0     327.80
  Sat 20°C      20           1     390.30
  Sun 12°C      12           1     243.10

 temp_c  is_weekend  rentals   y_hat  residual
     20           0      330  327.80      2.20
     20           1      395  390.30      4.70
     12           1      248  243.10      4.90
     12           0      186  180.60      5.40

RMSE = 4.59 rentals

The RMSE of 4.59 rentals is far lower than the one-feature model's 10.2 on the eight-point dataset — but note that these are different datasets of different sizes, so the comparison is not meaningful. The point of this example is the interpretation of the coefficients, not the fit quality.

6 Interactive activity

7 Self-check questions

Answer each question in your own words before revealing the answer.