Part III · Models · Chapter 3C

Fitting a model

Closed-form solutions and gradient descent

1 · The loss: what fitting minimizes

Once a model is specified, you need to fit its parameters: find the weight and bias, in the AR(1) case, that minimize the prediction error on training data. The error is measured by a loss function, most often the mean squared error of Chapter 3A:

Each term is the squared gap between a return and its prediction, averaged over $n$ pairs of previous return and return.

With $b = 0$, expanding the squares for these four pairs gives the whole loss curve at once:

2 · The slope of the loss

To find the best weight you need the slope of the loss curve. The derivative of $L$ with respect to $w$, written $dL/dw$, is the slope of the curve at a point: how fast $L$ changes per unit change in $w$, for a very small change. Three rules cover everything in this chapter: $w^2$ has derivative $2w$, $c\,w$ has derivative $c$ and a constant has derivative zero. Applied to (3C.2), they give

To check it with numbers, note that at $w = 0$ the formula gives a slope of 0.5. Moving from $w = 0$ to $w = 0.01$ raises the loss from 0.5 to 0.50505, a rise of 0.00505 for a step of 0.01, which is 0.505 per unit.

At the bottom of the curve the slope is zero, so the best weight solves $w + 0.5 = 0$, which gives $w = -0.5$. With two parameters, the partial derivative $\partial L/\partial w$ is the slope in the $w$ direction with $b$ held fixed, while $\partial L/\partial b$ is the slope in the $b$ direction. Together they form the gradient, which points in the direction in which the loss rises most steeply.

3 · The closed form: ordinary least squares

For linear regression with squared-error loss there is a one-line analytical solution, ordinary least squares (OLS). Stack the data in a design matrix $X$, whose rows are observations and whose columns are features, with a column of ones for the bias. The targets go in a vector $y$. The OLS estimator of the vector of weights $\beta$ is

Here $X^\top$ is the transpose of $X$, its rows turned into columns, while $(\cdot)^{-1}$ is the matrix inverse, the matrix version of dividing. The result is exact and fast for small datasets, which is why it is used inside scikit-learn's LinearRegression, R's lm() and basically every statistical package's linear regression function.

For the AR(1) case with one feature, (3C.4) collapses to a particularly clean form, with bars for sample means:

That is two numbers, with no iteration and no learning rate: compute the sample covariance and the sample variance, and divide.

Where the AR(1) formulas come from

Step 1: the slope in $b$. Write $x_t = r_{t-1}$ and $y_t = r_t$. The derivative of a square $u^2$ is $2u$ times the derivative of $u$. Since the derivative of $y_t - w x_t - b$ with respect to $b$ is −1, the slope in $b$ is

$$\frac{\partial L}{\partial b} = -\frac{2}{n}\sum_t \big(y_t - w x_t - b\big)$$ (3C.6)

Step 2: set it to zero. Dividing the sum by $n$ turns it into means, so $\bar{y} - w\bar{x} - b = 0$, which gives $b = \bar{y} - w\bar{x}$, the second line of (3C.5).

Step 3: the slope in $w$. The derivative of $y_t - w x_t - b$ with respect to $w$ is $-x_t$, so

$$\frac{\partial L}{\partial w} = -\frac{2}{n}\sum_t x_t\big(y_t - w x_t - b\big)$$ (3C.7)

Step 4: substitute and solve. Put in $b = \bar{y} - w\bar{x}$, so each gap becomes $(y_t - \bar{y}) - w(x_t - \bar{x})$. Setting the slope to zero leaves $\sum (x_t - \bar{x})(y_t - \bar{y}) = w \sum (x_t - \bar{x})^2$, where the leading $x_t$ has become $x_t - \bar{x}$ because deviations from a mean sum to zero. Dividing both sides by $n$ turns them into the sample covariance and variance. So $w$ is their ratio, the first line of (3C.5).

Applied to the five returns, the means of the previous returns and of the returns are both zero. The covariance is $(-0.72 - 0.48 + 0.08 + 0.12)/4 = -0.25$ and the variance of the previous returns is $(0.36 + 1.44 + 0.16 + 0.04)/4 = 0.5$. So $\hat{w} = -0.25/0.5 = -0.5$ and $\hat{b} = 0$, the minimum found from the slope.

Where the closed form breaks

The flaw of OLS is that it requires inverting a matrix of size features × features. While for a model with 5 features and 1,000 observations this is trivial, for a model with 50,000 features and 100 million observations, typical of large-scale machine learning, the matrix inversion becomes infeasible. You cannot load all of the data into memory at once: 100 million rows of 50,000 numbers is $5 \times 10^{12}$ numbers, 40 terabytes at 8 bytes each. And inversion is $O(n^3)$ in the feature count, where $n$ here counts features: its work grows with the cube of their number. Five features cost on the order of 125 operations, while 50,000 cost on the order of $1.25 \times 10^{14}$.

4 · Gradient descent

Gradient descent is iterative, repeating four actions at each step:

1. Compute the current loss on the training data, or on a mini-batch of it.
2. Compute the gradient of the loss with respect to each parameter.
3. Take a small step in the direction opposite to the gradient, downhill on the loss surface.
4. Repeat until the loss stops falling.

The update rule for a parameter $w$ at step $i$ is

where $\eta$ (eta) is the learning rate, the step size. Too large a learning rate and the optimizer overshoots, oscillates and may diverge entirely, while too small a rate makes convergence painfully slow. Tuning $\eta$ is part of the art of optimization: getting it right is most of the challenge in training modern machine learning models.

On the five returns, starting at $w = 0$ with $\eta = 0.5$, the slope is 0.5, so the first step moves $w$ to $0 - 0.5 \times 0.5 = -0.25$. At the new slope of 0.25 the next step reaches −0.375, halving the remaining distance to −0.5 as every step does.

That halving is no accident, and the figure below shows why. It runs gradient descent on the loss $L(w) = (w - 0.5)^2 + 0.2$, whose minimum is at $w = 0.5$ and whose slope is $2(w - 0.5)$. Subtract 0.5 from both sides of (3C.8) to see what one step does to the distance from the minimum:

Every step multiplies the distance by $1 - 2\eta$, so starting from $w = -1$, a distance of 1.5 from the minimum, the distance after $n$ steps is $1.5\,|1 - 2\eta|^n$. Five cases follow from the size of $\eta$:

• Below 0.5, the factor lies between 0 and 1, so the steps shrink and stay on one side.
• At exactly 0.5, the factor is zero and the first step lands on the minimum.
• Between 0.5 and 1, the factor lies between −1 and 0, so each step overshoots and the path zigzags inward.
• At exactly 1, the factor is −1, so the path bounces between −1 and 2 forever.
• Above 1, the factor is below −1, so each overshoot is bigger than the last and the path diverges.

On the five returns the slope, $w + 0.5$, is the distance from −0.5 itself rather than twice it. There each step multiplies the distance by $1 - \eta$, which at $\eta = 0.5$ is the halving you saw.

The figure has two sliders, the learning rate $\eta$ from 0.01 to 1.2 (starting at 0.05) and the number of steps from 1 to 30 (starting at 15). The readouts give the start, where the last step lands and its distance to the optimum. Try learning rates of 0.01, 0.5, 0.9, 1.0 and 1.1 in turn.

learning rate η

0.05

steps

15

start w = −1.000

end w = 0.191

distance to optimum = 0.3088

At the starting settings the last step lands at $w = 0.191$, a distance of 0.3088, which is $1.5 \times 0.9^{15}$. At 0.01 the path crawls to −0.608 in 15 steps. At 0.9 it zigzags to 0.553, the same 0.0528 from the optimum that 0.1 reaches from the other side, because both multiply the distance by 0.8 in size. At 1.0 it ends at 2.000, while at 1.1 it is at 23.6, off the chart.

Mini-batches: stochastic gradient descent

The crucial property of gradient descent is that it does not require loading all the data at once. You can compute the gradient on a mini-batch, a small random subset of the training data, and take a step. The mini-batch gradient is a noisy but unbiased estimate of the full one, so the steps wander but head downhill on average. This is stochastic gradient descent (SGD), which scales to arbitrarily large datasets. Modern variants such as Adam (Kingma and Ba, 2014), RMSProp (Tieleman and Hinton, 2012) and SGD with momentum (Polyak, 1964) all build on this idea. Instead of inverting a giant matrix, they take many small steps with smart adaptations.

5 · The shape of the loss

A loss surface is convex when it is bowl-shaped, so that the straight line between any two points on it lies on or above it, as squared error on a linear model does. For convex loss functions, gradient descent converges to the global optimum from any starting point, provided the learning rate is appropriate. Convex problems are easy.

Non-convex loss functions include everything in deep learning, plus most realistic trading objectives. On them, gradient descent converges to some local minimum that depends on the starting point. Real-world loss surfaces have four hazards:

• Local minima: points where every nearby direction increases the loss, but the global optimum lies elsewhere.
• Saddle points: points where some directions go up and others go down. These are vastly more common than local minima in high-dimensional spaces.
• Plateaus: large regions where the gradient is nearly zero, slowing progress dramatically.
• Cliffs: regions where the gradient is enormous, causing standard SGD to take huge destabilizing steps.

6 · Modern optimizers

The plain SGD update is rarely used in practice, because modern variants address its specific failure modes.

SGD with momentum (Polyak, 1964) accumulates a running average of past gradients and uses it to dampen oscillation and escape saddle points. With $v$ the velocity and $\gamma$ (gamma) a number between 0 and 1:

The velocity keeps you moving in directions that have consistently pointed downhill, even when the current local gradient is noisy. Under a steady gradient $g$, the velocity settles where $v = \gamma v + \eta g$, that is at $v = \eta g/(1 - \gamma)$. With $\gamma = 0.9$, that is $10\,\eta g$: steps ten times the plain ones along a consistent slope.

AdaGrad (Duchi et al., 2011) adapts the learning rate per parameter, dividing it by the square root of the running sum of squared past gradients. Parameters that have already seen big updates get smaller updates from then on.

RMSProp (Tieleman and Hinton, 2012) is like AdaGrad but uses an exponential moving average instead of a cumulative sum, which prevents the learning rate from shrinking to zero.

Adam (Kingma and Ba, 2014) combines momentum with RMSProp. It is the default choice in most modern machine learning codebases: robust, fast and in need of minimal tuning.

You do not need to know the internals of these. You need to know that the optimizer is doing something smarter than vanilla gradient descent and that the choice of optimizer can dramatically affect training speed and final performance. For most trading applications with linear or simple nonlinear models, vanilla SGD or Adam with default hyperparameters works fine.

7 · Why optimization matters for trading

The optimization step is where most of the computational expense in modern quant research lives. Backtesting a strategy means fitting models, generating predictions, simulating execution, computing performance and iterating. Fitting a model on millions of (feature, target) pairs is the slow step. Modern quant firms run massive distributed-training infrastructure precisely because they are optimizing thousands of model variants in parallel.

For retail-scale work, fitting a model on a few thousand observations, optimization is essentially free. The bottleneck is data acquisition, feature engineering and avoiding overfitting, not the optimization step itself.

8 · Regularization

Regularization is a core technique that addresses overfitting at the optimization stage. It adds a penalty term to the loss function that discourages large parameter values. The two most common forms penalize the squared weights (L2 regularization, or ridge regression) or their absolute values (L1 regularization, or the lasso):

Here $L_{\text{data}}$ is the loss of (3C.1) and the $w_i$ are the model's weights. The hyperparameter $\lambda$ (lambda) controls the strength of regularization. Higher $\lambda$ means smaller fitted weights, which means a smoother model with less overfitting risk.

L2 keeps all features but shrinks their weights toward zero. L1 has the additional property that it zeros out many weights entirely, performing feature selection automatically. For trading models with many candidate features, L1 is often the right choice, because it produces sparse, interpretable models.

Ridge is the shrinkage toward zero that Chapter 3A's zero-forecast example argued for, a little bias bought with a large cut in variance. Choose $\lambda$ by walk-forward testing, never by the in-sample fit, which always prefers $\lambda = 0$.

For AR(1), with a single feature, regularization is unnecessary, whereas for multivariate models with dozens of features it is essential.

9 · Check yourself