Free preview

192 lessons

Essential Calculus for ML

Free preview

Essential Calculus for ML · 192 lessons

Master the calculus behind how ML models learn

No surprise gaps

Actually remember it

Skip what you know

One subscription. All learning paths included.

Our content is best on a larger screen

Tangent planes and linear approximation

The tangent plane to a surface

Explanation

A curve has one slope at a point, and the tangent line there has the curve's height and that slope. A surface $z = f(x,y)$ has a cross-section through a point in each input direction, so matching the surface at that point requires a height and two slopes.

The diagram below builds this plane step by step.

0,0

az = 1.08

el = 0.42

bank = 0.00

x

y

z

Step 1: The point on the surface

The surface $z = x^{2} + 2 y^{2}$ has height $f (1, 1) = 3$ above $(1, 1)$ , so it passes through the point $(1, 1, 3)$ .

Drag the scene to rotate it.

Step 2: The cross-section at $y = 1$

The grey plane $y = 1$ cuts the surface in the curve $z = x^{2} + 2$ . The slope of its tangent line at the point is

f_{x} (1, 1) = 2

Drag the scene to rotate it.

Step 3: The cross-section at $x = 1$

The grey plane $x = 1$ cuts the surface in the curve $z = 1 + 2 y^{2}$ . The slope of its tangent line at the point is

f_{y} (1, 1) = 4

Drag the scene to rotate it.

Step 4: The plane through both tangent lines

The plane containing both tangent lines passes through $(1, 1, 3)$ with slope $2$ in the $x$ -direction and slope $4$ in the $y$ -direction. In point-slope form it is

z = 3 + 2 (x - 1) + 4 (y - 1)

Drag the scene to rotate it.

Tangent plane

Definition

The tangent plane to $z = f(x,y)$ at $(a,b)$ is the plane containing the tangent lines to the cross-sections $x = a$ and $y = b$ through the point $(a, b, f(a,b))$ .

Equation of the tangent plane

Theorem

The tangent plane to $z = f(x,y)$ at $(a,b)$ has equation

z = f(a,b) + f_x(a,b)(x-a) + f_y(a,b)(y-b).

This is the point-slope form of a plane: the surface's height at the point, plus one slope term for each input direction.

Example

For $f(x,y) = xy^2 + 3y$ , write the equation of the tangent plane at $(1,2)$ .

Solution

We need three numbers at $(1,2)$ : the height $f(1,2)$ , and the two cross-section slopes $f_x(1,2)$ and $f_y(1,2)$ .

For the height we evaluate $f$ at the point:

\begin{align*} f(1,2) &= 1(2)^2 + 3(2) \\ &= 10. \end{align*}

Holding $y$ fixed we find $f_x = y^2$ , and we substitute the point:

\begin{align*} f_x(1,2) &= 2^2 \\ &= 4. \end{align*}

Holding $x$ fixed we find $f_y = 2xy + 3$ , and we substitute the point again:

\begin{align*} f_y(1,2) &= 2(1)(2) + 3 \\ &= 7. \end{align*}

With $a = 1$ and $b = 2$ , we substitute the three numbers into

z = f(a,b) + f_x(a,b)(x-a) + f_y(a,b)(y-b)

to get

z = 10 + 4(x-1) + 7(y-2).

Practice questions

4 questions

A function $f$ of two variables has $f(-1,4) = 5$ , $f_x(-1,4) = -3$ and $f_y(-1,4) = 2$ . Write the equation of the tangent plane to $z = f(x,y)$ at $(-1,4)$ .

Select the correct answer:

$z = 5 - 3(x-1) + 2(y-4)$

$z = -1 + 5(x+1) + 4(y-4)$

$z = 5 - 3(x+1) + 2(y-4)$

$z = 5 + 2(x+1) - 3(y-4)$

+ 3 more questions

Linear approximation in two variables

Explanation

Near a point where $f$ is differentiable, the graph of a function of one variable lies very close to its tangent line. In the same way, near $(a,b)$ the surface $z = f(x,y)$ lies very close to its tangent plane. A plane's height takes only a few multiplications and additions to evaluate, so near $(a,b)$ we can work with the tangent plane in place of $f$ .

The diagram below zooms in on a point of a surface and its tangent plane.

0,0

az = 1.08

el = 0.42

bank = 0.00

x

y

z

0

1

1

2

2

The surface $z = x^{2} + 2 y^{2}$ and its tangent plane at the point $(1, 1, 3)$ . Drag the slider to zoom in on the point: close to $(1, 1)$ , the surface flattens onto the plane.

Largest gap between the surface and the plane in view: 3.002.742.502.282.081.891.731.571.441.311.191.090.9930.9060.8260.7540.6870.6270.5720.5210.4750.4340.3950.3610.3290.3000.2740.2500.2280.2080.1890.1730.1570.1440.1310.1190.1090.09930.09060.08260.07540.06870.06270.05720.05210.04750.04340.03950.03610.03290.03000.02740.02500.02280.02080.01890.01730.01570.01440.01310.01190.01090.009930.009060.008260.007540.006870.006270.005720.005210.004750.004340.003950.003610.003290.003000.002740.002500.002280.002080.001890.001730.001570.001440.001310.001190.001090.0009930.0009060.0008260.0007540.0006870.0006270.0005720.0005210.0004750.0004340.0003950.0003610.0003290.000300 $3.00$ $2.74$ $2.50$ $2.28$ $2.08$ $1.89$ $1.73$ $1.57$ $1.44$ $1.31$ $1.19$ $1.09$ $0.993$ $0.906$ $0.826$ $0.754$ $0.687$ $0.627$ $0.572$ $0.521$ $0.475$ $0.434$ $0.395$ $0.361$ $0.329$ $0.300$ $0.274$ $0.250$ $0.228$ $0.208$ $0.189$ $0.173$ $0.157$ $0.144$ $0.131$ $0.119$ $0.109$ $0.0993$ $0.0906$ $0.0826$ $0.0754$ $0.0687$ $0.0627$ $0.0572$ $0.0521$ $0.0475$ $0.0434$ $0.0395$ $0.0361$ $0.0329$ $0.0300$ $0.0274$ $0.0250$ $0.0228$ $0.0208$ $0.0189$ $0.0173$ $0.0157$ $0.0144$ $0.0131$ $0.0119$ $0.0109$ $0.00993$ $0.00906$ $0.00826$ $0.00754$ $0.00687$ $0.00627$ $0.00572$ $0.00521$ $0.00475$ $0.00434$ $0.00395$ $0.00361$ $0.00329$ $0.00300$ $0.00274$ $0.00250$ $0.00228$ $0.00208$ $0.00189$ $0.00173$ $0.00157$ $0.00144$ $0.00131$ $0.00119$ $0.00109$ $0.000993$ $0.000906$ $0.000826$ $0.000754$ $0.000687$ $0.000627$ $0.000572$ $0.000521$ $0.000475$ $0.000434$ $0.000395$ $0.000361$ $0.000329$ $0.000300$

Drag the scene to rotate it.

Zoom ×1×1.1×1.2×1.3×1.4×1.5×1.6×1.7×1.8×1.9×2×2.1×2.2×2.3×2.4×2.5×2.6×2.8×2.9×3×3.2×3.3×3.5×3.6×3.8×4×4.2×4.4×4.6×4.8×5×5.2×5.5×5.8×6×6.3×6.6×6.9×7.2×7.6×7.9×8.3×8.7×9.1×9.5×10×11×12×13×14×15×16×17×18×19×20×21×22×23×24×25×26×28×29×30×32×33×35×36×38×40×42×44×46×48×50×52×55×58×60×63×66×69×72×76×79×83×87×91×95×100

\times 1

\times 1.1

\times 1.2

\times 1.3

\times 1.4

\times 1.5

\times 1.6

\times 1.7

\times 1.8

\times 1.9

\times 2

\times 2.1

\times 2.2

\times 2.3

\times 2.4

\times 2.5

\times 2.6

\times 2.8

\times 2.9

\times 3

\times 3.2

\times 3.3

\times 3.5

\times 3.6

\times 3.8

\times 4

\times 4.2

\times 4.4

\times 4.6

\times 4.8

\times 5

\times 5.2

\times 5.5

\times 5.8

\times 6

\times 6.3

\times 6.6

\times 6.9

\times 7.2

\times 7.6

\times 7.9

\times 8.3

\times 8.7

\times 9.1

\times 9.5

\times 10

\times 11

\times 12

\times 13

\times 14

\times 15

\times 16

\times 17

\times 18

\times 19

\times 20

\times 21

\times 22

\times 23

\times 24

\times 25

\times 26

\times 28

\times 29

\times 30

\times 32

\times 33

\times 35

\times 36

\times 38

\times 40

\times 42

\times 44

\times 46

\times 48

\times 50

\times 52

\times 55

\times 58

\times 60

\times 63

\times 66

\times 69

\times 72

\times 76

\times 79

\times 83

\times 87

\times 91

\times 95

\times 100

The tangent plane's equation gives its height at every point $(x,y)$ . Read as a function of $(x,y)$ , it is the linearisation.

The linearisation of

f

(a,b)

Definition

The linearisation of $f$ at $(a,b)$ is the function giving the tangent plane's height,

L(x,y) = f(a,b) + f_x(a,b)(x-a) + f_y(a,b)(y-b).

For $(x,y)$ near $(a,b)$ , $f(x,y) \approx L(x,y)$ .

We estimate a value of $f$ in three steps, as with one input.

Estimating

f(x,y)

by linear approximation

Procedure

Take a point $(a,b)$ near $(x,y)$ where $f(a,b)$ , $f_x(a,b)$ and $f_y(a,b)$ are known exactly.
Build the linearisation $L(x,y) = f(a,b) + f_x(a,b)(x-a) + f_y(a,b)(y-b)$ .
Evaluate $L$ at $(x,y)$ , which gives the estimate $f(x,y) \approx L(x,y)$ .

The estimate is exact at $(a,b)$ , since $L(a,b) = f(a,b)$ , and near $(a,b)$ it usually gets worse as $(x,y)$ moves further away.

Example

For $f(x,y) = x^3 + xy^2$ , estimate $f(2.02, 0.97)$ using the linearisation at $(2,1)$ , then compare the estimate with the true value.

Solution

We start with the height at the base point:

\begin{align*} f(2,1) &= 2^3 + 2(1)^2 \\ &= 10. \end{align*}

Holding $y$ fixed we find $f_x = 3x^2 + y^2$ , and holding $x$ fixed we find $f_y = 2xy$ . At the base point:

\begin{align*} f_x(2,1) &= 3(2)^2 + 1^2 = 13, \\ f_y(2,1) &= 2(2)(1) = 4. \end{align*}

With $a = 2$ and $b = 1$ , we assemble those three numbers into the linearisation:

L(x,y) = 10 + 13(x-2) + 4(y-1).

Substituting $(2.02, 0.97)$ into $L$ :

\begin{align*} L(2.02, 0.97) &= 10 + 13(2.02 - 2) + 4(0.97 - 1) \\ &= 10 + 13(0.02) + 4(-0.03) \\ &= 10 + 0.26 - 0.12 \\ &= 10.14. \end{align*}

So $f(2.02, 0.97) \approx 10.14$ .

To compare, a calculator gives $f(2.02, 0.97) = 10.14303$ to five decimal places. Our estimate differs from the true value by about $0.003$ , at a point only $0.02$ and $0.03$ away from $(2,1)$ in the two inputs.

Practice questions

4 questions

For $f(x,y) = x^2y$ , estimate $f(1.98, 3.01)$ working from the point $(2,3)$ .

Select the correct answer:

$12.04$

$11.72$

$12.28$

$11.80$

+ 3 more questions

Part of

Optimisation Foundations

See all 10 lessons, what comes first, and what this unlocks.

Master the calculus behind how ML models learn

Tangent planes and linear approximation

Explanation

Step 1: The point on the surface

Step 2: The cross-section at y=1

Step 3: The cross-section at x=1

Step 4: The plane through both tangent lines

Example

Explanation

Example

Optimisation Foundations

Step 2: The cross-section at $y = 1$

Step 3: The cross-section at $x = 1$