University Mathematics — Year 1 · Bachelor Year 1

25Functions of Two Variables

The year ends with a first walk into higher dimension: functions $f(x, y)$ of two real variables. Everything generalizes — limits, continuity, derivatives, extrema — but each notion gains a twist: limits can be approached along every direction at once, derivatives split into partials, and the gradient points the way uphill. The full theory (differentials, general $\R^n$ , submanifolds) belongs to the second year; here we set the vocabulary and the first honest theorems.

25.1 The plane $\R^2$ ; continuity

Definition 25.1

On $\R^2$ , use the Euclidean norm $\norm{(x,y)} = \sqrt{x^2 + y^2}$ (Chapter 23). Open balls, neighborhoods, open subsets of $\R^2$ are defined exactly as in Chapter 12, with balls in place of intervals. A function $f \colon U \to \R$ ( $U \subseteq \R^2$ open) is continuous at $a \in U$ when

\forall\varepsilon > 0,\ \exists\delta > 0, \quad \norm{X - a} \leq \delta \implies \abs{f(X) - f(a)} \leq \varepsilon,

with the same sequential characterization as in one variable. Sums, products, quotients and compositions with continuous one-variable functions preserve continuity; the coordinate maps are continuous, hence so are polynomials in $(x,y)$ .

Example 25.2 (The polar bound, the clean way to prove a limit)

Show that $f(x, y) = \dfrac{x^2y^2}{x^2 + y^2}$ (with $f(0,0) = 0$ ) is continuous at the origin. In polar coordinates $x = \rho\cos\theta$ , $y = \rho\sin\theta$ :

\abs{f} = \frac{\rho^4\cos^2\theta\sin^2\theta}{\rho^2} = \rho^2\,(\cos\theta\sin\theta)^2 \leq \frac{\rho^2}{4} \xrightarrow[\rho \to 0]{} 0 ,

a bound independent of $\theta$ : whatever the direction of approach, the values are squeezed to $0$ . That uniformity in $\theta$ is the whole point — a bound like $\abs g = \abs{\cos\theta\sin\theta}$ (no $\rho$ left) proves nothing, and indeed that $g$ is the discontinuous radial trap of the next example.

Example 25.3 (The radial trap)

Let $f(x, y) = \dfrac{xy}{x^2 + y^2}$ for $(x,y) \neq (0,0)$ , $f(0, 0) = 0$ . Along each axis, $f = 0 \to 0$ ; but along the diagonal $y = x$ , $f(x, x) = \frac12 \not\to 0$ . No limit at the origin: approaching along every line, and even finding the same limit along each, is not enough (here the line limits disagree; worse examples agree along all lines yet fail along a parabola, Exercise 25.3). Continuity in each variable separately does not imply continuity.

25.2 Partial derivatives

Definition 25.4

The partial derivatives of $f$ at $(a, b)$ are the one-variable derivatives along the axes:

\frac{\partial f}{\partial x}(a,b) = \lim_{h \to 0} \frac{f(a + h, b) - f(a,b)}{h}, \qquad \frac{\partial f}{\partial y}(a,b) = \lim_{k \to 0} \frac{f(a, b + k) - f(a,b)}{k}.

$f$ is of class $C^1$ on $U$ when both exist and are continuous on $U$ . The gradient is $\nabla f(a,b) = \bigl(\frac{\partial f}{\partial x},\, \frac{\partial f}{\partial y}\bigr)(a,b)$ .

Theorem 25.5 ( $C^1$ implies a tangent plane)

Let $f$ be $C^1$ on $U$ and $(a,b) \in U$ . Then, as $(h, k) \to (0,0)$ :

f(a + h, b + k) = f(a, b) + h\,\frac{\partial f}{\partial x}(a,b) + k\,\frac{\partial f}{\partial y}(a,b) + o\bigl(\norm{(h,k)}\bigr) .

In particular $f$ is continuous, and the graph $z = f(x,y)$ has at each point the tangent plane read off the formula.

Example 25.6 (Linear approximation in action)

Estimate $f(1.02,\ 0.99)$ for $f(x, y) = x^3y^2$ . At $(1, 1)$ : $f = 1$ , $\frac{\partial f}{\partial x} = 3x^2y^2 = 3$ , $\frac{\partial f}{\partial y} = 2x^3y = 2$ , so Theorem 25.5 gives

f(1.02,\ 0.99) \approx 1 + 3\,(0.02) + 2\,(-0.01) = 1.04 ,

against the true value $1.02^3 \times 0.99^2 = 1.04006\dots$ — the error is of second order in the increments, as the $o(\norm{(h,k)})$ promises. The tangent plane to the graph at $(1, 1, 1)$ is $z = 1 + 3(x - 1) + 2(y - 1)$ , the equation implicit in the estimate.

Proof. Move one coordinate at a time:

f(a+h, b+k) - f(a,b) = \bigl[f(a+h, b+k) - f(a, b+k)\bigr] + \bigl[f(a, b+k) - f(a,b)\bigr].

By the one-variable mean value theorem (Theorem 14.9), the first bracket is $h\, \frac{\partial f}{\partial x}(a + \theta h,\, b + k)$ for some $\theta \in \intoo{0}{1}$ , and the second is $k\,\frac{\partial f}{\partial y}(a,\, b + \theta' k)$ . Continuity of the partials at $(a,b)$ lets us write each as (value at $(a,b)$ ) $+$ (error $\to 0$ ); the total error is $h\,\varepsilon_1 + k\,\varepsilon_2 = o(\norm{(h,k)})$ since $\abs h, \abs k \leq \norm{(h,k)}$ . ∎

Theorem 25.7 (Chain rule)

Let $f$ be $C^1$ on $U$ and $t \mapsto (x(t), y(t))$ be $C^1$ from an interval into $U$ . Then $g(t) = f\bigl(x(t), y(t)\bigr)$ is $C^1$ , with

g'(t) = x'(t)\,\frac{\partial f}{\partial x}\bigl(x(t),y(t)\bigr) + y'(t)\,\frac{\partial f}{\partial y}\bigl(x(t),y(t)\bigr) = \bigl\langle \nabla f,\ (x', y')\bigr\rangle .

Proof. Apply Theorem 25.5 at $(x(t), y(t))$ with $(h, k) = (x(t+s) - x(t),\, y(t+s) - y(t))$ : as $s \to 0$ , one-variable differentiability gives $h = s\,x'(t) + o(s)$ and $k = s\,y'(t) + o(s)$ , so $\norm{(h, k)} = O(s)$ and

g(t+s) - g(t) = h\,\frac{\partial f}{\partial x} + k\,\frac{\partial f}{\partial y} + o\bigl(\norm{(h,k)}\bigr) = s\,\Bigl(x'\,\frac{\partial f}{\partial x} + y'\,\frac{\partial f}{\partial y}\Bigr) + o(s) ,

the final error absorbing both the $o(s)$ ’s of $h$ and $k$ (multiplied by the fixed values of the partials) and the $o(O(s))$ of the tangent-plane estimate. Divide by $s$ and let $s \to 0$ . Continuity of $g'$ follows from that of all ingredients. ∎

Example 25.8 (The chain rule, verified both ways)

Let $f(x, y) = x^2 y$ and $g(t) = f(t, t^2)$ . Directly: $g(t) = t^2\cdot t^2 = t^4$ , so $g'(t) = 4t^3$ . By the chain rule: $\frac{\partial f}{\partial x} = 2xy$ and $\frac{\partial f}{\partial y} = x^2$ , evaluated along the curve $(t, t^2)$ :

g'(t) = 1\cdot 2t\cdot t^2 + 2t\cdot t^2 = 2t^3 + 2t^3 = 4t^3 .

The two computations agree, and the split is meaningful: $2t^3$ of the growth comes from moving rightward through the $x$ -slope, $2t^3$ from moving upward through the $y$ -slope. On curves where no closed form for $g$ exists, only the second computation survives — that is the theorem’s point.

Remark 25.9 (Reading the gradient)

Along a unit direction $u$ , the chain rule on $t \mapsto f(a + tu)$ gives the directional derivative $\langle \nabla f(a), u\rangle$ : maximal when $u$ points along $\nabla f(a)$ (by Cauchy–Schwarz, Theorem 23.4). The gradient is the direction of steepest ascent, and it is orthogonal to the level curves $\{f = c\}$ (differentiate $f$ along a curve drawn in a level set: the chain rule gives $\langle\nabla f,\, \text{tangent}\rangle = 0$ ).

Example 25.10 (Level curves and gradients, on one function)

Take $f(x, y) = x^2 - y^2$ . Its level sets: $\{f = c\}$ is a hyperbola opening left-right for $c > 0$ , up-down for $c < 0$ , and the crossed pair of lines $y = \pm x$ for $c = 0$ — the contour map of a mountain pass, with the saddle point at the origin where the two zero-level lines cross. Gradient: $\nabla f = (2x, -2y)$ . At the point $(2, 1)$ (on the level $c = 3$ ): $\nabla f = (4, -2)$ , while the tangent vector of the level curve, parametrized near that point by $\bigl(t, \sqrt{t^2 - 3}\bigr)$ , is $\bigl(1, \frac{t}{\sqrt{t^2 - 3}}\bigr) = (1, 2)$ at $t = 2$ — and indeed

\langle (4, -2),\ (1, 2)\rangle = 4 - 4 = 0 :

gradient perpendicular to the contour, pointing toward higher values of $f$ (here: away from the $y$ -axis). Two more readings: the gradient vanishes exactly at the saddle, where the contour map pinches; and the tangent line to the level curve at $(2,1)$ is $4(x - 2) - 2(y - 1) = 0$ , i.e. $2x - y = 3$ — the equation “ $\langle \nabla f,\ M - M_0\rangle = 0$ ” that generalizes the ellipse tangent of Exercise 24.11.

Theorem 25.11 (Schwarz)

If $f$ is of class $C^2$ (partials of partials exist and are continuous), then

\frac{\partial^2 f}{\partial x\,\partial y} = \frac{\partial^2 f}{\partial y\,\partial x} .

Proof. Admitted at this level. ∎

25.3 Local extrema

Method 25.12 (Extremum studies, organized)

Solve $\nabla f = 0$ completely. Factor each partial whenever possible (products of linear factors split the system into transparent cases, as in Example 25.16 below); a forgotten case is a forgotten critical point.
Classify each point with the Monge data $r, s, t$ — recomputed at each point, never once for all.
If $rt - s^2 = 0$ , examine $f$ directly along well-chosen curves through the point (lines first, then parabolas), hunting either for two signs (no extremum) or for a locked sign with an argument covering all directions.
Step back for the global picture: check the behavior at infinity (a local minimum may be no global one), and if the domain is not open, treat its boundary separately (Exercise 25.12) — the critical point theorem sees interior points only.

Theorem 25.13 (Critical points)

If $f$ ( $C^1$ on the open set $U$ ) has a local extremum at $(a,b) \in U$ , then $\nabla f(a,b) = (0,0)$ : the point is critical.

Proof. The one-variable functions $x \mapsto f(x, b)$ and $y \mapsto f(a, y)$ have interior local extrema at $a$ , resp. $b$ : Proposition 14.7 kills both partials. ∎

Method 25.14 (Second-order test (Monge notation))

At a critical point of a $C^2$ function, set

r = \frac{\partial^2 f}{\partial x^2}, \qquad s = \frac{\partial^2 f}{\partial x \partial y}, \qquad t = \frac{\partial^2 f}{\partial y^2} \qquad (\text{values at the point}).

If $rt - s^2 > 0$ : local extremum — minimum for $r > 0$ , maximum for $r < 0$ ;
if $rt - s^2 < 0$ : no extremum (a saddle point);
if $rt - s^2 = 0$ : the test is silent; examine directly.

(Justification — a Taylor–Young expansion at order $2$ and the sign study of the quadratic form $r h^2 + 2shk + tk^2$ — is carried out in the second year; the test is used here as a working tool.)

Example 25.15

$f(x,y) = x^3 + y^3 - 3xy$ . Critical points: $\nabla f = (3x^2 - 3y,\; 3y^2 - 3x) = 0$ gives $y = x^2$ and $x = y^2$ , so $x = x^4$ : $x \in \{0, 1\}$ : points $(0,0)$ and $(1,1)$ .

Second derivatives: $r = 6x$ , $s = -3$ , $t = 6y$ . At $(0,0)$ : $rt - s^2 = -9 < 0$ : saddle. At $(1,1)$ : $rt - s^2 = 36 - 9 > 0$ , $r = 6 > 0$ : local minimum, $f(1,1) = -1$ . (Not global: $f(x, 0) = x^3 \to -\infty$ .)

Example 25.16 (A four-point study, in full)

$f(x, y) = xy\,(3 - x - y) = 3xy - x^2y - xy^2$ . Gradient:

\frac{\partial f}{\partial x} = y\,(3 - 2x - y), \qquad \frac{\partial f}{\partial y} = x\,(3 - x - 2y).

Critical points: if $y = 0$ , the second equation gives $x \in \{0, 3\}$ ; if $x = 0$ , the first gives $y \in \{0, 3\}$ ; if $xy \neq 0$ , solve $2x + y = 3$ , $x + 2y = 3$ : $x = y = 1$ . Four points: $(0,0)$ , $(3,0)$ , $(0,3)$ , $(1,1)$ . Second derivatives: $r = -2y$ , $s = 3 - 2x - 2y$ , $t = -2x$ .

$(1,1)$ : $r = -2$ , $s = -1$ , $t = -2$ : $rt - s^2 = 3 > 0$ , $r < 0$ : local maximum, $f(1,1) = 1$ .
$(0,0)$ : $r = t = 0$ , $s = 3$ : $rt - s^2 = -9 < 0$ : saddle; likewise $(3, 0)$ ( $s = -3$ ) and $(0, 3)$ : three saddles.

The maximum is only local: $f(-T, -T) = T^2(3 + 2T) \to +\infty$ . Symmetry check: $f(x, y) = f(y, x)$ , and indeed the critical set and the classification are symmetric in $x \leftrightarrow y$ . Interpretation: among rectangles-with-slack $x, y \geq 0$ , $x + y \leq 3$ , the product $xy(3 - x - y)$ of the three “parts” of $3$ is largest when the parts are equal — a two-variable shadow of the AM–GM principle.

The two curves f = 0 for f = x3 + y3 - 3xy intersect at the critical points (0,0) (saddle) and (1,1) (local minimum). — The two curves $\nabla f = 0$ for $f = x^3 + y^3 - 3xy$ intersect at the critical points $(0,0)$ (saddle) and $(1,1)$ (local minimum).

Remark 25.17 (Common pitfalls)

Partial derivatives may exist at a point of discontinuity: the radial trap $g(x,y) = \frac{xy}{x^2+y^2}$ of Example 25.3 has $\frac{\partial g}{\partial x}(0,0) = \frac{\partial g}{\partial y}(0,0) = 0$ (both axis restrictions vanish identically), yet $g$ has no limit at the origin — partials probe two directions only, continuity needs all of them; only the $C^1$ hypothesis restores order (Theorem 25.5). Line limits never suffice: Exercise 25.3’s function has limit $0$ along every line and still no limit — always try parabolas (or polar bounds valid uniformly in $\theta$ ). Critical is necessary, not sufficient: saddles abound (three of four points in Example 25.16); and the theorem holds on open sets only — on a closed disc, extrema may sit on the boundary with nonzero gradient (Exercise 25.12). The silent case $rt - s^2 = 0$ is genuinely silent: $x^4 + y^4$ (minimum) and $x^3 + y^3$ (neither) both have $r = s = t = 0$ at the origin; only a direct sign study decides (Exercise 25.6, function $h$ ). The gradient is orthogonal to level curves, not along them: to follow a contour line, move perpendicular to $\nabla f$ ; to climb fastest, move along it — mixing the two reverses the geometry of every contour map.

Remark 25.18 (Where two variables lead)

This chapter is a doorway. The gradient and the chain rule extend verbatim to $n$ variables in the Year 2 volume, where the $o(\norm{(h,k)})$ of Theorem 25.5 becomes the differential and the Monge test is proved in full via the order-two Taylor formula and quadratic forms. The special case that can be settled this year — quadratic functions, for which the second-order expansion is exact — is the subject of the weekend problem, and it happens to be the case that runs the world’s data fitting: least squares regression. Constrained extrema (Exercise 25.5 was a preview) become Lagrange multipliers in Year 2; harmonic functions (Exercise 25.7) return in the complex analysis of the Year 3 volume.

Remark 25.19 (Perspectives inside Book 3: the year, closed)

This chapter is where the volume’s two halves shake hands. The analysis half supplied its tools one derivative at a time: the mean value theorem drives Theorem 25.5, Taylor expansions drive the extremum tests, and the $\varepsilon$ ’s of Chapter 12 came back with balls instead of intervals. The algebra half supplied the geometry: the gradient is read through the inner product of Chapter 23 (Cauchy–Schwarz makes it the steepest direction), the Monge data $(r, s, t)$ is a symmetric matrix of Chapter 21 with the determinant test of Chapter 22, and the weekend problem runs orthogonal projection on data vectors. Even the curves of Chapter 24 return as level sets. A reader who can reconstruct why each of these five hand-offs works has, in effect, revised the entire year — which is the real purpose of this final chapter.

Least squares in one picture: four data points, the regression line y = 1.4x - 0.1, and the vertical residuals (dashed) whose squares the line minimizes — total 0.2, the smallest achievable. The weekend problem computes this line, proves it is the unique minimizer, and identifies the whole construction with an orthogonal projection in ℝ4. — Least squares in one picture: four data points, the regression line $y = 1.4x - 0.1$ , and the vertical residuals (dashed) whose squares the line minimizes — total $0.2$ , the smallest achievable. The weekend problem computes this line, proves it is the unique minimizer, and identifies the whole construction with an orthogonal projection in $\R^4$ .

25.4 Exercises

Exercise 25.1 ★

Compute the partial derivatives: $f(x,y) = x^2 y + \eu^{xy}$ ; $\;g(x,y) = \ln(x^2 + y^2)$ (on $\R^2\setminus\{0\}$ ); $\;h(x,y) = \arctan\frac yx$ (on $x > 0$ ).

Solution

Solution of Exercise 25.1.

$\dfrac{\partial f}{\partial x} = 2xy + y\,\eu^{xy}$ , $\dfrac{\partial f}{\partial y} = x^2 + x\,\eu^{xy}$ .

$\dfrac{\partial g}{\partial x} = \dfrac{2x}{x^2+y^2}$ , $\dfrac{\partial g}{\partial y} = \dfrac{2y}{x^2+y^2}$ .

$\dfrac{\partial h}{\partial x} = \dfrac{-y/x^2}{1 + y^2/x^2} = \dfrac{-y}{x^2+y^2}$ , $\dfrac{\partial h}{\partial y} = \dfrac{x}{x^2+y^2}$ .

Exercise 25.2 ★

Study the continuity at $(0,0)$ (with value $0$ there) of:

f(x,y) = \frac{x^2 y}{x^2 + y^2}, \qquad g(x,y) = \frac{xy}{x^2 + y^2}, \qquad h(x,y) = \frac{x^3 + y^3}{x^2 + y^2}.

(Polar coordinates $x = \rho\cos\theta$ , $y = \rho\sin\theta$ help: bound by a function of $\rho$ alone when possible.)

Solution

Solution of Exercise 25.2.

In polar coordinates ( $\rho \to 0$ ):

$\abs{f} = \dfrac{\rho^3\abs{\cos^2\theta\sin\theta}}{\rho^2} \leq \rho \to 0$ : continuous.

$g = \cos\theta\sin\theta$ : independent of $\rho$ , taking different values along different rays (cf. Example 25.3): no limit, not continuous.

$\abs h \leq \dfrac{\rho^3(\abs{\cos^3\theta} + \abs{\sin^3\theta})}{\rho^2} \leq 2\rho \to 0$ : continuous.

Exercise 25.3 ★★

Let $f(x,y) = \dfrac{x^2 y}{x^4 + y^2}$ ( $f(0,0) = 0$ ). Prove that $f$ has limit $0$ at the origin along every straight line, but that $f\bigl(x, x^2\bigr) = \frac12$ : $f$ is not continuous at $(0,0)$ .

Solution

Solution of Exercise 25.3.

Along $y = mx$ : $f(x, mx) = \dfrac{m x^3}{x^4 + m^2 x^2} = \dfrac{mx}{x^2 + m^2} \to 0$ (for $m \neq 0$ ; along $y = 0$ and the $y$ -axis, $f = 0$ ). So every straight-line limit is $0$ . But on the parabola $y = x^2$ :

f(x, x^2) = \frac{x^4}{x^4 + x^4} = \frac12 :

the sequence $\bigl(\frac1n, \frac{1}{n^2}\bigr) \to (0,0)$ has $f \to \frac12 \neq 0$ . Not continuous: lines do not suffice to test two-variable limits.

Exercise 25.4 ★

Verify Schwarz’s theorem by hand on $f(x, y) = x^3 y^2 + \sin(xy)$ .

Solution

Solution of Exercise 25.4.

$\frac{\partial f}{\partial x} = 3x^2y^2 + y\cos(xy)$ ; then

\frac{\partial^2 f}{\partial y\,\partial x} = 6x^2 y + \cos(xy) - xy\sin(xy) .

$\frac{\partial f}{\partial y} = 2x^3 y + x\cos(xy)$ ; then

\frac{\partial^2 f}{\partial x\,\partial y} = 6x^2 y + \cos(xy) - xy\sin(xy) :

equal, as Schwarz promises.

Exercise 25.5 ★★

Let $f$ be $C^1$ on $\R^2$ and $g(t) = f(\cos t, \sin t)$ . Express $g'(t)$ via the chain rule. Deduce that $f$ restricted to the unit circle attains extrema at points where $\nabla f$ is parallel to the radius vector.

Solution

Solution of Exercise 25.5.

By Theorem 25.7 with $(x, y) = (\cos t, \sin t)$ :

g'(t) = -\sin t\,\frac{\partial f}{\partial x}(\cos t, \sin t) + \cos t\,\frac{\partial f}{\partial y}(\cos t, \sin t) = \bigl\langle \nabla f,\ (-\sin t, \cos t)\bigr\rangle .

At an extremum of $g$ , $g'(t) = 0$ : $\nabla f$ is orthogonal to the tangent vector $(-\sin t, \cos t)$ of the circle, hence parallel to the radius vector $(\cos t, \sin t)$ (the plane orthogonal complement of a unit vector is the line it spans, taken perpendicular). This is the simplest instance of a Lagrange multiplier.

Exercise 25.6 ★★

Find and classify the critical points of:

f(x, y) = x^2 + xy + y^2 - 3x, \qquad g(x, y) = x^2 - y^2 + 4y, \qquad h(x, y) = x^4 + y^4 - 2(x - y)^2 .

(For $h$ , the determinant test is silent at the origin: examine $h(x, x)$ and $h(x, -x)$ .)

Solution

Solution of Exercise 25.6.

$f$ : $\nabla f = (2x + y - 3,\; x + 2y) = 0$ : $y = -\frac x2$ and $2x - \frac x2 = 3$ : $x = 2$ , $y = -1$ . Second order: $r = 2$ , $s = 1$ , $t = 2$ : $rt - s^2 = 3 > 0$ , $r > 0$ : local (indeed global — quadratic) minimum at $(2, -1)$ , value $f(2,-1) = -3$ .

$g$ : $\nabla g = (2x,\; -2y + 4) = 0$ : point $(0, 2)$ ; $r = 2$ , $s = 0$ , $t = -2$ : $rt - s^2 = -4 < 0$ : saddle.

$h$ : $\nabla h = \bigl(4x^3 - 4(x - y),\; 4y^3 + 4(x-y)\bigr) = 0$ . Adding: $x^3 + y^3 = 0$ , so $y = -x$ ; substituting: $4x^3 - 8x = 0$ : $x \in \{0, \pm\sqrt2\}$ . Critical points: $(0,0)$ , $(\sqrt2, -\sqrt2)$ , $(-\sqrt2, \sqrt2)$ . At $(\pm\sqrt2, \mp\sqrt2)$ : $r = 12x^2 - 4 = 20$ , $s = 4$ , $t = 20$ : $rt - s^2 > 0$ , $r > 0$ : local minima (value $h = 4 + 4 - 16 = -8$ ). At $(0,0)$ : $r = t = -4$ , $s = 4$ : $rt - s^2 = 0$ : silent. Examine: $h(x, x) = 2x^4 > 0$ and $h(x, -x) = 2x^4 - 8x^2 < 0$ for small $x \neq 0$ : both signs in every neighborhood — no extremum at the origin.

Exercise 25.7 ★★

A function $f$ is harmonic when $\frac{\partial^2 f}{\partial x^2} + \frac{\partial^2 f}{\partial y^2} = 0$ . Check that $x^2 - y^2$ , $xy$ , $\eu^x\cos y$ and $\ln(x^2 + y^2)$ (off the origin) are harmonic.

Solution

Solution of Exercise 25.7.

$x^2 - y^2$ : second partials $2$ and $-2$ : sum $0$ . $xy$ : both pure second partials vanish. $\eu^x\cos y$ : $\frac{\partial^2}{\partial x^2} = \eu^x\cos y$ , $\frac{\partial^2}{\partial y^2} = -\eu^x\cos y$ : sum $0$ . $\ln(x^2+y^2)$ : from Exercise 25.1,

\frac{\partial^2}{\partial x^2}\ln(x^2+y^2) = \frac{2(x^2+y^2) - 4x^2}{(x^2+y^2)^2} = \frac{2(y^2 - x^2)}{(x^2+y^2)^2},

and the $y$ -version is its opposite: sum $0$ .

Exercise 25.8 ★★★

Find the point of the plane $\{x + 2y - z = 4\} \subseteq \R^3$ closest to the origin, two ways: by orthogonal projection (Chapter 23), and by minimizing the two-variable function $f(x, y) = x^2 + y^2 + (x + 2y - 4)^2$ obtained by eliminating $z$ .

Solution

Solution of Exercise 25.8.

Projection: the plane $P$ has normal $n = (1, 2, -1)$ and passes through $A = (4, 0, 0)$ . The closest point to the origin is $O$ projected: $p = O + \frac{\langle A - O, n\rangle}{\norm n^2}\,n = \frac{4}{6}(1,2,-1) = \bigl(\frac23, \frac43, -\frac23\bigr)$ , at distance $\frac{4}{\sqrt 6}$ . (Formula: distance from origin to plane $\langle X, n \rangle = c$ is $\frac{\abs c}{\norm n}$ with $c = 4$ .)

Minimization: $f(x,y) = x^2 + y^2 + (x + 2y - 4)^2$ . Writing $w = x + 2y - 4$ for the last factor:

\nabla f = \bigl(2x + 2w,\; 2y + 4w\bigr) = 0 \iff x = -w \text{ and } y = -2w .

Substituting into the definition of $w$ : $w = -w - 4w - 4$ , so $w = -\frac23$ , giving $x = \frac23$ , $y = \frac43$ , and $z = w = -\frac23$ . Same point as by projection, at distance $\sqrt{\frac49 + \frac{16}{9} + \frac49} = \frac{2\sqrt 6}{3} = \frac{4}{\sqrt6}$ ; it is a minimum since $f \to \infty$ at infinity (a positive-definite quadratic plus linear terms).

Exercise 25.9 ★★★

(Laplacian in polar coordinates, first contact) Let $f$ be $C^2$ on $\R^2 \setminus \{0\}$ and radial: $f(x, y) = \varphi\bigl(\sqrt{x^2+y^2}\bigr)$ with $\varphi$ of class $C^2$ on $\intoo{0}{+\infty}$ . Prove that

\frac{\partial^2 f}{\partial x^2} + \frac{\partial^2 f}{\partial y^2} = \varphi''(\rho) + \frac{\varphi'(\rho)}{\rho}, \qquad \rho = \sqrt{x^2 + y^2},

and find all radial harmonic functions on the punctured plane.

Solution

Solution of Exercise 25.9.

With $\rho = \sqrt{x^2+y^2}$ : $\frac{\partial \rho}{\partial x} = \frac{x}{\rho}$ , so

\frac{\partial f}{\partial x} = \varphi'(\rho)\,\frac{x}{\rho}, \qquad \frac{\partial^2 f}{\partial x^2} = \varphi''(\rho)\,\frac{x^2}{\rho^2} + \varphi'(\rho)\,\Bigl(\frac{1}{\rho} - \frac{x^2}{\rho^3}\Bigr),

(quotient and chain rules). Adding the symmetric $y$ -expression: the $\varphi''$ terms collect $\frac{x^2 + y^2}{\rho^2} = 1$ , the $\varphi'$ terms collect $\frac{2}{\rho} - \frac{x^2 + y^2}{\rho^3} = \frac{1}{\rho}$ :

\Delta f = \varphi'' + \frac{\varphi'}{\rho} .

Radial harmonic: solve $\varphi'' + \frac{\varphi'}{\rho} = 0$ : the first-order equation $u' + \frac u\rho = 0$ for $u = \varphi'$ gives $u = \frac{c}{\rho}$ (Theorem 5.2), then $\varphi = c\ln\rho + d$ . The radial harmonic functions are $c\,\ln\sqrt{x^2 + y^2} + d$ — the logarithmic potential of Exercise 25.7 and the constants, nothing else.

Exercise 25.10 ★★

Give the equation of the tangent plane to the graph $z = xy$ at the point $(1, 1, 1)$ . Then find all points of the graph of $f(x,y) = x^3 + y^3 - 3xy$ where the tangent plane is horizontal, and relate the answer to Example 25.15.

Solution

Solution of Exercise 25.10.

For $z = xy$ at $(1,1)$ : partials $y = 1$ and $x = 1$ , tangent plane $z = 1 + (x - 1) + (y - 1) = x + y - 1$ . Horizontal tangent plane means both partials vanish, i.e. $\nabla f = 0$ : by Example 25.15, exactly the critical points $(0, 0)$ and $(1, 1)$ , with horizontal planes $z = 0$ and $z = -1$ . “Horizontal tangent plane” and “critical point” are the same notion, seen on the graph and in the formula.

Exercise 25.11 ★★

Let $f$ be $C^1$ on $\R^2$ .

If $\frac{\partial f}{\partial x} = 0$ everywhere, prove that $f(x, y)$ depends only on $y$ .
Find all $C^1$ solutions of the equation $\frac{\partial f}{\partial x} = \frac{\partial f}{\partial y}$ on $\R^2$ . (Set $g(u, v) = f\bigl(\frac{u+v}2, \frac{u-v}2\bigr)$ and compute $\frac{\partial g}{\partial v}$ by the chain rule.)

Solution

Solution of Exercise 25.11.

For fixed $y$ , the one-variable function $x \mapsto f(x,y)$ has zero derivative on $\R$ , hence is constant (Theorem 14.9): $f(x, y) = f(0, y)$ for all $x$ : $f$ depends only on $y$ .
Let $g(u, v) = f\bigl(\frac{u+v}2, \frac{u-v}2\bigr)$ . By the chain rule (Theorem 25.7, applied in the variable $v$ with $u$ frozen):

$\frac{\partial g}{\partial v} = \frac12\,\frac{\partial f}{\partial x} - \frac12\,\frac{\partial f}{\partial y} = 0 .$

By (1), $g$ depends only on $u$ : $g(u, v) = \varphi(u)$ with $\varphi$ of class $C^1$ , and reversing the change of variables ( $u = x + y$ , $v = x - y$ ),

$f(x, y) = \varphi(x + y).$

Conversely every such $f$ satisfies $f_x = f_y = \varphi'$ : the solutions are exactly the $C^1$ functions of $x + y$ .

Exercise 25.12 ★★★

Find the global maximum and minimum of $f(x, y) = xy$ on the closed disc $x^2 + y^2 \leq 1$ . (Admit the two-variable extreme value theorem: a continuous function on the closed disc attains its bounds — proved in the Year 2 volume. Treat the open disc by critical points and the boundary circle by the parametrization of Exercise 25.5.)

Solution

Solution of Exercise 25.12.

On the open disc, an extremum would be critical: $\nabla f = (y, x) = 0$ only at $(0,0)$ , where $f = 0$ ; it is a saddle ( $f(\pm \varepsilon, \pm\varepsilon) = \varepsilon^2 > 0 > -\varepsilon^2 = f(\pm\varepsilon, \mp\varepsilon)$ ), so no extremum there. By the admitted extreme value theorem the bounds are attained, necessarily on the boundary circle. There, with Exercise 25.5,

f(\cos t, \sin t) = \cos t\sin t = \frac{\sin 2t}{2} \in \intcc{-\tfrac12}{\tfrac12},

with maximum $\frac12$ at $t = \frac\pi4, \frac{5\pi}4$ (points $\pm\frac{1}{\sqrt2}(1,1)$ ) and minimum $-\frac12$ at $t = \frac{3\pi}4, \frac{7\pi}4$ (points $\pm\frac{1}{\sqrt2}(1,-1)$ ). Global maximum $\frac12$ , global minimum $-\frac12$ .

25.5 Problem: least squares and the regression line

Problem 25.1

Given $n$ data points, which straight line passes “closest” to all of them? Legendre and Gauss answered: the line minimizing the sum of the squared vertical errors — because that minimization is exactly solvable by linear algebra. This problem first proves the Monge test of Method 25.14 honestly for quadratic functions (the one case where the second-order expansion is exact), then builds the normal equations, the regression line, and the correlation coefficient on top of the Euclidean geometry of Chapter 23.

Part I — Quadratic functions: the Monge test, proved. Fix reals $r, s, t$ and let $q(h, k) = r h^2 + 2s hk + t k^2$ .

Suppose $r \neq 0$ . Establish the completed-square form

$q(h, k) = r\Bigl(h + \frac{s}{r}k\Bigr)^{\!2} + \frac{rt - s^2}{r}\,k^2 ,$

and deduce: if $rt - s^2 > 0$ , $q$ has the strict sign of $r$ off the origin; if $rt - s^2 < 0$ , $q$ takes both signs.
Settle the remaining cases: $r = 0$ , $t \neq 0$ (symmetric); and $r = t = 0$ ( $q = 2shk$ ). Conclude: $q$ takes both signs iff $rt - s^2 < 0$ , and $q$ vanishes only at the origin iff $rt - s^2 > 0$ .
Let now $f(x, y) = \frac12\bigl(r x^2 + 2s xy + t y^2\bigr) + \beta x + \gamma y + \delta$ be a quadratic function with a critical point $X_0 = (a, b)$ . Prove the exact expansion

$f(X_0 + (h,k)) = f(X_0) + \tfrac12\,q(h, k) \qquad (\text{no remainder}),$

and deduce the Monge classification for quadratic functions: strict global minimum if $rt - s^2 > 0$ , $r > 0$ ; strict global maximum if $rt - s^2 > 0$ , $r < 0$ ; saddle if $rt - s^2 < 0$ .
Suppose $rt - s^2 > 0$ and $r > 0$ (so also $t > 0$ ). Prove the explicit coercivity bound

$q(h, k) \geq c\,(h^2 + k^2), \qquad c = \frac{rt - s^2}{r + t} > 0 .$

(Show that the quadratic form $q - c(h^2 + k^2)$ , with coefficients $r - c$ , $s$ , $t - c$ , still has nonnegative discriminant-test data.)
Deduce that a quadratic function with positive definite quadratic part tends to $+\infty$ as $\norm{(x,y)} \to \infty$ , and therefore has a unique global minimizer: its critical point. Contrast with Example 25.15, where a local minimum of a nonquadratic function was not global.

Part II — The normal equations. Let $C_1, C_2, b \in \R^n$ (canonical inner product), and

f(u, v) = \norm{u\,C_1 + v\,C_2 - b}^2 .

Expand $f$ and show it is a quadratic function of $(u, v)$ with quadratic-part coefficients $r = 2\norm{C_1}^2$ , $s = 2\langle C_1, C_2\rangle$ , $t = 2\norm{C_2}^2$ .
Show that $rt - s^2 > 0$ if and only if $(C_1, C_2)$ is free (the equality case of Cauchy–Schwarz, Theorem 23.4); assume this from now on.
Show that the critical equations $\nabla f = 0$ are the normal equations

$\begin{pmatrix} \norm{C_1}^2 & \langle C_1, C_2\rangle\\ \langle C_1, C_2\rangle & \norm{C_2}^2 \end{pmatrix} \begin{pmatrix} u\\ v\end{pmatrix} = \begin{pmatrix} \langle C_1, b\rangle\\ \langle C_2, b\rangle\end{pmatrix}$

— the Gram matrix of Exercise 23.11 on the left — and that they say exactly: $b - (uC_1 + vC_2) \perp C_1, C_2$ . Conclude with Part I and Theorem 23.10: the unique minimizer gives $p = uC_1 + vC_2 =$ the orthogonal projection of $b$ onto $\operatorname{Vect}(C_1, C_2)$ .
Show that the minimal value is $\norm{b}^2 - \langle p, b\rangle = \norm{b - p}^2$ , and draw the Pythagorean picture.

Part III — The regression line. Data points $(x_1, y_1), \dots, (x_n, y_n)$ , not all $x_i$ equal. Minimize

E(\alpha, \beta) = \sum_{i=1}^{n}\bigl(y_i - \alpha - \beta x_i\bigr)^2 .

Write $\bar x = \frac1n\sum x_i$ , $\bar y = \frac1n\sum y_i$ , $v_x = \frac1n\sum x_i^2 - \bar x^2$ , $v_y = \frac1n\sum y_i^2 - \bar y^2$ , $c_{xy} = \frac1n\sum x_i y_i - \bar x\bar y$ .

Recognize Part II with $C_1 = (1, \dots, 1)$ , $C_2 = (x_1, \dots, x_n)$ , $b = (y_1, \dots, y_n)$ , and write the normal equations

$n\alpha + \Bigl(\sum x_i\Bigr)\beta = \sum y_i, \qquad \Bigl(\sum x_i\Bigr)\alpha + \Bigl(\sum x_i^2\Bigr)\beta = \sum x_i y_i .$
Solve them:

$\beta = \frac{c_{xy}}{v_x}, \qquad \alpha = \bar y - \beta\,\bar x ,$

and observe that the regression line $y = \alpha + \beta x$ passes through the mean point $(\bar x, \bar y)$ .
Check $v_x > 0$ exactly when the $x_i$ are not all equal, and match this with question 7.
Prove that the minimal error is

$E_{\min} = n\Bigl(v_y - \frac{c_{xy}^2}{v_x}\Bigr) = n\,v_y\,(1 - \rho^2), \qquad \rho = \frac{c_{xy}}{\sqrt{v_x v_y}} \quad (v_y \neq 0).$

Deduce $\abs\rho \leq 1$ , with $\abs\rho = 1$ if and only if the data are perfectly aligned.
Worked example: for the data $(0,0)$ , $(1,1)$ , $(2,3)$ , $(3,4)$ , compute $\bar x, \bar y, v_x, c_{xy}$ , the regression line, the four residuals and $E_{\min}$ .
Prove that the residuals $\varepsilon_i = y_i - \alpha - \beta x_i$ of the optimal line satisfy $\sum_i \varepsilon_i = 0$ and $\sum_i x_i\varepsilon_i = 0$ , and interpret both via orthogonality.

Part IV — Variations and applications.

(Best constant) Show that the constant $\alpha$ minimizing $\sum_i (y_i - \alpha)^2$ is the mean $\bar y$ , and that the minimal value is $n\,v_y$ : the variance measures the failure of the data to be constant.
(Line through the origin) Show that the slope minimizing $\sum_i (y_i - \beta x_i)^2$ is $\beta_0 = \frac{\sum x_iy_i}{\sum x_i^2}$ , and give a condition on the data for $\beta_0$ to coincide with the slope $c_{xy}/v_x$ of question 11.
(Distance to a line, again) For a point $M$ and the line $D$ through $P_0$ directed by the unit vector $w$ , minimize $g(\tau) = \norm{P_0 + \tau w - M}^2$ and deduce $d(M, D)^2 = \norm{M - P_0}^2 - \langle M - P_0, w\rangle^2$ ; recover the formula $d = \frac{\abs{a x_0 + b y_0 + c}}{\sqrt{a^2 + b^2}}$ for the line $ax + by + c = 0$ in the plane.
(Analysis of variance) Prove the decomposition $v_y = \beta^2 v_x + \frac{E_{\min}}n$ : the variance of the $y_i$ splits into the part explained by the line plus the residual variance.
(Parabolic fit) To fit $y = a + bx + cx^2$ , show that the normal equations are the $3 \times 3$ system with the moment matrix

$\begin{pmatrix} n & \sum x_i & \sum x_i^2\\ \sum x_i & \sum x_i^2 & \sum x_i^3\\ \sum x_i^2 & \sum x_i^3 & \sum x_i^4 \end{pmatrix},$

a Gram matrix which is invertible as soon as three of the $x_i$ are distinct (freeness of $(1, X, X^2)$ sampled at the data, Exercise 23.11; compare the moment matrices of the weekend problem of Chapter 22).

Part V — Robustness, and synthesis.

Classify the critical points of the three quadratic functions

$x^2 + xy + y^2, \qquad x^2 + 3xy + y^2, \qquad x^2 + 2xy + y^2 ,$

by Part I, treating the degenerate third case by hand (where is the minimum attained?).
(Outlier experiment) Append the point $(10, 0)$ to the data of question 14 and recompute the slope $\beta$ . What happened, and why is the squared error so sensitive to one distant point?
(Weighted least squares) Given weights $w_i > 0$ , minimize $\sum_i w_i(y_i - \alpha - \beta x_i)^2$ : show the solution is given by the same formulas as question 11 with weighted means, variance and covariance (define them).
Show that for the optimal line, $E_{\min} = 0$ forces all points to lie exactly on it, and connect with the equality case $\abs\rho = 1$ of question 13; test on the aligned data $(1,1), (2,3), (3,5)$ .
Synthesis, in four sentences: why quadratic functions are the one class where this chapter’s second-order test needs no admitted theorem; how the normal equations identify the analytic minimization with the orthogonal projection of Chapter 23; what the correlation coefficient measures and which inequality bounds it; and which pieces of Chapters 18–23 (bases, Gram and moment matrices, projections) reappeared. Name the method and the equations studied in Parts II–III.

Solution

Solution of Problem 25.1.

1. Expanding the right-hand side:

r\Bigl(h + \frac srk\Bigr)^{\!2} + \frac{rt - s^2}{r}k^2 = rh^2 + 2shk + \frac{s^2}{r}k^2 + \frac{rt - s^2}{r}k^2 = q(h,k) .

If $rt - s^2 > 0$ : both squares carry the factor sign of $r$ , and $q(h,k) = 0$ forces $k = 0$ then $h = 0$ : strict sign of $r$ off the origin. If $rt - s^2 < 0$ : $q(1, 0) = r$ while $q\bigl(-\frac sr, 1\bigr) = \frac{rt - s^2}{r}$ has the opposite sign: both signs occur.

2. If $r = 0 \neq t$ , exchange the roles of $h$ and $k$ (the completed square in $k$ ), with the same conclusions; note $rt - s^2 = -s^2 < 0$ as soon as $s \neq 0$ , and indeed $q = 2shk + tk^2$ takes both signs then ( $k$ small against $h$ ). If $r = t = 0$ : $q = 2shk$ , both signs iff $s \neq 0$ , i.e. iff $rt - s^2 = -s^2 < 0$ . Summary: $q$ takes both signs $\iff rt - s^2 < 0$ ; $q$ vanishes only at the origin (definite) $\iff rt - s^2 > 0$ (which forces $r \neq 0$ , since $r = 0$ gives $q(1,0) = 0$ ).

3. With $f_x = rx + sy + \beta$ and $f_y = sx + ty + \gamma$ , direct expansion gives

f(a + h, b + k) = f(a, b) + h\,f_x(a,b) + k\,f_y(a,b) + \tfrac12 q(h, k),

with no higher terms (the function is a polynomial of degree $2$ ). At a critical point the linear part vanishes: $f(X_0 + H) - f(X_0) = \frac12 q(H)$ exactly, so the sign study of questions 1–2 classifies: strict global minimum ( $rt - s^2 > 0$ , $r > 0$ ), strict global maximum ( $rt - s^2 > 0$ , $r < 0$ ), saddle — both signs in every neighborhood — when $rt - s^2 < 0$ . This proves Method 25.14 for quadratic functions.

4. The form $q - c(h^2 + k^2)$ has data $r - c$ , $s$ , $t - c$ . With $c = \frac{rt - s^2}{r + t}$ (note $t > 0$ since $rt > s^2 \geq 0$ and $r > 0$ ):

(r - c)(t - c) - s^2 = rt - s^2 - c(r + t) + c^2 = c^2 \geq 0,

and $r - c \geq 0$ ( $c \leq r \iff rt - s^2 \leq r^2 + rt$ , true). If $r - c > 0$ , the completed square of question 1 shows $q - c(h^2 + k^2) \geq 0$ ; if $r - c = 0$ , then $s = 0$ (from $-s^2 = -c^2 \leq \dots$ forcing the displayed quantity $\geq 0$ with first factor $0$ ) and the form is $(t - c)k^2 \geq 0$ . In both cases $q(h,k) \geq c(h^2 + k^2)$ .

5. By questions 3–4, $f(X) = f(X_0) + \frac12 q(X - X_0) \geq f(X_0) + \frac c2\norm{X - X_0}^2 \to +\infty$ as $\norm X \to \infty$ : $f$ is coercive, and the inequality is strict for $X \neq X_0$ : the critical point is the unique global minimizer. For the cubic $f = x^3 + y^3 - 3xy$ of Example 25.15, no such conclusion holds: $f(x,0) = x^3 \to -\infty$ , and the local minimum at $(1,1)$ is not global — exactness of the quadratic expansion is what failed.

6. Expanding the squared norm:

f(u,v) = u^2\norm{C_1}^2 + 2uv\,\langle C_1, C_2\rangle + v^2\norm{C_2}^2 - 2u\,\langle C_1, b\rangle - 2v\,\langle C_2, b\rangle + \norm b^2 ,

a quadratic function of $(u, v)$ whose quadratic part is $\frac12(ru^2 + 2suv + tv^2)$ with $r = 2\norm{C_1}^2$ , $s = 2\langle C_1, C_2\rangle$ , $t = 2\norm{C_2}^2$ .

7. $rt - s^2 = 4\bigl(\norm{C_1}^2\norm{C_2}^2 - \langle C_1, C_2\rangle^2\bigr) \geq 0$ by Cauchy–Schwarz (Theorem 23.4), with equality exactly when $C_1, C_2$ are proportional (or one is zero), i.e. when the pair is linked. So $rt - s^2 > 0 \iff (C_1, C_2)$ free.

8. $\nabla f = 0$ reads

\norm{C_1}^2 u + \langle C_1, C_2\rangle v = \langle C_1, b\rangle, \qquad \langle C_1, C_2\rangle u + \norm{C_2}^2 v = \langle C_2, b\rangle,

the normal equations with the Gram matrix on the left; they say $\langle uC_1 + vC_2 - b,\ C_i\rangle = 0$ for $i = 1, 2$ , i.e. $b - p \perp \operatorname{Vect}(C_1, C_2)$ for $p = uC_1 + vC_2$ . By Part I (questions 3, 5) the unique critical point is the unique global minimizer, and by Theorem 23.10 the characterization “ $p \in F$ , $b - p \perp F$ ” identifies $p$ as the orthogonal projection of $b$ onto $F = \operatorname{Vect}(C_1, C_2)$ .

9. $\norm{b - p}^2 = \norm b^2 - 2\langle b, p\rangle + \norm p^2$ , and $\langle p, b - p\rangle = 0$ gives $\norm p^2 = \langle p, b\rangle$ : the minimum equals $\norm b^2 - \langle p, b\rangle$ . Pythagoras: $\norm b^2 = \norm p^2 + \norm{b - p}^2$ — the data vector splits into its explained part $p$ and its residual part $b - p$ , orthogonal to each other.

10. $E(\alpha, \beta) = \norm{\alpha C_1 + \beta C_2 - b}^2$ with the stated $C_1, C_2, b$ ; the normal equations of question 8 are, entrywise,

n\alpha + \Bigl(\sum x_i\Bigr)\beta = \sum y_i, \qquad \Bigl(\sum x_i\Bigr)\alpha + \Bigl(\sum x_i^2\Bigr)\beta = \sum x_i y_i .

11. Dividing by $n$ : $\alpha + \beta\bar x = \bar y$ and $\alpha\bar x + \beta\bigl(v_x + \bar x^2\bigr) = c_{xy} + \bar x\bar y$ . Substituting $\alpha = \bar y - \beta\bar x$ into the second: $\beta v_x = c_{xy}$ , so

\beta = \frac{c_{xy}}{v_x}, \qquad \alpha = \bar y - \beta\bar x ,

and the first equation says precisely that $(\bar x, \bar y)$ lies on the line.

12. $v_x = \frac1n\sum(x_i - \bar x)^2 \geq 0$ , zero iff every $x_i$ equals $\bar x$ . And $rt - s^2 = 4\bigl(n\sum x_i^2 - (\sum x_i)^2\bigr) = 4n^2 v_x$ : question 7’s freeness condition is $v_x > 0$ , i.e. the $x_i$ not all equal.

13. With $\alpha = \bar y - \beta\bar x$ , center the data ( $\tilde x_i = x_i - \bar x$ , $\tilde y_i = y_i - \bar y$ ):

E(\alpha, \beta) = \sum_i(\tilde y_i - \beta\tilde x_i)^2 = n\,v_y - 2\beta\,n\,c_{xy} + \beta^2 n\,v_x ,

minimized at $\beta = c_{xy}/v_x$ with value $E_{\min} = n\bigl(v_y - \frac{c_{xy}^2}{v_x}\bigr) = n v_y(1 - \rho^2)$ . Since $E_{\min} \geq 0$ and $v_y > 0$ : $\rho^2 \leq 1$ , and $\abs\rho = 1$ iff $E_{\min} = 0$ , i.e. iff every residual vanishes: the points lie exactly on the line.

14. $n = 4$ : $\bar x = \frac64 = 1.5$ , $\bar y = 2$ , $\sum x_i^2 = 14$ so $v_x = 3.5 - 2.25 = 1.25$ , $\sum x_iy_i = 0 + 1 + 6 + 12 = 19$ so $c_{xy} = 4.75 - 3 = 1.75$ . Hence

\beta = \frac{1.75}{1.25} = 1.4, \qquad \alpha = 2 - 1.4\times1.5 = -0.1 : \qquad y = 1.4\,x - 0.1 .

Fitted values $-0.1,\ 1.3,\ 2.7,\ 4.1$ ; residuals $0.1,\ -0.3,\ 0.3,\ -0.1$ ; $E_{\min} = 0.01 + 0.09 + 0.09 + 0.01 = 0.2$ (check: $v_y = \frac{26}4 - 4 = 2.5$ and $4(2.5 - \frac{1.75^2}{1.25}) = 4(2.5 - 2.45) = 0.2$ ).

15. The two normal equations of question 10 are exactly $\sum_i\varepsilon_i = 0$ and $\sum_i x_i\varepsilon_i = 0$ : the residual vector is orthogonal to $C_1 = (1, \dots, 1)$ and to $C_2 = (x_i)$ — to the whole model space. In particular the optimal line always balances its errors: they sum to zero.

16. $\frac{\dd}{\dd\alpha}\sum(y_i - \alpha)^2 = -2\sum(y_i - \alpha) = 0$ gives $\alpha = \bar y$ (and the second derivative $2n > 0$ makes it the global minimum, the function being a coercive quadratic in one variable). Minimal value $\sum(y_i - \bar y)^2 = n v_y$ : the variance is the irreducible quadratic error of a constant model.

17. $\frac{\dd}{\dd\beta}\sum(y_i - \beta x_i)^2 = -2\sum x_i(y_i - \beta x_i) = 0$ gives $\beta_0 = \frac{\sum x_iy_i}{\sum x_i^2}$ . Writing both slopes over centered quantities: $\beta_0 = \frac{c_{xy} + \bar x\bar y}{v_x + \bar x^2}$ equals $\frac{c_{xy}}{v_x}$ iff $c_{xy}\bar x^2 = \bar x\bar y\,v_x$ , i.e. iff $\bar x = 0$ (centered abscissas) or $\bar y = \beta\bar x$ — the latter meaning $\alpha = 0$ : the full regression line already passes through the origin.

18. $g(\tau) = \norm{M - P_0}^2 - 2\tau\langle M - P_0, w\rangle + \tau^2$ is minimal at $\tau^* = \langle M - P_0, w\rangle$ , with value $d(M,D)^2 = \norm{M - P_0}^2 - \langle M - P_0, w\rangle^2$ . In the plane, complete $w$ into an orthonormal basis $(w, n)$ with $n = \frac{(a, b)}{\sqrt{a^2+b^2}}$ (unit normal of $D$ ): then $\norm{M - P_0}^2 = \langle M - P_0, w\rangle^2 + \langle M - P_0, n\rangle^2$ , so $d(M, D) = \abs{\langle M - P_0, n\rangle}$ , and with $aP_{0x} + bP_{0y} = -c$ :

d = \frac{\abs{a x_0 + b y_0 + c}}{\sqrt{a^2 + b^2}} .

19. From $E_{\min}/n = v_y - \frac{c_{xy}^2}{v_x}$ and $\beta^2 v_x = \frac{c_{xy}^2}{v_x}$ :

v_y = \beta^2 v_x + \frac{E_{\min}}n :

total variance $=$ variance along the fitted line $+$ residual variance. (Dividing by $v_y$ : $1 = \rho^2 + (1 - \rho^2)$ , the share of variance “explained” by the line is $\rho^2$ .)

20. The model space is $\operatorname{Vect}\bigl(C_1, C_2, C_3\bigr)$ with $C_1 = (1)$ , $C_2 = (x_i)$ , $C_3 = (x_i^2)$ ; minimizing $\norm{aC_1 + bC_2 + cC_3 - b}^2$ leads, exactly as in question 8, to the $3\times3$ Gram system, whose matrix has entries $\langle C_i, C_j\rangle = \sum_k x_k^{\,i+j-2}$ : the displayed moment matrix. It is invertible iff $(C_1, C_2, C_3)$ is free (Exercise 23.11); a relation $a + bx_k + cx_k^2 = 0$ for all $k$ makes every $x_k$ a root of one polynomial of degree $\leq 2$ , impossible with three distinct values unless $a = b = c = 0$ . Compare the moment matrices of the weekend problem of Chapter 22, where their determinants were squared Vandermonde values.

21. First: $r = 2, s = 1, t = 2$ , $rt - s^2 = 3 > 0$ , $r > 0$ : strict global minimum at the origin. Second: $r = 2, s = 3, t = 2$ , $rt - s^2 = -5 < 0$ : saddle. Third: $r = s = t = 2$ , $rt - s^2 = 0$ : degenerate — but $x^2 + 2xy + y^2 = (x + y)^2 \geq 0$ vanishes on the whole line $y = -x$ : a (non-strict) global minimum attained along a line, invisible to the determinant test.

22. New sums ( $n = 5$ ): $\bar x = 3.2$ , $\bar y = 1.6$ , $\sum x_iy_i = 19$ so $c_{xy} = 3.8 - 5.12 = -1.32$ , $\sum x_i^2 = 114$ so $v_x = 22.8 - 10.24 = 12.56$ . New slope:

\beta = \frac{-1.32}{12.56} \approx -0.105 :

one point turned a clearly increasing trend ( $\beta = 1.4$ ) into a slightly decreasing one. The squared error charges a residual $\varepsilon^2$ , so a single distant point — large $\abs{x_{i} - \bar x}$ and large residual — dominates both $c_{xy}$ and $v_x$ : least squares is efficient but not robust.

23. Set $W = \sum w_i$ and define $\bar x_w = \frac1W\sum w_ix_i$ , $\bar y_w$ likewise, $v_x^w = \frac1W\sum w_ix_i^2 - \bar x_w^2$ , $c^w_{xy} = \frac1W\sum w_ix_iy_i - \bar x_w\bar y_w$ . The map $\langle u, v\rangle_w = \sum_i w_iu_iv_i$ is an inner product on $\R^n$ ( $w_i > 0$ gives definiteness), so Part II applies verbatim and the normal equations divide by $W$ into

\alpha + \beta\bar x_w = \bar y_w, \qquad \alpha\bar x_w + \beta(v^w_x + \bar x_w^2) = c^w_{xy} + \bar x_w\bar y_w ,

whence $\beta = c^w_{xy}/v^w_x$ and $\alpha = \bar y_w - \beta\bar x_w$ : the same formulas, with every average weighted.

24. $E_{\min} = \sum\varepsilon_i^2 = 0$ forces every $\varepsilon_i = 0$ : all points exactly on the line; by question 13 this is the case $\abs\rho = 1$ . Test: for $(1,1), (2,3), (3,5)$ : $\bar x = 2$ , $\bar y = 3$ , $v_x = \frac{14}3 - 4 = \frac23$ , $c_{xy} = \frac{22}3 - 6 = \frac43$ : $\beta = 2$ , $\alpha = -1$ , and indeed $y_i = 2x_i - 1$ for all three points; $v_y = \frac{35}3 - 9 = \frac83$ and $\rho^2 = \frac{(4/3)^2}{(2/3)(8/3)} = 1$ .

25. (i) For quadratic functions the order-two expansion is an identity, so the sign study of the quadratic form — pure algebra, questions 1–2 — classifies critical points with no admitted Taylor theorem. (ii) The normal equations say “residual orthogonal to the model space”, so the analytic minimum is the orthogonal projection of the data vector: calculus and Euclidean geometry compute the same object. (iii) The correlation $\rho = c_{xy}/\sqrt{v_xv_y}$ measures the share $\rho^2$ of the variance explained by the line, and Cauchy–Schwarz bounds it: $\abs\rho \leq 1$ , with equality only for aligned data. (iv) Bases and freeness (Chapter 18), Gram and moment matrices and their determinants (Chapters 22–23), and orthogonal projection (Chapter 23) all reappeared as the working parts of one algorithm. Parts II–III develop the method of least squares (Legendre, Gauss) through its normal equations.

Terms defined in this chapter

See all 395 terms in the glossary

25.1 The plane R2\R^2R2; continuity

25.2 Partial derivatives

25.3 Local extrema

25.4 Exercises

25.5 Problem: least squares and the regression line

Terms defined in this chapter

25.1 The plane $\R^2$ ; continuity