The Linear Least Squares Problem

Target function:

let the gradient to zero, we get:

Normal equations(NEs):

Then we have the following important results:

Above results lead to(since

Theorem 1

Thus

The solutions for the NEs are solutions for

All the solutions of NEs can be given by:

Theorem 2 If

Proof

If we have

thus we have the corrected SSR is

and we also have (holds even

hence we have

Reparameterization

Linear models equivalent or reparameterizations iff

If

If

Gram-Schmidt Orthonormalization

For any matrix

Such QR factorization also related to the Cholesky factorization:

So

Estimability and Least Squares EstimatorsAssumptions for the Linear Mean Model

For the linear model

We now assume that the error have zero mean:

Such Assumption is also equivalent to assume

Estimator and Estimability

Definition:

An estimator unbiased for linear iff estimable iff there exists a linear unbiased estimator for it, otherwise the function is called nonestimabel.

The following statements are equivalent:

We can prove this follow:

Thus when

Suppose

If

We often choose

Conditions for a Unique Solution

Suppose we impose a condition on the solution to NEs:

now the NEs become:

where

Thus have

And the system of equations above is equivalent to

since

Then we have the following important results:

Proof We only prove 5, consider

Note that for idempotent matrix

The proof is finished since invertible idempotent matrix is identity.