PCA = Eigenvectors of Covariance

Principal components are eigenvectors of the covariance matrix — equivalently, the SVD of the centered data.

PCA = Eigenvectors of Covariance

Every cloud of data has directions along which it spreads out most. Those directions are the eigenvectors of a single matrix, the covariance, and finding them is all PCA does.

Covariance & its Eigenvectors

Drag points and watch the covariance, eigenvalues, and arrows update. Stretch the cloud into a cigar, then spread it into a ring where no direction wins.
covariance Σ = (1/n) Xᵀ X
13.957.417.414.53
eigenvalues (variance per PC)
λ₁ = 18.025 (97.5% of variance)
λ₂ = 0.461 (2.5% of variance)
PC1 angle: 28.8°

The teal arrow is the first principal component — the direction of maximum variance. It is the unit eigenvector of Σ with the larger eigenvalue. The violet arrow is PC2, always orthogonal because Σ is symmetric. Arrow length is ±2σ along that axis. Drag a point to deform the cloud and watch how Σ, its eigenvalues, and the principal axes update in real time.

Go deeper:Eigenvectors of covariance

Principal Component Analysis sounds like an algorithm. It is really a sentence in linear algebra: the principal components of a dataset are the eigenvectors of its covariance matrix. Center the data so its mean is zero, form Σ = (1/n) Xᵀ X, and diagonalize. The eigenvectors point along the directions of greatest variance. The eigenvalues are those variances. That is the entire method.

  • Centered data matrix X: n rows of d-dimensional samples, with the column mean subtracted. Centering matters — it's why PCA captures variance rather than absolute position.
  • Covariance Σ = (1/n) Xᵀ X: a d × d symmetric positive semidefinite matrix. The diagonal entries are per-feature variances, the off-diagonals measure how features co-vary.
  • Eigendecomposition Σ = V Λ Vᵀ: by the spectral theorem, Σ is diagonalizable in an orthonormal basis. The columns of V are the principal axes; the diagonal entries of Λ are the variances along them.
  • Each eigenvalue is the variance along that PC. Their sum is the total variance (the trace of Σ).

SVD on an Image

Slide k to keep more singular components and watch the image sharpen. The scree plot shows how fast the singular values fall off.

SVD on a centered data matrix is PCA. Here the "data" is a 32×32 grayscale image. Keeping only the top k singular values gives the best rank-k approximation in the Frobenius norm — the Eckart–Young theorem.

original (rank 32)
reconstruction (rank 4)
Frobenius error
10.4%
‖A − A_k‖ / ‖A‖
variance kept
98.9%
Σᵢ₌₁ᵏ σᵢ² / Σ σᵢ²
storage ratio
0.25×
k(m+n+1) / mn
singular value spectrum (scree plot)

The scree plot shows how singular values decay. Real-world images and datasets are typically low-rank in this sense — a handful of components captures most of the variance. PCA exploits exactly this fact: project onto the top-k right singular vectors and you preserve the maximum possible variance for any k-dimensional linear projection.

Go deeper:The SVD view

Equivalently — and this is the form that scales — the principal components are the right singular vectors of the centered data matrix X. The Singular Value Decomposition X = U Σ Vᵀ packages everything: the columns of V are the principal directions, and the squared singular values σₖ² are n times the variances. Among all rank-k approximations of X, U_k Σ_k V_kᵀ minimizes the Frobenius error — the Eckart–Young theorem makes this optimality precise. PCA is the unique linear projection onto a k-dimensional subspace that preserves the maximum amount of variance.

  • SVD X = U Σ_s Vᵀ: the same V appears here. The connection is algebraic: Σ = (1/n) V Σ_s² Vᵀ, so eigenvalues of Σ are σₖ² / n.
  • Eckart–Young: truncating to the top k singular components gives the best rank-k approximation in any unitarily invariant norm — Frobenius and spectral norm included.
  • The SVD form is preferred numerically: never form Xᵀ X explicitly — that squares the condition number.

Project a 3D Cloud to 2D

Orbit the 3D cloud, then project it onto the plane of PC1 and PC2. The variance along the third axis is what you give up.

A 3D point cloud shaped like an elongated ellipsoid. The three colored arrows are the principal axes — eigenvectors of the 3×3 covariance matrix, sorted by eigenvalue. Drag to orbit. Toggle the projection to collapse the cloud onto the PC1-PC2 plane.

top-2 PCs preserve 98.3% of variance
PC1
λ = 16.034
85.3% variance
PC2
λ = 2.437
13.0% variance
PC3
λ = 0.317
1.7% variance

Optimal linear dimensionality reduction means: of all linear maps from R³ → R², the projection onto PC1 and PC2 loses the least variance. The third eigenvalue is the variance you throw away when you collapse onto that plane.

Go deeper:The math objects
  • Whitening: mapping data with V Λ⁻¹ᐟ² Vᵀ leaves a cloud with identity covariance — useful as preprocessing for many downstream methods.
  • PCA is linear. Real data often lies on curved manifolds; that's where kernel PCA, autoencoders, and t-SNE pick up the story.

Key takeaways

  • Principal components = eigenvectors of the covariance = right singular vectors of X.
  • Each eigenvalue is the variance along its principal component.
  • Keeping the top k SVD components gives the best rank-k approximation.
  • PCA is linear: it finds flat summaries and cannot follow curves.

Spot a mistake or have an idea?

Search