I3 · Mathematics for AI

Vectors and Matrices

A scalar is one number. A vector is an ordered list of numbers. A matrix is a grid of numbers with rows and columns. A tensor is the general idea: an array of numbers with any number of dimensions, so a scalar is a 0D tensor, a vector 1D and a matrix 2D.

10 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

How can a photo or a sentence go into a formula? It has to become numbers first.

A scalar is one number. A vector is an ordered list of numbers. A matrix is a grid of numbers with rows and columns. A tensor is the general idea: an array of numbers with any number of dimensions, so a scalar is a 0D tensor, a vector 1D and a matrix 2D.

In simple words

One student's marks in one subject is a scalar: 85. That student's marks in three subjects is a vector: [85, 70, 92]. The whole class's marks form a matrix, with one row per student and one column per subject. Marks for every class in a school, in every exam, make a tensor.

Overview

  • Linear algebra is the language AI uses to store and transform data.
  • A scalar is a single number; a vector is an ordered list of numbers; a matrix is a rectangular grid; a tensor generalises these to any number of axes.
  • The shape lists the length of each axis, such as a 28 × 28 image or a 32 × 224 × 224 × 3 batch.
  • Each data row is a vector of features; a data set is a matrix of samples × features.
  • Neural-network weights are matrices, and their inputs and outputs are tensors.
  • NumPy arrays and PyTorch tensors implement all of these objects.

Why it matters

Every model works only on numbers. A customer becomes a vector, a table of customers becomes a matrix, and a colour image becomes a 3D tensor. All the training code you'll ever write comes down to operations on these objects, and most errors in real projects are shape errors.

Key terms

Scalar
A single number, such as a temperature or a learning rate.
Vector
An ordered list of numbers; it can represent a point, a direction or a set of features.
Matrix
A rectangular grid of numbers with rows and columns.
Tensor
An array of numbers with any number of dimensions (axes).
Shape
The size along each axis, such as (3, 4) for 3 rows and 4 columns.
Feature vector
A vector describing one example, with one number per feature.

Types

Linear-algebra objects come in several types by their number of axes, and some matrices have special forms. These are the types you will meet.

1. Scalar (rank 0)

A single number with no axes.

  • How it works: Scalars multiply vectors and matrices element by element and set quantities such as a learning rate.
  • Example: A temperature of 31°C, a learning rate of 0.001 or a bias term b.
  • When to use it: Use scalars for single settings, constants and results such as a loss value.

2. Vector (rank 1)

An ordered list of numbers, with one axis.

  • How it works: A vector has a length (magnitude) and a direction; vectors add element by element and are compared with dot products and distances.
  • Example: A house as [area 1200, rooms 3, age 10]; a word embedding of 300 numbers.
  • When to use it: Use vectors for one sample's features, embeddings and a model's weights for one output.

3. Matrix (rank 2)

A rectangular grid of numbers with rows and columns.

  • How it works: An m × n matrix can store a data set, or represent a transformation that turns n-dimensional vectors into m-dimensional ones.
  • Example: A 1,000 × 10 table of 1,000 customers and 10 features; a greyscale image.
  • When to use it: Use matrices for tabular data, images with one channel and the weights of a neural-network layer.

4. Tensor (rank 3 and above)

An array with three or more axes.

  • How it works: Each axis has a meaning, such as batch, height, width and channel; operations work along chosen axes.
  • Example: A colour image is height × width × 3; a batch of videos is batch × frames × height × width × 3.
  • When to use it: Use tensors for colour images, video, audio spectrograms and batches in deep learning.

5. Special matrices

Matrices with structure: square, identity, diagonal, symmetric, zero and sparse.

  • How it works: The identity matrix I leaves vectors unchanged; a diagonal matrix scales each axis; a symmetric matrix equals its transpose; a sparse matrix is mostly zeros and is stored compactly.
  • Example: A covariance matrix is symmetric; a bag-of-words matrix for 10,000 documents is sparse.
  • When to use it: Recognise them because they allow faster algorithms and save memory, especially sparse text data.

Explained in detail

Everything becomes numbers

Machine learning works only with numbers, so every kind of data has to be turned into vectors, matrices or tensors. A house might become the vector [3, 1200, 15, 2]: bedrooms, area in square feet, age in years and floor number. A table of 1,000 houses becomes a matrix with 1,000 rows and 4 columns, one row per house. A grayscale image of 28 × 28 pixels is a matrix of brightness values; a colour image adds a third axis for red, green and blue, making a 3D tensor; a batch of 64 colour images is a 4D tensor of shape (64, 28, 28, 3). A sentence becomes a sequence of token vectors. Learning to think in these shapes is the foundation of all practical AI.

Ranks and shapes

ObjectNumber of axesExample shapeAI example
Scalar0()A loss value, a learning rate
Vector1(4,)One house's features, one word embedding
Matrix2(1000, 4)A data table, a layer's weights
3D tensor3(28, 28, 3)One colour image
4D tensor4(64, 28, 28, 3)A batch of images

Vectors as points and arrows

A vector with two numbers can be drawn as a point on a graph, or as an arrow from the origin to that point. A vector with three numbers is a point in 3D space. Machine learning routinely uses vectors with hundreds or thousands of numbers. We cannot picture 768-dimensional space, but the same ideas carry over exactly: distance between points, the angle between arrows, adding and scaling. Word embeddings place words in such a space so that words with similar meanings, such as "king" and "queen", end up as nearby points.

Basic operations

  • Adding vectors of the same length adds element by element: [1, 2] + [3, 4] = [4, 6].
  • Multiplying by a scalar scales every element: 3 × [1, 2] = [3, 6].
  • Shapes must be compatible; adding vectors of lengths 3 and 4 is undefined, and NumPy raises an error.
  • NumPy broadcasting stretches smaller arrays to match when it is unambiguous, such as adding one vector to every row of a matrix.

The word tensor gave its name to Google's TensorFlow library, and PyTorch's central object is torch.Tensor. In code, most bugs in deep learning are shape mismatches, so a good habit is to print x.shape at every step until the flow of shapes through your program is clear.

How it works

ObjectDimensionsShape exampleIn AI
Scalar0()A loss value, a learning rate
Vector1(3,)One customer's features
Matrix2(100, 3)100 customers, 3 features each
3D tensor3(224, 224, 3)A colour image: height, width, RGB
4D tensor4(32, 224, 224, 3)A batch of 32 colour images

Notation you'll see in papers

  • Lowercase italic x is a scalar. Lowercase bold x is a vector. Uppercase A is a matrix.
  • x_i is element i of a vector. A_ij is the element in row i and column j of a matrix.
  • The symbol ℝ means the real numbers, so ℝⁿ is the set of all vectors with n numbers.

The maths

A vector and a matrix
How many numbers the vector holds, or how many columns the matrix has
How many rows the matrix has
The number in row i, column j

In code

shapes.py
import numpy as np

scalar = np.array(85)
vector = np.array([85, 70, 92])
matrix = np.array([[85, 70, 92], [60, 75, 88]])
tensor = np.zeros((32, 224, 224, 3))

for name, value in [("scalar", scalar), ("vector", vector), ("matrix", matrix), ("tensor", tensor)]:
    print(name, "dimensions:", value.ndim, "shape:", value.shape)

print("Row 1, column 2 of the matrix:", matrix[0, 1])

Output

scalar dimensions: 0 shape: ()
vector dimensions: 1 shape: (3,)
matrix dimensions: 2 shape: (2, 3)
tensor dimensions: 4 shape: (32, 224, 224, 3)
Row 1, column 2 of the matrix: 70

Common mistakes

  • Mixing up rows and columns. In ML data, rows are almost always examples and columns are features.
  • Confusing a vector of shape (3,) with a matrix of shape (3, 1). They look alike but behave differently in code.
  • Counting from 1 in code. Maths usually starts at 1, while Python starts at 0.

Check yourself

Question 1

1. A dataset has 500 rows and 8 columns. What is its shape?

Question 2

2. How many dimensions does a batch of colour images have?

Question 3

3. Which is a tensor?

Summary

  • Scalar: 0D. Vector: 1D. Matrix: 2D. Tensor: any number of dimensions.
  • All data must become these before a model can use it.
  • Shape tells you what the data is. Rows are examples, columns are features.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning