I2 · Working with Data: NumPy, pandas, SQL and Visualisation

Series and DataFrames

pandas is Python's main library for tabular data. A Series is a one-dimensional labelled array: values plus an index of labels. A DataFrame is a two-dimensional table of columns, each a Series that can have its own type, sharing one row index. pandas adds labelled selection, alignment by label, missing-data handling and many tools for reading, cleaning and summarising data.

10 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

A NumPy array knows that its third column is numbers, but not that it is called "science". What if your data could keep names for its rows and columns, and mix text, dates and numbers in one table?

pandas is Python's main library for tabular data. A Series is a one-dimensional labelled array: values plus an index of labels. A DataFrame is a two-dimensional table of columns, each a Series that can have its own type, sharing one row index. pandas adds labelled selection, alignment by label, missing-data handling and many tools for reading, cleaning and summarising data.

In simple words

A DataFrame is a spreadsheet you control with code: named columns, a row for each record, and commands such as "average of the maths column" or "rows where city is Pune". A Series is a single column of it.

Overview

  • pandas, created by Wes McKinney in 2008, is Python's main library for tables of data.
  • A Series is one labelled column; a DataFrame is a table of Series that share an index.
  • Each column has one data type, but different columns can have different types.
  • The index labels rows; columns are reached by name: df["marks"].
  • pandas is built on NumPy, so vectorised operations and broadcasting still apply.
  • head(), info(), describe() and shape give a first view of any table.

Why it matters

Almost every data science project begins by loading data into a DataFrame. pandas is how analysts explore, clean, combine and summarise data before modelling, and scikit-learn and plotting libraries accept DataFrames directly.

Key terms

Series
A one-dimensional labelled array.
DataFrame
A table of labelled columns sharing a row index.
Index
The labels for rows (or for a Series' values), such as 0, 1, 2 or student IDs.
dtype
The type of each column, such as int64, float64, str (text) or datetime64.
Alignment
pandas matching values by label, not position, when combining Series.
Vectorised column operation
Arithmetic on whole columns, such as df["total"] = df["maths"] + df["science"].

Types

pandas has a few core data structures, and its columns come in several types. Knowing them avoids most beginner errors.

1. Series

A one-dimensional labelled array: values plus an index.

  • How it works: pd.Series([78, 85], index=["Asha", "Ravi"]) lets you read s["Asha"]; arithmetic aligns values by label, not position.
  • Example: A Series of daily rainfall indexed by date.
  • When to use it: Use it for one variable: a single column, a time series or a result per group.

2. DataFrame

A two-dimensional table of labelled rows and columns.

  • How it works: Build it from a dictionary of lists, a list of dictionaries or a file; df.loc selects by label and df.iloc by position.
  • Example: pd.DataFrame({"name": ["Asha", "Ravi"], "maths": [78, 66]}).
  • When to use it: Use it for any data set with rows of records and columns of features.

3. Index types

The row labels: a default RangeIndex (0, 1, 2…), labels such as names, a DatetimeIndex or a MultiIndex with several levels.

  • How it works: set_index("date") makes a column the index; reset_index() moves it back; a DatetimeIndex allows df.loc["2024-03"] to select a month.
  • Example: Sales indexed by (state, city) as a MultiIndex for grouped reports.
  • When to use it: Use meaningful indexes for lookups and time series; keep the default for plain tables.

4. Numeric column types

int64, float64 and nullable Int64 columns.

  • How it works: Missing numbers become NaN, which forces an integer column to float64 unless the nullable Int64 type is used.
  • Example: A marks column read from CSV is int64, but becomes float64 if one mark is blank.
  • When to use it: Use them for measurements and counts; check dtypes after loading.

5. Text, category, boolean and datetime column types

object or string columns for text, category for repeated labels, bool for flags and datetime64 for dates.

  • How it works: astype("category") stores each label once and saves memory; pd.to_datetime turns text into dates that support .dt.year and date arithmetic.
  • Example: A state column with 28 values repeated over a million rows becomes a small category column.
  • When to use it: Convert columns to their proper types early: it saves memory and enables the right operations.

Explained in detail

Origins

Wes McKinney began writing pandas in 2008 while working at the investment firm AQR Capital, because he needed tools in Python for the kind of labelled, messy financial data that spreadsheets and the R language handled well. It was open-sourced in 2009; the name comes from "panel data", an economics term for multi-dimensional data. pandas grew into one of the most used Python libraries, and its DataFrame idea has been copied by many newer tools. Version 2.0 in 2023 added faster, more memory-efficient data types backed by Apache Arrow, and pandas 3.0 continues that direction.

Building and inspecting

CodeWhat it does
pd.DataFrame({"name": [...], "maths": [...]})Build a table from a dictionary of columns
df.head(), df.tail()Show the first or last five rows
df.shape(rows, columns)
df.info()Column names, types and non-missing counts
df.describe()Count, mean, std, min, quartiles and max for numeric columns
df["maths"]One column as a Series
df[["name", "maths"]]Several columns as a DataFrame

Labels change everything

The key difference from NumPy is that pandas works with labels. If one Series has marks indexed by student names and another has attendance indexed by the same names in a different order, adding them matches Riya with Riya, not position 0 with position 0. Labels missing from one side produce NaN, which pandas then helps you find and handle. Columns can have different types: names as text, marks as integers, dates of birth as datetimes. Behind each numeric column there is still a fast array, so column arithmetic such as df["total"] = df["maths"] + df["science"] is vectorised just as in NumPy.

Good first habits

  • Always run df.shape, df.head() and df.info() right after loading data.
  • Prefer column operations to loops over rows; iterrows() is slow and rarely needed.
  • Use df.to_numpy() when a library needs a plain array.
  • Set a meaningful index, such as a student ID, with df.set_index("id") when it helps lookups.

Selecting rows by label uses df.loc, and by position df.iloc; the next pages cover both in detail, along with reading files, filtering, grouping and merging, the everyday tasks that make pandas the workhorse of data analysis. A useful mental model: each column is a NumPy array with a name, and the DataFrame is a dictionary of those named columns sharing one row index.

How it works

  1. import pandas as pd.
  2. Create or load a DataFrame.
  3. Inspect it with shape, head, info and describe.
  4. Select columns by name and compute new columns with vectorised arithmetic.
  5. Check types and missing values before analysing.

In code

dataframes.py
import pandas as pd

df = pd.DataFrame({
    "name": ["Riya", "Arjun", "Sara", "Kabir", "Meera"],
    "city": ["Pune", "Delhi", "Pune", "Jaipur", "Delhi"],
    "maths": [91, 78, 85, 62, 74],
    "science": [88, 81, 90, 58, 79],
})
print(df.head(3))
print("Shape:", df.shape)
print(df.dtypes)

df["total"] = df["maths"] + df["science"]
print(df[["name", "total"]])
print(df[["maths", "science"]].describe().round(1))

attendance = pd.Series({"Sara": 0.95, "Riya": 0.92, "Arjun": 0.88})
marks = df.set_index("name")["maths"]
print("Aligned by name:\n", (marks * attendance).round(1))

Output

    name   city  maths  science
0   Riya   Pune     91       88
1  Arjun  Delhi     78       81
2   Sara   Pune     85       90
Shape: (5, 4)
name         str
city         str
maths      int64
science    int64
dtype: object
    name  total
0   Riya    179
1  Arjun    159
2   Sara    175
3  Kabir    120
4  Meera    153
       maths  science
count    5.0      5.0
mean    78.0     79.2
std     11.1     12.7
min     62.0     58.0
25%     74.0     79.0
50%     78.0     81.0
75%     85.0     88.0
max     91.0     90.0
Aligned by name:
 name
Arjun    68.6
Kabir     NaN
Meera     NaN
Riya     83.7
Sara     80.8
dtype: float64

Try it yourself

Labels to the rescue

Two teachers send data: one sends marks, the other attendance, listed in different orders.

Match the method to its purpose

What does each tell you?

  • df.shape
  • df.info()
  • df.describe()
  • df.head()

Common mistakes

  • Looping over rows instead of using column operations.
  • Assuming position-based matching when pandas aligns by label.
  • Skipping df.info() and missing wrong types or blanks.

Check yourself

Question 1

1. What is a pandas Series?

Question 2

2. When adding two Series, pandas matches values by…

Question 3

3. Which shows column types and non-missing counts?

Summary

  • A DataFrame is a labelled table of typed columns; a Series is one column.
  • pandas aligns by label and makes missing data visible.
  • Inspect with shape, head, info and describe, and use column operations, not loops.

Finished reading?

Sources: pandas: 10 minutes to pandas · Python for Data Analysis, 3rd edition (Wes McKinney)