Definition
A NumPy array knows that its third column is numbers, but not that it is called "science". What if your data could keep names for its rows and columns, and mix text, dates and numbers in one table?
pandas is Python's main library for tabular data. A Series is a one-dimensional labelled array: values plus an index of labels. A DataFrame is a two-dimensional table of columns, each a Series that can have its own type, sharing one row index. pandas adds labelled selection, alignment by label, missing-data handling and many tools for reading, cleaning and summarising data.
In simple words
A DataFrame is a spreadsheet you control with code: named columns, a row for each record, and commands such as "average of the maths column" or "rows where city is Pune". A Series is a single column of it.
Overview
- pandas, created by Wes McKinney in 2008, is Python's main library for tables of data.
- A Series is one labelled column; a DataFrame is a table of Series that share an index.
- Each column has one data type, but different columns can have different types.
- The index labels rows; columns are reached by name: df["marks"].
- pandas is built on NumPy, so vectorised operations and broadcasting still apply.
- head(), info(), describe() and shape give a first view of any table.
Why it matters
Almost every data science project begins by loading data into a DataFrame. pandas is how analysts explore, clean, combine and summarise data before modelling, and scikit-learn and plotting libraries accept DataFrames directly.
Key terms
- Series
- A one-dimensional labelled array.
- DataFrame
- A table of labelled columns sharing a row index.
- Index
- The labels for rows (or for a Series' values), such as 0, 1, 2 or student IDs.
- dtype
- The type of each column, such as int64, float64, str (text) or datetime64.
- Alignment
- pandas matching values by label, not position, when combining Series.
- Vectorised column operation
- Arithmetic on whole columns, such as df["total"] = df["maths"] + df["science"].
Types
pandas has a few core data structures, and its columns come in several types. Knowing them avoids most beginner errors.
1. Series
A one-dimensional labelled array: values plus an index.
- How it works: pd.Series([78, 85], index=["Asha", "Ravi"]) lets you read s["Asha"]; arithmetic aligns values by label, not position.
- Example: A Series of daily rainfall indexed by date.
- When to use it: Use it for one variable: a single column, a time series or a result per group.
2. DataFrame
A two-dimensional table of labelled rows and columns.
- How it works: Build it from a dictionary of lists, a list of dictionaries or a file; df.loc selects by label and df.iloc by position.
- Example: pd.DataFrame({"name": ["Asha", "Ravi"], "maths": [78, 66]}).
- When to use it: Use it for any data set with rows of records and columns of features.
3. Index types
The row labels: a default RangeIndex (0, 1, 2…), labels such as names, a DatetimeIndex or a MultiIndex with several levels.
- How it works: set_index("date") makes a column the index; reset_index() moves it back; a DatetimeIndex allows df.loc["2024-03"] to select a month.
- Example: Sales indexed by (state, city) as a MultiIndex for grouped reports.
- When to use it: Use meaningful indexes for lookups and time series; keep the default for plain tables.
4. Numeric column types
int64, float64 and nullable Int64 columns.
- How it works: Missing numbers become NaN, which forces an integer column to float64 unless the nullable Int64 type is used.
- Example: A marks column read from CSV is int64, but becomes float64 if one mark is blank.
- When to use it: Use them for measurements and counts; check dtypes after loading.
5. Text, category, boolean and datetime column types
object or string columns for text, category for repeated labels, bool for flags and datetime64 for dates.
- How it works: astype("category") stores each label once and saves memory; pd.to_datetime turns text into dates that support .dt.year and date arithmetic.
- Example: A state column with 28 values repeated over a million rows becomes a small category column.
- When to use it: Convert columns to their proper types early: it saves memory and enables the right operations.
Explained in detail
Origins
Wes McKinney began writing pandas in 2008 while working at the investment firm AQR Capital, because he needed tools in Python for the kind of labelled, messy financial data that spreadsheets and the R language handled well. It was open-sourced in 2009; the name comes from "panel data", an economics term for multi-dimensional data. pandas grew into one of the most used Python libraries, and its DataFrame idea has been copied by many newer tools. Version 2.0 in 2023 added faster, more memory-efficient data types backed by Apache Arrow, and pandas 3.0 continues that direction.
Building and inspecting
| Code | What it does |
|---|---|
| pd.DataFrame({"name": [...], "maths": [...]}) | Build a table from a dictionary of columns |
| df.head(), df.tail() | Show the first or last five rows |
| df.shape | (rows, columns) |
| df.info() | Column names, types and non-missing counts |
| df.describe() | Count, mean, std, min, quartiles and max for numeric columns |
| df["maths"] | One column as a Series |
| df[["name", "maths"]] | Several columns as a DataFrame |
Labels change everything
The key difference from NumPy is that pandas works with labels. If one Series has marks indexed by student names and another has attendance indexed by the same names in a different order, adding them matches Riya with Riya, not position 0 with position 0. Labels missing from one side produce NaN, which pandas then helps you find and handle. Columns can have different types: names as text, marks as integers, dates of birth as datetimes. Behind each numeric column there is still a fast array, so column arithmetic such as df["total"] = df["maths"] + df["science"] is vectorised just as in NumPy.
Good first habits
- Always run df.shape, df.head() and df.info() right after loading data.
- Prefer column operations to loops over rows; iterrows() is slow and rarely needed.
- Use df.to_numpy() when a library needs a plain array.
- Set a meaningful index, such as a student ID, with df.set_index("id") when it helps lookups.
Selecting rows by label uses df.loc, and by position df.iloc; the next pages cover both in detail, along with reading files, filtering, grouping and merging, the everyday tasks that make pandas the workhorse of data analysis. A useful mental model: each column is a NumPy array with a name, and the DataFrame is a dictionary of those named columns sharing one row index.
How it works
- import pandas as pd.
- Create or load a DataFrame.
- Inspect it with shape, head, info and describe.
- Select columns by name and compute new columns with vectorised arithmetic.
- Check types and missing values before analysing.
In code
import pandas as pd
df = pd.DataFrame({
"name": ["Riya", "Arjun", "Sara", "Kabir", "Meera"],
"city": ["Pune", "Delhi", "Pune", "Jaipur", "Delhi"],
"maths": [91, 78, 85, 62, 74],
"science": [88, 81, 90, 58, 79],
})
print(df.head(3))
print("Shape:", df.shape)
print(df.dtypes)
df["total"] = df["maths"] + df["science"]
print(df[["name", "total"]])
print(df[["maths", "science"]].describe().round(1))
attendance = pd.Series({"Sara": 0.95, "Riya": 0.92, "Arjun": 0.88})
marks = df.set_index("name")["maths"]
print("Aligned by name:\n", (marks * attendance).round(1))Output
name city maths science
0 Riya Pune 91 88
1 Arjun Delhi 78 81
2 Sara Pune 85 90
Shape: (5, 4)
name str
city str
maths int64
science int64
dtype: object
name total
0 Riya 179
1 Arjun 159
2 Sara 175
3 Kabir 120
4 Meera 153
maths science
count 5.0 5.0
mean 78.0 79.2
std 11.1 12.7
min 62.0 58.0
25% 74.0 79.0
50% 78.0 81.0
75% 85.0 88.0
max 91.0 90.0
Aligned by name:
name
Arjun 68.6
Kabir NaN
Meera NaN
Riya 83.7
Sara 80.8
dtype: float64Try it yourself
Labels to the rescue
Two teachers send data: one sends marks, the other attendance, listed in different orders.
Match the method to its purpose
What does each tell you?
- df.shape
- df.info()
- df.describe()
- df.head()
Common mistakes
- Looping over rows instead of using column operations.
- Assuming position-based matching when pandas aligns by label.
- Skipping df.info() and missing wrong types or blanks.
Check yourself
Summary
- A DataFrame is a labelled table of typed columns; a Series is one column.
- pandas aligns by label and makes missing data visible.
- Inspect with shape, head, info and describe, and use column operations, not loops.
Finished reading?
Sources: pandas: 10 minutes to pandas · Python for Data Analysis, 3rd edition (Wes McKinney)