Definition
Every minute, people around the world send millions of messages, make huge numbers of online payments and upload hundreds of hours of video. Who could possibly look at all of it?
Big data means collections of data so large, so fast-growing or so varied that ordinary tools such as a single spreadsheet cannot handle them. It is often described by three Vs: volume (how much), velocity (how fast it arrives) and variety (how many different kinds). Big data is one of the main reasons modern AI became possible.
In simple words
Counting the mangoes in one basket is easy. Counting every mango sold in every market in India, every hour, as trucks keep arriving, and also reading the farmers' notes and photos of each crate, needs a completely different system. That is the jump from data to big data.
Why it matters
Big data powers maps, recommendations, fraud detection, weather forecasts and AI chatbots. It also raises serious questions about privacy and power, because whoever holds huge amounts of data about people can learn a great deal about them.
Key terms
- Big data
- Data too large, fast or varied for ordinary tools.
- Volume
- The sheer amount of data.
- Velocity
- How fast new data arrives.
- Variety
- The different forms of data: numbers, text, images, sound, video, locations.
- Data centre
- A building full of computers that store and process large amounts of data.
- Gigabyte and terabyte
- Units of data size. A terabyte is about a thousand gigabytes.
Explained in detail
The three Vs
| V | Meaning | Example |
|---|---|---|
| Volume | Huge amounts | Every UPI payment in India in a year: many billions of transactions |
| Velocity | Arriving very fast | Location pings from every phone using a maps app during rush hour |
| Variety | Many kinds mixed together | A hospital's records: numbers, doctors' notes, X-ray images and scans |
Some people add more Vs, such as veracity (how trustworthy the data is) and value (whether it is useful). Veracity is important: a huge pile of messy, biased or wrong data is still messy, biased and wrong. More data helps only when it is good data.
Where big data comes from
- Phones: locations, app use, photos and messages.
- Payments: every card swipe, UPI transfer and online order.
- Sensors: weather stations, satellites, traffic cameras, smart meters and fitness bands.
- The internet: web pages, videos, posts, searches and clicks.
- Organisations: hospital records, school results, government surveys.
How it is handled
No single computer can store or process data at this scale, so the work is split across thousands of computers in data centres working at the same time, each handling a small piece. The results are then combined. This idea of dividing a huge job into many small ones that run side by side is what makes big data, and training large AI models, possible. It is also why data centres use so much electricity and water, as you learned in the Responsible AI module.
Big data and AI
Machine learning improves with more examples. A model that learns to recognise speech from a few hundred recordings will be poor; one that learns from many thousands of hours across many accents will be far better. Large language models were trained on enormous collections of text. This is why the companies with the most data often have an advantage in AI, and why questions of who owns data, who agreed to its use and who benefits from it matter so much.
Big data is also used for public good: tracking the spread of disease, predicting floods and planning bus routes from travel patterns.
How it works
- Ask how much data there is, how fast it arrives and what kinds it includes.
- Ask whether it is accurate and representative, not just large.
- Ask who collected it, with whose consent, and for what purpose.
Try it yourself
How a maps app knows about the traffic jam
You open a maps app and a road is shown in red. Nobody reported a jam. How does it know?
Which V is it?
Which V of big data does each example show best?
A farm app collecting soil numbers, crop photos and farmers' voice notes
Every book in a national library, scanned
All the photos ever uploaded to a social media site
Stock prices changing many times a second
Live readings from smart electricity meters across a city
Common mistakes
- Thinking more data always means better answers.
- Forgetting that big data about people raises privacy questions.
- Assuming big data is only about size, and ignoring speed and variety.
Check yourself
Summary
- Big data is too large, fast or varied for ordinary tools: volume, velocity and variety.
- It is processed by many computers working together, and it powers modern AI.
- Quality, consent and privacy matter as much as size.
Finished reading?
Sources: Open Government Data Platform India · Digital Personal Data Protection Act, 2023 (MeitY)