Data Cleaning

Raw data kinda sucks. It usually has problems like:

• Missing values: NaN, None, empty strings
• Duplicates: the same record shows up more than once
• Wrong types: numbers stored as strings
• Inconsistent formats: 2024-01-15 vs. 01/15/2024 vs. Jan 15, 2024
• Invalid data: ages of 200, negative prices, future dates

Before you analyze anything, you need to make the data consistent enough to trust. That's data cleaning: finding problems, deciding what to do about them, and making those fixes repeatable.

Cleaning Is Not Optional

"Garbage in, garbage out" is the basic rule of data work. If the input is dirty, so's the output.

The more you can standardize and automate your cleaning with tools like Pandas or Polars, the easier it is to produce usable results.

You'll spend 60–80% of your time cleaning data. Get it right, and make it repeatable, so you don't keep fixing the same messes by hand.

I have 3 rules:

1. Handle missing data explicitly in code
2. Always validate after cleaning
3. Keep the original data by only cleaning a copy