We're sorry but this app doesn't work properly without JavaScript enabled. Please enable it to continue.

This lesson's interactive features are locked, please to keep using them

Handling Missing Values

Missing data is everywhere. People skip fields, systems fail to populate values, imports go awry.

You generally have three options: remove it, fill it in, or leave it. Which one is right depends on the situation. Sometimes a missing value is actually meaningful, but sometimes it's an artifact of system failure.

Detecting Missing Values

Pandas uses several markers for missing data, including NaN, NaT, and None, depending on the column type.

The nice part: isna() and isnull() treat all of those values as missing. isna() is more common, but they do basically the same thing:

# Check for nulls
df.isna()  # Returns a boolean DataFrame
df.isnull()  # Same result

# Count nulls per Series
df.isna().sum()

# Count nulls in full DataFrame
df.isna().sum().sum()

Drop Missing Values

The dropna() method removes rows (or columns) with nulls:

# Drop rows with any nulls
df = df.dropna()

# Drop rows where specific column is null
df = df.dropna(subset=["customer_id"])

# Drop columns with any nulls
df = df.dropna(axis="columns")

Fill-In Missing Values

The fillna() method replaces nulls with a given value:

# Fill with zero
df["discount_amount"] = df["discount_amount"].fillna(0)

# Fill with the mean (i.e. average) value of the column
df["order_total"] = df["order_total"].fillna(df["order_total"].mean())

# Forward fill (reuse the last valid value)
df["shipment_status"] = df["shipment_status"].ffill()

# Fill different values per column
df = df.fillna(
    {
        "discount_amount": 0,
        "shipment_status": "unknown",
        "order_total": df["order_total"].median(),
    }
)

Assignment

SnackStack's analytics team is getting incomplete records from a smart fridge on the manufacturing floor. Some records are missing device IDs or temperatures.

Complete the clean_device_data function.