Handling Duplicates

"At-least-once delivery" is a common cause, but other sources include:

• An Excel jockey accidentally copies and pastes the same rows twice
• A Database export has duplicate rows due to an erroneous join
• Webhook retries create repeated records

Pandas gives you the duplicated() method, which returns a boolean Series: each True value indicates a duplicate row (or duplicate key) in the DataFrame. So, you can use it to filter them out:

duplicates_mask = df.duplicated()
# [False, False, True, False, True, ...]

df = df[duplicates_mask]  # Keep only the duplicates

Alternatively, you can only check for duplicates based on specific columns:

duplicates_mask = df.duplicated(subset=["order_id", "purchase_time"])

Or, you can keep the last occurrence of each duplicate instead of the first (default):

duplicates_mask = df.duplicated(keep="last")

The drop_duplicates() outright removes duplicate rows:

df = df.drop_duplicates()

Similarly, it has some useful options:

# keep last occurrence instead
df = df.drop_duplicates(keep="last")

# keep NO occurrences
df = df.drop_duplicates(keep=False)

# only consider specific columns for duplicates
df = df.drop_duplicates(subset=["order_id", "purchase_time"])

SnackStack has duplicate device records, making the team think more devices "passed" QA than actually did.

Complete the get_batch_pass_rate function. It accepts a device DataFrame (with device_id and status columns), removes duplicate device records, and returns the accurate batch "pass rate" as a percentage.

1. Remove duplicate device records, keeping only the first occurrence for each device_id.
2. Count the total number of unique devices (the number of rows in the device_id column).
3. Count how many devices have a status of "passed".
4. Calculate and return the pass rate as a percentage: (passed / total) * 100.