removing duplicates

Data Cleaning With Pandas: A Step by Step Walkthrough on a Messy File

by

Every pandas tutorial starts with a clean dataset. Real work never does. Real work starts with a file called sales_final_v3_USE_THIS.xlsx where the header is on row four, prices are stored as text with currency symbols, half the dates are in the wrong format, and the region column contains “North”, “north”, “N” and an empty cell. … Read more

Cleaning Messy Data in SQL: The Patterns You Will Use Every Week

by

Nobody warns you about this part. You learn joins, you learn aggregation, you land a job, and then you discover that the field called district contains “Pune”, “pune”, ” Pune “, “PUNE”, “Pune ” and “Punr”. And a number. And, for reasons nobody can explain, the word “test”. Cleaning data is the majority of analytics … Read more