Every pandas tutorial starts with a clean dataset. Real work never does. Real work starts with a file called sales_final_v3_USE_THIS.xlsx where the header is on row four, prices are stored as text with currency symbols, half the dates are in the wrong format, and the region column contains “North”, “north”, “N” and an empty cell. … Read more
Nobody warns you about this part. You learn joins, you learn aggregation, you land a job, and then you discover that the field called district contains “Pune”, “pune”, ” Pune “, “PUNE”, “Pune ” and “Punr”. And a number. And, for reasons nobody can explain, the word “test”. Cleaning data is the majority of analytics … Read more
The brief The client came to me with a simple sounding problem: “our sales numbers do not match and we do not know why.” They had been running the business for a few years. Data had accumulated in the way it usually does, which is to say one file per month, exported from different systems … Read more