Coincidentally, something strange happened to the California COVID-in-wastewater data yesterday. When I reloaded my little dashboard this morning, the COVID-in-wastewater line jumped up. I don't just mean that there was a high number yesterday; the historical numbers jumped up, too.
The little computer script that that computes this dashboard's data, when it grabbed data and saved it, just saved the data in the same place each time. Thus, today's data file "clobbered" yesterday's data file on my hard drive. In theory this is fine; because the file that gets grabbed today includes yesterday's data, too. But what if there are a bunch of mysterious historical changes, too? Then clobbering yesterday's file with today's file is not fine.
On a previous occasion, I emailed the California Open Data Portal people and begged them for whatever backup data they could spare me. This time… this time I might not bother?
I'm not going to try to debug today's weird changes. I ?think? it also affected the data that the California Department of Public Health uses; I bet those people know how to debug this better than I do.
But I did change my little computer script so that it doesn't "clobber" yesterday's file with today's data. So the next time there's a mysterious change that goes back into the historical data, maybe I'll have an easier time figuring out what changed.
