Help fix thousands of malformed date tags

[Minh_Nguyen]

start_date=* and end_date=* are the most important keys in OHM, controlling the appearance of a feature using the time slider or a similar date filter. We’re stricter about the format than OpenStreetMap. Uncertainty and approximation go in separate start_date:edtf=* and end_date:edtf=* keys.

Unfortunately, some JOSM users have been adding malformed dates that aren’t compatible with OHM tools, probably due to familiarity with OSM. At one point, this was causing the renderer to get stuck. Now the OHM renderer merely ignores these malformed tags and assumes the feature has existed since the beginning of time and is everlasting. This still leads to some unfortunate rendering quirks:

Fortunately, it’s easy to find these malformed tags and correct them. This SPARQL query (documented here) returns 10,342 features with malformed dates, updating daily. Click “Execute” to run the query, then “Map view” to see it on a map:

If you zoom into a hotspot, you can click on a feature to see its details and a link to it on the OHM site:

Open the feature in iD to reveal the errors and a suggested fix. Clicking the “Tag as” suggestion will populate the Extended Date/Time Format subfield (that is, start_date:edtf=* or end_date:edtf=*). Don’t forget to also set the main Start Date or End Date to a resolved date within the date range.

In some cases, iD won’t be able to guess what the tag was supposed to indicate, so you may need to consult the source or contact the original mapper to find out. But at a glance, very many are straightforward, so there’s plenty to do even if you aren’t in the mood to do that research.

Someday we should have our own OHM-specific JOSM validation rules to minimize this kind of error:

Publish JOSM validator rules

In the meantime, remember this activity when you need a distraction from your deep research or delicate mapping exercise.

[Lance]

Perhaps a newbie question here. But…

So, the issue is the dates are in a bad format, which causes an error?

Could we run the query, extract the data, transform the bad dates, and give the file back to you to apply? I realize we may not know what the date SHOULD be, but could we do something like pick an arbitrary date in the correct format. That way it would at least allow the file to render, even if with the wrong date. Is a wrong date better than a bad date?

Then is there a way in the data that we can join the user who created it, to create a mailing list that could be sent out with instruction on what to fix. Oh, and of course, WHAT to fix. Important detail there.

[Minh_Nguyen]

@Lance

Could we run the query, extract the data, transform the bad dates, and give the file back to you to apply? I realize we may not know what the date SHOULD be, but could we do something like pick an arbitrary date in the correct format. That way it would at least allow the file to render, even if with the wrong date. Is a wrong date better than a bad date?

As things stand, malformed dates cause the feature to appear when it shouldn’t. Personally, I’m a bit of an eventualist, so I’d rather see us work towards the correct date than fudge things to disappear from the map. Better to make an error stick out like a sore thumb than invisible and undiscoverable, but even better to fix the error.

We could do some mass transformations to fix some common patterns and upload those changes in bulk – anyone could do that. However, there’s a lot of other malformed dates that don’t fit any patterns. I suspect it’s because OSM has a different format for start_date=* and doesn’t really use it for much, so JOSM presents it as a freeform field and that’s how some mappers have traditionally thought of it.

@Lance

Then is there a way in the data that we can join the user who created it, to create a mailing list that could be sent out with instruction on what to fix. Oh, and of course, WHAT to fix. Important detail there.

Due to the OSM/OHM data model, it’s difficult to query for who added a tag in the first place, but here’s a modified version of the earlier SPARQL query that finds all the elements with malformed dates, groups them by which user last edited them, and ranks the users. It’s also possible to query for only the elements last modified by an individual user.

Unfortunately, some of the users at the top of this chart haven’t been active in OHM for a while, and most of those who are still active haven’t joined this forum yet. So we’d need to reach out to them directly via private messages on openhistoricalmap.org.

[jeffmeyer]

@Lance - just wanted to double-back here and see if @Minh_Nguyen 's answer made sense, etc.

Minh & I talked today and our OHM to-do lists are quite long and they get longer every time either of us looks at the map.

If there’s any way that you (or someone else) could help tackle some of the “common patterns,” that would be uh-mazing. I think the key would be a very quick documentation of what you were solving (like writing a ticket) and your approach for solving it, and I’m sure we could turn around a community review very quickly. I’d be most of the people listed on Minh’s query would be grateful, too!