Topic: Race condition when exporting the database

Posted under Site Bug Reports & Feature Requests

I am currently developing an e621 client that downloads CSV database exports from https://e621.net/db_export/ and performs searches on them locally to provide some search features the site doesn't. I noticed that today's (2023-11-13) export of the post database contains a tag in the tag_string field (kamu33) of one post (post #4411730) that does not appear in the export of the tag database. I suspect this is due to a race condition -- the tag was created after exporting the tag database but before exporting the post database.

I don't see code on the GitHub for exporting the database, so I can't see how it works for sure and check whether this would work, but if creating a CoW copy of the database for the duration of the export should prove infeasible, I think an easy way to fix it would be to switch the order in which the post database export and tag database exports are generated, since the tag database is more or less append-only and a race condition would have a lot less of an impact that way.

For what it's worth, I never try to archive the database from the last listed (as in, current) day from the database export page on archive.org. I wonder how this effects consistency of archives of previous day, though, since shouldn't that same error exist, still?

Ouch, timing out trying to archive it. Internal errors on viewing snapshots, etc. Gonna try again later this month I guess.

Updated

  • 1