07/19/2018 (Thu) 01:32:27
MongoDB shat its pants in a way that Lynxchan couldn't recover from, probably because it stepped in a document that didn't exist because it wasn't salvaged from the encryption clusterfuck. It didn't get caught immediately because both of us have day jobs. The 500 errors are probably related, since I've checked the server while the site is in 500 and it usually happens when mongod is eating obscene amounts of CPU time. (Of course it could also be 500ing because mongod is a hog and we're running on a hybrid VPS, so large operations take longer than lynxchan's database timeout and trick lynxchan into retrying with a new DB connection.)
To keep this shit from continuing we need to copy the valid documents (your boards, posts, accounts, and media) from our old broken database to a fresh database that doesn't have huge fucking holes in it. This would require experimentation to find how to do it without copying the same broken references over and would almost certainly require more downtime while the copy happens, so we haven't been able to do it yet. We will
do it, though, because we hate this shit too.