Yeah, this is tricky. I still think your best answer is to wait for me to implement a proper xml/html parser for sidecars, or do the parsing yourself in an external script (e.g. with python) and then convert your html into nicer .txt files that hydrus can suck up easier. You could also filter your undesired tags better there.

If the 'description' string is always in the same location, and that location is always the first or last index of your list of strings, then the 'string selector' might help. It does list slicing like in programming, if you are familiar, like "my_list[4:6]". If you aren't familiar with that, or the description is in the middle of the list here, you are correct in trying a string match, which is basically a filter.

If the description line has a classname in the html, you might be able to exclude it with a string match before you Split/Convert all the html garbage away.

I am not totally sure what you mean by being unable to import by pngs, but if I click 'import->from pngs' (pic related is from the 'sidecars' tab of the 'add tags/urls with the import' dialog after you drop some files on the client), I get a file dialog that allows me to import a png like this. Do you get different?

Update:

(i am using a crucial mx500 500gb with luks and ext4, if that helps)

I was looking at launch options to fix the storage issue, do you think that --no_db_temp_files would help? I have 16gb ram, is that enough?

>>1610
>>1611
>>1612
Yeah, the write is most likely temp storage. I've heard that in some situations, the actual amount written to disk is less since some temp-file magic means some of the shortest-lived data is purged from the write cache before it can actually be committed, but I don't have good numbers on the topic.

The PTR does a lot of database work. The no_db_temp_files parameter will reduce it by a good amount. Not sure how much, but it would reduce it.

Increasing db_transaction_commit_period may also reduce it, I think, since it will reduce the commit frequency (at the cost of increasing per-transaction size, which will strain your temp folder and slow processing time).

A user also wrote this document with a lot of related info, if it helps: https://hydrusnetwork.github.io/hydrus/Fixing_Hydrus_Random_Crashes_Under_Linux.html

I'm not a Linux expert myself, so I can't talk super confidently here. Let me know how you get on!

I had a good week. A bunch of system predicates now support customisable +/- ranges, both by absolute and percentage values.

The release should be as normal tomorrow.

I had a good week. I fixed some bugs and made some shortcuts a bit nicer to deal with, and there are a bunch of macOS improvements.

The release should be as normal tomorrow.

In future I hope to have an xml/html parser for sidecars, but I can't promise when it will be ready. So, if you have time, I think save these for later. Otherwise, parse them yourself and make JSON/txt files, or just run the download again if possible.

alright, thanks for your help!

I had a tepid week. I fixed some bugs and improved some quality of life, that's all!

The release should be as normal tomorrow.

misc

In v559, it turns out I did make a '54 years ago' mistake, on moving files from one local file domain to another. This is fixed, and on database update, any borked timestamps that were saved should be corrected too. Let me know if you encounter any more issues.

Mr Bones and the File History Chart are now under the database menu.

There's a new checkbox under 'files and trash', 'Remove files from view when they are moved to another local file domain'. This actually re-introduces the unintended behaviour we had a couple weeks ago, which was happening under 'remove when trashed', but targeted to the correct logical trigger.

The gallery downloader cog button now has a 'don't add new dupes' option, which discards the query_text/source pairs you add if they are already in the download list. If you need to paste a hundred queries from a big mishmash list, I think this will help.

As a very brief test, I've added 'sort files by pixel hash'. If you have some pixel hashes in your current query, this will make them stand out, but the rest of the sort data is meaningless. I had wanted to try out some similar hacky new tech with 'sort by perceptual hash', but thumbnails don't have quick access to that data yet, so it'll have to wait.

I wasn't sure when my internet was going to come back up, so I dumped a bunch of time into some longer cleanup jobs. There's a ton of updated code behind the scenes, particularly around the way files get informed about new metadata, but nothing you need to care about. I have some new plans here and can see the path to getting all sorts of new metadata commands integrated into the shortcut and undo systems. I changed a couple thousand lines, so there may be a typo bug somewhere--let me know if you try to set any unusual content changes and it throws an error!

next week

Now we have tech that relies on file order, I think it is time we get the ability to conveniently reorder thumbnails. This will also help when I figure out cascading tagging in 'manage tags'. I'd love to have reordering by simple drag and drop, but that may be too complicated to slip into one week. I'll see what I can do.

Hey, I screwed up a line in all the rewrites and one error in the duplicate filter is particularly bad, sometimes causing crashes. If this has hit you, there's a hotfix here:

https://github.com/hydrusnetwork/hydrus/releases/tag/v560a

I had a mixed week. I added thumbnail rearrangement, and I cleared some bugs and quality of life stuff, but I didn't get as much as I wanted done.

The release should be as normal tomorrow.

thanks dad love you

Hey lads, hydev here, my internet died and it could be a week before it is fixed. I don't have a good solution for my dev logins to talk to the internet, so no release or posts for a bit, will update later!

My internet came back today, and despite the problems, I had a great couple of weeks of work. The manage times dialog can now operate on many files at once, and it can even set incrementally increasing times to files to force particular file sorts. There are also some miscellanious bug fixes and quality of life improvements.

The release should be as normal tomorrow.

Sorry, question here. Is there a way to change a file's hash? To make hydrus associate all of the tags and logs of one file with another?

>>1583
Not in the convenient way you are thinking, unfortunately. 'This file has this hash is core to the whole database, so if you start messing around with ids, I can't promise that some maintenance routine won't notice the problem later and throw a bunch of warnings and errors.

I can do something though. Most of this tech is in the 'duplicates' system, where you can copy tags and ratings and URLs from one file to another, but it is clunky to work with and doesn't copy every possible thing. I plan to keep working on it because we are running into more and more situations where we want clean easy and automatable sharing/duplicating of metadata across similar files.

Here's the general help, if you haven't seen it. You can access all the 'set these files as same quality duplicates' commands outside of the duplicate filter through the normal thumbnail menu when you have multiple files selected, but you'll want to set up your 'default duplicate metadata merge options' in a duplicates processing page first, so your client knows what to merge where in these situations.

https://hydrusnetwork.github.io/hydrus/duplicates.html

>>1585
Thank for your answering my question.

>>1586
*Thank you for answering my question.

I had a great week. I completely overhauled how file timestamps are stored across the program, adding millisecond resolution so 'sort by import time' is more reliable, and I then added timestamp editing to the Client API.

The release should be as normal tomorrow.

>>1576
Not yet, but in the future I expect to link clients together via the Client API this way. Fingers crossed, you'll be able to browse another client's files as simply as selecting it as the location in the normal file service selector.

The copy service key button is just there because I think I have it set as available for everything. It mostly gets used for weird debug purposes atm.

>>1577
That will be good to have. In the meantime, is it possible to use the same db for both a client and server?
e.g. The client is all set up, I shutdown the client, point a server to the same db folder. Would this cause any problems, especially if I then stopped using the server and booted the client?
If it matters, it's dockerized, I'll probably just make a backup to play with, but I wondered if it's expected for the two to play nice with the underlying database.

Also really like what you're doing with the system predicates.
If this is already possible let me know, but I'd like a way to set custom shorthand for specific system predicates... like system:landscape for system:ratio=landscape, for instance, or something even shorter - it didn't seem like siblings work for system predicates (technically they're not tags anyways, so that part makes sense to me)

I had an ok week. I fixed some bugs, improved some quality of life, and figured out a basic first version of 'system:number of urls'.

The release should be as normal tomorrow.

>>1578
Unfortunately the client and server databases are not compatible. Even if you point both exes at the same directory, the client makes client.db and server makes server.db. They are very different programs and databases and outside of the normal repository syncing, they cannot talk to each other or do each other's jobs, I'm afraid. This has come up before, so some of the discord guys wrote up this document:

https://hydrusnetwork.github.io/hydrus/youDontWantTheServer.html

I quite like the idea of 'system:landscape' style shorthands. I'm in the midst of fleshing out the system predicate parsing system right now, and there's a ton I'd like to do with it once things are a little less thrown together (I'm rearranging a bunch of system preds, like system:hash last week, so they parse easier). For now, you might like to play with the 'favourite searches' button (the star beside the normal search input text box), which let's you save full searches, or templates of them, for quick loading of quite complicated search setups in just a couple clicks.

>>1579
That page is exactly why I am using the client and not the server.

I believe that page is written from a high-and-mighty misleading perspective, not to mention incredibly lacking in information.

So, my other original question. That page explicitly states that you can run multiple clients without the server and sync files between them.
>Do you want to use multiple clients and have everything synced between them? You don't want the server.

How do you accomplish this without a complex export/import process between the clients?

>>1580
>How do you accomplish this without a complex export/import process between the clients?
Unfortunately, there isn't a good way right now. I plan to, over the next year or two, buff up the Client API and add a way for clients to browse each other. You'll be able to load up another client's file services in the normal search and it'll all just happen over the Client API instead. Once that tech works, I'll be recommending people have a master client that does all the subscriptions and stuff, and then other computers can dial into that one and read its content live.

Although it sucks to say, I know some users who simply set up VPN situations to browse their master client from another machine or tablet.

If you have a relatively slender client, let's say less than 50GB total 'db' directory, another option is simply to have a master client and then FreeFileSync its whole install dir to your other computers once a week, simply cloning it.

birthday and year summary

The first non-experimental beta of hydrus was released on December 14th, 2011. We are now going on twelve years.

Like many, I had an imperfect 2023. I've no complaints, but IRL problems from 2022 cut into my free time and energy, and I regret that it impacted my hydrus work time. I had hoped to move some larger projects forward this year, but I was mostly treading water with little features and optimisations. That said, looking at the changelog for the year reveals good progress nonetheless, including: multiple duplicate search and filter speed and accuracy improvements, and the 'one file in this search, the other in this search' system; significant Client API expansions, in good part thanks to a user, including the duplicates system, more page inspections, multiple local file domains, and http headers; new sidecar datatypes and string processing tools; improvements to 'related tags' search; much better transparency support, including 'system:has transparency'; more program stability, particularly with mpv; much much faster tag autocomplete results, and faster tag and file search cancelling; the inc/dec rating service; better file timestamp awareness and full editing capability; the SauceNAO-style image search under 'system:similar files'; blurhashes; more and better system predicate parsing, and natural system predicate parsing in the normal file search input; a background database table delete system that relieves huge jobs like 'delete the PTR'; more accurate Mr. Bones and File History, and both windows now taking any search; and multiple new file formats, like HEIF and gzip and Krita, and thumbnails and full rendering for several like PSD and PDF, again in good part thanks to a user, and then most recently the Ugoira and CBZ work.

I'm truly looking forward to the new year, and I plan to keep working and putting out releases every week. I deeply appreciate the feedback and help over the years. Thank you!

next week

I have only one more week in the year before my Christmas holiday, so I'll just do some simple cleanup and little fixes.

Hey, my dev machine died today (its GPU actually caught fire when I turned it on!), so no release tomorrow. I'll get a new machine over the holiday and restart for the new year, meaning v557 will be on January 3rd.

Thanks everyone, and 𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had a good week back after the holiday. I fixed some bugs and improved some system:hash predicate parsing.

The release should be as normal tomorrow.