/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Magrathea | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 568 Anonymous Board owner 03/27/2024 (Wed) 23:13 Id: 6ee685 [Preview] No. 1621 [Reply] [Last 50 Posts]

Message too long. Click here to view full text.



Anonymous Board owner 03/27/2024 (Wed) 23:15 Id: 6ee685 [Preview] No.1622 del
other highlights

I also added a 'tag in reverse' checkbox to the new 'manage tags' 'incremental tagger' thing. It adds tags like 5, 4, 3, 2, 1 instead of 1, 2, 3, 4, 5.

And all new system:url predicates have new labels. They are all a bit simpler, and they should copy/paste into the system predicate parsing system. All existing system:url predicates still have their old labels, so if this is a big deal, you'll want to recreate them and re-save your session/search.

Thanks to a user, the new docx, xlsx, and pptx file formats get some nicer thumbnails and a little metadata. It should all recalculate soon after update.

The Client API is now more careful about which files you can undelete, and it also now lets you clear file deletion records.

next week

I want to put proper time into getting a 'future build' working. Last time I tried, I ran into some technical problems related to the newer libraries I wanted to bundle, so I'll see if I can fix it all and have a test release for people to try. Otherwise, I just want to clear out some small jobs that this URL work boshed.


Anonymous 03/29/2024 (Fri) 02:03 [Preview] No.1623 del
Anyone knows if I can download files (through url downloader, non-booru site) whilst keeping their original filenames somehow (as tags or whatever)? And in a way that I can have the files renamed when exporting? I found plenty of places saying it can be done but I've tried messing with the tag import options among others and nothing seems to work.


Anonymous Board owner 03/30/2024 (Sat) 20:10 Id: 5e2328 [Preview] No.1624 del
>>1623
Servers typically give that filename in the 'http header' of a file download. Hydrus generally does not parse this in the same way that your browser would and does not push it into the general metadata pipeline of a file import, so I'm afraid the only way you can figure this out is to parse it from the html file or similar, as you would for other metadata like tags or related URLs.

In general, also, most downloader makers do not parse the filename. The imageboard watchers like the 4chan parser does parse for the filename, and gives it a 'filename:' namespace. By default, I have these set not to add in the 'tag import options' for watchers, since most users don't want them.

If you feel clever and brave, you can try to add a 'content parser' to a downloader you are interested in to try and grab filename from the boorus you like, but beyond that, I'm sorry to say I don't have a good answer for you. I've thought a few times about making a 'filename' tag service that remembers hard drive import filenames, and could potentially get server-set filenames too, but every time I return to the idea, I realise it'll probably just get overwhelmed by 'Image.jpg' garbage that isn't typically useful.

Although I can't give you a nice answer, you might like to read my FAQs on this question, just so you see better where I am coming from and why I don't want to put much time into this:

https://hydrusnetwork.github.io/hydrus/faq.html#filenames
https://hydrusnetwork.github.io/hydrus/faq.html#external_files (somewhat related)

And if you would like to dabble with making/editing your own downloaders, check out the help here:

https://hydrusnetwork.github.io/hydrus/downloader_intro.html


Anonymous 04/02/2024 (Tue) 07:49 [Preview] No.1625 del
I'd seen those pages already, thank you. I understand about filenames but this time I really needed them. Over 90,000 work-related images that were numbered in a specific order, and that ordering had to be kept. I've found another solution but thank you for your work and your reply.


Release Tomorrow! Anonymous Board owner 04/03/2024 (Wed) 01:51 Id: 35070d [Preview] No.1626 del
I had a good week. I fixed a bunch of small issues, and I figured out the problems with the 'future' build, so I'll also have a special version for more advanced users to test out.

The release should be as normal tomorrow.

>>1625
Great, I am glad you found a solution!



Version 566 Anonymous Board owner 03/13/2024 (Wed) 21:24 Id: ed4447 [Preview] No. 1617 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=PlvK2pabBqI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Linux.-.Executable.tar.zst

I had a good week. The long-awaited incremental tagger is ready, and the program supports some more document types. You will get a yes/no on update, but it isn't a big deal.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

incremental tagger

When you open the manage tags dialog on several thumbnails, it now has a '±' button that lets you tag the files 'page:1', 'page:2', 'page:3' and so on. You can set the namespace (or no namespace), the starting point (so you can start at 'page:18' if you need to), the 'step' (so you can count by +2, or even -1 to decrement), and even prefix/suffix for the number if you need to decorate with 'page:x (wip)' or something.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 03/20/2024 (Wed) 02:10 Id: 31ddea [Preview] No.1618 del
I had an ok week. I reworked some URL handling in ways that are mostly important behind the scenes and cleared some small jobs. It'll just be a simple release.

The release should be as normal tomorrow.


Anonymous Board owner 03/20/2024 (Wed) 22:39 Id: 3c4933 [Preview] No.1619 del
The v567 release is cancelled! We found a problem and I need to do more work on it. I will move it on to next Wednesday, sorry for the inconvenience!


Release Tomorrow! Anonymous Board owner 03/27/2024 (Wed) 02:25 Id: a17fcc [Preview] No.1620 del
I had a great week. I tightened up the URL storage/handling improvements that I was not confident in last week, so I am very happy to put out the release tomorrow. There are also advanced new tools for downloader makers for handling 'ephemeral token' parameters along with new quality of life UI in the manage URL Classes dialog. For normal users, there are also several bug fixes, file handling improvements, and a couple little things in system predicates, emoji tag presentation, and reversed tags in the new incremental tagger.

The release should be as normal tomorrow.



Version 565 Anonymous Board owner 03/06/2024 (Wed) 22:58 Id: 3a7407 [Preview] No. 1614 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=lgpD6OsHCKU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Linux.-.Executable.tar.zst

I had a simple week. Lots of small changes today. The update step may take a couple of minutes.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

You are going to get a couple yes/no dialogs on update this week talking about deleting some mis-parsed URLs. If you do not manually store weird data in your 'known urls' store, just click yes. If you have lots of URLs, the work will take a couple of minutes.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 03/13/2024 (Wed) 02:40 Id: 15d85c [Preview] No.1616 del
I had a good week. The 'incremental tagger' system is working, allowing you to tag page:3, page:4, page:5 ... page:17 and similar to a selection of files, and .docx files are now importable.

The release should be as normal tomorrow.



Version 564 Anonymous Board owner 02/28/2024 (Wed) 22:22 Id: 1d3ccb [Preview] No. 1609 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Oo0o84-TJTU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Linux.-.Executable.tar.zst

I had a good week, several system predicates have better range-based searching.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The system predicates for width, height, num_notes, num_words, num_urls, num_frames, duration, and framerate now support two different kinds of approximate equals (≈): absolute (±x), and percentage (±x%). Previously, the ≈ secretly just did ±15% in all cases, but now you set how and how far they go.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 03/06/2024 (Wed) 05:13 Id: 13cf98 [Preview] No.1613 del
I had a simple week, mostly fixing a bunch of bugs.

The release should be as normal tomorrow.



html sidecar attempt Anonymous 02/17/2024 (Sat) 15:00 Id: 897e41 [Preview] No. 1603 [Reply] [Last 50 Posts]
Hey,

Since my last message, I recently had time to work with Hydrus, and I developed a semi-functional parser for my HTML files. However, I don't want to include the description string in the tags, so I need to remove it. However, I'm not sure how to do this. I assume that it has something to do with the "string selector/slicer" component, but I'm not sure how that component works. I managed to do an incredibly crude filter by regex filtering by ^[^<]+$, which blocks the < in html tags, but this will presumably fail for any description that doesn't include html tags, for whatever reason. I would really appreciate your advice on this. If you need to see my (bad) code, I have attached the code to copy as a text file.

Thanks!

Ps: I can't import sidecar parsers through pngs, as the file selection dialogue doesn't show png files, just folders. Is there something I'm doing wrong?


Anonymous Board owner 02/24/2024 (Sat) 19:22 Id: 7fe2bb [Preview] No.1607 del
Yeah, this is tricky. I still think your best answer is to wait for me to implement a proper xml/html parser for sidecars, or do the parsing yourself in an external script (e.g. with python) and then convert your html into nicer .txt files that hydrus can suck up easier. You could also filter your undesired tags better there.

If the 'description' string is always in the same location, and that location is always the first or last index of your list of strings, then the 'string selector' might help. It does list slicing like in programming, if you are familiar, like "my_list[4:6]". If you aren't familiar with that, or the description is in the middle of the list here, you are correct in trying a string match, which is basically a filter.

If the description line has a classname in the html, you might be able to exclude it with a string match before you Split/Convert all the html garbage away.

I am not totally sure what you mean by being unable to import by pngs, but if I click 'import->from pngs' (pic related is from the 'sidecars' tab of the 'add tags/urls with the import' dialog after you drop some files on the client), I get a file dialog that allows me to import a png like this. Do you get different?


Anonymous 03/02/2024 (Sat) 17:37 Id: f3d388 [Preview] No.1610 del
(202.89 KB 1600x900 highdiskusage.png)
(128.77 KB 1600x900 pngparserbug.png)
Hey,

Sorry that it took me a while to respond. I figured out a solution to my html parsing issue. Apparently, the parser first splits sidecars by the specified string, then passes the split strings onto the postprocessor, breaking the regex splitting I used in the process, before then passing it back to the parser, where it is then split by the string again and saved. I figured out a solution to this problem, but thanks for your help anyways. To answer your questions about the parser import, I am using Linux Mint 21.3 Cinnamon with qt6.5.2, rather than windows, and the pngs do not appear in the window. Maybe the apis are different? I attached a screenshot of this, if that helps.

Thanks for your help!

Ps: why is this client writing TBs of data to my ssd while processing the PTR? I understand if it writes a lot, but multiple TBs (according to some math I did, I haven't gotten that far yet, just ~750gb) still seems a bit excessive. I attached a screenshot of this too.


Anonymous 03/02/2024 (Sat) 23:17 Id: f3d388 [Preview] No.1611 del
(199.57 KB 1600x900 hydrusdiskwtf.png)
Update:

(i am using a crucial mx500 500gb with luks and ext4, if that helps)


Anonymous 03/04/2024 (Mon) 17:39 Id: 545671 [Preview] No.1612 del
I was looking at launch options to fix the storage issue, do you think that --no_db_temp_files would help? I have 16gb ram, is that enough?


Anonymous Board owner 03/06/2024 (Wed) 23:40 Id: 1be395 [Preview] No.1615 del
>>1610
>>1611
>>1612
Yeah, the write is most likely temp storage. I've heard that in some situations, the actual amount written to disk is less since some temp-file magic means some of the shortest-lived data is purged from the write cache before it can actually be committed, but I don't have good numbers on the topic.

The PTR does a lot of database work. The no_db_temp_files parameter will reduce it by a good amount. Not sure how much, but it would reduce it.

Increasing db_transaction_commit_period may also reduce it, I think, since it will reduce the commit frequency (at the cost of increasing per-transaction size, which will strain your temp folder and slow processing time).

A user also wrote this document with a lot of related info, if it helps: https://hydrusnetwork.github.io/hydrus/Fixing_Hydrus_Random_Crashes_Under_Linux.html

I'm not a Linux expert myself, so I can't talk super confidently here. Let me know how you get on!



Version 563 Anonymous Board owner 02/21/2024 (Wed) 22:06 Id: fc1ef5 [Preview] No. 1605 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=XXlzWhBvYwg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Linux.-.Executable.tar.zst

I had a good week. There's a mix of small fixes and improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

macOS

A user has improved the macOS release in many ways, mostly in brushing up the App to normal macOS standards. The menubar should now plug into native global bar, with some standardised command labels. The program icon is better, some colours should be improved, dialog menus are no longer a crazy hack by default, and the system tray icon is turned on. In a side thing, I also added Cmd+W to close pretty much any dialog or non-main-gui window in macOS, just like hitting Escape.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 02/28/2024 (Wed) 02:26 Id: 3a4f95 [Preview] No.1608 del
I had a good week. A bunch of system predicates now support customisable +/- ranges, both by absolute and percentage values.

The release should be as normal tomorrow.



Version 562 Anonymous Board owner 02/14/2024 (Wed) 22:16 Id: eddf56 [Preview] No. 1601 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=XnwHtzBf-c8 [Embed]
(go to 4:44:00 for a wild fight)
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Linux.-.Executable.tar.zst

I had a tepid week, but there's some decent fixes and quality of life improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

all misc this week

I fixed a stupid typo error in the manage times dialog when you go into a single time and try to copy/paste the timestamp. The buttons also add millisecond data now.

Message too long. Click here to view full text.

Edited last time by hydrus_dev on 02/14/2024 (Wed) 22:17.


Release Tomorrow! Anonymous Board owner 02/21/2024 (Wed) 01:32 Id: 140d0d [Preview] No.1604 del
I had a good week. I fixed some bugs and made some shortcuts a bit nicer to deal with, and there are a bunch of macOS improvements.

The release should be as normal tomorrow.



html sidecars - possible? Anonymous 02/10/2024 (Sat) 00:47 Id: d856bf [Preview] No. 1598 [Reply] [Last 50 Posts]
Hey,

I was recently trying to import some files into hydrus that I had downloaded using gallery-dl. Thanks to a config-file mistake, all the tag files were saved as html pages. Is there a way to use regex to extract the tags? I attached one of the html files, if that helps.

Thanks!


Anonymous Board owner 02/14/2024 (Wed) 22:15 Id: 4f6ce1 [Preview] No.1600 del
In future I hope to have an xml/html parser for sidecars, but I can't promise when it will be ready. So, if you have time, I think save these for later. Otherwise, parse them yourself and make JSON/txt files, or just run the download again if possible.


Anonymous 02/16/2024 (Fri) 03:51 Id: d856bf [Preview] No.1602 del
alright, thanks for your help!



Version 561 Anonymous Board owner 02/07/2024 (Wed) 22:12 Id: cd0789 [Preview] No. 1597 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=GuVnZZ1sFIc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Linux.-.Executable.tar.zst

I had a mixed week. Thumbnail rearranging is added, and some bugs and quality of life issues are cleared.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

thumbnail rearranging

So, if you right-click any thumbnail, you'll now see a 'move' menu. This makes it simple to rearrange your thumbnails. You can move your current selection to the start, the end, left one, right one, or 'to here' if you have multiple selected. If your multiple selection is non-contiguous, it will be made so on move, with the move focusing around the position of the first selected item.

Message too long. Click here to view full text.



Simple Release Tomorrow! Anonymous Board owner 02/14/2024 (Wed) 01:54 Id: e507ce [Preview] No.1599 del
I had a tepid week. I fixed some bugs and improved some quality of life, that's all!

The release should be as normal tomorrow.



Version 560 Anonymous Board owner 01/31/2024 (Wed) 22:22 Id: 99f7d2 [Preview] No. 1593 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=UBYThAP_C5A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Linux.-.Executable.tar.zst

I was suddenly without internet for a while, so there was no release last week, but everything is back to normal now, and I had a great couple weeks' work. The 'edit times' dialog can now handle multiple files.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

editing times for multiple files at once

If you launch 'edit times' on a single file, everything is the same. If you open it on a larger selection, it will now show, in a summarised way, the range of each particular time type, and how many files in the selection have or do not have a time set there. The controls all work as before, but in general, when you set a time, all files are now locked to that new time. Give it a play and you'll see how it all works.

Message too long. Click here to view full text.



Anonymous Board owner 01/31/2024 (Wed) 22:23 Id: 99f7d2 [Preview] No.1594 del
misc

In v559, it turns out I did make a '54 years ago' mistake, on moving files from one local file domain to another. This is fixed, and on database update, any borked timestamps that were saved should be corrected too. Let me know if you encounter any more issues.

Mr Bones and the File History Chart are now under the database menu.

There's a new checkbox under 'files and trash', 'Remove files from view when they are moved to another local file domain'. This actually re-introduces the unintended behaviour we had a couple weeks ago, which was happening under 'remove when trashed', but targeted to the correct logical trigger.

The gallery downloader cog button now has a 'don't add new dupes' option, which discards the query_text/source pairs you add if they are already in the download list. If you need to paste a hundred queries from a big mishmash list, I think this will help.

As a very brief test, I've added 'sort files by pixel hash'. If you have some pixel hashes in your current query, this will make them stand out, but the rest of the sort data is meaningless. I had wanted to try out some similar hacky new tech with 'sort by perceptual hash', but thumbnails don't have quick access to that data yet, so it'll have to wait.

I wasn't sure when my internet was going to come back up, so I dumped a bunch of time into some longer cleanup jobs. There's a ton of updated code behind the scenes, particularly around the way files get informed about new metadata, but nothing you need to care about. I have some new plans here and can see the path to getting all sorts of new metadata commands integrated into the shortcut and undo systems. I changed a couple thousand lines, so there may be a typo bug somewhere--let me know if you try to set any unusual content changes and it throws an error!

next week

Now we have tech that relies on file order, I think it is time we get the ability to conveniently reorder thumbnails. This will also help when I figure out cascading tagging in 'manage tags'. I'd love to have reordering by simple drag and drop, but that may be too complicated to slip into one week. I'll see what I can do.

Message too long. Click here to view full text.



Anonymous Board owner 02/03/2024 (Sat) 19:48 Id: 2880b1 [Preview] No.1595 del
Hey, I screwed up a line in all the rewrites and one error in the duplicate filter is particularly bad, sometimes causing crashes. If this has hit you, there's a hotfix here:

https://github.com/hydrusnetwork/hydrus/releases/tag/v560a


Simple Release Tomorrow! Anonymous Board owner 02/07/2024 (Wed) 03:48 Id: 53c657 [Preview] No.1596 del
I had a mixed week. I added thumbnail rearrangement, and I cleared some bugs and quality of life stuff, but I didn't get as much as I wanted done.

The release should be as normal tomorrow.