/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Magrathea | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 470 Anonymous Board owner 01/19/2022 (Wed) 22:25:03 Id: 9aac08 [Preview] No. 1206 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=_SSPdeoAHsM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Linux.-.Executable.tar.gz

I had a great week. The big rewrites are finally doing something interesting, and I got some other stuff done as well.

cleverer search

First of all, I finished my file domain rewrites. File search, tag search, and all the UI that displays it now works on n file domains at once rather than just one. Also, any of those n can be the deleted files of a service now. You can't make new local file services yet, obviously, but all search is now ready for it.

Since adding deleted files search was easy, I am adding it this week. Users in advanced mode will now see a list of deleted file domains on the tag autocomplete file domain button menu. While it is rare you ever want to do this, it was never actually possible to search these domains completely before, and definitely not quickly. Give it a go and let me know if you run into any trouble!

Message too long. Click here to view full text.



Anonymous Board owner 01/19/2022 (Wed) 22:25:23 Id: 9aac08 [Preview] No.1207 del
full list

- multiple file services:
- I finished the conversion of all UI search to the new multiple location object. everything from back- to frontend now supports cleverer search. since searching deleted files is simple to add, users in advanced mode will now see 'deleted from...' in a new list in the tag autocomplete dropdown file domain button
- the next step is writing a widget that allows multiple selection, and then all this should work right out of the box, and we'll be an important step closer to allowing multiple local file services
- .
- misc:
- the video parsing routine is better at detecting when a present audio track is actually silent (and hence when it should mark a video as 'no audio'). all video with audio will be requeued for a metadata reparse in the files maintenance system on update
- fixed an error from last week when trying to create a new page from the tags (e.g. middle-clicking them) in the active search list
- added 'audio mkv' format to the client, to represent mkvs without a video track. I think most of the time this is going to be audio track webms from youtube-dl and similar
- added 'file relationships: set files as potential duplicates' command to the 'media actions' shortcut set
- I expanded the 'backing up' section in 'installing and updating' help
- I wrote an 'anti-virus' section for 'installing and updating' help, since I kept writing the same basic spiel about false positives. please feel free to point people there in future to relieve their concerns
- improved some shutdown tests, the client and server should exit faster in some cases (e.g. when a hydrus repository network job is hanging on reconnection attempts, holding up the 'synchronise_repository' daemon shutdown)
- the 'file was xxx at (y timestamp), which was (z time units) before this check' line in file import notes now always puts 'z time units' as that, ignoring the 'always show ISO time' setting, which was just substituting it with 'y timestamp' again. let me know if you spot other bad grammar with this setting on, I'll fix it!
- fingers crossed, images in the LAB colourspace _should_ now normalise to sRGB with the correct whitepoint. thanks to the user who provided example test tiff images here. this now uses the new PIL-based colourspace conversion I used to make ICC profiles work, just on LAB->sRGB. as far as I understand, OpenCV uses a fixed whitepoint of D65, resulting in yellow/warm conversions for some formats, but PIL may be able to figure out if D50 is needed??? if you have some crazy LUV or YPbPr or YIQ image that shows up wrong, please send it in and I'll see what I can do!


Anonymous Board owner 01/19/2022 (Wed) 22:27:55 Id: 9aac08 [Preview] No.1208 del
- boring rewrites and cleanup while doing file service work:
- many more UI objects now store and do file service logic using a more complicated 'location context', which can store a mix of multiple services and 'deleted from service' data. all the search code that works on this can now propagate to display:
- the management objects behind every page now store a multiple location object, not a single file service id
- all media panels (the thumbnail grid on a page) are now instantiated by a multiple location object, and when they serve a highlighted downloader, they now inherit that from the file import options, which in future will dictate import destinations
- all canvases are now the same, inheriting their new location context from their parents
- all tag lists are the same. mostly they don't care much about file domain, but when you middle-click to create new pages from the autocomplete dropdown list or active search list, it can matter, so they now propagate it along
- the underlying medialist objects are now the same, and various delete logic (e.g. 'should we remove this thumb we just deleted?') is updated to work on complex domains
- some duplicate lookup code now works on location context
- renamed 'location search context' object to 'location context' since it is used all over now and put it in its own file. also wrote it some neater initialisation and meta object code
- mr bones now gives duplicate data based on the union of all non-trash local services sans update files (another case of now supporting n services but n is fixed for the moment at 1, 'my files')
- a bunch of places across the program that used to default to 'my files' or 'all local files' (which is everything on disk, including trash and repository update files) now default to this new union of all non-trash local media services
- when doing page-to-page file drag and drops, the location context is now preserved (previously, the new page would always be 'my files')
- whole heap of other cleanup in these systems
- when a thumbnail cannot be provided (for deleted files or many 'all known files' situations), the thumbnail cache now provides the hydrus icon stand-in instantly, no delayed waterfall
- fixed an unusual situation where the file search could not provide a file in a tagless search when that file had no detailed file info row in the database. this seems to effect a legacy borked row or two in the new deleted file domain searches
- removed some ancient dumper status code from thumbnail objects

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 01/26/2022 (Wed) 02:21:48 Id: 7b4235 [Preview] No.1209 del
I had a great week doing some fun stuff along with catching up on older issues. I managed to add tracking for 'last viewed time' for files, along with search and sort capability just like for import and modified times. There's also the start of custom colours for the video scanbar using the new style system.

The release should be as normal tomorrow.



Version 468 Anonymous Board owner 01/05/2022 (Wed) 23:02:52 Id: 26f6f8 [Preview] No. 1197 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=UGnbiBDmuBI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Linux.-.Executable.tar.gz

I had a good first week back. A mix of bug fixes and little improvements.

couple of highlights, otherwise all misc this week

Thanks for the quick issue reports over the holiday. The scanbar had an annoying thing where the new single pixel border was making it awkward to drag when in borderless fullscreen--you'd move your mouse to the bottom of the screen, but then you'd click the border and not the scanbar, and the whole video would move--this I have now fixed. Also an issue with new imports' 'pixel duplicates' data not being saved correctly--I have fixed the problem and scheduled all affected files to regen their pixel duplicate data.

I also put some time into multiple local file services. It was more background work, this time to do with system predicate fetching, so there isn't much neat stuff to show off unless you are an advanced user. Almost all the background work is done now, though, so I hope to start spinning up more complex search UI and actually adding new file services pretty soon. One neat thing, now that I have new file filtering tools, I was able to expand 'system:file service' to search for deleted and petitioned files.

Message too long. Click here to view full text.

3 posts omitted.


Simple Release Tomorrow! Anonymous Board owner 01/12/2022 (Wed) 02:43:48 Id: 98831b [Preview] No.1201 del
I had a mixed week. The tag autocomplete rewrite was more work than I expected. While I am happy with what I did, most of my time was spent on boring background and cleanup, so there is little on the front end to show for it. Beyond that, I fixed the Linux/macOS write permissions checking problem and did a couple small things in the Client API and downloader search log. I will recommend the release just to people interested in those issues.

The release should be as normal tomorrow.

>>1199
>>1200
Thank you, I am sorry for the trouble! I think I have fixed this for tomorrow's release, please try it out and let me know if you have any more problems.


Release Tomorrow! Anonymous Board owner 01/19/2022 (Wed) 04:21:03 Id: 759469 [Preview] No.1202 del
Hey, sorry, Endchan was not working for me at post time last week, so we missed 469 here. I will continue with 470 tomorrow, fingers crossed.

I had a great week. I finished the multiple file service rewrites and can finally start extending search. First off, for tomorrow's release, all advanced users will be able to quickly search their deleted files. I also fixed some bugs and improved some file handling, including the test that detects whether a video has an audio stream or not.

The release should be as normal tomorrow.


Anonymous 01/19/2022 (Wed) 08:41:38 Id: 2bca88 [Preview] No.1203 del
>>1202
I just wanted to say thank you for your work.


Anonymous 01/19/2022 (Wed) 09:53:22 Id: 9dd40f [Preview] No.1204 del
>>1201
Same anon from >>1199, everything seems to be working fine now on 469. Thanks.


Release Tomorrow! Anonymous Board owner 01/19/2022 (Wed) 22:24:16 Id: 4f1262 [Preview] No.1205 del
>>1203
>>1204
Thanks lads, keep on pushing.



Version 467 Anonymous Board owner 12/22/2021 (Wed) 23:09:46 Id: 9ab3aa [Preview] No. 1193 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=MrweXsImVhg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Linux.-.Executable.tar.gz

I had a good week cleaning some things up for end of year.

couple of scanbar things

I polished the new autohiding video scanbar. A bunch of the layout and coordinate detection code of the scanbar and the media canvas behind it is less jank. If you had some flicker or weird mouse-popup behaviour last week, I hope it is better now!

If you are a macOS user however, I held back one of the changes. There's a background-erasing hack I put in a couple years ago because without it macOS media viewers went to 100% CPU. I hope this is no longer true, but I'm not certain, so you still have the flag on by default. Please hit _help->debug->gui actions->macos anti-flicker test_ and then try browsing some images and video in a new viewer. Does your client lock up, or is it ok now? If everything is good, I'll remove the flag and you can get some nicer anti-flicker tech too.

Message too long. Click here to view full text.



Anonymous Board owner 12/22/2021 (Wed) 23:10:03 Id: 9ab3aa [Preview] No.1194 del
- misc:
- the 'file log' and 'search log' buttons are now a new widget class that puts an arrow on the side that opens a menu. the secret right-click menus of these buttons is now exposed for all
- fixed a bug affecting some greyscale pngs with ICC profiles--they were coming out pure white due to a colourspace conversion problem
- fixed an import problem when PIL could not load a file (due to file malformation) but OpenCV could. this was causing a failed import from the new ICC profile detection code
- when the downloader hits a broken image file that cannot be imported due to malformation, the status is now 'error' instead of the incorrect 'ignored'
- fixed the duplicate file filesize comparison statement sometimes showing > in one direction and ≈ in the other. it happened when the larger file was between 20/19 and 21/20 times the size of the smaller, just a logic typo (issue #1028)
- the trash maintenance daemon is moved from the old threaded daemon system to the new repeating job worker pool. this is the last daemon cleaned up, so I am retiring the old and mostly defunct 'no_daemons' launch argument. a variety of other daemon infrastructure for things like shutdown checks is similarly removed. the program also now waits for the newer daemon jobs to finish working on shutdown
- moved most client daemon jobs like repository sync and dirty object save down so they start after the first session is loaded rather than right after boot
- if a file is called to regen its thumbnail but currently has no dimension, this is now a no-op rather than an error. in the situation where users force thumb regen before metadata regen and encounter this, it is sorted out later when the metadata regen recognises new dimensions and reschedules the thumb regen
- added an extensive user-written guide to the --db_synchronous_override launch argument to the launch arguments help page. it is possible and safe to run the program with synchronous=0 as long as certain caveats are obeyed. thanks to the user who figured this out and wrote it up
- the downloader engine now discards source urls in an import job if they have the same domain as any existing primary url. this will ensure that if a booru has a link back to itself as a source url, when the 'source' is really an alternate rather than a dupe, it won't be added in hydrus as a known url for that imported file
- misc cleanup in downloader system and file/search log UI
- fixed a type bug in the file and search log 'import from png' action. if you have existing pngs previously exported from here, they will import ok now
- refactored the various hydrus compression code to a new HydrusCompression file
- exported serialisable data pngs such as from file or search log that hold simple Strings now always compress the data before embedding it in the png. existing pngs that hold uncompressed strings should still load ok
- the payload in an exported png is now always compressed, and the payload description always states the uncompressed size
- sped up client shutdown when network traffic has been paused the whole time and a repo sync job might have wanted to run. these jobs also do not hang on a thread worker if network traffic is paused, but they should wake immediately when it is unpaused

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 01/05/2022 (Wed) 03:44:47 Id: b342a2 [Preview] No.1196 del
I had a good week. I fixed bugs--including the recent annoying scanbar border clicking issue--rewrote some bad code, and added a couple of advanced file operations to search and the client api.

The release should be as normal tomorrow.



(10.80 KB 397x237 92384659.png)
Malformed Drive -- Help Anonymous 01/02/2022 (Sun) 12:49:54 Id: c590d9 [Preview] No. 1195 [Reply] [Last 50 Posts]
I've got multiple drives I've tested Hydrus on, and can't figure out why I'm having issues. All my drives are healthy and run without problems. At first I thought Window's built in drive compression might be fucking with Hydrus, but I decompressed my whole SSD, and I'm still getting errors. Can someone suggest a possible reason for my issue? Will provide logs if needed, Thanks in advance.



Version 466 Anonymous Board owner 12/15/2021 (Wed) 23:18:40 Id: 960c24 [Preview] No. 1188 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=kWXAL9bHciQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Linux.-.Executable.tar.gz

I had a great week. The duplicate filter now supports filtering by pixel duplicates, and videos now fit better in the media viewer.

better duplicate search

Now we have pixel duplicate data stored in the database, I can search it. The duplicate filter page now has a dropdown that lets you select 'must be pixel dupes', 'can be pixel dupes', and 'must not be pixel dupes'. It does exactly what you think, and with luck it should be pretty fast no matter what you select.

Note that you may not have all your old pixel dupe data calculated yet. I started it for everyone last week, but even if it is doing 10,000 files a day, if you have a big client it will take a little longer. It may also still be working on the ICC profile queue. Hit up database->file maintenance->manage scheduled jobs to see how it is doing. You can rush it there if you like.

Message too long. Click here to view full text.



Anonymous Board owner 12/15/2021 (Wed) 23:19:45 Id: 960c24 [Preview] No.1189 del
full list

- video scanbar autohide:
- the scanbar that shows below audio and video is now embedded inside the video frame, and it show/hides based on how close your mouse is to it
- I've wanted to do this for a long time, since it will allow you to watch 16:9 videos at true 100% in borderless fullscreen, but the hackery of how the media viewer works behind the scenes means this took more work than you'd think and is still a little jank. there's a small amount of flicker when it pops in and out, which I will work on in future. in any case, please have a play with it and let me know what you think. I expect to add some more options, like for the activation padding area around it, and I will be tidying up more layout stuff throughout the media viewer
- if you are a mostly keyboard user, please check out the new 'global' shortcut to flip on/off a 'force the animation scanbar to show' mode
- I don't really want to bring back the always-on hanging-below scanbar that just takes up space, but if you try this new embedded scanbar and really hate it, we'll see what we can do
- .
- more duplicate filter search options:
- the duplicates page now has a dropdown on the search for 'must be/can be/excludes pixel dupes'!
- the duplicates page now has a number control on the search for what distance the pair was found at! I am not sure how accurate this thing is in all cases, but it seems I started tracking this data some time ago and forgot I even had it
- these new options are remembered in your session and _should_ remain fast in most normal cases. I put time into some complicated database work this week to get this going, please let me know if you have any trouble with it
- .
- misc:
- when the export filename pattern in the export files dialog means many of the files share the same base and hence need to do 'filename (5)'-style suffixes to be unique, the number here is now calculated much more efficiently. opening this dialog on 10,000 files with an oft-duplicate pattern should now be a jolt of lag but not many minutes
- when you choose to 'separate' a subscription with more than 100 queries, you are no longer forced to break it into half
- when you do break a subscription in half, it now makes sure to sort the query texts before separating

Message too long. Click here to view full text.



Anonymous Board owner 12/15/2021 (Wed) 23:20:37 Id: 960c24 [Preview] No.1190 del
- boring cleanup and little fixes:
- when you set the checker options in the edit subscription dialog, the queries now recalculate their file velocity better. previously, they would just set 'unknown' and recalc on the next run, but now they will actually recalculate if the query container is loaded into memory or otherwise put a status that says 'will calculate on next run'
- removed the 'should be namespaced' reason from the manage tags quick petition reasons. this is now all handled by siblings, tidying up storage tags manually is busywork
- when you click 'copy traceback' on an error popup, it also copies the software version, your platform, and if you are on a frozen build or running from source
- the logger now prints version number for every block, just before the timestamp
- cleaned up a variety of media viewer UI code while working on the scanbar, fixing some misc display bugs
- moved pixel hash storage responsibility from 'file metadata' to 'similar files' module
- the similar files system now searches pixel hashes when it is called to do any similar files search. they count as 'exact match' distance
- when a file gets a new pixel hash, it now sees if any other files have that same hash. if so, it now gets queued up again in the similar files search system, ensuring this match is not missed
- misc nomenclature cleanup--since we now have both 'pixel hashes' and 'phashes', phashes are now referred to as 'perceptual hashes' everywhere
- massively refactored the primary table join that drives potential duplicates search. it should work a bit faster now and it is much easier to work with
- I added pixel dupe and distance search to the standard search results version of the join and the 'system:everything' version, which has several optimisations
- silenced some shutdown handling in file maintenance that was being printed to log as an error
- fixed some 'broken object load' error handling to print the timestamp of the specific bad object, not whatever timestamp was requested. this error handling now also prints the full dump name and version to the log, and version to the exported filename. I was working with a user who had broken subs this week, and lacking this full info made things just a little trickier to put back together
- fixed some drag and drop handling where it was possible to drop thumbnails on a certain location of a page of pages that held an empty page of pages but it would not create a new child media page to hold them
- misc serverside db code cleanup
- fixed python 3.10 type bugs in window coordinate saving and Qt image generation from buffer (issue #1027)

Message too long. Click here to view full text.



Anonymous 12/20/2021 (Mon) 09:27:00 Id: 1f98ee [Preview] No.1191 del
>>1188
not sure where you will see this first, so ill post in both places.

as far as the media scrollbar is concerned, I GREATLY prefer to have it below and always on, I am not using hydrus to watch shows or long videos, for me the videos are essentially just replacing gifs at this point. I love being able to see how much longer they have, and being able to scrub through them without covering a good portion of the video.

If there is a way to have a toggle between visible and hidden, on video or below video, I think that would be best, because as it stands now, it's annoying to use to scrub things because it's not always there, and at least for me it's also annoying to have to keep my mouse there to see how long the video will last.

I should also mention that the way I use hydrus is I have a 4k screen, so I have the left half as media viewer area, and the right half as either the hydrus thumbnails, or a youtube video, I never have it just fullscreen.


Release Tomorrow! Anonymous Board owner 12/22/2021 (Wed) 02:34:07 Id: 129cd6 [Preview] No.1192 del
I had a good week. I fixed some small bugs and brushed up a little UI. Nothing too ambitious, but it should make for a nice clean release before I break for my holiday.

The release should be as normal tomorrow.

>>1191
Thank you. I untangled all this UI code this week and I feel better about adding options like 'hang below' in future. I have saved your thoughts and will try to enable this again soon into the new year.



Version 464 Anonymous Board owner 12/01/2021 (Wed) 23:09:01 Id: e04c8d [Preview] No. 1179 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=mn34GKTVqMM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Linux.-.Executable.tar.gz

I had a good week. Some images get better colour, and the software now deletes files more neatly.

ICC profiles

tl;dr: Some image colours are better now.

Some images come with an 'ICC profile'. A scan of one ICC-heavy client showed that 10% of all images had them. ICCs are basically colour correction data like 'actually, the blue should be this much more intensely blue than is stored' in a variety of sometimes complicated ways. Some nice cameras attach a fast ICC based on lighting conditions or known defects in the camera, and some image editing software does similar. ICCs are also used for broader and more complicated colour conversion tasks like in the migration from sRGB to HDR formats if you have a new tv/monitor.

Message too long. Click here to view full text.



Anonymous Board owner 12/01/2021 (Wed) 23:09:26 Id: e04c8d [Preview] No.1180 del
better file delete

tl;dr: Files delete from your disk better now.

For a long time, actually physically deleting a file that leaves the client's trash has been a bit hacky. To reduce UI lag, the recycle or delete action has to be deferred to the future, but since there was no permanent record of what to delete, the deferred job was always a bit rushed and in a couple of instances (e.g. if you cleared trash and closed the client soon after) it could leave file orphans behind. The system has always been fail safe, never deleting too much, but it could lead to surplus cruft in file storage. Secondly, it turns out the server never got physical file delete tech added! The file repository has never been used much, and it just never came up.

So, this week I overhauled the whole way it works. Now both client and server keep records of the files they should delete, print delete summaries to the log, use neater delete daemons that smooth out the work to be non-interrupting, have sophisticated 'is this an orphan now?' logic to ensure that edge cases are kept in storage until they are truly no longer needed, and correct themselves in odd cases (like if you re-import a file just a couple of seconds after deleting it from the trash). They are also more aware of repository update files, which is another thing neither have been good at clearing out in the past.

So, you don't have to do anything here, but with luck doing things like 'clear trash' and other large forced physical deletes should just work a bit nicer now. Users who run a server may see it clear out a couple of files after booting, and in a few weeks I'd like to roll out a 'clear orphan files' server database action for the admin menu just like the client's so you can clear out legacy files from deleted services.

full list

- image icc:
- images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
- the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
- images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
- future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display


Anonymous Board owner 12/01/2021 (Wed) 23:11:51 Id: e04c8d [Preview] No.1181 del
- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/08/2021 (Wed) 03:31:48 Id: 95e96e [Preview] No.1182 del
I had a good week. I fixed a variety of small bugs, added some quality of life options, and made several behind the scenes database improvements that will improve search now and in the future.

The release should be as normal tomorrow.



Version 463 Anonymous Board owner 11/24/2021 (Wed) 22:52:19 Id: f27ebb [Preview] No. 1174 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Xm3mrwyR2pw [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Linux.-.Executable.tar.gz

I had a great week. I was able to catch up on the work I wanted and do some neat stuff besides.

import presentation

I completely overhauled the importer 'presentation' options, which you can see under all 'file import options'. These have always been a bit advanced and confusing, so I rewrote it to specifically say a more human-friendly summary like 'show new files' and added text and tooltips to better explain what was going on. It also works a bit faster and cleaner.

You can also now say 'show inbox files', regardless of 'successful' or 'already in db', which I think may be neat as the default for a watcher page that you process and revisit several times--every time you were to re-highlight a watcher, it would filter out anything you previously archived.

Message too long. Click here to view full text.



Anonymous Board owner 11/24/2021 (Wed) 22:53:58 Id: f27ebb [Preview] No.1175 del
full list

- misc:
- ogv files (ogg with a video stream) are now recognised by the client! they will get resolution, duration, num frames and now show in the media viewer correctly as resizeable videos. all your existing ogg files will be scheduled for a rescan on update
- wrote new downloader objects to fix deviant art tag search. all clients will automatically update and should with luck just be working again with the same old 'deviant art tag search' downloader
- added prototype copy/paste buttons to the manage ratings dialog. the copy button also grabs 'null' ratings, let me know how you get on here and we'll tweak as needed
- file searches with namespaced and unnamespaced tags should now run just a little faster
- most file searches with multiple search predicates that include normal tags should now run just a little faster
- the file log right-click menu now shows 'delete x yyy files from the queue' for deleted, ignored, failed, and skipped states separately
- the tag siblings + parents display sync manager now forces more wait time before it does work. it now waits for the database and gui to be free of pending or current background work. this _should_ stop slower clients getting into hangs when the afterwork updates pile up and overwhelm the main work
- the option 'warn at this many pages' under _options->gui pages_ now has a max value of 65535, up from 500. if you are a madman or you have very page-spammy subscriptions, feel free to try boosting this super high. be warned this may lead to resource limit crashes
- the 'max pages' value that triggers a full yes/no dialog on page open is now set as the maximum value of 500 and 2 x the 'warn at this many pages' value
- the 'max pages' dialog trigger now only fires if there are no dialogs currently open (this should fix a nested dialog crash when page-publishing subscriptions goes bananas)
- improved error reporting for unsolveable cloudflare captcha errors
- added clarification text to the edit subscription query dialog regarding the tag import options there
- added/updated a bunch of file import options tooltips
- .

Message too long. Click here to view full text.



Anonymous Board owner 11/24/2021 (Wed) 22:54:31 Id: f27ebb [Preview] No.1176 del
- boring presentation import options stuff:
- presentation options are now in their own object and will be easier to update in future
- the 'should I present' code is radically cleaned up and refactored to a single central object
- presentation filtering in general has more sophisticated logic and is faster when used on a list (e.g. when you highlight a decent sized downloader and it has to figure out which thumbs to show). it is now careful about only checking for inbox status on outstanding files
- presentation now always checks file domain, whereas before this was ad-hoc and scattered around (and in some buggy cases lead to long-deleted files appearing in import views)
- added a full suite of unit tests to ensure the presentation import options object is making the right decisions and filtering efficiently at each stage
- .
- boring multiple local file services work:
- I basically moved a bunch of file search code from 1 file services to n file services:
- the file storage module can now filter file ids to a complex location search context
- namespace:anything searches of various sorts now use complex location search contexts
- wildcard tag searches now use complex location search contexts
- counted tag searches now use complex location search contexts
- search code that uses complex location search contexts now cross-references its file results in all cases
- I now have a great plan to add deleted files search and keep it working quick. this will be the next step, and then I can do efficient complex-location regular tag search and hopefully just switch over the autocomplete control to allow deleted files search

next week

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/01/2021 (Wed) 05:59:04 Id: 5a29c1 [Preview] No.1178 del
I had a good week. I managed to add support for embedded ICCs, which will improve some images' colours, and overhauled how files are deleted from the client and serevr so the system is smoother and more reliable.

The release should be as normal tomorrow.



Version 462 Anonymous Board owner 11/17/2021 (Wed) 22:40:54 Id: e7725a [Preview] No. 1169 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Gu2Iayi3AvM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Linux.-.Executable.tar.gz

I had a good week with a variety of fixes and optimisations. New users also get a couple of user-friendly services to start with.

highlights

A recent update to the mpv seekbar accidentally caused bad errors during audio playback. I have now fixed it. I regret the regression, and I have added tests to make sure this does not happen again.

All 'file log' import queues now search and count themselves faster. Any heavy import session, like a very large downloader or many medium size downloaders working at once, should use less CPU just to do normal work.

Message too long. Click here to view full text.



Anonymous Board owner 11/17/2021 (Wed) 22:41:23 Id: e7725a [Preview] No.1170 del
- network redirect improvements:
- gallery page jobs now give their child 'next gallery page' jobs themselves as a referrer
- when the gallery downloader gets a 3XX redirect, the file import objects and next gallery pages it makes now get the redirected URL as referral url (previously, it used the original gallery url)
- when the post downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the post downloader gets a 3XX redirect, child import objects and direct file downloads now get the redirect URL as referral url (previously, it used the original post url)
- when the raw file downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the raw file downloader gets a 3XX redirect to a Post URL, it now tries to queue that URL back up in the file log, just like when a gallery fetch comes back with a Post URL. some safety code stops potential loops here as well
- .
- new services:
- a new client now starts with a second local tag service called 'downloader tags'. default tag import options are now initialised in a fresh client to pull all file post tags to this service. this should relieve new users' confusion about setting up default tag import options
- similarly, a new client now starts with a like/dislike rating service called 'favourites'. existing users who have no rating services will be asked if they want to get it on update. many users are unaware of the rating system, so this is a chance to play with it
- the 'getting started with downloading' 'and '...with ratings' help has some updated screenshots and talk about the new default services and parsing
- .
- database fixes:
- fixed a very slow database query that was involved with file search using unnamespaced tags when other search predicates had already limited the scope of search
- fixed a similar slow query when the 'bad' search predicate was 'namespace:anything', particularly when the namespace is a wildcard
- fixed the 'clear orphan tables' database maintenance routine. it had decayed due to bit rot and was deleting new repository processing tracking tables. the routine is now plugged directly into the new database modules system, and any module now can be upgraded to discover surplus service tables. the system has been updated to permit the detection and removal of duplicate tables in the wrong database file, and it also makes a statement if no orphan tables were found

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 11/24/2021 (Wed) 06:38:50 Id: e01736 [Preview] No.1173 del
I had a great week. As well as some little fixes and quality of life work, I improved support for ogv files (oggs with video), and completely reworked the 'presentation' options of importers, making the whole system faster and more user-friendly.

The release should be as normal tomorrow.



Version 461 Anonymous Board owner 11/10/2021 (Wed) 22:47:22 Id: 10414b [Preview] No. 1163 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=oiP5bKlJ0fM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Linux.-.Executable.tar.gz

I had a great week doing a lot of background cleanup work, and I also fixed some things and improved some quality of life.

highlights

I brushed up last week's .clip file metadata parsing. Some clip files were getting too-small resolutions and thumbnails due to a 'canvas unit' issue, basically some clips store resolution in mm or inches etc... rather than pixels. Thanks to a user who also provided some great examples, we managed to pin down a solution to get actual pixel resolution, and I also figured out duration and number of frames for animated clips. All clips will reparse again on update and generate nicer size thumbs if they need it.

Also, if you ever got a webm that seemed normal except it somehow had a crazy 7 hour duration, I think I figured out the problem here too. All superlong-but-actually-small files will be reparsed on update and should get a more accurate duration.

Message too long. Click here to view full text.



Anonymous Board owner 11/10/2021 (Wed) 22:48:27 Id: 10414b [Preview] No.1164 del
full list

- misc:
- the text on the animation scanbar is now center-vertically aligned and should look better on taller and thinner scanbars (issue #998)
- the scanbar now reports better frame number and current time for the mpv player when the current video is very short or has very few frames. screamer gifs should now report 2/2 frames if you scan to the right, not like 97/2
- fixed using the mpv player with an embed button (it previously was staying hidden even after embed was clicked) (issue #999)
- the 'search' submenu when you right-click on tags in certain locations now shows add/exclude namespace:anything if all the selected tags share the same namespace
- as an experiment that I think will be bulked out into proper shortcuts later, and maybe actual +/- buttons like you'll see on a booru, if you activate the 'selection tags' listbox (double-click or enter key) while ctrl is down, it now excludes the selected tags from the current query
- when you paste query texts into the edit subscription dialog, those queries already in the sub _and_ DEAD will now be revived (it does a 'check now' on them). the dialog reports this
- when editing subscriptions, the way it waits for the current subs to stop running is improved. it is now separate from the global 'pause subs' variable, so big delays here (e.g. waiting a long time to open the dialog, then hitting 'pause' on the network menu, which was secretly a logically messy unpause) should be less able to run into trouble
- watchers now sort DEAD and 404 separately when sorting the status column (previously they were sorted by their now-defunct 'next check time')
- I think I improved the speed of the new subscriptions guaranteed shutdown. I think I also fixed a shutdown hang on some lagging async jobs. there are a couple of reports of hanging shutdown, so let me know if this changes
- I moved the autocomplete options from 'gui pages' to 'search', and I brushed up the layout and tooltips there generally
- .
- file parsing:
- clip files with canvas size units in mm, cm, inches, or points are now parsed correctly! thanks to the user who helped here! turns out a point is 1/72 (two grossths :^)) of an inch
- clip animations now get the number of frames and duration of the first timeline!

Message too long. Click here to view full text.



Anonymous Board owner 11/10/2021 (Wed) 22:49:02 Id: 10414b [Preview] No.1165 del
- client api:
- /get_files/file_metadata now has an optional boolean parameter, 'hide_service_names_tags', default False, which will hide the old 'service_names_to...' tag Objects
- a unit test tests hide_service_names_tags
- client api help documentation now talks about hide_service_names_tags
- client api version is now 22
- .
- boring code refactoring and cleanup:
- tl;dr: about 60KB total code moved out of client database!
- moved most combined sibling+parent database code to a new 'tag display' module
- moved autocomplete counts cache database code to a new 'mappings counts' module, and refactored a whole lot of of misc old a/c table creation and reference code into that module, cleaning things up
- the 'mappings counts' module is plugged into new repair code and on error repopulates itself as efficiently as regen code currently allows
- moved tag cache code to a new 'tag search' module, and similar related decoupling refactoring
- the 'tag search' module is plugged into new repair code and on error repopulates itself very efficiently
- the 'local tags cache' module can now regenerate itself on boot
- regenerating the local tags cache now works a little faster and takes less memory
- I _believe_ client.caches.db can now regenerate all of itself automatically, with no subsequent user actions needed
- the boot database repair notifications have some quality of life improvements. modules now say whether they think they can recover everything, and there is more guidance on what to do in the different situations

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 11/17/2021 (Wed) 06:55:16 Id: 777cc4 [Preview] No.1168 del
I had a good week. I fixed some bugs (including the new mpv audio file playback issue), reduced CPU load during heavy import sessions, improved network redirect support in the downloader, added new-user-friendly services to new clients, and massively sped up some complicated file searches that include unnamespaced tags.

The release should be as normal tomorrow.



Version 460 Anonymous Board owner 11/03/2021 (Wed) 22:19:15 Id: 4aa4e4 [Preview] No. 1160 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=gP5LXd6DFx4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Linux.-.Executable.tar.gz

I had an ok but light week of various work.

misc

When I improved the 'apng' file parsing the other week, there was a mistake in how I hooked it up and not all the file metadata was being propagated to the database. A couple of users got some weird '1 frame' apngs as a result. I have fixed this problem, and improved the parsing to deal with more sorts of apngs, and scheduled a complete rescan of all your apngs to get some nicer 'number of frames' counts.

In a related thing, a user sent in a way to get some cool metadata from 'clip' files. These now get resolution and a rough thumbnail. I have scheduled all your clip files, if you have any, to update too.

Message too long. Click here to view full text.



Anonymous Board owner 11/03/2021 (Wed) 22:20:47 Id: 4aa4e4 [Preview] No.1161 del
- misc:
- right-clicking a selection tag and choosing 'select->files with x tag' now obeys the current tag domain (previously, it forced 'all known tags'). so, if you want to quickly select just the files with 'samus aran' on 'my tags', it should be doable
- the new 'partial' download resumer system will now tolerate two successive empty chunks before throwing a 'this chunk was empty' error. it seems some servers will randomly give an empty chunk at times during 206 transfers
- cancelling the slideshow custom time dialog no longer raises an error
- after the build boot problem last week, updated the opencv version in requirements.txts--we are now officially >=4.0.0, <=4.5.3.56. it looks like pyinstaller needs a patch for 4.5.4.58 to work, so we'll wait for that. I am improving my weekly test routine to try to catch this in future
- also, the windows build no longer includes two copies of an opencv dll. turns out PyInstaller finds this dll ok now (putting it in another location) and it doesn't need to be explicitly added during build
- added a new help file to the db dir for users who experience crashes as soon as they load videos in mpv when using WASAPI or ASIO drivers. thanks greatly for the user who figured out the mpv.conf solution here (issue #973)

next week

More small work like this, and I would like to do some behind the scenes database code cleanup too.


Release Tomorrow! Anonymous Board owner 11/10/2021 (Wed) 07:01:40 Id: 1a1362 [Preview] No.1162 del
I had a great week. As well as a heap of good background cleanup work in the database, I fixed several bugs, improved .clip and video file metadata parsing, added some new quick tag search actions to the taglist, and cleaned up some bad animation display.

The release should be as normal tomorrow.