/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Magrathea | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 472 Anonymous Board owner 02/02/2022 (Wed) 23:42:09 Id: cd0314 [Preview] No. 1215 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=tGgmv1yEI-U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Linux.-.Executable.tar.gz

I had a good week. There are some neat new UI features and several bug fixes.

new stuff

Although it isn't super useful yet, searching in multiple file domains is now complete. If you click the file domain button on a search page's tag autocomplete dropdown, there is now a 'multiple locations' entry. This lets you select a union of several domains. New users will probably only see 'my files' and 'trash', but advanced users will see more, and deleted domains too. Please feel free to play with this. Autocomplete counts should have accurate ranges and the union file searches, limited though they currently are, do work. We are a step closer to fully supporting multiple 'my files' services.

Under options->thumbnails, there is now a dropdown for 'thumbnail scaling'. The default remains 'scale down only', but you can now 'scale to fit' (which scales small things up) or 'scale to fill', which crops and scales up so the thumbnail fills the whole thumbnail space. Give it a go--the animation as one format changes to another is accidentally one of the coolest things I have done.

Message too long. Click here to view full text.

4 posts and 1 image omitted.


Anonymous 02/06/2022 (Sun) 12:16:31 Id: b3e0e1 [Preview] No.1220 del
>>1219
I've narrowed down the error to the update from 438 to 439


Anonymous Board owner 02/09/2022 (Wed) 00:56:30 Id: 669aed [Preview] No.1221 del
>>1218
Thank you, I am glad you like it!

>>1219
>>1220
Thank you for this report. I am sorry for the trouble. I know that around that time (441), we moved to builds on Github. This was because my old macOS laptop (I think it was 10.13 High Sierra) was so old the things it made were incompatible with the new Big Sur. We now build on a Big Sur machine on the cloud. 439 isn't exactly when that happened, but I know I started work on newer version requirements for libraries around that time, so I am afraid I think this might just be a version incompatibility problem. Searching "LSOpenURLsWithRole 10810" shows a variety of similar versioning problems.

However there are also some things like this: https://www.mathworks.com/matlabcentral/answers/460920-why-do-i-get-the-error-lsopenurlswithrole-failed-with-error-10810-when-trying-to-install-matlab

Maybe this is related to security? Is your macOS set up with default security rules. I know you normally have to enable 'Allow apps downloaded from anywhere' as the Readme in the App dmg says.

I am not a macOS expert however. I thought that Mojave (10.14?) was compatible with Big-Sur-built Apps. I may be wrong, or it may just be more complicated than I understand. Do you happen to have a special version of python installed on your machine? Some of the searching I did on that error suggested a conflicting python install might cause it somehow.

I assume if you just run the App by double-clicking its icon (which would normally make a database in ~/Library I think), that doesn't work either?

If it seems like this is just a versioning problem--that Mojave (or your Mojave) is too old to run the Big Sur build, I think your best shot is to see about updating your machine to a later macOS. When we move to Qt 6 over this year, I am afraid I will have to say the same about Windows 7. Unfortunately, macOS seems to retire older OSes faster than Windows. Another potential answer here, if you have some python experience, although it sucks, is to run from source. It may be possible to keep using an older version of Qt, I imagine around 5.12 (check your help->about in v438 to see what Qt seems ok for you), if you are willing to go through the work of setting up a virtual environment and testing out different library versions. Here is my page on that, if you are interested: https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

Message too long. Click here to view full text.



Simple Release Tomorrow! Anonymous Board owner 02/09/2022 (Wed) 00:56:52 Id: 669aed [Preview] No.1222 del
I had a mixed week--unfortunately, some IRL stuff reduced my work time. I did however clean some code, fix a couple bugs, and integrate a cool new quick-navigation widget that a user wrote.

The release should be as normal tomorrow.


Anonymous 02/09/2022 (Wed) 20:26:22 Id: 1ee91b [Preview] No.1223 del
(2.97 KB 383x64 Screenshot_4361.png)
How do I give it the cookie?
Why does it need the cookie to log in when I gave it the login info already?
v464, if that matters


Anonymous 02/09/2022 (Wed) 22:49:52 Id: 977a09 [Preview] No.1226 del
>>1223
Hmm, this error basically means that as it tried to log in, the site did not give a cookie that the login script expected to be given. This suggests the site has changed its login process since that script was written, or maybe an unusual error like your login password was wrong or something or the login page now has a captcha, so the login just failed.

For all complicated login situations, I recommend Hydrus Companion now. It lets you copy your cookies right from your logged-in browser to Hydrus using the Client API:

https://gitgud.io/prkc/hydrus-companion



Version 471 Anonymous Board owner 01/26/2022 (Wed) 22:40:18 Id: 858b1b [Preview] No. 1210 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=HSdNxvFCZj4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Linux.-.Executable.tar.gz

I had a great week. The program can now track the last time you viewed a file.

last viewed time

By default, file viewing statistics are on. You can control how they work and turn them off under options->file viewing statistics and if you like clear all your records under the database menu.

If you have them on, hydrus now records the last time you saw a file in the media viewer and preview viewer. A 'last viewed' here counts with the same rules as recording view time (typically you need to look at the file a few seconds), so if you just scroll through ten files in a second, those won't be recorded as views.

Message too long. Click here to view full text.



Anonymous Board owner 01/26/2022 (Wed) 22:40:53 Id: 858b1b [Preview] No.1211 del
full list

- times:
- if you have file viewing stats turned on (by default it is), the client will now track the 'last viewed time' of your files, both in preview and media viewers. a record is only made assuming they pass the viewtime checks under _options->file viewing statistics_ (so if you scroll through really quick but have it set to only record after five seconds of viewing, it will not save that as the last viewed time). this last viewed time is shown on the right-click menu with the normal file viewing statistics
- sorting by 'import time' and 'modified time' are moved to a new 'time' subgroup in the sort button menu
- also added to 'time' is 'last viewed time'. note that this has not been tracked until now, so you will have to look at a bunch of things for a few seconds each to get some data to sort with
- to go with 'x time' pattern, 'time imported' is renamed to 'import time' across the program. both should work for system predicate parsing
- system:'import time' and 'modified time' are now bundled into a new 'system:time' stub in the system predicates list. the window launched from here is an experimental new paged panel. I am not sure I really like it, but let's see how it works IRL
- 'system:last view time' is added to search the new field! give it a go once you have some data
- also note that the search and sort of last viewed time works on the 'media viewer' number. those users who use preview or combined numbers for stuff, let me know if and how you would like that to work here--sort/search for both media and preview, try to combine based on the logic in the options, or something else?
- .
- loading serialised pngs:
- the client can now load serialised downloader-pngs if they are a perfect RGB conversion of an original greyscale export.
- the pngs don't technically have to be pngs anymore! if you drag and drop an image from firefox, the temporary bitmap exported and attached to the DnD _should_ work!
- the lain easy downloader import now has a clipboard paste button. it can take regular json text, and now, bitmap data!
- the 'import->from clipboard' button action in many multiple column lists across the program (e.g. manage parsers) (but not every list, a couple are working on older code) also now accepts bitmap data on the clipboard
- the various load errors here are also improved

Message too long. Click here to view full text.



Anonymous Board owner 01/26/2022 (Wed) 22:42:28 Id: 858b1b [Preview] No.1212 del
- misc:
- mouse release events no longer trigger a command in the shortcuts system if the release happens more than about 20 pixels from the original mouse down. this is tricky, so if you are into clever shortcuts, let me know how it works for you
- the file maintenance manager (which has been getting a lot of work recently with icc profiles, pixel dupes, some thumb regen, and new audio channel checks), now saves its work and publishes updates faster to the UI, at least once every ten seconds
- the sort entries in the page sort control are now always sorted according to their full (type, name) string, and the mouse-wheel-to-navigate is now fixed to always mirror this
- improved some 'delete file reason' handling. currently, a file deletion reason should only be applied when a file is entering trash. there was a bug that force-physical-deleting files from trash would overwrite their original deletion reason. this is now fixed. the advanced delete files dialog now disables the whole reason panel better when needed, never sends a file reason down to the database when there should be no reason, disables the panel if all the files are in the trash, and at the database level when file deletion reasons are being set, all files are filtered for status beforehand to ensure none are accidentally set by other means. I am about to make trash more intelligent as part of multiple local file services, so I expect to revisit this soon
- the new ICC Profile conversion no longer occurs on I or F mode files. there are weird 32/64 bit monochrome files, and mode/ICC conversion goes whack with current PIL code
- replaced the critical hamming test in the duplicate files system with a different bit-counting strategy that works about 9% faster. hamming test is used in all duplicate file searching, so this should help out a tiny bit in a lot of places
- .
- boring cleanup:
- cleaned up how media viewer canvas type is stored and tested in many places
- all across the program, file viewing statistics are now tracked by type rather than a hardcoded double of preview & media viewer. it will take a moment to update the database to reflect this this week
- cleaned up a ton of file viewing stats code
- cleared out the last twenty or so uses of the old 'execute many select' database access routine in favour of the new lower-overhead and more query-optimisable temporary integer tables method

next week

Back to multiple local file services. A widget to select multiple services in file search, and I'll start on a trash that can handle deletes from and undeletes back to multiple locations.


Release Tomorrow! Anonymous Board owner 02/02/2022 (Wed) 05:00:41 Id: 28a2f8 [Preview] No.1214 del
I had a good week. I fixed a heap of bugs, added a couple new ways to scale thumbnails, and finished the basics of multiple domain file search.

The release should be as normal tomorrow.



Parser Sharing Thread Anonymous 01/28/2022 (Fri) 10:22:28 Id: a83177 [Preview] No. 1213 [Reply] [Last 50 Posts]
Can't see one here, maybe I'm blind, I'll start.



Version 470 Anonymous Board owner 01/19/2022 (Wed) 22:25:03 Id: 9aac08 [Preview] No. 1206 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=_SSPdeoAHsM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Linux.-.Executable.tar.gz

I had a great week. The big rewrites are finally doing something interesting, and I got some other stuff done as well.

cleverer search

First of all, I finished my file domain rewrites. File search, tag search, and all the UI that displays it now works on n file domains at once rather than just one. Also, any of those n can be the deleted files of a service now. You can't make new local file services yet, obviously, but all search is now ready for it.

Since adding deleted files search was easy, I am adding it this week. Users in advanced mode will now see a list of deleted file domains on the tag autocomplete file domain button menu. While it is rare you ever want to do this, it was never actually possible to search these domains completely before, and definitely not quickly. Give it a go and let me know if you run into any trouble!

Message too long. Click here to view full text.



Anonymous Board owner 01/19/2022 (Wed) 22:25:23 Id: 9aac08 [Preview] No.1207 del
full list

- multiple file services:
- I finished the conversion of all UI search to the new multiple location object. everything from back- to frontend now supports cleverer search. since searching deleted files is simple to add, users in advanced mode will now see 'deleted from...' in a new list in the tag autocomplete dropdown file domain button
- the next step is writing a widget that allows multiple selection, and then all this should work right out of the box, and we'll be an important step closer to allowing multiple local file services
- .
- misc:
- the video parsing routine is better at detecting when a present audio track is actually silent (and hence when it should mark a video as 'no audio'). all video with audio will be requeued for a metadata reparse in the files maintenance system on update
- fixed an error from last week when trying to create a new page from the tags (e.g. middle-clicking them) in the active search list
- added 'audio mkv' format to the client, to represent mkvs without a video track. I think most of the time this is going to be audio track webms from youtube-dl and similar
- added 'file relationships: set files as potential duplicates' command to the 'media actions' shortcut set
- I expanded the 'backing up' section in 'installing and updating' help
- I wrote an 'anti-virus' section for 'installing and updating' help, since I kept writing the same basic spiel about false positives. please feel free to point people there in future to relieve their concerns
- improved some shutdown tests, the client and server should exit faster in some cases (e.g. when a hydrus repository network job is hanging on reconnection attempts, holding up the 'synchronise_repository' daemon shutdown)
- the 'file was xxx at (y timestamp), which was (z time units) before this check' line in file import notes now always puts 'z time units' as that, ignoring the 'always show ISO time' setting, which was just substituting it with 'y timestamp' again. let me know if you spot other bad grammar with this setting on, I'll fix it!
- fingers crossed, images in the LAB colourspace _should_ now normalise to sRGB with the correct whitepoint. thanks to the user who provided example test tiff images here. this now uses the new PIL-based colourspace conversion I used to make ICC profiles work, just on LAB->sRGB. as far as I understand, OpenCV uses a fixed whitepoint of D65, resulting in yellow/warm conversions for some formats, but PIL may be able to figure out if D50 is needed??? if you have some crazy LUV or YPbPr or YIQ image that shows up wrong, please send it in and I'll see what I can do!


Anonymous Board owner 01/19/2022 (Wed) 22:27:55 Id: 9aac08 [Preview] No.1208 del
- boring rewrites and cleanup while doing file service work:
- many more UI objects now store and do file service logic using a more complicated 'location context', which can store a mix of multiple services and 'deleted from service' data. all the search code that works on this can now propagate to display:
- the management objects behind every page now store a multiple location object, not a single file service id
- all media panels (the thumbnail grid on a page) are now instantiated by a multiple location object, and when they serve a highlighted downloader, they now inherit that from the file import options, which in future will dictate import destinations
- all canvases are now the same, inheriting their new location context from their parents
- all tag lists are the same. mostly they don't care much about file domain, but when you middle-click to create new pages from the autocomplete dropdown list or active search list, it can matter, so they now propagate it along
- the underlying medialist objects are now the same, and various delete logic (e.g. 'should we remove this thumb we just deleted?') is updated to work on complex domains
- some duplicate lookup code now works on location context
- renamed 'location search context' object to 'location context' since it is used all over now and put it in its own file. also wrote it some neater initialisation and meta object code
- mr bones now gives duplicate data based on the union of all non-trash local services sans update files (another case of now supporting n services but n is fixed for the moment at 1, 'my files')
- a bunch of places across the program that used to default to 'my files' or 'all local files' (which is everything on disk, including trash and repository update files) now default to this new union of all non-trash local media services
- when doing page-to-page file drag and drops, the location context is now preserved (previously, the new page would always be 'my files')
- whole heap of other cleanup in these systems
- when a thumbnail cannot be provided (for deleted files or many 'all known files' situations), the thumbnail cache now provides the hydrus icon stand-in instantly, no delayed waterfall
- fixed an unusual situation where the file search could not provide a file in a tagless search when that file had no detailed file info row in the database. this seems to effect a legacy borked row or two in the new deleted file domain searches
- removed some ancient dumper status code from thumbnail objects

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 01/26/2022 (Wed) 02:21:48 Id: 7b4235 [Preview] No.1209 del
I had a great week doing some fun stuff along with catching up on older issues. I managed to add tracking for 'last viewed time' for files, along with search and sort capability just like for import and modified times. There's also the start of custom colours for the video scanbar using the new style system.

The release should be as normal tomorrow.



Version 468 Anonymous Board owner 01/05/2022 (Wed) 23:02:52 Id: 26f6f8 [Preview] No. 1197 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=UGnbiBDmuBI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Linux.-.Executable.tar.gz

I had a good first week back. A mix of bug fixes and little improvements.

couple of highlights, otherwise all misc this week

Thanks for the quick issue reports over the holiday. The scanbar had an annoying thing where the new single pixel border was making it awkward to drag when in borderless fullscreen--you'd move your mouse to the bottom of the screen, but then you'd click the border and not the scanbar, and the whole video would move--this I have now fixed. Also an issue with new imports' 'pixel duplicates' data not being saved correctly--I have fixed the problem and scheduled all affected files to regen their pixel duplicate data.

I also put some time into multiple local file services. It was more background work, this time to do with system predicate fetching, so there isn't much neat stuff to show off unless you are an advanced user. Almost all the background work is done now, though, so I hope to start spinning up more complex search UI and actually adding new file services pretty soon. One neat thing, now that I have new file filtering tools, I was able to expand 'system:file service' to search for deleted and petitioned files.

Message too long. Click here to view full text.

3 posts omitted.


Simple Release Tomorrow! Anonymous Board owner 01/12/2022 (Wed) 02:43:48 Id: 98831b [Preview] No.1201 del
I had a mixed week. The tag autocomplete rewrite was more work than I expected. While I am happy with what I did, most of my time was spent on boring background and cleanup, so there is little on the front end to show for it. Beyond that, I fixed the Linux/macOS write permissions checking problem and did a couple small things in the Client API and downloader search log. I will recommend the release just to people interested in those issues.

The release should be as normal tomorrow.

>>1199
>>1200
Thank you, I am sorry for the trouble! I think I have fixed this for tomorrow's release, please try it out and let me know if you have any more problems.


Release Tomorrow! Anonymous Board owner 01/19/2022 (Wed) 04:21:03 Id: 759469 [Preview] No.1202 del
Hey, sorry, Endchan was not working for me at post time last week, so we missed 469 here. I will continue with 470 tomorrow, fingers crossed.

I had a great week. I finished the multiple file service rewrites and can finally start extending search. First off, for tomorrow's release, all advanced users will be able to quickly search their deleted files. I also fixed some bugs and improved some file handling, including the test that detects whether a video has an audio stream or not.

The release should be as normal tomorrow.


Anonymous 01/19/2022 (Wed) 08:41:38 Id: 2bca88 [Preview] No.1203 del
>>1202
I just wanted to say thank you for your work.


Anonymous 01/19/2022 (Wed) 09:53:22 Id: 9dd40f [Preview] No.1204 del
>>1201
Same anon from >>1199, everything seems to be working fine now on 469. Thanks.


Release Tomorrow! Anonymous Board owner 01/19/2022 (Wed) 22:24:16 Id: 4f1262 [Preview] No.1205 del
>>1203
>>1204
Thanks lads, keep on pushing.



Version 467 Anonymous Board owner 12/22/2021 (Wed) 23:09:46 Id: 9ab3aa [Preview] No. 1193 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=MrweXsImVhg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Linux.-.Executable.tar.gz

I had a good week cleaning some things up for end of year.

couple of scanbar things

I polished the new autohiding video scanbar. A bunch of the layout and coordinate detection code of the scanbar and the media canvas behind it is less jank. If you had some flicker or weird mouse-popup behaviour last week, I hope it is better now!

If you are a macOS user however, I held back one of the changes. There's a background-erasing hack I put in a couple years ago because without it macOS media viewers went to 100% CPU. I hope this is no longer true, but I'm not certain, so you still have the flag on by default. Please hit _help->debug->gui actions->macos anti-flicker test_ and then try browsing some images and video in a new viewer. Does your client lock up, or is it ok now? If everything is good, I'll remove the flag and you can get some nicer anti-flicker tech too.

Message too long. Click here to view full text.



Anonymous Board owner 12/22/2021 (Wed) 23:10:03 Id: 9ab3aa [Preview] No.1194 del
- misc:
- the 'file log' and 'search log' buttons are now a new widget class that puts an arrow on the side that opens a menu. the secret right-click menus of these buttons is now exposed for all
- fixed a bug affecting some greyscale pngs with ICC profiles--they were coming out pure white due to a colourspace conversion problem
- fixed an import problem when PIL could not load a file (due to file malformation) but OpenCV could. this was causing a failed import from the new ICC profile detection code
- when the downloader hits a broken image file that cannot be imported due to malformation, the status is now 'error' instead of the incorrect 'ignored'
- fixed the duplicate file filesize comparison statement sometimes showing > in one direction and ≈ in the other. it happened when the larger file was between 20/19 and 21/20 times the size of the smaller, just a logic typo (issue #1028)
- the trash maintenance daemon is moved from the old threaded daemon system to the new repeating job worker pool. this is the last daemon cleaned up, so I am retiring the old and mostly defunct 'no_daemons' launch argument. a variety of other daemon infrastructure for things like shutdown checks is similarly removed. the program also now waits for the newer daemon jobs to finish working on shutdown
- moved most client daemon jobs like repository sync and dirty object save down so they start after the first session is loaded rather than right after boot
- if a file is called to regen its thumbnail but currently has no dimension, this is now a no-op rather than an error. in the situation where users force thumb regen before metadata regen and encounter this, it is sorted out later when the metadata regen recognises new dimensions and reschedules the thumb regen
- added an extensive user-written guide to the --db_synchronous_override launch argument to the launch arguments help page. it is possible and safe to run the program with synchronous=0 as long as certain caveats are obeyed. thanks to the user who figured this out and wrote it up
- the downloader engine now discards source urls in an import job if they have the same domain as any existing primary url. this will ensure that if a booru has a link back to itself as a source url, when the 'source' is really an alternate rather than a dupe, it won't be added in hydrus as a known url for that imported file
- misc cleanup in downloader system and file/search log UI
- fixed a type bug in the file and search log 'import from png' action. if you have existing pngs previously exported from here, they will import ok now
- refactored the various hydrus compression code to a new HydrusCompression file
- exported serialisable data pngs such as from file or search log that hold simple Strings now always compress the data before embedding it in the png. existing pngs that hold uncompressed strings should still load ok
- the payload in an exported png is now always compressed, and the payload description always states the uncompressed size
- sped up client shutdown when network traffic has been paused the whole time and a repo sync job might have wanted to run. these jobs also do not hang on a thread worker if network traffic is paused, but they should wake immediately when it is unpaused

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 01/05/2022 (Wed) 03:44:47 Id: b342a2 [Preview] No.1196 del
I had a good week. I fixed bugs--including the recent annoying scanbar border clicking issue--rewrote some bad code, and added a couple of advanced file operations to search and the client api.

The release should be as normal tomorrow.



(10.80 KB 397x237 92384659.png)
Malformed Drive -- Help Anonymous 01/02/2022 (Sun) 12:49:54 Id: c590d9 [Preview] No. 1195 [Reply] [Last 50 Posts]
I've got multiple drives I've tested Hydrus on, and can't figure out why I'm having issues. All my drives are healthy and run without problems. At first I thought Window's built in drive compression might be fucking with Hydrus, but I decompressed my whole SSD, and I'm still getting errors. Can someone suggest a possible reason for my issue? Will provide logs if needed, Thanks in advance.



Version 466 Anonymous Board owner 12/15/2021 (Wed) 23:18:40 Id: 960c24 [Preview] No. 1188 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=kWXAL9bHciQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Linux.-.Executable.tar.gz

I had a great week. The duplicate filter now supports filtering by pixel duplicates, and videos now fit better in the media viewer.

better duplicate search

Now we have pixel duplicate data stored in the database, I can search it. The duplicate filter page now has a dropdown that lets you select 'must be pixel dupes', 'can be pixel dupes', and 'must not be pixel dupes'. It does exactly what you think, and with luck it should be pretty fast no matter what you select.

Note that you may not have all your old pixel dupe data calculated yet. I started it for everyone last week, but even if it is doing 10,000 files a day, if you have a big client it will take a little longer. It may also still be working on the ICC profile queue. Hit up database->file maintenance->manage scheduled jobs to see how it is doing. You can rush it there if you like.

Message too long. Click here to view full text.



Anonymous Board owner 12/15/2021 (Wed) 23:19:45 Id: 960c24 [Preview] No.1189 del
full list

- video scanbar autohide:
- the scanbar that shows below audio and video is now embedded inside the video frame, and it show/hides based on how close your mouse is to it
- I've wanted to do this for a long time, since it will allow you to watch 16:9 videos at true 100% in borderless fullscreen, but the hackery of how the media viewer works behind the scenes means this took more work than you'd think and is still a little jank. there's a small amount of flicker when it pops in and out, which I will work on in future. in any case, please have a play with it and let me know what you think. I expect to add some more options, like for the activation padding area around it, and I will be tidying up more layout stuff throughout the media viewer
- if you are a mostly keyboard user, please check out the new 'global' shortcut to flip on/off a 'force the animation scanbar to show' mode
- I don't really want to bring back the always-on hanging-below scanbar that just takes up space, but if you try this new embedded scanbar and really hate it, we'll see what we can do
- .
- more duplicate filter search options:
- the duplicates page now has a dropdown on the search for 'must be/can be/excludes pixel dupes'!
- the duplicates page now has a number control on the search for what distance the pair was found at! I am not sure how accurate this thing is in all cases, but it seems I started tracking this data some time ago and forgot I even had it
- these new options are remembered in your session and _should_ remain fast in most normal cases. I put time into some complicated database work this week to get this going, please let me know if you have any trouble with it
- .
- misc:
- when the export filename pattern in the export files dialog means many of the files share the same base and hence need to do 'filename (5)'-style suffixes to be unique, the number here is now calculated much more efficiently. opening this dialog on 10,000 files with an oft-duplicate pattern should now be a jolt of lag but not many minutes
- when you choose to 'separate' a subscription with more than 100 queries, you are no longer forced to break it into half
- when you do break a subscription in half, it now makes sure to sort the query texts before separating

Message too long. Click here to view full text.



Anonymous Board owner 12/15/2021 (Wed) 23:20:37 Id: 960c24 [Preview] No.1190 del
- boring cleanup and little fixes:
- when you set the checker options in the edit subscription dialog, the queries now recalculate their file velocity better. previously, they would just set 'unknown' and recalc on the next run, but now they will actually recalculate if the query container is loaded into memory or otherwise put a status that says 'will calculate on next run'
- removed the 'should be namespaced' reason from the manage tags quick petition reasons. this is now all handled by siblings, tidying up storage tags manually is busywork
- when you click 'copy traceback' on an error popup, it also copies the software version, your platform, and if you are on a frozen build or running from source
- the logger now prints version number for every block, just before the timestamp
- cleaned up a variety of media viewer UI code while working on the scanbar, fixing some misc display bugs
- moved pixel hash storage responsibility from 'file metadata' to 'similar files' module
- the similar files system now searches pixel hashes when it is called to do any similar files search. they count as 'exact match' distance
- when a file gets a new pixel hash, it now sees if any other files have that same hash. if so, it now gets queued up again in the similar files search system, ensuring this match is not missed
- misc nomenclature cleanup--since we now have both 'pixel hashes' and 'phashes', phashes are now referred to as 'perceptual hashes' everywhere
- massively refactored the primary table join that drives potential duplicates search. it should work a bit faster now and it is much easier to work with
- I added pixel dupe and distance search to the standard search results version of the join and the 'system:everything' version, which has several optimisations
- silenced some shutdown handling in file maintenance that was being printed to log as an error
- fixed some 'broken object load' error handling to print the timestamp of the specific bad object, not whatever timestamp was requested. this error handling now also prints the full dump name and version to the log, and version to the exported filename. I was working with a user who had broken subs this week, and lacking this full info made things just a little trickier to put back together
- fixed some drag and drop handling where it was possible to drop thumbnails on a certain location of a page of pages that held an empty page of pages but it would not create a new child media page to hold them
- misc serverside db code cleanup
- fixed python 3.10 type bugs in window coordinate saving and Qt image generation from buffer (issue #1027)

Message too long. Click here to view full text.



Anonymous 12/20/2021 (Mon) 09:27:00 Id: 1f98ee [Preview] No.1191 del
>>1188
not sure where you will see this first, so ill post in both places.

as far as the media scrollbar is concerned, I GREATLY prefer to have it below and always on, I am not using hydrus to watch shows or long videos, for me the videos are essentially just replacing gifs at this point. I love being able to see how much longer they have, and being able to scrub through them without covering a good portion of the video.

If there is a way to have a toggle between visible and hidden, on video or below video, I think that would be best, because as it stands now, it's annoying to use to scrub things because it's not always there, and at least for me it's also annoying to have to keep my mouse there to see how long the video will last.

I should also mention that the way I use hydrus is I have a 4k screen, so I have the left half as media viewer area, and the right half as either the hydrus thumbnails, or a youtube video, I never have it just fullscreen.


Release Tomorrow! Anonymous Board owner 12/22/2021 (Wed) 02:34:07 Id: 129cd6 [Preview] No.1192 del
I had a good week. I fixed some small bugs and brushed up a little UI. Nothing too ambitious, but it should make for a nice clean release before I break for my holiday.

The release should be as normal tomorrow.

>>1191
Thank you. I untangled all this UI code this week and I feel better about adding options like 'hang below' in future. I have saved your thoughts and will try to enable this again soon into the new year.



Version 464 Anonymous Board owner 12/01/2021 (Wed) 23:09:01 Id: e04c8d [Preview] No. 1179 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=mn34GKTVqMM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Linux.-.Executable.tar.gz

I had a good week. Some images get better colour, and the software now deletes files more neatly.

ICC profiles

tl;dr: Some image colours are better now.

Some images come with an 'ICC profile'. A scan of one ICC-heavy client showed that 10% of all images had them. ICCs are basically colour correction data like 'actually, the blue should be this much more intensely blue than is stored' in a variety of sometimes complicated ways. Some nice cameras attach a fast ICC based on lighting conditions or known defects in the camera, and some image editing software does similar. ICCs are also used for broader and more complicated colour conversion tasks like in the migration from sRGB to HDR formats if you have a new tv/monitor.

Message too long. Click here to view full text.



Anonymous Board owner 12/01/2021 (Wed) 23:09:26 Id: e04c8d [Preview] No.1180 del
better file delete

tl;dr: Files delete from your disk better now.

For a long time, actually physically deleting a file that leaves the client's trash has been a bit hacky. To reduce UI lag, the recycle or delete action has to be deferred to the future, but since there was no permanent record of what to delete, the deferred job was always a bit rushed and in a couple of instances (e.g. if you cleared trash and closed the client soon after) it could leave file orphans behind. The system has always been fail safe, never deleting too much, but it could lead to surplus cruft in file storage. Secondly, it turns out the server never got physical file delete tech added! The file repository has never been used much, and it just never came up.

So, this week I overhauled the whole way it works. Now both client and server keep records of the files they should delete, print delete summaries to the log, use neater delete daemons that smooth out the work to be non-interrupting, have sophisticated 'is this an orphan now?' logic to ensure that edge cases are kept in storage until they are truly no longer needed, and correct themselves in odd cases (like if you re-import a file just a couple of seconds after deleting it from the trash). They are also more aware of repository update files, which is another thing neither have been good at clearing out in the past.

So, you don't have to do anything here, but with luck doing things like 'clear trash' and other large forced physical deletes should just work a bit nicer now. Users who run a server may see it clear out a couple of files after booting, and in a few weeks I'd like to roll out a 'clear orphan files' server database action for the admin menu just like the client's so you can clear out legacy files from deleted services.

full list

- image icc:
- images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
- the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
- images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
- future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display


Anonymous Board owner 12/01/2021 (Wed) 23:11:51 Id: e04c8d [Preview] No.1181 del
- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/08/2021 (Wed) 03:31:48 Id: 95e96e [Preview] No.1182 del
I had a good week. I fixed a variety of small bugs, added some quality of life options, and made several behind the scenes database improvements that will improve search now and in the future.

The release should be as normal tomorrow.



Version 463 Anonymous Board owner 11/24/2021 (Wed) 22:52:19 Id: f27ebb [Preview] No. 1174 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Xm3mrwyR2pw [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Linux.-.Executable.tar.gz

I had a great week. I was able to catch up on the work I wanted and do some neat stuff besides.

import presentation

I completely overhauled the importer 'presentation' options, which you can see under all 'file import options'. These have always been a bit advanced and confusing, so I rewrote it to specifically say a more human-friendly summary like 'show new files' and added text and tooltips to better explain what was going on. It also works a bit faster and cleaner.

You can also now say 'show inbox files', regardless of 'successful' or 'already in db', which I think may be neat as the default for a watcher page that you process and revisit several times--every time you were to re-highlight a watcher, it would filter out anything you previously archived.

Message too long. Click here to view full text.



Anonymous Board owner 11/24/2021 (Wed) 22:53:58 Id: f27ebb [Preview] No.1175 del
full list

- misc:
- ogv files (ogg with a video stream) are now recognised by the client! they will get resolution, duration, num frames and now show in the media viewer correctly as resizeable videos. all your existing ogg files will be scheduled for a rescan on update
- wrote new downloader objects to fix deviant art tag search. all clients will automatically update and should with luck just be working again with the same old 'deviant art tag search' downloader
- added prototype copy/paste buttons to the manage ratings dialog. the copy button also grabs 'null' ratings, let me know how you get on here and we'll tweak as needed
- file searches with namespaced and unnamespaced tags should now run just a little faster
- most file searches with multiple search predicates that include normal tags should now run just a little faster
- the file log right-click menu now shows 'delete x yyy files from the queue' for deleted, ignored, failed, and skipped states separately
- the tag siblings + parents display sync manager now forces more wait time before it does work. it now waits for the database and gui to be free of pending or current background work. this _should_ stop slower clients getting into hangs when the afterwork updates pile up and overwhelm the main work
- the option 'warn at this many pages' under _options->gui pages_ now has a max value of 65535, up from 500. if you are a madman or you have very page-spammy subscriptions, feel free to try boosting this super high. be warned this may lead to resource limit crashes
- the 'max pages' value that triggers a full yes/no dialog on page open is now set as the maximum value of 500 and 2 x the 'warn at this many pages' value
- the 'max pages' dialog trigger now only fires if there are no dialogs currently open (this should fix a nested dialog crash when page-publishing subscriptions goes bananas)
- improved error reporting for unsolveable cloudflare captcha errors
- added clarification text to the edit subscription query dialog regarding the tag import options there
- added/updated a bunch of file import options tooltips
- .

Message too long. Click here to view full text.



Anonymous Board owner 11/24/2021 (Wed) 22:54:31 Id: f27ebb [Preview] No.1176 del
- boring presentation import options stuff:
- presentation options are now in their own object and will be easier to update in future
- the 'should I present' code is radically cleaned up and refactored to a single central object
- presentation filtering in general has more sophisticated logic and is faster when used on a list (e.g. when you highlight a decent sized downloader and it has to figure out which thumbs to show). it is now careful about only checking for inbox status on outstanding files
- presentation now always checks file domain, whereas before this was ad-hoc and scattered around (and in some buggy cases lead to long-deleted files appearing in import views)
- added a full suite of unit tests to ensure the presentation import options object is making the right decisions and filtering efficiently at each stage
- .
- boring multiple local file services work:
- I basically moved a bunch of file search code from 1 file services to n file services:
- the file storage module can now filter file ids to a complex location search context
- namespace:anything searches of various sorts now use complex location search contexts
- wildcard tag searches now use complex location search contexts
- counted tag searches now use complex location search contexts
- search code that uses complex location search contexts now cross-references its file results in all cases
- I now have a great plan to add deleted files search and keep it working quick. this will be the next step, and then I can do efficient complex-location regular tag search and hopefully just switch over the autocomplete control to allow deleted files search

next week

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/01/2021 (Wed) 05:59:04 Id: 5a29c1 [Preview] No.1178 del
I had a good week. I managed to add support for embedded ICCs, which will improve some images' colours, and overhauled how files are deleted from the client and serevr so the system is smoother and more reliable.

The release should be as normal tomorrow.