/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Magrathea | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 427 Anonymous Board owner 01/27/2021 (Wed) 23:14:50 Id: 031065 [Preview] No. 953 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=E97ftpTjeC4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Linux.-.Executable.tar.gz

I had an ok week. I mostly fixed and cleaned things.

more ghosts

There was another 'ghost' pending tag issue. Sometimes, on committing pending tags with siblings up to the PTR, the pending record and its count was not clearing properly. I reproduced the problem and fixed it, and a new test ensures it will not happen again. Thank you to the reports about this, they helped me to figure it out.

Since we have had some trouble with pending tags in particular, I have written a new maintenance routine to recalculate pending tags and their counts efficiently without having to wait for siblings/parents re-sync. This will run on update, so with luck, your tags and autocompletes should be fixed up (until the next problem!).

Message too long. Click here to view full text.



Anonymous Board owner 01/27/2021 (Wed) 23:15:23 Id: 031065 [Preview] No.954 del
- autocomplete shortcuts:
- there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over
- the tag autocomplete input text box is now plugged into the new shortcut system and uses this set
- migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults):
- - force search now, for when you have automatic searching turned off (ctrl+space)
- - enable IME-friendly mode (insert)
- - if input empty, move left/right a tab (left/right arrow)
- - if input empty, move left/right a service page (up/down arrow)
- - if input empty and on media viewer, move to previous/next media (page up/down)
- misc improvements to my shortcut handler
- misc shortcut code cleanup
- .
- the rest:
- I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser
- thanks to a user report, wrote a new url class for 420chan's newer thread url format
- sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately
- fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788)

Message too long. Click here to view full text.



Anonymous 02/02/2021 (Tue) 06:29:36 Id: 3cd043 [Preview] No.955 del
>>953
I hate to bitch, but can we change the nitter downloader to a new instance or something? They error out pretty much constantly anymore.


No Release Tomorrow! Anonymous Board owner 02/03/2021 (Wed) 01:06:39 Id: ec785a [Preview] No.956 del
I had an ok week. I was thinking of taking two weeks so I could do the taglist improvements nicely, and since that would give me a little extra time, I decided to pull the trigger on finally upgrading my main dev computer as well. I spent a bit of this week sorting that out, replacing my six year old office computer with a nice mini pc with an SSD. Beyond that, I only have some little bug fixes and cleanup done, so I will spend tomorrow as a normal work day and put off the release.

There will be no release tomorrow. 428 should be on the 10th!

>>955
I will see about updating the defaults, thank you for the report.


Release Tomorrow! Anonymous Board owner 02/10/2021 (Wed) 09:05:42 Id: 6195ce [Preview] No.966 del
I had a good couple of weeks. As well as fixing a bunch of bugs and improving quality of life, I was able to finish an important backend update to the taglists used across the program. Implied parents and siblings are now shown nicely in the manage tags dialog.

There is a whole bunch to test, so the release may be late tomorrow.



(498.15 KB 1847x1080 i cant read.jpg)
Anonymous 01/17/2021 (Sun) 03:53:39 Id: 0bdfaf [Preview] No. 918 [Reply] [Last 50 Posts]
Is it me or is this UI text ant-sized? I've been using hydrus on ubuntu and got used to bigger windows but now im on w10 and this is some small print, so small I dont know how I first got used to it and almost think I've done something wrong. I'm just on 1920x1080. Can I do something about this?


Anonymous 02/08/2021 (Mon) 17:43:37 Id: bbbe9a [Preview] No.964 del
Looks good to me chief, have you been to an optician lately?



Version 426 Anonymous Board owner 01/20/2021 (Wed) 23:28:21 Id: f78aa4 [Preview] No. 944 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=SpXsBNBtqP0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Linux.-.Executable.tar.gz

I had a great week. I mostly fixed and cleaned things.

8kun board deleted

The 8kun /hydrus/ board is now deleted. I was planning last week to move to Endchan as the new primary, but Codexx of 8chan.moe kindly offered to host me on 8chan.moe /t/, in a new Hydrus Network General. The thread is here https://8chan.moe/t/res/2219.html . The old /hydrus/ board is also migrated to 8chan.moe /hydrus/, if you would like to search it or recover something important to you. All the links across the program and help are updated to reflect the changes. I appreciate the help from Codexx in getting this going. I have never been a comfortable or great BO, so this arrangement is a better fit for my situation for the primary place to talk Anon about Hydrus. Endchan will remain as a bunker, and I will keep making release posts here. Please feel free to use whichever you are most comfortable with.

downloaders from the future

Message too long. Click here to view full text.



Anonymous Board owner 01/20/2021 (Wed) 23:32:15 Id: f78aa4 [Preview] No.945 del
the rest

I also updated the gelbooru pools downloader. Some users get different pool 'gallery' markup from the site, we are not sure why, but it should be fixed now.

Last week's network object breakup went well overall. There was one unfortunate but harmless error popup that could appear on client shutdown, typically when certain websites had downloads start in the last five minutes of the program, which I have fixed.

Also fixed is the new fast tag search sometimes dropping subtag results. If you noticed that 'samus aran' was not appearing in results, even though 'character:samus aran' existed and could appear in a 'character:anything' query, this is now fixed. Thank you to the users who reported on this and helped me figure it out. A new maintenance routine will run on update to fill in any gaps that may have appeared for you. It shouldn't take more than a minute, even if you sync with the PTR.

I added a new 'profile' mode to the help->debug menu, 'callto profile mode'. This one tracks mostly UI-level jobs that work in the background. If I have been talking to you recently about heavy downloaders or similar making the UI juddery, please give it a go and send me the profile logs as usual. Also, profiles in general should be less spammy with the popups.

The 'bandwidth used this session' section of the main window's status bar is now more accurate. In certain cases, after a delay, it could report bandwidth used in a session in the hours or days before boot, but now it is exclusive to this session. The difference here is usually not noticeable, but if you restarted a client after pausing all network traffic and then left it open for three hours, it could be confusing!

full list

- misc:
- thanks to help from Codexx at 8chan.moe, the old 8kun board is completely migrated and archived at 8chan.moe /hydrus/. going forward I will be maintaining a Hydrus Network General there on /t/ for merged release posts, Q&A, and Bug Reports. the plan is that whenever it fills up, it will be moved to the /hydrus/ archive. the links across the program and help are updated, please let me know if I missed any. Endchan /hydrus/ remains as a bunker
- fixed a bug where subtag entries in the new tag fast search cache were being deleted for all namespaces when a single namespaced version was went to count 0. it meant some autocomplete results were not appearing, often after some sibling changes. a new 'repopulate' job has been added to the database regenerate menu to fix this efficiently if something like it happens again. this routine will be run on update to fix all users, it shouldn't take long (issue #785)

Message too long. Click here to view full text.



Anonymous Board owner 01/20/2021 (Wed) 23:34:42 Id: f78aa4 [Preview] No.946 del
- when an archive/delete filter finishes, it now fires off all its changes in one go. previously they would go in ~64-file chunks over the next few hundred milliseconds. this will add a small amount of 'refresh lag', delaying page refreshes etc..., on bigger filter jobs for some clients, but it will guarantee that if you hit F5 real quick after finishing filtering on a processing page with non-random sort, you won't see the same files again at the top, only for them to be swiftly archived/deleted as you watch. trash file performance is much better these days, let me know how this goes for you if you do megafilters
- the tag import options whitelist now checks subtags of parsed tags. if you add 'samus aran' to the whitelist, but the site delivers 'character:samus aran', this now passes the whitelist
- thanks to a user's submission, the gelbooru 0.2.5 post parser is updated and should get tags again, for those users who stopped getting them last week--however, I never experienced this myself, so please let me know if you still have trouble. there could be something more complicated going on here
- updated the gelbooru 0.2.x gallery parser to handle an alternate form of gelbooru pools--we did not figure out why different users are being given different markup, it wasn't as simple as being logged in or not, but there is a difference for some. this parser is folded in on update, so the gelb pool downloader should be fixed for users who had trouble with it
- also updated the gelbooru pool gallery url class to infer next page url, as in the alternate form the next page is difficult to parse
- the 'clear all closed pages' command under the 'undo' menu now asks for yes/no confirmation
- added a 'callto' profile mode, which will be very useful in diagnosing GPU lag in future. the 'callto' jobs are little off-main-thread things like image rendering and async panel preparation. should help us figure out where big download pages etc... are eating up CPU
- the different profile modes in the debug menu now all show popup messages, but only when their job exceeds the particular profile's interesting time, usually 3-20ms. this should reduce spam
- the 'this session' bandwidth tracker on the status bar is now a special tracker that only includes data from boot. previously, it was using the 'global' tracker, which after certain time intervals (four minutes, three hours, three days), will compress bandwidth history into larger time windows to save space. if one of these windows covered time before the client started, it could spookily report a little bandwidth used on a client started with network traffic paused
- bandwidth data usage in times shorter than the last ten seconds (which are smoothed out to avoid bumps) now also get the 'don't get bandwidth from the future on motherboards that had a briefly crazy system clock' fix from last week
- fixed some disengaged database tuning that was leading to worse cancel times on certain jobs
- updated a whole bunch of the help so section headings are links with nice #fragment/anchor ids, making it easy to link other users to a particular section. I will continue this work, and future help will follow this new format
- fixed some bad character encodings in the changelog document, siblings help, and tagging schema help. these should now be utf-8 valid


Anonymous Board owner 01/20/2021 (Wed) 23:38:02 Id: f78aa4 [Preview] No.947 del
- object load improvements:
- the client now detects serialisable (saveable) objects that were generated in a future version format your client does not yet support. this mostly affects downloader objects like parsers, where you might import an object a user in a much newer version of the client made. for instance, this week some users imported a fixed gelbooru parser in an older client, which was then saved and double-updated later on, and that caused other problems down the line. downloader imports deal with this situation cleanly, but otherwise it mostly makes a popup notifying you of the problem and asking to contact me. there are about 170 places in the program where objects are deserialised and I am not ready to make this a fullblown error until I know more about people's IRL situations. let's hope this is not widespread. if you run into this, please let me know!
- if you were running an older client and manually imported the updated gelbooru parser that was going around, and then you got errors about 'md5', hex' or 'additional_info' something, it _should_ be automatically fixed on update. you should be able to update from previous to ~422, see it in network->downloader components->manage parsers, and it should just work. many users will have the entry overwritten anyway in the above gelb update I am rolling in. if any of this does still give you trouble, please delete and re-import the affected object(s)
- importing one of these future-versioned serialised objects using the import/export buttons on a multi-column list, either clipboard, json, or png, will cleanly discard future objects with a non-spammy notification
- the Lain drag-and-drop easy downloader import does the same
- the parser 'show what this can parse in nice text' routine now fails gracefully
- multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells
- .
- network server stuff
- fixed being able to delete an account type in the server admin menu
- the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
- did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements

next week

I started on the network updates this week. I will be cleaning more server code and reworking my ancient unit tests, and getting some older admin commands that were 'temporarily' turned off working again. I'll also continue reformatting the help to make sure all the headers have #fragment links, to help external linking to specific sections.


Release Tomorrow! Anonymous Board owner 01/27/2021 (Wed) 09:27:34 Id: 76d418 [Preview] No.952 del
I had an ok week. I mostly fixed and cleaned things. Another instance of 'ghost' pending tags is fixed, with bad counts and tags corrected nice and quickly with a new maintenance routine. Also, the various undocumented shortcuts in the tag autocomplete search box are now exposed and customisable.

The release should be as normal tomorrow.



Version 425 Anonymous Board owner 01/13/2021 (Wed) 22:39:14 Id: eb79da [Preview] No. 897 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=r1nn-tp26KE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Linux.-.Executable.tar.gz

I had a good week. I optimised and fixed several core systems.

faster

I messed up last week with one autocomplete query, and as a result, when searching the PTR in 'all known files', which typically happens in the 'manage tags' dialog, all queries had 2-6 seconds lag! I figured out what went wrong, and now autocomplete should be working fast everywhere. My test situation went from 2.5 seconds to 58ms! Sorry for the trouble here, this was driving me nuts as well.

I also worked on tag processing. Thank you to the users who have sent in profiles and other info since the display cache came in. A great deal of overhead and inefficient is reduced, so tag processing should be faster for almost all situations.

Message too long. Click here to view full text.

5 posts omitted.


Anonymous 01/17/2021 (Sun) 05:27:54 Id: 96dee4 [Preview] No.919 del
>>906
>>906
>I'd like to have an imageboard somewhere. Somewhere with anon image posting, so lads can talk somewhat privately and make bug reports on nsfw pictures and so on.
An idea. Perhaps the board /1ntr/ (internets) of /mlpol/ would be suitable. The fags are friendly and the mods are legit anons and refugees from 4cuck.


Anonymous 01/19/2021 (Tue) 17:05:20 Id: c130d4 [Preview] No.927 del
Sorry for being blunt but how long are you going to leave the manage tags window unusable? Some of us do actually use it, you know.

I just spent 5 minutes trying to find which tag parented in the gender:male tag on an image with no males in it, and I wasn't successful.

Please don't leave important features half-finished.


Anonymous Board owner 01/20/2021 (Wed) 05:04:19 Id: d36528 [Preview] No.935 del
>>906
>>919 Thanks m8, we'll see how this goes!

The plan to move to Endchan only has changed!

Sorry for the sudden change, this all came together this week!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.


Release Tomorrow! Anonymous Board owner 01/20/2021 (Wed) 05:08:35 Id: d36528 [Preview] No.936 del
I had a great week. I fixed up some issues with new systems; one where tags were sometimes being forgotten by the new tag search cache, and another where late-created network sessions were sometimes causing harmless but annoying errors on client shutdown. There are also some fixes for the gelbooru downloaders, improved handling for when you try to import a donwloader your client cannot yet support, and some help and code cleanup.

The release should be as normal tomorrow.

>>927
I am sorry for the trouble. It has been tricky to fit this in, as it involves a complicated new expansion to the taglist control. Since the network object breakup and the new tag cache seem to be doing ok, my next 'medium size' job week is free to do anything. I think that will be for 428. I agree this is currently bad, so how about I aim to make it nicer for then? If I can't get nice ghost tags, I think I can get something good going.


Anonymous 01/22/2021 (Fri) 00:39:33 Id: c130d4 [Preview] No.949 del
>>936
>I think that will be for 428. I agree this is currently bad, so how about I aim to make it nicer for then? If I can't get nice ghost tags, I think I can get something good going.

Thank you. Sorry if I was rude, I was just frustrated. Even if you can't get the ghost tags you have planned functioning a less elegant temporary solution would be good just to make it usable for now. I need to be able to see the parents and siblings added on the manage tags window, or else it's just a huge slow down on my work flow.



No discord hydrus shenanigans for now. Anonymous 10/31/2019 (Thu) 08:01:33 Id: e94922 [Preview] No. 252 [Reply] [Last 50 Posts]
Locked out of discord because of their shitty mobile verification, so doing my part tagging paywalled furry art ยฏ\_(ใƒ„)_/ยฏ

> https://youtube.com/watch?v=SMTz9nIUkGc [Embed]


Anonymous 01/19/2021 (Tue) 02:48:34 Id: cb6acd [Preview] No.924 del
>>252
I really wish Hydrus would disband Discord, or at least provide a privacy friendly alternative for users who don't want to use that fucking trash.

At least give us a Matrix or IRC chat.
They could also add a bridged channel that bridges to a channel on the Discord.

https://usemumble.neocities.org/
The top piece of this provides an excellent overview on Discord being dogshit.



Version 388 Anonymous Board owner 03/11/2020 (Wed) 22:23:32 Id: 415127 [Preview] No. 561 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=X-vdjur459c [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v388.tar.gz

I had a great week. The client can now save and load searches.

favourite searches

Every tag autocomplete input text box that searches for files--the most obvious being the one on normal search pages--now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!

Message too long. Click here to view full text.

13 posts and 1 image omitted.


Anonymous 04/01/2020 (Wed) 04:25:11 Id: d89d54 [Preview] No.605 del
Is using hydrus for managing a collection that includes doujins a terrible idea?

Or would I simply have to script auto-tagging of those doujins with proper author & title + page index? (Of course I have that info in .json files made from extracted metadata of the source sites)


Anonymous 04/01/2020 (Wed) 04:26:46 Id: d89d54 [Preview] No.606 del
>>605
I suppose I'd need to compile a metadata/hash associative DB before importing anything.


Anonymous Board owner 04/05/2020 (Sun) 19:14:29 Id: af0af2 [Preview] No.617 del
>>605
>>606
It is a mix. It works, but I don't like how hydrus manages paged/chaptered media at the moment.

If you have nice cbr/cbz files, I'd say don't import their files to hydrus for now, and just use another comic reader software. I expect to have actual cbr/cbz support in the next two years, with hydrus able to go into the archive and read through the separate file pages in the media viewer like any other reader. I am now generally of the opinion that treating chapters/volumes as single files is better for comics than treating pages as separate files.

That said, if you can get nice page/chapter/series tags to your files using filename-import-parsing, hydrus does work, and it is usually easy to reverse the operation as you can export the files again with nice filenames. It just takes that extra effort, and you have to get the page tags correct or it is a clusterfuck to try to fix atm.

Maybe you can try importing one chapter or volume, without deleting the original file(s), and seeing how you like to read that in hydrus, with the different sort/collect choices. If it works better than a proper comic reader, then yeah I'd say do more, otherwise hold off a bit.


Anonymous Board owner 04/05/2020 (Sun) 19:33:23 Id: af0af2 [Preview] No.618 del
>>595
Hmm, that's odd. I was writing out the rules here, which involve calculating the 'cull' time period, and realised your fixed scheduling may be producing a weird one. Normally, as long as there are more than 250 urls in a query's file import cache, then urls that have a source time more than twice the sub's death period are deleted. The death period is the '90' part in 'sub query is dead if less than 1 url per 90 days', so typically 180 days.

If your death period is set to 360 days or something, then even if there are a thousand import objects in the sub, they may not be being culled down to 250 because they are probably all still less than two years old and hence relatively new according to the sub's timer and could be involved in future timing decisions.

This of course matters less for you with the static checking timing. It doesn't matter if 17 files came in in the past m days or if 3 did, you are still going to check every n days.

I think I'll do two things here--make the cull period calculation a bit cleverer than just the safe 'twice' the death period, and also cap the cull period to 90 days or so when the sub has static check timing as you do.

If your subs have very high death periods, even if you have the '1' part of '1 url per n days' set to '0' to remove the dead check altogether, I recommend you reduce it to 90 or 180 days, especially for subs that reliably give new files.


Anonymous 01/16/2021 (Sat) 21:36:33 Id: 15a72f [Preview] No.907 del
>>618

I never followed up on this change. It is working now. All my old queries were trimmed to (if my memory serves me correctly) 250 successful URLs plus erroneous.

Thanks!



Version 424 Anonymous Board owner 01/07/2021 (Thu) 02:06:19 Id: 75b7b3 [Preview] No. 890 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=YnU_j_ZA-tc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Linux.-.Executable.tar.gz

I had a good week. There are some quality of life improvements and faster tag search across the board.

The update will take some time this week to update a cache. If you do not sync to the PTR, it will be just a few seconds. If you sync to the PTR, expect about 5-15 minutes.

faster tag search

In the second half of 2020, I tried several times to tune the database for different sorts of wildcard tag search, which is used in all autocomplete lookups and many file searches. I was sometimes able to get small clients always running well, or complicated large systems running, well, but I failed to get it good for all situations with code alone--the structure of the database tag lookup cache made the tuning difficult.

Message too long. Click here to view full text.

1 post omitted.


Anonymous Board owner 01/07/2021 (Thu) 02:12:06 Id: 75b7b3 [Preview] No.892 del
- other tag cache info:
- the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
- on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
- an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
- I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
- thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
- while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
- unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
- unified how autocomplete counts are fetched across different service domains
- optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
- optimised display autocomplete count cache updates for tags with multiple siblings or parents
- optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
- merged 'exact match' autocomplete tag searching code into generalised wildcard search
- misc autocomplete and other tag code cleanup and harmonisation
- ditched some old mass UNION queries that were not cancelling well


Anonymous Board owner 01/07/2021 (Thu) 02:13:29 Id: 75b7b3 [Preview] No.893 del
- the rest:
- when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
- the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
- when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
- updated a couple petition reason suggestions in manage tags and parents
- added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
- updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
- sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
- the default e621 parser now pulls meta tags, thank you to a user for providing this
- the default nitter timeline url classes are updated, thank you to a user for providing this
- the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
- the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
- an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
- the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
- for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
- improved the way a job thread was pulling new jobs (issue #750)

Message too long. Click here to view full text.



Anonymous 01/07/2021 (Thu) 02:18:52 Id: adbf98 [Preview] No.894 del
(89.86 KB 1280x720 maxresdefault (4).jpg)
>>890
Upset that I'm angry again Modeus? Every fucking time, all alone, literal God tier effort, but you and the rest cannot accept less than the gold. It won't work if you're abusive. You've lost how many good outcomes looking for a constructed ideal instead of ....just, consider the people wiped fro m your extended family tree because of this. I went a full day without this exhausting rage in my chest. Fucks wrong with you? You're making me too jumpy... just... I dunno...don't fuck my life up worse.


Anonymous Board owner 01/08/2021 (Fri) 20:25:26 Id: 3f26ed [Preview] No.895 del
While the new search cache is working great on normal search pages, on 'all known files'/'PTR' domain, which you usually see in 'manage tags' for the PTR, performance is bad. Often 2-6 seconds to fetch results. I regret this, and I am sorry for the inconvenience. I have identified the slowdown to one link in the chain, and will work on it next week.


Release Tomorrow! Anonymous Board owner 01/13/2021 (Wed) 06:32:03 Id: c867bb [Preview] No.896 del
I had a good week. I was not able to fit much interesting fun stuff in, but I fixed the slow tag autocomplete search in the manage tags dialog, sped up tag processing, fixed some 'ghost' tag bugs, and reduced some wasted CPU in the network engine. Overall, the client should be a bit neater and faster in 425.

The release should be as normal tomorrow.



Version 423 Anonymous Board owner 12/23/2020 (Wed) 23:40:02 Id: 2a5e2c [Preview] No. 882 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=SvsHVu3xt6A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz

๐•ธ๐–Š๐–—๐–—๐–ž ๐•ฎ๐–๐–—๐–Ž๐–˜๐–™๐–’๐–†๐–˜!

I had a good week making some small fixes and improvements to finish up the year. This is the last release of the year. There is a large poll on what 'big thing' to work on next:

poll

Here is the poll on what large work to go for next:

Message too long. Click here to view full text.



Anonymous Board owner 12/23/2020 (Wed) 23:41:11 Id: 2a5e2c [Preview] No.883 del
work is all misc this week

The settings for autocomplete auto-search and character threshold have been moved from options->speed and memory to tags->manage tag display and search. They are also now service-specific! So, you can now set 'all known tags' or 'PTR' to autocomplete after, say, 5 characters but your 'my tags' to always get all results.

I worked on the recently re-activated '*' and 'namespace:*' advanced searches, which were running slow in the new pipeline on larger clients. I have improved some situations, and I think I have reduced the worst case scenario, but some large clients will still have trouble. I am not happy here, nor with other namespace- and subtag- lookup speed across the program, so I have made a plan to make some clever new indices here. As I have simplified and cleaned much of the tag logic over the past year, I now see the next holes that I should fill.

The new siblings and parents tag right-click menu proved a little tall after all, so I have reworked it to group items by all the services that share them. It is denser to look at, but I think it'll highlight unusual exceptions when you are trying to fix application. The 'this is the ideal' line is also removed.

You can now sort a row of pages by name from their right-click menu.

Advanced users: The parser edit UI's test panel now shows more data, and it formats JSON prettily. It isn't anything close to what a browser's developer mode will give you of course, but it is nicer. That panel also detects if you hit a jpeg or something and says so, rather than dumping garbage to the test panel or just throwing an error.

full list

- tag autocomplete searches:
- the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
- getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')

Message too long. Click here to view full text.



Anonymous Board owner 12/23/2020 (Wed) 23:42:17 Id: 2a5e2c [Preview] No.884 del
- the rest:
- the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
- this menu also no longer has the 'this is the ideal tag' line
- added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
- the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
- the parsing test panel now shows json in prettier indented form
- when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
- the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
- the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
- the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
- added --db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
- updated to cloudscraper 1.2.50

next week

I am now on vacation. I have some family Christmas things going on, and otherwise I am looking forward to grinding away at lategame X3 TC with some Wagner. I hope you can have a good break as well. I'll be back to normal in the new year, and 424 should be out on the 6th of January.


Release Tomorrow! Anonymous Board owner 01/06/2021 (Wed) 05:40:29 Id: b24a0c [Preview] No.888 del
I had a good work week. I did a variety of small fixes and quality of life improvements, and I finished redesigning the part of the database that does wildcard tag searches. Autocomplete lookups and file searches that rely on complicated tags are running faster across the board, and the old design's sudden lag spikes (e.g. with namespace:*anything* searches) are entirely eliminated.

The release should be as normal tomorrow. There will be some database work on update. I will have a better idea tomorrow, but I estimate users who sync to the PTR with an SSD can expect it to take 5-15 minutes.


Qwertyius 01/06/2021 (Wed) 05:45:31 Id: b16059 [Preview] No.889 del
(20.26 KB 474x352 (Drink Prunes).jpeg)
>>888
Why do Joss Whedon's cronies all post here? Doyle really was the only good man.



Version 422 Anonymous Board owner 12/16/2020 (Wed) 23:38:46 Id: bcfea2 [Preview] No. 876 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=AHBJ68icJ_4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Linux.-.Executable.tar.gz

๐ŸŽ‰๐ŸŽ‰ It was hydrus's birthday this week! ๐ŸŽ‰๐ŸŽ‰

I had a great week. I mostly fixed bugs and improved quality of life.

tags

It looks like when I optimised tag autocomplete around v419, I accidentally broke the advanced 'character:*'-style lookups (which you can enable under tags->manage tag display and search. I regret this is not the first time these clever queries have been broken by accident. I have fixed them this week and added several sets of unit tests to ensure I do not repeat this mistake.

Message too long. Click here to view full text.



Anonymous Board owner 12/16/2020 (Wed) 23:39:13 Id: bcfea2 [Preview] No.877 del
misc

Some websites have a 'redirect' optimisation where if a gallery page has only one file, it moves you straight to the post page for that file. This has been a problem for hydrus for some time, and particularly affected users who were doing md5: queries on certain sites, but I believe the downloader engine can now handle it correctly, forwarding the redirect URL to the file queue. This is working on some slightly shakey tech that I want to improve more in future, but let me know how you get on with it.

The UPnPc executables (miniupnp, here https://miniupnp.tuxfamily.org/) are no longer bundled in the 'bin' directory. These files were a common cause of anti-virus false positives every few months, and are only used by a few advanced users to set up servers and hit network->data->manage upnp, so I have decided that new users will have to install it themselves going forward. Trying to perform a UPnP operation when the exe cannot be found now gives a popup message talking about the situation and pointing to the new readme in the bin directory.

After working with a user, it seems that some clients may not have certain indices that speed up sibling and parent lookups. I am not totally sure if this was due to hard drive damage or broken update logic, but the database now looks for and heals this problem on every boot.

parsing (advanced)

String converters can now encode or decode by 'unicode escape characters' ('\u0394'-to-'ฮ”') and 'html entities' ('&'-to-'&'). Also, when you tell a json formula to fetch 'json' rather than 'string', it no longer escapes unicode.

The hydrus downloader system no longer needs the borked 'bytes' decode for a 'file hash' content parser! These content parsers now have a 'hex'/'base64' dropdown in their UI, and you just deliver that string. This ugly situation was a legacy artifact of python2, now finally cleared up. Existing string converters now treat 'hex' or 'base64' decode steps as a no-op, and existing 'file hash' content parsers should update correctly to 'hex' or 'base64' based on what their string converters were doing previously. The help is updated to reflect this. hex/base64 encodes are still in as they are used for file lookup script hash initialisation, but they will likely get similar treatment in future.


Anonymous 12/16/2020 (Wed) 23:42:44 Id: bcfea2 [Preview] No.878 del
birthday

๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

On December 14th, 2011, the first non-experimental beta of hydrus was released. This week marks nine years. It has been a lot of work and a lot of fun.

Looking back on 2020, we converted a regularly buggy and crashy new Qt build to something much faster and nicer than we ever had with wx. Along with that came mpv and smooth video and finally audio playing out of the client. The PTR grew to a billion mappings(!), and with that came many rounds of database optimisation, speeding up many complicated tag and file searches. You can now save and load those searches, and most recently, search predicates are now editable in-place. Siblings and parents were updated to completely undoable virtual systems, resulting in much faster boot time and thumbnail load and greatly improved tag relationship logic. Subscriptions were broken into smaller objects, meaning they load and edit much faster, and several CPU-heavy routines no longer interrupt or judder browsing. And the Client API expanded to allow browsing applications and easier login solutions for difficult sites.

There are still a couple thousand things I would like to do, so I hope to keep going into 2021. I deeply appreciate the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

full list

- advanced tags:
- fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
- expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again

Message too long. Click here to view full text.



Anonymous 12/16/2020 (Wed) 23:45:00 Id: bcfea2 [Preview] No.879 del
- misc:
- the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
- miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
- the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
- UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
- the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
- the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
- added the --boot_debug launch switch, which for now prints additional splash screen texts to the log
- the global pixmaps object is no longer initialised in client model boot, but now on first request
- fixed type of --db_synchronous_override launch parameter, which was throwing type errors
- updated the client file readwrite lock logic and brushed up its unit tests
- improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
- the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects
- .
- downloaders and parsing (advanced):
- started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
- added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'ฮ”', and the latter does '&'-to-'&'

Message too long. Click here to view full text.



Anonymous 12/16/2020 (Wed) 23:47:40 Id: bcfea2 [Preview] No.880 del
- in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
- updated the make-a-downloader help and some screenshots regarding the new hash decoding
- when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc...)
- duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
- importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
- added unit tests for hex and base64 string converter encoding

next week

Last week of the year. I could not find time to do the network updates I wanted to this week, so that would be nice. Otherwise I will try and clean and fix little things before my week off over Christmas. The 'big thing to work on next' poll will go up next week with the 423 release posts.

Note:

It looks like 'namespace:*' searches, despite now working, may be running super slow or doing too much database work, in IRL PTR-syncing situations. I will look into it next week! If you are hit by this, please turn them off for now under tags->manage tag display and search.


Release Tomorrow! Anonymous Board owner 12/23/2020 (Wed) 03:28:32 Id: f29211 [Preview] No.881 del
I had a good week making some small fixes and improvements to finish up the year. Autocomplete works a bit faster, and some quality of life is improved.

The release should be as normal tomorrow. It will have the 'next big job' poll and be the last release for the year.



Version 421 Anonymous Board owner 12/09/2020 (Wed) 23:12:23 Id: 76cc77 [Preview] No. 868 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=m9r7VRNUyR8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Linux.-.Executable.tar.gz

I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works betterโ„ข.

highlights

I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.

When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:12:47 Id: 76cc77 [Preview] No.869 del
I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

full list

- misc:
- thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
- fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
- fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
- prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
- when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
- the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
- the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
- fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
- I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
- when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
- fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
- reduced overhead of tag filtering, which should improve display speed of taglist for very large pages

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:14:28 Id: 76cc77 [Preview] No.870 del
- duplicates search improvements:
- potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
- the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do
- all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
- misc cleanup for duplicates page
- .
- database modes:
- a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
- added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
- ensured --db_synchronous_override was hooked up correctly
- the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
- fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
- db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
- .
- technical database nonsense:
- PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
- hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:15:01 Id: 76cc77 [Preview] No.871 del
- when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
- after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
- the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
- improved launch switch code generally
- boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
- the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

next week

There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.

Since a 'next big work' poll is coming up on Christmas, on Saturday I'll post here and in discord about the current big list and ask for more things to add.


Anonymous Board owner 12/12/2020 (Sat) 23:17:19 Id: 14eaa4 [Preview] No.873 del
Hey, here are the preliminary 'big job poll' items. This poll will go up on December 23rd with my release posts, for the new year. You will be able to vote on multiple items. Please feel free to ask questions and request big jobs be added to it.
As many issues are being handled by the github issue team, here is a neat link that lets you review them in more detail:
https://github.com/hydrusnetwork/hydrus/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+milestone%3A%22Major+jobs%22
The poll:
Add CBZ/CBR support (including framework for multi-page format)
Add Ugoira support (including optional mp4/webm conversion)
Add a carousel for the media viewer to see previous/next files in the current navigation thing
Add an incremental number tagging dialog for thumbnails (for adding page:n etc... to a sequence of files)
Add animated thumbnails for videos (animating on mouseover)
Add blurhash for quick thumbnail presentation
Add duplicate video detection
Add file note parsing to the download system
Add import any file support (giving it 'unknown' mime but preserving file extension)
Add more commands to the undo system
Add multiple local file services (which will enable true nsfw/sfw partition)
Add popular/favourite tag cloud controls for better 'browsing' search
Add ratings import/export, and add 'rating import options' to auto-rate imports

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/16/2020 (Wed) 08:26:59 Id: c9ec18 [Preview] No.875 del
I had a great week. I mostly fixed bugs, including the recently broken 'namespace:*' autocomplete lookups, extended database recovery code, improved some quality of life, and added some advanced tools for downloader makers.

The release should be as normal tomorrow.