advanced users

I added a String Selector/Slicer object to the parsing system. You can select the nth item or the mth to nth. m can be 'start', n can be 'end', and either can be a negative index. Try mixing it with the new String Sorter! The String Processing UI has also had an update pass to better support multiple strings--it now shows everything its parent formula had, so you can better test sorting or how processing is changing the third string in the list or similar.

The Server/Client API now support single 'range' requests for files (basically anything not html/json). Apparently Safari was having trouble getting videos from the Client API, let's see if this fixes it. If not, I can try supporting the (more complicated) multiple range requests. The server's error handling is also improved--on certain 'emergency' errors, it was sometimes hanging without closing the connection properly.

full list

- tag sorting:
- the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
- my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
- when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
- I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
- the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
- fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
- if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
- .

- advanced string processing:
- added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
- the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
- the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
- the new String Slicer edit UI similarly gets all the strings at that stage of processing
- future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
- .
- server/client api core improvements:
- I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
- the client now understands 416 ("can't do that requested range m8") errors
- I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
- .
- boring:
- fixed up a handful of typo-borked unit tests
- fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
- started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
- updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance

I had a good week fixing bugs and adding a little quality of life.

The release should be as normal tomorrow. I expect it to be the last release for a few weeks so I can work on a larger network overhaul.

- the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
- the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
- entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
- fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
- fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
- updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists
- .
- new string processing sort step (for advanced users):
- string processing objects have a new processing step: String Sorter
- this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
- it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
- content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
- the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
- I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...
- .
- boring code cleanup:
- refactored the refresh call in filename tagging dialog to nicer Qt signals

I am trying to contribute to making the PTR better by adding siblings/parent and removing garbage, but this doesn't help. In manage tags window, I sibling a tag like this:

tag1 (displays as tag2)

How do I now add parents to tag2? I can't. Also the sibling change does not appear until I commit the pending tags, which means I'll have to close the manage tags window and lose my place in the tag list (I often have thousands of files open at once so the list will be long). Annoying.

I had a great week. I caught up on some small fixes and improvements, finished off a new parsing tool for advanced users, improved behind the scenes server code, and reworked the tag sorting dropdown to be easier to use.

The release should be as normal tomorrow.

>>1018
Thank you for your feedback. I will extend the tag menu when there is a sibling displayed like that to allow for commands on the sibling.

I would also like to work on the sibling/parent and general tag mapping edit logic, pulling it out of their specific dialogs, so I can better support quicker 'just do a quick change' actions for all these things. I don't like having to boot a big dialog just to add one thing either.

full list

- misc:
- fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)
- to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future
- absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe
- updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x
- fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change
- the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information
- fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'
- .
- taglists:
- you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags
- you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags
- the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes
- fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)
- gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus

- boring code cleanup:
- cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird
- the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service
- the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service
- the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)
- refactored client object serialisation access routines to a new db module
- refactored database transaction code and status tracking to a separate object
- refactored some more tag definition routines to the master tag module

next week

I am having trouble fitting in the network improvements. This coming week is supposed to be a 'medium size' job week, where I would usually push hard on something heavy, but I'll save it for the network stuff. I'll look at my big network improvements job, triage what is reasonable to do, and try to just grind some proper work out.

I had a challenging week, but I did get some nice work done. I mostly fixed bugs and updated a couple of downloaders. For advanced users, there is also a new sorting system for parsers.

The release should be as normal tomorrow.

- boring db cleanup:
- wrote a local hashes cache to store hashes for all the files on your disk, much like the tag one. this should speed up all normal searches and other common file lookups in the db
- the raw storage mapping tables are spun off to their own module
- basic file info and inbox is spun off to its own module
- improved and sped up some inboxing file count calculations
- cleaned up some more misc file metadata and inbox code
- improved logic in local tags cache

next week

Some IRL stuff has been eating my time and energy recently. That should be easing up, so I'll be back at network improvements and hopefully some autocomplete improvements for short inputs.

I am crushed for time atm. Sorry for the delays, I will catch up with Endchan thread replies on Saturday.

Help I'm scared

>>982
Often times anti viruses/firewalls don't like hydrus due to its autism code so sometimes you'll have to whitelist it. Some will even ignore your whitelist and still continue to block it. If that happens and you still can't white list hydrus then you'll have to actually summit the program to your anti virus over to their lab and have them white list it. Usually they'll have a section on their site where you can summit false positives.

>>982
>>987
I am very sorry about this. We've had several of these the past weeks. I am not sure what specifically is doing it. My code is unusual, unsigned, and packaged with duct tape and pyinstaller, and I do network and file stuff and a little encryption code, so I guess some libraries and calls look suspicious. So far, every one of these has been a false positive. I appreciate the notice--if ever one of these is real, I want to know asap.

The 'this program is dangerous and executes commands from an attacker' is a new one they have been showing. Maybe they just started recognising the client api, and it looks like a virus command server?

The warnings usually go away after a couple of weeks once new anti-virus definitions roll out, if you would prefer to wait. Or you can just whitelist the exe/directory or whatever. If you know python, the nuclear solution for this is to run from source, but if you aren't familiar with python, it would be a whole bunch of work.

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

I had an ok week. I fixed the tumblr downloader, which accidentally broke recently, and some lagging text display in the download UI, added show/hide options for the new tag parents in the manage tags dialog (along with some other quality of life), and made it so short tag autocomplete inputs can match odd characters, so 'a' will give a result of '/a/'.

The release should be as normal tomorrow.

Thanks for the update. I'm looking forward to expand/collapse options for the taglist because it indeed can get kind of cluttered. A quickly accessible checkbox to make the lists behave as they did in v427 would be appreciated

I had a good week fixing and cleaning little issues. The annoying popup errors some people got with the new taglists are fixed, thread watchers and multi-column lists are nicer to work with, .clip files are supported, and many file database lookups are faster.

I have some IRL going on, shuffling my day about, so I expect the release to be out several hours early.

Sorry, I am crushed for time. I'll get to these messages and the rest of the board on Saturday.

This is a slightly trickier proposition. The new downloaders are for the mirrors, so you would have to be browsing nitter.eu or nitter.nixnet.services, and then HC would recognise that as the nitter mirror and line you up correct.

If you know the downloader system well, you could probably hook up an 'API link' from nitter.com to one of the mirrors, but it would be easier just to browse a mirror instead.

>>973
Yeah, as long as you are using the 'Install' option in the installer. In the second screen I think, it has a dropdown for 'install' or 'extract only'. Most people will click through it, but only the former does the cleanup.

>>972
>>975
Thanks, I agree. I'll see if I can do some this week.

i'm on the latest version

- autocomplete shortcuts:
- there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over
- the tag autocomplete input text box is now plugged into the new shortcut system and uses this set
- migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults):
- - force search now, for when you have automatic searching turned off (ctrl+space)
- - enable IME-friendly mode (insert)
- - if input empty, move left/right a tab (left/right arrow)
- - if input empty, move left/right a service page (up/down arrow)
- - if input empty and on media viewer, move to previous/next media (page up/down)
- misc improvements to my shortcut handler
- misc shortcut code cleanup
- .
- the rest:
- I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser
- thanks to a user report, wrote a new url class for 420chan's newer thread url format
- sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately
- fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788)

>>953
I hate to bitch, but can we change the nitter downloader to a new instance or something? They error out pretty much constantly anymore.

I had an ok week. I was thinking of taking two weeks so I could do the taglist improvements nicely, and since that would give me a little extra time, I decided to pull the trigger on finally upgrading my main dev computer as well. I spent a bit of this week sorting that out, replacing my six year old office computer with a nice mini pc with an SSD. Beyond that, I only have some little bug fixes and cleanup done, so I will spend tomorrow as a normal work day and put off the release.

There will be no release tomorrow. 428 should be on the 10th!

>>955
I will see about updating the defaults, thank you for the report.

I had a good couple of weeks. As well as fixing a bunch of bugs and improving quality of life, I was able to finish an important backend update to the taglists used across the program. Implied parents and siblings are now shown nicely in the manage tags dialog.

There is a whole bunch to test, so the release may be late tomorrow.

Looks good to me chief, have you been to an optician lately?

the rest

I also updated the gelbooru pools downloader. Some users get different pool 'gallery' markup from the site, we are not sure why, but it should be fixed now.

Last week's network object breakup went well overall. There was one unfortunate but harmless error popup that could appear on client shutdown, typically when certain websites had downloads start in the last five minutes of the program, which I have fixed.

Also fixed is the new fast tag search sometimes dropping subtag results. If you noticed that 'samus aran' was not appearing in results, even though 'character:samus aran' existed and could appear in a 'character:anything' query, this is now fixed. Thank you to the users who reported on this and helped me figure it out. A new maintenance routine will run on update to fill in any gaps that may have appeared for you. It shouldn't take more than a minute, even if you sync with the PTR.

I added a new 'profile' mode to the help->debug menu, 'callto profile mode'. This one tracks mostly UI-level jobs that work in the background. If I have been talking to you recently about heavy downloaders or similar making the UI juddery, please give it a go and send me the profile logs as usual. Also, profiles in general should be less spammy with the popups.

The 'bandwidth used this session' section of the main window's status bar is now more accurate. In certain cases, after a delay, it could report bandwidth used in a session in the hours or days before boot, but now it is exclusive to this session. The difference here is usually not noticeable, but if you restarted a client after pausing all network traffic and then left it open for three hours, it could be confusing!

full list

- misc:
- thanks to help from Codexx at 8chan.moe, the old 8kun board is completely migrated and archived at 8chan.moe /hydrus/. going forward I will be maintaining a Hydrus Network General there on /t/ for merged release posts, Q&A, and Bug Reports. the plan is that whenever it fills up, it will be moved to the /hydrus/ archive. the links across the program and help are updated, please let me know if I missed any. Endchan /hydrus/ remains as a bunker
- fixed a bug where subtag entries in the new tag fast search cache were being deleted for all namespaces when a single namespaced version was went to count 0. it meant some autocomplete results were not appearing, often after some sibling changes. a new 'repopulate' job has been added to the database regenerate menu to fix this efficiently if something like it happens again. this routine will be run on update to fix all users, it shouldn't take long (issue #785)

- when an archive/delete filter finishes, it now fires off all its changes in one go. previously they would go in ~64-file chunks over the next few hundred milliseconds. this will add a small amount of 'refresh lag', delaying page refreshes etc..., on bigger filter jobs for some clients, but it will guarantee that if you hit F5 real quick after finishing filtering on a processing page with non-random sort, you won't see the same files again at the top, only for them to be swiftly archived/deleted as you watch. trash file performance is much better these days, let me know how this goes for you if you do megafilters
- the tag import options whitelist now checks subtags of parsed tags. if you add 'samus aran' to the whitelist, but the site delivers 'character:samus aran', this now passes the whitelist
- thanks to a user's submission, the gelbooru 0.2.5 post parser is updated and should get tags again, for those users who stopped getting them last week--however, I never experienced this myself, so please let me know if you still have trouble. there could be something more complicated going on here
- updated the gelbooru 0.2.x gallery parser to handle an alternate form of gelbooru pools--we did not figure out why different users are being given different markup, it wasn't as simple as being logged in or not, but there is a difference for some. this parser is folded in on update, so the gelb pool downloader should be fixed for users who had trouble with it
- also updated the gelbooru pool gallery url class to infer next page url, as in the alternate form the next page is difficult to parse
- the 'clear all closed pages' command under the 'undo' menu now asks for yes/no confirmation
- added a 'callto' profile mode, which will be very useful in diagnosing GPU lag in future. the 'callto' jobs are little off-main-thread things like image rendering and async panel preparation. should help us figure out where big download pages etc... are eating up CPU
- the different profile modes in the debug menu now all show popup messages, but only when their job exceeds the particular profile's interesting time, usually 3-20ms. this should reduce spam
- the 'this session' bandwidth tracker on the status bar is now a special tracker that only includes data from boot. previously, it was using the 'global' tracker, which after certain time intervals (four minutes, three hours, three days), will compress bandwidth history into larger time windows to save space. if one of these windows covered time before the client started, it could spookily report a little bandwidth used on a client started with network traffic paused
- bandwidth data usage in times shorter than the last ten seconds (which are smoothed out to avoid bumps) now also get the 'don't get bandwidth from the future on motherboards that had a briefly crazy system clock' fix from last week
- fixed some disengaged database tuning that was leading to worse cancel times on certain jobs
- updated a whole bunch of the help so section headings are links with nice #fragment/anchor ids, making it easy to link other users to a particular section. I will continue this work, and future help will follow this new format
- fixed some bad character encodings in the changelog document, siblings help, and tagging schema help. these should now be utf-8 valid

- object load improvements:
- the client now detects serialisable (saveable) objects that were generated in a future version format your client does not yet support. this mostly affects downloader objects like parsers, where you might import an object a user in a much newer version of the client made. for instance, this week some users imported a fixed gelbooru parser in an older client, which was then saved and double-updated later on, and that caused other problems down the line. downloader imports deal with this situation cleanly, but otherwise it mostly makes a popup notifying you of the problem and asking to contact me. there are about 170 places in the program where objects are deserialised and I am not ready to make this a fullblown error until I know more about people's IRL situations. let's hope this is not widespread. if you run into this, please let me know!
- if you were running an older client and manually imported the updated gelbooru parser that was going around, and then you got errors about 'md5', hex' or 'additional_info' something, it _should_ be automatically fixed on update. you should be able to update from previous to ~422, see it in network->downloader components->manage parsers, and it should just work. many users will have the entry overwritten anyway in the above gelb update I am rolling in. if any of this does still give you trouble, please delete and re-import the affected object(s)
- importing one of these future-versioned serialised objects using the import/export buttons on a multi-column list, either clipboard, json, or png, will cleanly discard future objects with a non-spammy notification
- the Lain drag-and-drop easy downloader import does the same
- the parser 'show what this can parse in nice text' routine now fails gracefully
- multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells
- .
- network server stuff
- fixed being able to delete an account type in the server admin menu
- the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
- did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements

next week

I started on the network updates this week. I will be cleaning more server code and reworking my ancient unit tests, and getting some older admin commands that were 'temporarily' turned off working again. I'll also continue reformatting the help to make sure all the headers have #fragment links, to help external linking to specific sections.

I had an ok week. I mostly fixed and cleaned things. Another instance of 'ghost' pending tags is fixed, with bad counts and tags corrected nice and quickly with a new maintenance routine. Also, the various undocumented shortcuts in the tag autocomplete search box are now exposed and customisable.

The release should be as normal tomorrow.