I am really pleased with this week's work, but there are some drawbacks: I did it quick, so I cannot promise it is good. The most obvious bug already is that at around 200-500% zoom you start to see tiling artifacts. I know what causes this (interpolation algorithms not getting full pixel neighbour data from my simple tesselating tiles) and have a plan to fix it (adding a tile border pre-resize, and then cropping). There is also an issue when the 'virtual' image exceeds about 32,000x32,000, so I hacked a zoom block for that. There may be some weird files that render with other stitching artifacts or bad tile data. Note also that hydrus's 'Animation' renderer (the soundless fallback if you do not have mpv support) does NOT use tiling yet, so it still sucks at zooming! Please let me know how you get on!

If you have a steam-powered GPU or a machine with only 4GB of ram, you might like to wait for 439 so I can address any surprise bugs or performance issues.

PTR and account permissions

The PTR is changing how its accounts work. The shared public account is transforming to a 'read-only' account that can only download, so if you want to upload, you'll be going to manage services to auto-create your own privileged account. This is being done to improve janitor workflow for the various petitions, which were all being merged together because of the shared account. With the recent network updates, it will soon be easier for janitors to send simple messages back to these individual accounts, like 'that proposed sibling was not approved because...'.

Unfortunately, various permission and account-management code has not been tested much until now, so as the PTR guys have been trying this stuff out, I have been working to improve bad notifications and workflows. This week I rounded out account permissions testing with uploading. Hydrus no longer tries to upload content the current account does not have permission for, and if you end up in that situation, popup messages now tell you what is going on. It also catches if your account is currently 'unsynced', with instructions to fix.

Similarly, under 'manage siblings/parents', you can now see and edit all tag repositories (previously, they were hidden if you currently had no permission), but you get a label telling you if you don't have permission.

full list

- media viewer:
- I have hacked in tile-based image rendering for the media viewer. this has always been planned as a larger, longer-term job, but the problem of large images is only getting worse, so I decided to just slam out a prototype in a week. if you have a steam-powered GPU or 4GB ram, you might like to wait until next week to update so I can iron out any surprise bugs or performance problems
- images are now cut into tiles that are rendered on demand, so whenever the image is zoomed larger than the media viewer window, only those tiles currently in view have CPU and memory spent on resizing and storage. as you pan around, new tiles are rendered as needed, and old discarded. this makes zooming in super fast and low memory, even for large images!

- futhermore, a new 'image tile' cache is added. you can customise size and timeout under _options->speed and memory_ like for images and thumbnails. this is a dedicated cache for remembering image resize computation across images and zooms. once you have seen both situations once, flicking back and forth between two images or zoom levels is now generally always instant! this new cache starts at a healthy default of 256MB. let's see how that amount works out IRL--I think it will be plenty
- I tuned the image renderer cache--it no longer caches huge images that eat more than 25% its total size--meaning these images only hang around as long as you are looking at them--and the prefetch call that pre-renders several files previous/next to the current image no longer occurs on images that would eat more than 10% the cache size. this should greatly reduce weird flicker and other lag when browsing through a series of mega-images (which before would stomp through the cache in quick succession, barging each other out of the way and wasting a bunch of CPU). in real world terms, this basically means that with an image cache of 200MB, you should have slower individual image performance but much better overall performance looking at images with more than about 5k resolution. the dreaded 14,000x12,000 png will still bonk you on the head to do the first render, but it won't try to uselessly prefetch or flush the whole cache any more
- if you are currently looking at a static image, neighbour prefetch now only starts once the image is rendered, giving the task in front of you a bit more CPU time
- new options for prefetch delay and previous/next distance are added to 'speed and memory'
- note this does not yet apply to the old hydrus animation renderer. that still sucks at high zoom!
- another future step here is to expand prefetch to tiles so the first view of the 'next' media is instant, but let's let all this breathe for a bit. if you get bugs, let me know!
- due to a Qt issue, I am stopping zoom-in events that would make the 'virtual' size of the image greater than 32,000x32,000
- .
- account permission improvements:
- to group sibling and parent petitions by uploader (and thus help janitor workflow), the PTR is moving to a system where the public account is download-only and accounts that can upload content are auto-generated in manage services. this code has not been tested much before, and it revealed some very bad reporting and handling of current permissions. I move this forward this week:
- if your repository account is currently unsynced from a serious previous error, any attempt to upload pending data will result in a little popup and the upload being abandoned
- manage tag siblings and parents will now show service tabs even if the account for those services does not seem currently able to upload tags or siblngs
- if your repository account is currently unsynced from a serious previous error, this is now noted in red text in manage siblings and manage parents
- if your repository account does not have sibling/parent upload permission, this is now noted in red text in manage siblings and manage parents. you will be able to pend and petition siblings and parents ok
- if your repository account does not have mapping/sibling/parent upload permission of the right kind, your client will no longer attempt to upload these content types, and if there is pending count for one of these types, a popup will note this on an upload attempt
- .
- the rest:

- fixed a secondary error when dropping non-list, non-downloader pngs on Lain's easy downloader import window, and fixed a 'no interesting objects' reporting test when dropping multiple pngs
- added a 'cache report mode' to help debug image and thumb caching issues
- refactored the media viewer code to a new 'canvas' submodule
- improved the error reporting when a thumbnail cannot be generated for a file being imported
- fixed an error in zoom center calculation when a change zoom event was sent in the split-second during media viewer initialisation
- I think I fixed an issue where pages could sometimes not automatically move on from 'loading initial files' statusbar text when initialising the session
- the requirements.txt now specifies 'requests' 2.23.0 exactly, as newer versions seemed to be giving odd urllib3 attribute binding errors (seems maybe a session thread safety thing) when recovering from connection failures. this should update the macOS build as well as anyone running from source who wants to re-run the requirements.txt. I hacked in a catch for this error case anyway, just a manual retry like a normal connection error, we'll see how it goes (issue #665)
- patched an unusual file import bug for a flash file with an inverted bounding box that resulted in negative reported resolution. flash now takes absolute values for width and height

next week

Back to multiple local file services. Mostly more backend cleanup and prepping File Import Options and the Client API for talking to multiple locations.

I had an ok week. I improved the new image rendering system, cleaning up instability and errors and mitigating the tiling artifacts. I also fixed some other issues and optimised some database queries.

The release should be as normal tomorrow.

full list

- misc:
- hydrus now keeps a track of when files were deleted! this information has never been recorded clientside, and it is sadly not retroactively recoverable, but it is stored for all deletes from now on. on occasion, when hydrus says 'this was deleted from xxx', it will now have 'at an unknown time' or a nice '3 days ago' string attached. it will take a few seconds to update this week as the new table data is created
- the 'trash' panel on review services now has an 'undelete all' button
- fixed a typo error in manage services when auto-creating a service account when more than one type of account can be created
- the thread watcher page now sorts the status column secondarily by next check time (previously, equal status would sort alphabetically by subject as a fallback secondary sort)
- I have renamed some network concepts across the program. before we had access keys, account keys, and registration keys--now we have access keys (secret password for account), account ids (identifier for account that jannies may need), and registration tokens (one-time token used to create a new account). I hope this reduces some confusion
- reduced some overhead when fetching media results for a search, and when refreshing their tags on major content updates
- fixed a 'no such table: mem.temp_int_hash_id_1'-style database error state that could persist for 30 seconds or more after certain rare rollbacks
- fixed the FlipFlip link html in the client api help
- fingers crossed, I fixed that bad Applications shortcut in the new macOS release
- fixed a couple more instances of 'pulsing' progress gauges. now they should be blank
- .
- more efficient updates in sessions with collected media:
- several updates this week should reduce client UI lag when the session contains any pages with a lot of collected media, particularly when you are also running several downloaders (which spam all sorts of content updates across the client):
- the content update pipeline now tests collections for their files before content processing, and now filters down to process just the updates in a group that apply

- boring multiple local file services stuff:
- I cleaned a bunch of old hardcoded references to 'my files' and related code. it is not very interesting, but there are a few hundred references to clean up and convert to a system that supports 1-to-n local services, and this week I started hacking away, mostly presentation stuff, labels on menus and so on
- your 'my files' now has a separate deletion record to the 'all local files' domain. its count shows in 'review services', and for the moment will just be 'all local files' plus the count in trash, but this will become more important when you can have multiple 'my files'
- behind the scenes re-jiggering means that the deletion record now records deletion time and original import time. delete and undelete transitions are neater as a result
- logically, files are now generally no longer moved to the trash nor undeleted from there, they instead fall there when they are in 'all local files' but no longer in any local domain, and are undeleted back to a specific service. a bunch of awkwardness is cleaned up, and import/delete/undelete content updates are regeared and ready for multiple local file services
- a whole bunch of little things have been fixed and changed behind the scenes. I cleaned file service code in general as I went. examples of little things fixed:
- - a 'delete and do not keep a deletion record' action now correctly does not change the cached number of deleted files as reported in review services
- - the 'clear deletion record and try again' 'remove from trash' component now uses a unified and improved and UI-updating 'untrash' database action, with correct service count changes and UI-side status changes
- - the 'clear deletion record and try again' action on downloader import queues now handles mixes of actually deleted files and files just in trash more neatly
- - in the very odd situation that you are looking at a non-local file on 'all known files' and it is then imported using 'archive on import', its thumbnail and metadata now fade in correctly as archived
- added some unit tests to test the new file delete/undelete transitions
- cleaned up a bunch of hacky old db SELECT code

next week

Next week is a 'medium size' job week. I would like to try putting some time into the ancient image rendering pipeline and related systems like previous/next file prefetch. The basic media viewer has been jank and bad at zooming for too long. I am not sure I can make it beautiful, but I will try to clean some things up.

I had a great week focusing almost entirely on improving media viewer performance. I have overhauled the way images are zoomed and drawn to screen and completely eliminated the additional lag and memory bloat from zooming big images. Furthermore, the caching of rendered image data is greatly improved, so flicking back and forth between neighbouring images or different zooms is, for the most part, now instant.

The release should be as normal tomorrow.

full list

- macOS:
- I fixed an issue with last week's Big Sur compatible release where it wasn't finding your old database correctly--it was defaulting to a different location, so without a specific launch command otherwise, it started a fresh db and said 'hey, looks like first time you ran the program'. if you are a long-time user of hydrus, please install and run 436 as usual, it should figure out your old db location correctly as ~/Library/Hydrus without any launch command override needed
- If you never ran any of the old macOS builds, and you started using hydrus for the first time on macOS last week with the experimental Big Sur compatible build, your brand new database is in a funky location! don't update yet, or you will delete it! You will want to copy your .db files and the client_files folder from inside_the_435_app/Contents/MacOS/db to ~/Library/Hydrus, which should for most people be /Users/(YOU)/Library/Hydrus. feel free to ask for help if you can't figure this out
- fixed a 'this is macOS' platform check for newer macOS releases, which ensures the 'userpath' fallback is correctly initialised to ~/Library/Hydrus
- fixed the new macOS github workflow build script to tell hydrus that it is running from inside an App, so it knows to default to the userpath fallback correctly
- the macOS build now has the old filename
- it also has the ReadMeFirst.rtf file and Applications shortcut
- collected the new build-related files in static/build_files, which will likely see more files in future
- .
- pending tag cache regen:
- two new maintenance tasks are added to the database->regenerate menu--one that forces a recalc of your total 'pending' count as used in the pending menu, and one that recalculates the cached pending tag mappings for storage tags (just like the display one added some time ago, but one layer deeper). the menu entries are relabelled appropriately
- these routines will be run on database update, and should correct the bad pending menu counts many users discovered last week (the new efficient way that the pending count is calculated exposed some legacy bad cached pending storage mappings entries. we'll see if they come back, or if this is just clearing up bad counts hanging around from ages ago)
- the quick pending mapping cache regen routines take a little longer to initialise now, but they now clear out surplus tag data, rather than just regenerating the 'correct' tags
- .
- misc:

- the 'refresh account' button in _review services_ now works on the new async system. it presents errors nicely
- a repository's current update period is now stated in its review services panel
- review services now says 'checking for updates in...' rather than 'next update due...', which is more accurate and will matter more with small update times
- fixed some false positive instances of 'this server was not a tag repo' error in the network engine.
- the hydrus server now also outputs hydrus specific 'Server' header (rather than some twisted default) on 'unsupported request' 404s and any other unusual 'infrastructure' 4XX or 5XX
- if the repository updates in the filesystem are lacking some required file information when calculating what to process, the client now queues those files for a metadata regen maintenance job and raises a cleaner error
- just as a safety measure, if a repository ever happens to deliver a metadata update slice with a 'next update due' time that has already passed, the client now adds a buffer and checks tomorrow instead
- a new program launch argument, db_transaction_commit_time, lets you change how often the database's changes are committed to disk. default is 30 (seconds) for client, 120 for server
- altering the repository update period now prints a summary of the change to the log
- updated the ipfs links in the help
- updated the main help index.html and the github readme.md with the user-run repo and wiki at https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts

next week

I may or may not be tied up with IRL stuff for a bit. Once I am back to things, I will keep working on smaller issues and get started on the pre-work for multiple local file services. There are several hundred locations where the 'my files' service is hardcoded as the local file reference, so a decent part of this work, before I get to file service migration and new location import options, will just be putting some time into ancient code.

I had a good week fixing bugs and cleaning old code. Client performance is improved for large sessions, particularly those that have 'collected' page media, and I got started on multiple local file services by overhauling the core file delete/trash/undelete system. As a neat side-effect, the client now remembers when you delete files.

The release should be normal time tomorrow.

the rest

I had another go at multi column list 'last column' resizing. Dialogs shouldn't judder around so much when you resize them, and more instances of dialogs growing or shrinking slightly on the every re-open should be mitigated.

Tag upload to a repository has a new safety check for a particular error state. I know one user got stuck with some pending mappings that wouldn't clear--in fact it lead to an endless cycle of uploading!--which is now caught, and a new maintenance routine can fix it.

The Client API's file search now converts tags with asterisks into wildcard or 'namespace' search terms. Its version is now 16. Also added to the Client API help page is a link to https://ififfy.github.io/flipflip/#/ , an advanced slideshow engine that now supports hydrus as a source.

full list

- misc:
- a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice)
- the bottom-right corner of the regular media viewer canvas now also shows media zoom
- the StringSorter object now has a simple 'reverse' sort type
- the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch
- the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default
- the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window

- new server stuff:
- a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update
- any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better
- the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year
- let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time!
- the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch
- fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play
- brushed up metadata sync status string on review services
- cleaned misc server and network code
- cleaned up some old clientside service code
- the client api now supports wildcard and namespace tags in the file search call
- client api version is now 16
- added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page

next week

I have unavoidable IRL next week (jury duty), so I can't do the release on Wednesday. I will put a smaller one out on Tuesday!

I had a great few days mostly fixing bugs and cleaning things. The new macOS build is more polished, and I fixed an issue where it could have trouble finding your default database location. I believe I also fixed the issue some users had with 'sticky' pending menu counts, then added some UI quality of life, and wrote an experimental tag display mode that replaces underscores with spaces.

The release should be normal time tomorrow. I have unavoidable IRL on Wed, so it is moved up a day.

admin improvements

After a very long delay, account management is finally back. If you have permission to modify accounts, the 'modify an account' service admin menu item works again and launches a fully revamped window. You can also launch account modification from manage tags for a particular tag selection, the thumbnail right-click for file repository files, or the petition processing page.

The new window shows better data about accounts, and lets you zoom in on individual ones to see current account type, expiration, banned status, or message. It also shows if you are editing yourself!

The account modifications are: change account type, set/extend/clear account expiration, ban, unban, and the new 'set message'. All modifications now print janny and subject account key to the server log. I removed the old 'superban' (which deleted all that user's content contributions), but it will likely return in future.

The service admin menu also has a new 'review all accounts' entry that lets you see a simple summary list of all accounts, which you can then select for modification.

The manage account types dialog now lets you set up auto-account creation. It works a bit like subscriptions, with a 'account creation velocity' limit of 'x accounts per y time delta', like '10 per day'. This defaults to off obviously, but if you set some account types to auto-create, users will be able to see that in manage services.

future admin improvements

As always, there was much more I wanted to do than could get done. I wasn't able to finish improving petition processing workflow or add custom update duration, a serverside tag filter, or a server message, but I was able to get the important network stuff done I needed for these things. Since incrementing the network version is always such an obstacle, I am confident I will now be able to work on these things in normal weekly work. Please let me know how they work out for you as I roll them out.

full list

I had a good week. I fixed several bugs and added some quality of life, got a Big Sur build working for macOS users to try out, and for users who run smaller servers on a LAN or between friends, hydrus repositories now support changing their 'update periods'.

The release should be as normal tomorrow.

full list

- misc:
- thanks to the effort of a user, this week reintroduces a native twitter downloader! it now gets video! (and 'gifs', which on twitter are just mp4s) please experiment with it and move your nitter subs back to this as you find success. it is called 'twitter syndication' and uses a different access method to get tweet info. it should get the highest resolution videos and images. the search has limited lookup distance, perhaps 500 tweets, but should work for most subscription purposes. this is a first version and may have future updates
- on the main gui, middle-clicking and left-double-clicking to open the 'new page' dialog, and right-clicking to open the page menu, should now only work on the page tabs or page greyspace. middle-clicking on some random downloader page greyspace should no longer spawn these commands. also, tiny change--middle-clicking now activates here on click _release_ rather than _press_
- on the file sort widget, the asc/desc sort dropdown is now a 'scrollable' menu button. since sorts are just one of two values, you can now scroll either direction to flip it
- the 'collect/leave unmatched' in the collect control is also now a scrollable menu button
- the new tag sort dropdowns are now all scrollable menu buttons. go ham with them
- middle-clicking the collect-by dropdown now clears it
- namespace file sort now supports a-z and z-a sorting. files with none of the matching namespaces still count as 'less than a' in a-z terms, but since I am updating all this code, perhaps this could get more attention. I don't use this much, so if you do, let me know what you would like
- network job controls now show their jobs' current bandwidth limits on their cog menu, split up by network context. you can edit the bandwidth rules directly from this menu, and if it is using defaults, set a specific ruleset
- userpath generation routines used for database location fallback and default export directory determination now recover from failure in the case of undefined user directory. the client will now not boot if the userpath is needed but undefined
- the client api no longer prints empty lists for any tag statuses on file metadata calls, nor service entries that have no tags at all
- fixed the new subscription 'caught up to small initial sync' calculation, which last week was only firing properly after one page of results
- subscriptions are now better about saving interesting status notes on their gallery logs, rather than overwriting with boring 'no new urls found' messages
- fixed a typo bug in the new range header implementation for unended ranges
- .

God bless you Hydrusdev

I had a good week getting started on the network updates. I cleaned out ancient code and improved some admin-side workflow. There is plenty more to do.

The release is still expected to be another two weeks, for April 7th.

>>1032
Thanks m8, keep pushing.

I had another good week grinding on the network code. I fixed and cleaned old admin functions and started on future extensions for these systems. The main tasks remaining are improvements to the janitor petition processing UI and new features for core repository storage like a tag filter and dynamic update times.

Unless I really fall into a hole, the release is still expected for next week, for April 7th.

I had a great three weeks working on network functions and serverside admin controls I had neglected. The old account management controls are working again with many improvements, account types can now be set up as user-creatable straight from the client (no need for registration keys!), and much of the code is cleaner and handles errors better. In the rewrites, I also laid the groundwork for custom update periods, serverside tag filtering, and faster petition processing, which I hope to get done in the coming weeks.

Regular users who do not admin a server do not have so many changes, although I have fixed bugs and added more controls to repositories, so you can now pause account sync, update downloading, and update processing separately, and I fixed a logical problem with processing that was sticking some users with deleted siblings.

The 'network version' goes up tomorrow, and the PTR should be updating, so if you sync with the PTR, you will want to update some time to keep talking to it! There's no rush, but you'll get a little error popup if you try to talk to an updated server on an older client.

The release should be as normal tomorrow.

advanced users

I added a String Selector/Slicer object to the parsing system. You can select the nth item or the mth to nth. m can be 'start', n can be 'end', and either can be a negative index. Try mixing it with the new String Sorter! The String Processing UI has also had an update pass to better support multiple strings--it now shows everything its parent formula had, so you can better test sorting or how processing is changing the third string in the list or similar.

The Server/Client API now support single 'range' requests for files (basically anything not html/json). Apparently Safari was having trouble getting videos from the Client API, let's see if this fixes it. If not, I can try supporting the (more complicated) multiple range requests. The server's error handling is also improved--on certain 'emergency' errors, it was sometimes hanging without closing the connection properly.

full list

- tag sorting:
- the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
- my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
- when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
- I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
- the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
- fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
- if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
- .

- advanced string processing:
- added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
- the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
- the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
- the new String Slicer edit UI similarly gets all the strings at that stage of processing
- future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
- .
- server/client api core improvements:
- I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
- the client now understands 416 ("can't do that requested range m8") errors
- I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
- .
- boring:
- fixed up a handful of typo-borked unit tests
- fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
- started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
- updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance

I had a good week fixing bugs and adding a little quality of life.

The release should be as normal tomorrow. I expect it to be the last release for a few weeks so I can work on a larger network overhaul.

- the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
- the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
- entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
- fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
- fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
- updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists
- .
- new string processing sort step (for advanced users):
- string processing objects have a new processing step: String Sorter
- this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
- it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
- content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
- the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
- I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...
- .
- boring code cleanup:
- refactored the refresh call in filename tagging dialog to nicer Qt signals

I am trying to contribute to making the PTR better by adding siblings/parent and removing garbage, but this doesn't help. In manage tags window, I sibling a tag like this:

tag1 (displays as tag2)

How do I now add parents to tag2? I can't. Also the sibling change does not appear until I commit the pending tags, which means I'll have to close the manage tags window and lose my place in the tag list (I often have thousands of files open at once so the list will be long). Annoying.

I had a great week. I caught up on some small fixes and improvements, finished off a new parsing tool for advanced users, improved behind the scenes server code, and reworked the tag sorting dropdown to be easier to use.

The release should be as normal tomorrow.

>>1018
Thank you for your feedback. I will extend the tag menu when there is a sibling displayed like that to allow for commands on the sibling.

I would also like to work on the sibling/parent and general tag mapping edit logic, pulling it out of their specific dialogs, so I can better support quicker 'just do a quick change' actions for all these things. I don't like having to boot a big dialog just to add one thing either.

full list

- misc:
- fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)
- to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future
- absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe
- updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x
- fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change
- the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information
- fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'
- .
- taglists:
- you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags
- you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags
- the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes
- fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)
- gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus

- boring code cleanup:
- cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird
- the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service
- the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service
- the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)
- refactored client object serialisation access routines to a new db module
- refactored database transaction code and status tracking to a separate object
- refactored some more tag definition routines to the master tag module

next week

I am having trouble fitting in the network improvements. This coming week is supposed to be a 'medium size' job week, where I would usually push hard on something heavy, but I'll save it for the network stuff. I'll look at my big network improvements job, triage what is reasonable to do, and try to just grind some proper work out.

I had a challenging week, but I did get some nice work done. I mostly fixed bugs and updated a couple of downloaders. For advanced users, there is also a new sorting system for parsers.

The release should be as normal tomorrow.