/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Magrathea | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 460 Anonymous Board owner 11/03/2021 (Wed) 22:19:15 Id: 4aa4e4 [Preview] No. 1160 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=gP5LXd6DFx4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Linux.-.Executable.tar.gz

I had an ok but light week of various work.

misc

When I improved the 'apng' file parsing the other week, there was a mistake in how I hooked it up and not all the file metadata was being propagated to the database. A couple of users got some weird '1 frame' apngs as a result. I have fixed this problem, and improved the parsing to deal with more sorts of apngs, and scheduled a complete rescan of all your apngs to get some nicer 'number of frames' counts.

In a related thing, a user sent in a way to get some cool metadata from 'clip' files. These now get resolution and a rough thumbnail. I have scheduled all your clip files, if you have any, to update too.

Message too long. Click here to view full text.



Anonymous Board owner 11/03/2021 (Wed) 22:20:47 Id: 4aa4e4 [Preview] No.1161 del
- misc:
- right-clicking a selection tag and choosing 'select->files with x tag' now obeys the current tag domain (previously, it forced 'all known tags'). so, if you want to quickly select just the files with 'samus aran' on 'my tags', it should be doable
- the new 'partial' download resumer system will now tolerate two successive empty chunks before throwing a 'this chunk was empty' error. it seems some servers will randomly give an empty chunk at times during 206 transfers
- cancelling the slideshow custom time dialog no longer raises an error
- after the build boot problem last week, updated the opencv version in requirements.txts--we are now officially >=4.0.0, <=4.5.3.56. it looks like pyinstaller needs a patch for 4.5.4.58 to work, so we'll wait for that. I am improving my weekly test routine to try to catch this in future
- also, the windows build no longer includes two copies of an opencv dll. turns out PyInstaller finds this dll ok now (putting it in another location) and it doesn't need to be explicitly added during build
- added a new help file to the db dir for users who experience crashes as soon as they load videos in mpv when using WASAPI or ASIO drivers. thanks greatly for the user who figured out the mpv.conf solution here (issue #973)

next week

More small work like this, and I would like to do some behind the scenes database code cleanup too.


Release Tomorrow! Anonymous Board owner 11/10/2021 (Wed) 07:01:40 Id: 1a1362 [Preview] No.1162 del
I had a great week. As well as a heap of good background cleanup work in the database, I fixed several bugs, improved .clip and video file metadata parsing, added some new quick tag search actions to the taglist, and cleaned up some bad animation display.

The release should be as normal tomorrow.



Version 459 Anonymous Board owner 10/27/2021 (Wed) 21:40:28 Id: 758e47 [Preview] No. 1156 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=gZgbF2TRK2k [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Linux.-.Executable.tar.gz

Hey, the release I put out earlier had a problem! The links above point to the fixed version. If you got the old one within an hour of me putting this post up, please download and install/extract again and you'll be able to boot. Sorry for the trouble!

I had a great week mostly clearing out bugs.

highlights

I gave the new duplicate filter zoom locking calculation another pass. Thank you to the users who submitted examples of my last attempt not working right. It is now more careful and targeted towards fixing the 'a watermark banner spilled over my monitor edge' problem, and it deals with different combinations of portrait and landscape images and viewers better. Let me know if you still have any borked zooms!

Message too long. Click here to view full text.

Edited last time by hydrus_dev on 10/27/2021 (Wed) 23:57:58.


Anonymous Board owner 10/27/2021 (Wed) 21:41:59 Id: 758e47 [Preview] No.1157 del
full list

- main highlights:
- to help debugging from screenshots etc..., the client now puts its version name on every window title, like 'review services - hydrus client 459'. (issue #447)
- the 'main gui title' option is reset and replaced with 'application display name' this week. it now alters the 'hydrus client' part on every window title. the actual 'main gui title' is now "main" lmao
- wrote a new help document, 'help my media files are broke' in the db directory. this collects the different recovery routines I have developed while helping users after drive failure or other problems cause many missing files or a desynced database and file storage structure. I will be pointing people to this in future, please feel free to do the same
- two new file maintenance jobs are added: for 'presence' and 'integrity' checks, you can now do 'if has URL, then try to redownload, else remove record'. this tidily combines the two more specific jobs that are commonly run after a hard drive problem. the 'presence' version is now the default selected job and recommended for most simple situations
- a new easy-select button on the review file maintenance panel lets you select all media files
- I put some more time into the new duplicate filter zoom locking calculation. thank you to users who sent in examples of my code not working well--I have scaled back what it tried to do. now it will tend to heigh-lock for landscape images and width-lock for everything else _unless_ you are currently viewing the default zoom and that roughly fills the canvas on a dimension and doing the default lock would cause the next image to spill over the screen. the 'solution' here hence targets the 'watermark spilled over' problem more specifically and deals with all combinations of landscape/portrait A/B/canvas better. I'd still like to introduce some zoom locking options here for regular browsing, but pinning down what exactly is useful is trickier than I expected
- the edit tag import options panel now shows 'THIS CURRENTLY GETS NO TAGS' warning red text if it is non default and no tags are set to parse and there are no additional tags
- the status bar now shows '1,234 files in 20 collections' when you have just collections or just collections selected (previously, it wrongly said '1,234 collections') (issue #807)
- macOS clients will now show dialog-created menus in a debug dialog unless the new BUGFIX checkbox under 'gui' options page is unchecked. this _should_ help Big Sur users who are unable to interact with menus created in dialogs like manage tags or manage services. I threw this together, so let me know how this works for you! (issue #986, issue #858)
- the program now waits specifically for currently running subscriptions to stop work and save themselves before moving on with further shutdown tasks. hand in hand with this, subscriptions are now faster at stopping work on client exit, even when they have no popup message (through which some hackery dackery shutdown signals are sent otherwise) (issue #790)
- physically deleting thousands of files in one go should no longer lock up the file manager and other systems for ages--physical delete is now serialised and processed on a new threaded mainloop, so it doesn't matter how fast the requests come it, it will chunter at a polite speed and take breaks and should not choke other consumers and freeze up other 'things are great, you can start new work' status checks


Anonymous Board owner 10/27/2021 (Wed) 21:42:33 Id: 758e47 [Preview] No.1158 del
- network job improvements:
- hydrus network jobs now try to resume incomplete responses (previously they just dumped out and tried again from the beginning). if a server provides less content than it said it would, or it explicitly gives us a partial response, we now resume at that point! should fix dowmnloading of longer videos on 8chan.moe
- hydrus network jobs now send a range header by default, letting servers return 206 (partial content) if they wish
- SSL errors (cert verification and similar) are now caught in the network engine separately to generic connection errors. they will not be reattempted, and the failure note will display specific error info
- refactored some response header parsing code, cleaning up how some variables are initialised
- greatly improved the job reattempt system, resetting variables more neatly
- improved some response range and content length calculations
- .
- smaller items, mostly bug fixes:
- fixed a recent typo bug that caused the edit url class dialog to always spawn with 'file url' type set. sorry, this was stupid! (issue #982)
- the edit url class dialog now sends the 'normalised' url as the example text for the api and referral string converter edit panels
- fixed the new advanced file deletion 'remember last' checkboxes in _options->files and trash_. they weren't hooked up right, sorry!
- fixed the tag menu's siblings submenu's copy command where it says 'ideal is "xxx" on: yyy'. despite the correct label, this was sometimes copying a different service's ideal (issue #855)
- fixed the 'media zooms' text input under _options->media_ not turning off the 'red' invalid mode once its text is again valid
- when you cancel the 'edit parser' dialog, it shouldn't say 'it looks like you made changes' so much when you didn't make any. the 'has changes' test now ignores some background test data updates that may have happened (issue #875)
- if a JSON parsing formula is given HTML, the 'cannot parse' error now tries to detect this and present a better error text (issue #888)
- I _think_ I fixed a problem in the new bytes rendering calculation (where it goes 1018825 to "995KB") where on some unlucky edge-case numbers it could non-determinitively choose different sig figs (e.g. flipping between 994.9KB and 995KB)

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 11/03/2021 (Wed) 02:22:43 Id: 7c4376 [Preview] No.1159 del
I had an ok week. I fixed some bugs, updated some Client API commands, and improved file parsing for apngs and clip files.

The release should be as normal tomorrow.



Version 458 Anonymous Board owner 10/13/2021 (Wed) 21:06:06 Id: 8490e1 [Preview] No. 1151 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=45pF6YtMvQ4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Linux.-.Executable.tar.gz

I had a good week. I fixed some more bugs and improved some quality of life.

highlights

I fixed an html parsing problem that started in the past few weeks in the release build. I understand it broke Newgrounds searching, but it should be fixed now, so if this affected you please give it another go.

You can now set the client to always boot with network traffic paused under network->pause. Useful if you have an unusual network/vpn situation that is often not connected!

Message too long. Click here to view full text.



Anonymous Board owner 10/13/2021 (Wed) 21:06:34 Id: 8490e1 [Preview] No.1152 del
full list

- quality of life:
- under _options->files and trash_, you can now govern whether the advanced file deletion dialog remembers action and reason. being able to save action (trash, physical delete, vs physical delete and clear history) is new, default off for now, and won't _always_ save (if you are currently set to trash, but the next dialog doesn't have trash as an option, then it won't overwrite to physical delete). if you try it out, let me know how you like it
- a new option under 'network->pause' now lets you always boot the client with paused network traffic
- the main file import object now stores primary urls (such as post and file url) separately from source url (which is produced by many parsers and typically refers to another website). a new checkbox in 'file import options' (when in advanced mode) now allows you to not associate primary urls separately to source urls (which is useful in some one-time technical jobs that talk to some unusual proxy etc...)
- the new import object right-click menu that shows urls now separates primary and source urls, and also shows any referral url
- when you flip between two images in the dupe filter, the zoom preservation calculation, which previously only locked to the same width, now tries to choose width or height based on the relative ratios of the two images to keep both images completely in view on a canvas zoom start. it should ensure that lower watermark banners stay in view and don't accidentally spill over the bottom of your monitor
- moved popup toaster options from 'gui' options page to the new 'popup' page
- added options for whether the popup toaster should update while the client is minimised and while the mouse is on a different monitor than the main gui window. these options now default to false, so if you have any trouble, please try turning them back on
- a new shortcut action in the 'global' set now flips profile mode on and off. please note for now 'global' only works on main gui and media viewer--I will add a hook to every window in the future!
- .
- bug fixes:
- you now cannot start an 'upload pending' job for a service more than once at a time. the menu is now disabled for each service while uploading is happening
- fixed a bug in media load where if the file was not in a specific domain (i.e. somewhere in all known files), its tags would not show implied parents. for non-specific files, this calculation happens on the fly, and previously it was only doing siblings
- fixed a bug from the somewhat recent file deletion update that meant many files' custom deletion reasons were being overwritten to 'No reason given' when trash was clearing. I am sorry for the inconvenience!
- fixed an issue with parsing 'string' from html 'script' tags (and perhaps some other 'meta' tag types) on recent versions of the built hydrus release. this should fix the newgrounds gallery parser

Message too long. Click here to view full text.



Anonymous Board owner 10/13/2021 (Wed) 21:07:11 Id: 8490e1 [Preview] No.1153 del
- boring cleanup:
- cleaned some of the positioning code that places icons and text on thumbnails, removing hardcoded numbers and letting custom icons position better
- cleaned some import url tracking, checking, and association code
- refactored profile mode and query planner mode switching up out of UI code into the controller
- added a hefty unit test to test that siblings and parents are transitively applied to mappings correctly for files outside and inside specific file services, and for tag sync and the normal tag pipeline
- refactored some database file maintenance code to decouple the queue from the worker

next week

I will try to focus entirely on the bug reports on the github issue tracker.


No Release Tomorrow! Anonymous Board owner 10/19/2021 (Tue) 21:03:17 Id: 5cbd8e [Preview] No.1154 del
Hey, unfortunately I am super sick. I didn't get much work done, so I will put the release off a week.

469 should be next week, on the 27th!


Release Tomorrow! Anonymous Board owner 10/27/2021 (Wed) 02:49:40 Id: 027732 [Preview] No.1155 del
I had a great week. I fixed many different bugs and improved the network code to better handle and resume interrupted downloads.

The release should be as normal tomorrow.



Version 457 Anonymous Board owner 10/06/2021 (Wed) 21:40:53 Id: ee284d [Preview] No. 1147 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=T4D3Qmn0nuY [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Linux.-.Executable.tar.gz

I had a good week. Some UI jank should be much better!

highlights

I reworked how the menubar menus (file, network, pending etc...) update this week. They update labels and swap items in and out more efficiently. Most importantly, the 'pending' menu should now update with no flicker at all, so if you have a bunch of downloaders/subscriptions running and continually pending new tags, you should be able to commit and upload with no trouble! Let me know how it works for you.

I also removed some ancient and now obsolete safety checks that stop the client from updating certain UI while the program is minimised. With luck, when you restore the client--and more significantly, when you restore it from minimised to system tray--there should be less judder, and hopefully no freezing, since the client no longer has to catch up with hundreds or thousands of updates. If you minimise to system tray and have had hangs, let me know if the situation is better now. Also, if you suddenly get popup message sizing problems or similar UI jank after minimising or putting the client on a virtual desktop or similar, please let me know.

Message too long. Click here to view full text.



Anonymous Board owner 10/06/2021 (Wed) 21:41:11 Id: ee284d [Preview] No.1148 del
full list

- smoother menubar updates:
- improved the way the menubar menus update. rather than generating a whole new (e.g. 'pages') menu and replacing the existing out of date one, now there is a static menu skeleton that has subsections or labels updated in place. this means fewer objects changing, less flicker/jank, and should allow you to upload pending even if you have, say, a bunch of subscriptions running
- .
- misc:
- thanks to a user's help, the filetype parser now detects pngs (this mostly happens during import) much faster! the problem previously was determining if a png is actually an apng--figuring out if they are truly apngs is now done with very fast file header scanning, rather than the previous method that booted ffmpeg. this brings filetype parse time for pngs down from 50-150ms to 1-2ms
- getting apng metadata is also now faster. num_frames is now pulled from the file header, it no longer has to be manually counted by ffmpeg
- clicking the session weight item in the 'pages' menu now gives you more detailed info on your session weight, including on currently closed pages in the undo list
- stripped out a lot of ancient wx-era safety code that stops the client from doing certain UI work while it is minimised or minimised to tray. also brushed up some ugly update routines for menus refresh and modal message presentation that could lead to a pile-up of updates as soon as the client was unminimised, causing lag or freezes. with luck, the client should be better about restoring itself from minimised to system tray. if you minimise to tray, feedback on how this works out for you would be appreciated
- when a network job stalls with the 'this domain had some errors recently' message, the cog menu on the widget now allows you to 'scrub domain errors' and try again immediately
- if your search has system:limit, then any tag search you type in the autocomplete will now search the database, not your thumbnails. previously, the hack to enable this behaviour was to flip 'searching immediately' off. let's see if this new behaviour is ultimately confusing/annoying, I am mixed on it and think this subtle search option needs more thought and UI to make it more obvious and user friendly
- if you have autocomplete tag search typed, and results from thumbnails displayed, and you flip 'searching immediately' off, the search will now automatically update and give you full database numbers immediately
- .
- help:
- I moved 'searching with wildcards' from the advanced help to the 'more getting started with files' help here: https://hydrusnetwork.github.io/hydrus/help/getting_started_more_files.html
- I also wrote a more detailed description of what the autocomplete dropdown buttons do in that page

Message too long. Click here to view full text.



Anonymous Board owner 10/06/2021 (Wed) 21:41:35 Id: ee284d [Preview] No.1149 del
- boring code cleanup:
- cleaned some network job widget update calls
- improved some misc autocomplete search status tracking
- improved some account object permission checking and tests. accounts now never say they have permissions (e.g. if you click the 'see account permissions' button on review services) if they are banned or expired
- file and pages menus now uses the new update routine
- pending menu now uses the new update routine, with an emphasis on anti-jitter so you can interact while it is updating
- database, network, service, and undo menu now use newer async update code and also use the new update routine
- cleaned up help and tags menu init code
- the signal that causes the pending menu to update is now only sent on tag changes if the tag service is a repository (previously, local-only updates were janking this for no reason)
- the pending menu now updates its sibling/parent numbers when repository processing causes a clever row change to stuff you have pending
- also, some menubar items that only show when in advanced mode now update their visibility when advanced mode is flipped on or off
- misc menubar code cleanup and improvements

next week

I've got many little jobs in my immediate todo list, so I'd like to keep working like this. A grab bag of misc items, and some github bug reports.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 10/13/2021 (Wed) 00:36:21 Id: 959e70 [Preview] No.1150 del
I had a good week. I fixed a variety of bugs and added several small options to improve quality of life.

The release should be as normal tomorrow.



Version 456 Anonymous Board owner 09/29/2021 (Wed) 23:06:28 Id: 200b6e [Preview] No. 1144 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=MxaD8_Vazpo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Linux.-.Executable.tar.gz

I had an ok week. With luck, the client should have less UI lag, and I also fixed a bunch of stuff and improved some quality of life.

basic highlights

I removed some very hacky memory management code this week. It was eating far more CPU than it was worth, particularly for large clients. If you have a very heavy client, particularly if it has lots of heavy subscriptions, please let me know if you A) have fewer UI lockups, and B) see any crazy memory spikes while running subs. As bad as the old routine was, it was aggressive and effective at what it did, so I may have to revisit this.

All multi-column lists across the program now catch enter/return keystrokes and trigger an 'activate' call, as if you had double-clicked. Should be easy to navigate and highlight a downloader page list just with the keyboard now.

Message too long. Click here to view full text.



Anonymous Board owner 09/29/2021 (Wed) 23:06:54 Id: 200b6e [Preview] No.1145 del
full list

- misc:
- the client no longer regularly commits a full garbage collection during memory maintenance. this debug-tier operation can take up to 15s on very large clients, resulting in awful lag. various instances of forcing it after big operations complete (e.g. to encourage post-subscription memory cleanup), are now replaced with regular pauses to allow python to clean itself more granularly. this may result in temporary memory bloat for some very subscription-heavy clients, so feedback would be appreciated
- right-clicking on a single url import item in a 'file log' now shows you all the known hashes, parsed urls, and parsed tags for that item. I hope this will help debug some weird problems!
- all multi-column lists across the program now convert an enter/return key press into an 'activate' command, as if you had double-clicked. this should make it easier to, for instance, highlight a downloader or shift/ctrl select a bunch of sibling rows and mass-delete (issue #933)
- the subscription gap filler button now propagates file import options and tag import options from the subscription to the downloader it creates (issue #910)
- a new 'mpv report mode' now prints a huge amount of mpv debug information to the hydrus log when activated
- improved how mpv prints log messages to the hydrus log, including immediate log flushing
- fixed a bug that meant the hydrus server was not saving custom update period or anonymisation period for next boot. thank you for the reports, and sorry for the trouble! (issue #976)
- cleaned up some database savepoint handling after a serious transaction error occurs
- the client api now ignores any parameter with a value of null, as if it were not there, rather than moaning about invalid datatypes (issue #922)
- .
- url classes:
- the edit url class dialog is now broken into two notebook pages--'match rules', which strictly covers how to recognise a url, and 'options', which handles url storage, conversion, and normalisation
- url classes can now support single-value parameters (a parameter with just a value, not a key/value pair). if turned on, then at least one single-value parameter is required to match the url, and multiple are permitted. a checkbox in the dialog turns this on and a string match lets you determine if the url class matches the received single value params
- added unit tests to test the new single-value parameter matching

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 10/06/2021 (Wed) 01:33:25 Id: 2c550e [Preview] No.1146 del
I had a good week. There's more UI anti-jank work, with a smoother menubar and better minimise-to-tray support, faster png file import, and some quality of life.

The release should be as normal tomorrow.



Version 455 Anonymous Board owner 09/22/2021 (Wed) 22:03:37 Id: 9764e1 [Preview] No. 1138 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Gs069dndIYk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Linux.-.Executable.tar.gz

I had an ok couple of weeks. Some quality of life is improved, the UI should save sessions a bit quicker, and the database repairs itself more of itself.

Your client may warn you about a missing index on first boot. If it does, that's great--some new repair code is working!

session save

Some users with large sessions (typically around 2-3+ million 'weight') are experiencing severe UI lockups and general lag. This week I deal with part of that problem by greatly reducing the amount of CPU needed to save most sessions. Now pages know if they have been changed since being loaded, and if they have no changes, they can skip a bunch of session save work.

Message too long. Click here to view full text.



Anonymous Board owner 09/22/2021 (Wed) 22:04:44 Id: 9764e1 [Preview] No.1139 del
database repair

tl;dr: Boot checks are better, you don't have to do anything.

I have spent most of my 'cleanup' time this year breaking the client database into smaller modules. These modules are now responsible for their own repair (which happens on boot), and they do much of it in a neater automatic way, using the same set of definitions it uses to initialise on first start to check whether an existing database is missing anything now. Most of my old hardcoded 'oh, you are missing this table' code is now simplified and modularised, works more reliably, and will be much easier to maintain in future.

You may get a popup when you boot saying you are missing one or more indices. There are several legacy reasons why this might be the case, but most often it will be because you once recovered from former hard drive damage and a clone/recover was unable to rescue an index. Whatever the case, these can now be regenerated automatically (they store duplicated data, so can be regenerated 100%), so please let it run. You might even notice PTR processing working faster afterwards.

Many tables can also be regenerated, and some can be completely repopulated. It is now possible (though not a great idea!) to start a client without a client.caches.db or client.mappings.db file and have it recover to a bootable state. Error presentation around this is improved, with instructions on what to do next in the case of critical problems.

I will continue modularising more of the database and fleshing out this repair code to cover more of the schema!

full list

- misc:
- many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
- the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard

Message too long. Click here to view full text.



Anonymous Board owner 09/22/2021 (Wed) 22:08:42 Id: 9764e1 [Preview] No.1140 del
- ui freezes:
- session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
- for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
- a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
- the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
- misc session save code cleanup
- .
- database repair, mostly boring:
- a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
- all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
- essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
- lots of module repair and initialisation code is refactored and generally given a full pass
- the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
- master definition critical error detection code is no longer hardcoded!
- mapping storage repair code is no longer hardcoded!
- similar files repair code is no longer hardcoded!
- parent or sibling cache repair repopulation is no longer hardcoded!

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 09/29/2021 (Wed) 02:57:53 Id: bc88c5 [Preview] No.1141 del
I had an ok week. I eliminated more UI lag, added downloader support for a special kind of URL, and fixed some more bugs.


Anonymous 09/29/2021 (Wed) 11:36:19 Id: ce2e29 [Preview] No.1142 del
(47.12 KB 1356x621 457474.jpg)
Not sure if this is related to version 455 since I haven't used the site in awhile but I get this error when trying to add links from rule34.xxx about a missing login cookie and I don't see this type of cookie in my browser.


Anonymous Board owner 09/29/2021 (Wed) 23:00:02 Id: 866d21 [Preview] No.1143 del
>>1142
Hey, thank you for this report. It may be that the default login script is now out of date. I will investigate this, and if it is an easy fix, I will roll out an update to that login. You may want to deactivate that login for now and maybe try Hydrus Companion to copy your browser login cookies over directly using the Client API:

https://gitgud.io/prkc/hydrus-companion



Version 454 Anonymous Board owner 09/08/2021 (Wed) 22:27:35 Id: bd99d7 [Preview] No. 1134 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Ch24PpzMKog [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Linux.-.Executable.tar.gz

I had an ok week mostly working on making images look better.

warped tiles

tl;dr: Images look better now.

I put a lot of time into the tiled renderer this week, mostly fixing 'stretch-warped' tiles. These are most obvious in the duplicate viewer when you are flipping between two very similar images that have slightly different resolution--you would see one part of one of the images stretched one pixel horizontally or vertically compared to the other.

Message too long. Click here to view full text.



Anonymous Board owner 09/08/2021 (Wed) 22:27:54 Id: bd99d7 [Preview] No.1135 del
full list

- misc:
- when a downloader page fetch gives a 500 Server Error, it is now handled better, status numbers are updated to 'failed' quickly, and I believe the post-500 downloader deadlock issue (downloaders staying as 'pending', not 'working', after several of these 500s) should also be fixed (issue #898 maybe, but many other reports also)
- when finishing a very large archive/delete filter, the UI should not hang so much to commit the changes. the changes may be delayed a second or more, if your client is currently chugging, so if you are a user who hits F5 real quick after committing archive/delete, let me know how you get on. I've tried to mitigate for your situation, but it may not be perfect.(issue #845)
- the Edit URL Class dialog will now refuse to OK if the API Converter fails.
- the Edit URL Class dialog will now put up an 'are you sure?' messagebox if the URL Class matches its own example API URL
- fixed some gallery error handling when a gallery cannot be parsed
- improved 'cannot parse' error reporting text to include more information
- the client api /manage_pages/add_files command should now always preserve the sort of both the file_ids or hashes parameter
- fixed an instance where image bitmaps were being handled incorrectly (issue #876)
- the client will no longer report shutdown work due for a repository when the work is on a currently paused content type
- fixed some logic related to the advanced tag option of 'fetch page even if url recognised and already in db'
- improved the error message when an unlucky duplicate boot causes a locked database error
- I rearranged and clarified a couple of links in the 'advanced' area of the help index page
- added some help texts and tooltips to the edit checker options dialog and fixed some borked layout
- did a quick hack to fix some db repair code that wasn't dealing with some missing module information. I will brush the repair code up in the near future so that modules can repair themselves

Message too long. Click here to view full text.



Anonymous Board owner 09/08/2021 (Wed) 22:28:50 Id: bd99d7 [Preview] No.1136 del
- improved tile coordinate safety checks for extremely small images (e.g. 1x1) when blown up to ridiculous zooms so there are more than one tile per pixel
- this took a bunch of work, and I am happy that I figured out some solutions, but I believe it may be impossible to get perfect answers here. please try out this new version and let me know how it goes, particularly in the duplicate viewer where warping is obvious. I think ultimately I may replace this with a single tile system that goes over the borders of your screen, eliminating the stitching problems entirely, although this will eat more memory and CPU

next week

I still have plenty of old bug reports to clear from the issue tracker, so I will keep plugging away at them. I want to focus on the UI freezes an increasing number of users are seeing. I also want to do some database cleanup and database repair code improvements


No Release Tomorrow! Anonymous Board owner 09/15/2021 (Wed) 05:03:22 Id: 4d5701 [Preview] No.1137 del
I had a not great week. Some IRL landed on me, so I was not as productive as I wanted. I did manage to upgrade the database repair code in a pretty neat way, but it is not an exciting thing that most users will benefit from immediately, so rather than do a release tomorrow, I will catch up with regular work.

So, 455 will be next week, on the 22nd! I'll try and fix the UI freezes and clear out more issue tracker bugs!



Version 453 Anonymous Board owner 09/01/2021 (Wed) 21:52:24 Id: 4213b1 [Preview] No. 1130 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=YA6A8xohpxc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Linux.-.Executable.tar.gz

I had a great week. I ended up mostly fixing bugs. If you have a large client, the update may take a few minutes this week.

bug fixes

I discovered a semi-important tag processing bug last week--for some users, files that were imported before they added the PTR were not getting all the correct tag info and would not appear in some tag searches. I fixed the underlying file-tracking bug and have added a retroactive 'gap-filler' to this week's update. If you used the client for a long time but added/reset the PTR recently, you'll have more gaps so your update will take longer.

Another persistent issue has been the 'tiling artifacts' with my new tile-based image renderer, where at certain zooms an image would have one or more horizontal or vertical blurred/janked line of pixels. You'd notice it on cleaner vector images where a smooth curved black line would suddenly jag at one zoom. I had trouble reproducing this at first, but some users got it quite often, and with their help I was able to figure it out. I have rewritten the part of my renderer that was failing, and I think I now have the artifacts completely fixed for all 'normal' zooms. Let me know how you get on!

Message too long. Click here to view full text.



Anonymous Board owner 09/01/2021 (Wed) 21:52:49 Id: 4213b1 [Preview] No.1131 del
full list

- qol and misc:
- the network job status labels around waiting for 'subscription'/'download page'/'watcher' forced wait slots are reworded. now they just say a more plain 'waiting to work' with a time estimate, and if a job does not get a chance to work this check cycle, it says 'a different xxx got the chance to work' for a few seconds.
- if a network job does not get bandwidth on a check cycle, it now says 'a different network job got the bandwidth' for a few seconds
- when waiting on bandwidth or gallery work, network jobs should count down more smoothly, one second at a time, not skip a second so often
- network job widgets are now better about updating the layout of their two text labels. the status text on the left should take all the available pixels much better, sharing with the '64KB/s' speed text as it changes width and disappear
- added a new user-made darkmode QSS stylesheet called 'OledBlack' to default hydrus, try it out under _options->style_
- if the tag domain in a search page is other than 'all known tags', the 'selection tags' box, which limits itself to the current domain's tags, now explicitly labels itself with that domain
- consolidated and optimised the pre-work checks on all importers/downloaders in pages. pages with idling/finished/paused downloaders will consume just a little less CPU and need to talk to fewer important objects
- renamed the shortcut sets for viewer/preview media windows and clarified that they are mouse only for now. the new seek command works with these, but you'd have to map ctrl+right-click or something
- improved the system predicate unit tests to catch datatype problems like with last week's hotfix and system:time imported
- advanced archive/delete stuff: wrote up a neat idea I had about using local parents applied to the PTR to make fast multi-tag processing workflows here: https://hydrusnetwork.github.io/hydrus/help/advanced_parents.html#parent_favourites
- .
- bug fixes:
- an important tag search bug is fixed. for some users, files that were imported before a service was added were not appearing in some of that service's search results, or their tag counts were not added in certain tag autocomplete results. this file miscount is fixed, and holes will be filled on database update. it should not take too long to fix, although different users will have different situations
- this bug was leading to artificially fast PTR processing speeds on some clients as their older files were being skipped. if you have used the client for a long time but only added the PTR recently, sorry if you notice it slow down! it is now working correct!

Message too long. Click here to view full text.



Anonymous Board owner 09/01/2021 (Wed) 21:54:05 Id: 4213b1 [Preview] No.1132 del
- client api:
- OR predicates are now supported in the client api! Just nest within the tag list, and it'll bundle the nested list into an OR. there's an example in the client api help
- some permissions testing in file search is tightened up--now we have OR and system predicates, if you do not submit any regular positive tags, the search permissions have to be 'allow anything'
- fixed an issue where the client api would let you ask about sha256 hashes of incorrect length (and would ultimately make a master database id for these borked hashes, even the empty string!!). now the client api throws a 400
- fixed a bug in /manage_pages/get_pages where all pages were marked as 'selected'=true (issue #841)
- in the client api, if you use missing file_id(s) on a request for a file, thumbnail, metadata about a file, or when trying to add a files to a page, it now gives 404 correctly (rather than 500) (issue #961)
- added a section to the client api help on variable encoding, including an example of how to convert a python tag list to JSON+URL encoded string
- added new unit tests for OR pred parsing and the hash length check
- client api version is now 20

next week

Unfortunately, I did not spend time on notes parsing as I had planned. The people helping to run the github issue tracker pointed me to old bug reports that I had not been keeping track of, so I am going to spend a little time prioritising clearing out that queue (and generally trying to integrate important github issues into my weekly routine more, as I often have trouble naturally focusing on priorities). So, next week will likely be more like this!


Release Tomorrow! Anonymous Board owner 09/08/2021 (Wed) 02:56:53 Id: 6821b1 [Preview] No.1133 del
I had an ok week, mostly more bug fixes. Another issue with the image viewer--slightly warped tiles that were obvious in the duplicate viewer--is now much better, and I think I fixed an issue of downloaders sometimes getting stuck on 'pending' forever after serious network problems.

The release should be as normal tomorrow.



Version 452 Anonymous Board owner 08/25/2021 (Wed) 22:49:32 Id: 5a5d15 [Preview] No. 1127 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=iePWOFOSl_U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Linux.-.Executable.tar.gz

I had an ok week. I ended up doing a lot of boring behind the scenes prep, but there is also some nice new quality of life. If you sync with the PTR, update will take a few minutes this week.

all misc this week

After a very long delay, I finally have shortcuts for seeking video back/forwards in the media viewer. New users will get ctrl+left/right to seek back 2.5 seconds or forwards 5 seconds. Existing users need to add their own--please check it under the 'media viewers - all' shortcut set. You can set whatever seek distance you like, and even set multiple with different distance jumps. Sorry for how long it took to get this in--I had to update my shortcut system first!

I did some behind the scenes tag work this week. The database can now handle larger 'tag as number' searches. Update will take a few minutes to load these bigger numbers into the fast search cache.

Message too long. Click here to view full text.



Anonymous Board owner 08/25/2021 (Wed) 22:50:26 Id: 5a5d15 [Preview] No.1128 del
- vastly improved the cancel speed for searches in the realm of 'get the files that have any xxx tag', be that all tags or a namespace wildcard, and also some important search setup for various 'all known files' search pages. if you have ever tried to search the PTR raw and run into a three minute uncancellable initial setup lag, it should be gone now!
- when you right-click on files in a specific local file domain (e.g. trash or my files), the 'view this file's dupes' number check is now run on 'all local files' as well, and if the numbers differ, a second menu is shown for all local files. this should make it easier to chase dupes of trashed files that are still untrashed while also allowing a trash-only search
- fixed a critical bug in repository mapping processing that was not adding mappings to certain caches for files imported before the repo was added, and/or the new repository 'per content type' processing reset. this mostly manifested as these files not showing up in search results despite the tag being there. there is more work to do here, so it is top priority next week, and likely some maintenance to regen the bad caches
- .
- boring rewrites for multiple local file services:
- many users have asked for the ability, when multiple local file domains are available, to search multiple file domains at once. I spent time this week doing background work to support this, and a related concept of searching 'deleted' files, which is a current gap in the program and not always covered by 'all known files'. nothing significant changes, but almost all the file search code now works on n file domains rather than 1, but for now n=1 lmao
- made a new 'database search context' object to handle a virtualised but still simple and fast file 'location view' at the database level
- the primary file search call is converted to use this object. references to a single file service are replaced with the view or its components
- all duplicate file search code is moved to the new location search context
- searching by 'system:import time' now works over multiple domains, although it is a little muddled. in future, import time predicate will have an optional specific file service and do 'import time' vs 'deleted time'
- 'system:local' and 'system:not local' is adapted so it can still work fast with multiple file domains, sped up worst case 'local' time, and a new optimisation lets it run fast for 'deleted from local files' too
- sorting search files by import time is now only supported if the search domain is just one domain

next week

While I was working on database stuff this week, I discovered a problem with repository processing that meant files imported before a repository was added (or mappings reset, with the new 'reset by content type' function) would not get fully processed and may not show up in tag searches. I fixed half of that bug this morning, but there is more to do, so that's top priority.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 09/01/2021 (Wed) 02:36:16 Id: dd23a6 [Preview] No.1129 del
I had a great week, ending up focusing mostly on bug fixes. The 'jank line' tile artifact in the image viewer is fixed for all normal zooms, and I also fixed the long-time tag search issue I discovered last week. There's also some quality of life improvements, and OR searching is added to the Client API.

The release should be as normal tomorrow.



Bug Reports Anonymous Board owner 08/08/2019 (Thu) 00:20:53 Id: 58c04a [Preview] No. 9 [Reply] [Last 50 Posts]
Please feel free to submit any bug reports here.
64 posts and 12 images omitted.


Anonymous 03/25/2021 (Thu) 03:22:26 Id: ad3466 [Preview] No.1038 del
hey im trying to use the url downloader and it wont find parse this url e621.net/posts/2662369

where do I go or what do I do? who knows how many downloads were affected


Anonymous 03/25/2021 (Thu) 04:03:43 Id: ad3466 [Preview] No.1039 del
same with this e621 posts/2659888


Anonymous 04/27/2021 (Tue) 10:30:08 Id: 15cfd9 [Preview] No.1052 del
Not sure if this is a bug or if I'm missing something, but I screwed up and ran out of space on the disk I keep my DB on. It's not the first time this has happened, but this time it seems to have broken Hydrus somewhat.

It takes a long time to start now, and then it works slowly but normally for a while before it decides it's out of space again, at which point I can't make any changes, like deleting files to free up space.

I've used the moments where it does work to delete some of the bigger files to free up 25 GB. This is 5% of the entire disk, and although I think it's made it a little better, meaning it takes a bit longer for it to think it's run out of space, it hasn't actually fixed the problem.

Is there a way to fix this other than replacing the database file entirely and reimporting the nearly 500 GB worth of client files?


Anonymous 05/04/2021 (Tue) 23:04:07 Id: 69d0a8 [Preview] No.1057 del
(9.15 KB 574x116 Capture.PNG)
I'm following what this message is telling me to do. I don't know how or when this popped up but I just noticed it when I looked at Hyrdus today. I'm on v434, Win10 19042.


Anonymous 08/19/2021 (Thu) 03:01:22 [Preview] No.1125 del
I'd like to start this post off by saying that I'm using the AUR version of Hydrus and I know there isn't any real support for Linux builds in general, but I'm still posting this here in case someone else had the same problem, or possibly even knows what fucked up and how to fix it. Alright -

Dark mode seemed to just stop working for me one day. It looks exactly like the default white color scheme except the backdrop for image preview areas is the dark gray color that dark mode typically uses. I checked the "colours" in options and everything seems to be fine too (although I don't think you can change the color of the hydrus window itself through there anyway). This just kind of suddenly started happening after booting up Hydrus one day. It wasn't after updating or anything like that either.