>>1286
To add more information, I run Hydrus on an external HDD. I used crystaldisk to check the state of the drive and everything seems ok, no bad signs. It just seems like a file went corrupt and went missing from my database in the folder. The file was imported 19 days ago from now but not archived, I'm not sure if being archived does anything but I'll be on the lookout for any more issues.

I had a good simple week. I cleaned some code, improved some quality of life, and made multiple local file services ready for all users.

The release should be as normal tomorrow.

>>1286
>>1287
Hmm, this is a shame. I am not sure what happened here. The most likely culprit is a hard drive fault, although those tend to make a file 0KB more often than simply disappearing a file. Sometimes you can have anti-virus go bananas and remove an mp3 or something, but this seems to be just a jpeg.

If you are worried this may have happened to more files, the place to check is database->file maintenance->manage scheduled jobs. Under 'add new work', set up a job for 'all media files' for an appropriate job type. The default, 'if file missing, try to redownload, else remove record' is good.

If it is any help, I don't have that file, but I think I found it through its PTR tags here:

https://danbooru.donmai.us/posts/5084454

You can probably see that on the borked thumbnail's right-click menu too. Just enter that URL into a new URL downloader page and fingers crossed hydrus should be able to automatically fix the file for you. The above maintenance job will do this automatically.

Let me know if you have any more trouble, and make sure you have a good backup!

>>1288
Thanks, I encountered a few more. Same day from that same session. I'm not sure what happen that day, I don't think I badly closed out of the program or improperly dismounted the drive. It might be that I haven't been doing regular maintenance other than shutting down and I definitely haven't been defraging the drive like I probably should so its probably that and might be bad sectors or something.

Also the other few I mentioned I encountered gave me a slightly different message. I can view the thumbnail but whenever I tried to open it it gave me this message. It would then redownloaded the file but still wouldn't load it. I forgot to check the actual folder where the file is located and just set my HDD aside until I got a proper solution.

>>1289

Ah, for files that seem to be malformed, there are two answers:

1) The file might have been corrupted. In the 'file maintenance' system, there's a set of jobs called 'missing/incorrect' that scan for this. These jobs are expensive though (they have to load the whole file from disk), so don't add them for all files. Try to target with a tag search or system:hash search just to hit the file(s) you think were affected.

2) The file might have once been supported but now is not. This is rare, but a handful of truncated jpegs and other borked files used to load in hydrus until I turned some safety rules on (the lack of safety was causing crashes in some systems). Best answer here is to just delete the file, or export it, run it through a load/save cycle in Gimp or Photoshop, and then reimport the fixed file. Some boorus just have jank files unfortunately.

If you like, you can post that image here (if the site can accept it), or zip it up and email it to me, and I can have a look. But if it is private, no worries.

These bad files being from the same session makes sense that this was a one-time event. I would still suspect it is a hard drive issue, but it could be something else, especially since crystaldiskinfo was good for you. There's a very small chance that if you have a program crash or other hard turn-off that the client can become unsynced, but that typically ends up being the other way around: the file gets added to your hydrus file storage, but the client isn't aware of it. The client can lose up to 30 seconds of database work if it ends non-cleanly.

Let me know how you get on!

>>1290
I ran some checks and found no more issues. I did the "if the file is missing/incorrect" option. It seems like it was just 2 sessions from that day acting up. 1 tag session entry gave me 2 missing file errors and the other entry session gave me 1 corrupt file error. Maybe it was my anti-virus acting up and doing scans over the files I was just viewing. Those were just normal jpgs and png images, nothing really out of the ordinary.


I'll probably be more cautious and do backups just in case. I already have a NAS but never used it and don't know how I should set it up with hydrus but I'll probably make a new thread here(or somewhere more active) to discuss storage backups and safety. Thanks again.

full list

- multiple local file services:
- multiple local file services are now available for everyone! you no longer need to be in advanced mode to create them. all are welcome, but in terms of skill level, I most recommend it for users who are comfortable with tag siblings and parents
- the tl;dr: you can now have more than one 'my files', which lets you put things in isolated locations
- I wrote a proper help document on multiple local file services--what they are, how they work, my recommendations, and a bit of extra info about hydrus file search in general, right here: https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html
- file searches in 'multiple locations' on large clients are now massively faster in almost all situations. the only place multiple location searches are still slow is whenever the duplicates system (system:file relationships) comes into play
- .
- misc:
- in the page tab menu, you can now sort pages by total file size
- the 'force system:limit for all searches' option is moved from the 'speed and memory' to 'search' panel
- when files download from sites, if the raw file is served by cloudflare and has a timestamp radically different to a parsed source time, that CF timestamp is saved under a different domain rather than overwriting the original domain timestamp. this seemed to affect danbooru on about 1 in 10-20 files. note this does not change much at the moment, but when you can see and sort on individual domain modified dates, this should improve the sort
- updated the 'installing' help to talk about bad install locations for the database. network locations are bad, and thanks to user reports, we now know USB drives can be bad if the database is busy when the OS goes to sleep
- if a 'database is malformed' error occurs on boot, the client now recognises it and points the user to 'install_dir/db/help my db is broke.txt' for the next steps
- .
- boring code cleanup:
- another 60KB or so of code pulled out of ClientDB.py:

the rest

This is a two week release because I was ill earlier on and it cut into my work time. So, there is a mix of different small work. Updated downloaders, reworked sibling&parent help with some neat new charts, fixes and improvements to the duplicate filter, some quality of life in UI labelling and texts. Nothing super important, but some things should be a bit smoother!

full list

- multiple local file services:
- the multiple local file services feature is ready for advanced users to test out! it lets you have more than one 'my files' service to store things, which will give us some neat privacy and management tools in future. there is no nice help for this feature yet, and the UI is still a little non-user-friendly, so please do not try it out unless you have been following it. and, while this has worked great in all testing, I generally do not recommend it for heavy use on a real client either, just in case something does go wrong. with those caveats, hit up _manage services_ in advanced mode, and you can now add new 'local file domain' services. it is possible to search, import to, and migrate files between these and everything basically works. I need to do more UI work to make it clear what is going on (for instance, I think we'll figure out custom icons or similar to show where files are), and some more search tech, and write up proper help, and figure out easy client merging so users can combine legacy clients, but please feel free to experiment wildly on a fresh client or carefully on your existing one
- if you have more than one local file service, a new 'files' or 'local services' menu on thumbnail right-click handles duplicating and moving across local services. these actions will preserve original import times (e.g. if you move from A to B and then back to A), so they should be generally non-destructive, but we may want to add some advanced tools in future. let me know how this part goes--I think we'll probably want a different status than 'deleted from A' when you just move A->B, so as not to interfere with some advanced queries, but only IRL testing will show it
- if you have a 'file import options' that imports files to multiple local services but the file import is 'already in db', the file import job will now examine if and where the file is still needed and send content update calls to fill in the gaps
- the advanced delete files dialog now gives a new 'delete from all and send to trash' option if the file is in multiple local file domains
- the advanced delete files dialog now fully supports file repositories
- cleaned up some logic on the 'remember action' option of the advanced file deletion dialog. it also supports remembering specific file domains, not just the clever commands like 'delete and leave no record'. also this dialog no longer places the 'suggested' file service at the top of the radio button list--instead it selects that 'suggested' if there is no 'remember action' initial selection applicable. the suggested file service is now also set by the underlying thumbnail grid or media canvas if it has a simple one-service location context
- the normal 'non-advanced' delete files dialog now supports files that are in multiple local file services. it will show a part of the advanced dialog to let you choose where to delete from

- misc:
- thanks to user submissions, there is a bit more help docs work--for file search, and for some neat new 'mermaid' svg diagrams in siblings/parents, which are automatically generated from a markup and easy to edit
- with the new easy-to-edit mermaid diagrams, I updated the unhelpful and honestly cringe examples in the siblings and parents help to reflect real world PTR data and brushed up all the text in the top sections
- just a small thing--the 'pages' menu and the page picker dialog now both say 'file search' to refer to a page that searches files. previously, 'search' or 'files' was used in different places
- completely rewrote the queue code behind the duplicate filter. an ancient bad idea is now replaced with something that will be easier to work with in future
- you can now go 'back' in the duplicate filter even when you have only done skips so far
- the 'index string' of duplicate filters, where it says 53/100, now also says the number of decisions made
- fixed some small edge case bugs in duplicate filter forward/backward move logic, and fixed the recent problem with going back after certain decisions
- updated the default nijie.info parser to grab video (issue #1113)
- added in a user fix to the deviant art parser
- added user-made Mega URL Classes. hydrus won't support Mega for a long while, but it can recognise and categorise these URLs now, presenting them in the media viewer if you want to open them externally
- fixed Exif image rotation for images that also have ICC Profiles. thanks to the user who provided great test images here (issue #1124)
- hitting F5 or otherwise saying 'refresh' explicitly will now turn a search page that is currently in 'searching paused' to 'searching immediately'. previously it silently did nothing
- the 'current file info' in the media window's top hover and the status bar of the main window now ignores Deletion reason, and also file modified date if it is not substantially different from another timestamp already stated. this data can still be seen on the file's right-click menu's expanded info lines off the top entry. also, as a small cleanup, it now says 'modified' and 'archived' instead of 'file modified/archived', just to save some more space
- like the above 'show if interesting' check for modified date, that list of file info texts now includes the actual import time if it is different than other timestamps. (for instance, if you migrate it from one service to another some time after import)
- fixed a sort error notification in the edit parser dialog when you have two duplicate subsidiary parsers that both have vetoes
- fixed the new media viewer note display for PyQt5

- boring code cleanup:
- cleaned up search signalling UI code, a couple of minor bugs with 'searching immediately' sometimes not saving right should be fixed
- the 'repository updates' domain now has a different service type. it is now a 'local update file domain' rather than a 'local file domain', which is just an enum change but marks it as different to the regular media domains. some code is cleaned up as a result
- renamed the terms in some old media filtering code to make it more compatible with multiple local file services
- brushed up some delete code to handle multiple local file services better
- cleaned up more behind the scenes of the delete files dialog
- refactored ClientGUIApplicationCommand to the widgets module
- wrote a new ApplicationCommandProcessor Mixin class for all UI elements that process commands. it is now used across the program and will grow in responsibility in future to unify some things here
- the media viewer hover windows now send their application commands through Qt signals rather than the old pubsub system
- in a bunch of places across the program, renamed 'remote' to 'not local' in file status contexts--this tends to make more sense to people at out the gate
- misc little syntax cleanup

next week

Some small misc jobs and user-friendly-isation of multiple local file services.

I had an ok week. I fixed some bugs (including some bad trash delete, and an issue with the new duplicate filter queue skipping badly), improved some quality of life, and integrated the new multi-service 'add/move file' commands into the shortcuts system and the media viewer.

The release should be as normal tomorrow.

- code cleanup:
- about 80KB of code moved out of the main ClientDB.py file:
- refactored all combined files display mappings cache code from the code database to a new database module
- refactored all combined files storage mappings cache code from the code database to a new database module
- refactored all specific storage mappings cache code from the code database to a new database module
- more misc refactoring of tag count estimate, tag search, and other code down to modules
- hooked up specific display mappings cache to the repair system correctly--it had been left unregistered by accident
- some misc duplicate action options code cleanup
- migrated some ancient pause states--repository, subscriptions, import&export folders--to the newer options structure
- migrated the image and thumbnail cache sizes to the newer options structure
- removed some ancient db and dialog code from the retired dumper system

next week

I want to catch up on some github issues and do a little more multiple local file services work.

I was ill for the start of the week and am short on work time. Rather than put out a slim release, I will spend tomorrow doing some more normal work and put the release off a week.

483 should be on the 4th of May. Thanks everyone!

I had a good couple of weeks. There are a variety of small fixes and quality of life improvements and the first version of 'multiple local file services' is ready for advanced users to test.

The release should be as normal tomorrow.

- cleaned up some old focus test logic that weas used when hovers were separate windows
- rewrote how each note panel in the new hover is stored. a bunch of sizing and event handling code is less hacked
- significantly improved the accuracy of the 'how high should the note window be?' calculation, so notes shouldn't spill over so much or have a bunch of greyspace below
- right- or middle-clicking a note now hides its text. repeat on its name to restore. this should persist through an edit, although it won't be reflected in the background atm. let's see how it works as a simple way to quickly browse a whole stack of big notes
- a new 'notes' option panel lets you choose if you want the text caret to start at the beginning or end of the document when editing
- you can now double-click a note tab in 'edit notes' to rename the note. some styles may let you double-click in note greyspace to create a new note, but not all will handle this (yet)
- as an experiment, all the buttons on the media viewer hover windows now do not take focus when you click them. this should let you, for instance, click a duplicate filter processing button and then use the arrow keys and space to continue to navigate. previously, clicking a button would focus it, and navigation keys would be intercepted to navigate the 'form' of the buttons on the hover window. you can still focus buttons with tab. if this affects you, let me know how this goes!
- .
- misc:
- added checkboxes to _options->gui pages_ to control whether ctrl- and shift- selects will highlight media in the preview viewer. you can choose to only do it for files with no duration if you prefer
- the 'advanced mode' tag autocomplete dropdown now has 'OR' and 'OR*' buttons. the former opens a new empty OR search predicate in the edit dialog, the latter opens the advanced text parser as before
- the edit OR predicate panel now starts wider and with the text box having focus
- hydrus is now more careful about deciding whether to make a png or a jpeg thumbnail. now, only thumbnails that have an alpha channel with interesting data in it are saved to png. everything else is jpeg
- when uploading to a repository, the client will now slow down or speed up depending on how fast things are going. previously it would work on 100 mappings at a time with a forced 0.1s wait, now it can vary between 1-1,000 weight
- just to be clean, the current files line on the file history chart now initialises at 0 on your first file import time
- fixed a bug in 'if file is missing, remove record' file maintenance job. if none of the files yet scanned had any urls, it could error out since the 'missing and invalid files' directory was yet to be created
- linux users who seem to have mpv support yet are set to use the native viewer will get a one-time popup note on update this week just to let them know that mpv is stable on linux now and how to give it a go

- client api:
- added 'create_new_file_ids' parameter to the 'file_metadata' call. this governs whether the client should make a new database entry and file_id when you ask about hashes it has never seen before. it defaults to false, which is a change on previous behaviour
- added help talking about this
- added a unit test to test this
- added archive timestamp and hash hex sort enum definitions to the 'search_files' client api help
- client api version is now 31

next week

Next week is cleanup. Nothing too exciting, but I'd like to break the database code up a bit more.

I had a great week of simple work. I fixed some bugs--including the scrolled taglist selection issue--and improved some quality of life.

The release should be as normal tomorrow.

full list

- file notes and media viewer hover windows:
- file notes are now shown on the media viewer! this is a first version, pretty ugly, and may have font layout bugs for some systems, but it works. they hang just below the top-right hover, both in the canvas background and with their own hover if you mouseover. clicking on any note will open 'edit notes' on that note
- the duplicate filter's always-on hover _should_ slide out of the way when there are many notes
- furthermore, I rewrote the backend of hover windows. they are now embedded into the media viewer rather than being separate frameless toolbar windows. this should relieve several problems different users had--for instance, if you click a hover, you now no longer lose focus on the main media viewer window. I hacked some of this to get it to work, but along the way I undid three other hacks, so overall it should be better. please let me know how this works for you!
- fixed a long time hover window positioning bug where the top-right window would sometimes pop in for a frame the first time you moved the mouse to the top middle before repositioning and hiding itself again
- removed the 'notes' icon from the top right hover window
- refactored a bunch of canvas background code
- .
- client api:
- search_files/get_thumbnail now returns image/jpeg or image/png Content-Type. it _should_ be super fast, but let me know if it lags after 3k thumbs or something
- you can now ask for CBOR or JSON specifically by using the 'Accept' request header, regardless of your own request Content-Type (issue #1110)
- if you send or ask for CBOR but it is not available for that client, you now get a new 'Not Acceptable' 406 response (previously it would 500 or 200 but in JSON)
- updated the help regarding the above and wrote some unit tests to check CBOR/JSON requests and responses
- client api version is now 30
- .

I had a great week. I fixed some problems with last week's hover window rewrite, improved the notes display and editing, and did a mix of other small work.

The release should be as normal tomorrow.

- when you hit 'copy->hash' on a file right-click, it now shows the sha256 hash for quick review
- in the duplicate filter, the zoom locking tech now works better™ when one of the pair is portrait and the other landscape. it now tries to select either width or height to lock both when going AB and BA. it also chooses the 'better' of width or height by choosing the zoom that'll change the size less radically. previously, it could do width on AB and height on BA, which lead to a variety of odd situations. there are probably still some issues here, most likely when one of the files almost exactly fills the whole canvas, so let me know how you get on
- webps with transparency should now load correct! previously they were going crazy in the transparent area. all webps are scheduled a thumbnail regen this week
- when import folders run, the count on their progress bar now ignores previous failed and ignored entries. it should always start 0, like 0/100, rather than 20/120 etc...
- when import folders run, any imports where the status type is set to 'leave the file alone' is now still scanned at the end of a job. if the path does not exist any more, it is removed from the import list
- fixed a typo bug in the recent delete code cleanup that meant 'delete files after export' after a manual export was only working on the last file in the selection. sorry for the trouble!
- the delete files dialog now starts with keyboard focus on the action radiobox (it was defaulting to ok button since I added the recent panel disable tech)
- if a network job has a connection error or serverside bandwidth block and then waits before retrying, it now checks if all network jobs have just been paused and will not reattempt the connection if so (issue #1095)
- fixed a bug in thumbnail fallback rendering
- fixed another problem with cloudscraper's new method names. it should work for users still on an old version
- wrote a little 'extract version' sql and bat file for the db folder that simply pull the version from the client.db file in the same directory. I removed the extract options/subscriptions sql scripts since they are super old and out of date, but this general system may return in future
- .
- file history chart:
- added 'archive' line to the file history chart. this isn't exactly (current_count - inbox_count), but it pretty much is
- added a 'show deleted' checkbox to the file history chart. it will recalculate the y axis range on click, so if you have loads of deleted files, you can now hide them to see current better
- improved the way data is aggregated in the file history chart. diagonal lines should be reduced during any periods of client import-inactivity, and spikes should show better
- also bumped the number of steps up to 8,000, so it should look nice maximised on a 4k

- misc boring cleanup:
- cleaned up an old wx label patch
- cleaned up an old wx system colour patch
- cleaned up some misc initialisation code

next week

Next week is a medium sized job week. I would like move the 'notes' system forward. Top priority is to get some preview of notes on the media viewer, next to think about is duplicate file note merging and parsing notes from sites.

I had a good week. There's a simple first version of showing notes in the media viewer and several quality of life UI improvements.

The release should be as normal tomorrow.

- the 'set_notes' command now checks the type of the notes Object. it obviously has to be string-to-string
- the 'get_thumbnail' command should now never 404. if you ask for a pdf thumb, it gives the pdf default thumb, and if there is no thumb for whatever reason, you get the hydrus fallback thumbnail. just like in the client itself
- updated client api help to talk about these
- updated the unit tests to handle them too
- did a pass over the client api help to unify indent style and fix other small formatting issues
- client api version is now 28

next week

I am feeling good about multiple local file services. Most of the cleanup this week was for that, and now there are only about three things left to do before we can start playing with it for real--UI and some importer code to handle imports to multiple locations, UI to present deletes and undeletes for multiple locations, and UI and db code to do move/copy across locations. I'll push on these in the coming weeks.

Next week will be a 'small jobs' week, and I would like to catch up on github issues in particular.

I had an ok week with a variety of work. I fixed some bugs, tweaked some UI (including a neat change to shift+select of thumbnails), improved the new file history chart, and added to the Client API.

The release should be as normal tomorrow.

full list

- misc:
- the network engine now parses the 'last-modified' response header for raw files. if this time is earlier than any parsed source time, it is used as the source time and saved to the new 'domain modified time' system. this provides decent post time parsing for a bunch of sites by default, which will also help for subscription timing and similar
- to get better apng duration, updated the apng parser to count up every frame duration separately. previously, if ffmpeg couldn't figure it out, I was just defaulting to 24 fps and estimating. now it is calculated properly, and for variable framerate apngs too. all apngs are scheduled for a metadata regen this week. thanks to the user who submitted some long apngs where this problem was apparent
- fixed a bug in the network engine filter that figures out url class precedence. url classes with more parameters were being accidentally sorted above those with more path components, which was messing with some url class matching and automatic parser linking
- improved the message when an url class fails to match because the given url has too few path components
- fixed a time delta display bug where it could say '2 years, 12 months' and similar, which was due to a rounding issue on 30 day months and the, for example, 362nd day of the year
- fixed a little bug where if you forced an archive action on an already archived file, that file would appear to get a fake newer archived timestamp in UI until you restarted
- updated the default nitter parsers to pull a creator tag. this seemed to not have been actually done when previously thought
- the image renderer now handles certain broken files better, including files truncated to 0 size by disk problem. a proper error popup is made, and file integrity and rescan jobs are scheduled
- .
- file history chart:
- for a long time, a user has been generating some cool charts on file history (how many files you've had in your db over time, how many were deleted, etc...) in matplotlib. you may have run his script before on your own database. we've been talking a while about integrating it into the client, and this week I finally got around to it and implemented it in QtCharts. please check out the new 'view file history' underneath Mr Bones's entry in the help menu. I would like to do more in this area, and now I have learned a little more about QtCharts I'd like to revisit and polish up my old bandwidth charts and think more about drawing some normal curves and so on of other interesting data. let me know what you think!
- I did brush up a couple things with the bandwidth bar chart already, improving date display and the y axis label format
- .
- client api:

Here's my chart! Two notes:

1) My inbox is a little higher than it should be because I rescheduled everything for a second scan some years ago.
2) Guess when 'many queries on one subscription' was added. :^)

I had a good week. I cleaned a heap of code, fixed some bugs, brushed up the new file history graph, and wrote some small extensions to the Client API.

The release should be as normal tomorrow.