/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 490 Anonymous Board owner 06/29/2022 (Wed) 21:43:56 Id: 16eaa5 [Preview] No. 1312 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=LBzE9JMoCeE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Linux.-.Executable.tar.gz

I had a good week working on a variety of small jobs.

highlights

While working with a user, I discovered I had recently messed up the initialisation of the image caches, causing them to use default values and be too small for many power users. This is fixed, so I hope you will get some smoother media viewing, especially when it comes to very large images.

I tidied up some weird file service bugs and annoyances that came from the multiple local file services transition. Fixed some weird 'delete from x' entries on the thumbnail menu, stopped spamming 'all my files' in places it wasn't helpful, figured out better service ordering, just some simple workflow stuff.

Message too long. Click here to view full text.



Anonymous Board owner 06/29/2022 (Wed) 21:45:42 Id: 16eaa5 [Preview] No.1313 del
- orphan files:
- there's a persistent processing bug some users have where some update files are missing but they won't redownload correctly. I think I fix that this week naturally so existing maintenance routines will now be able to fix it themselves after another round
- fixed some issues related to deleting files from the repository updates file domain.
- the 'clear orphan file records' maintenance command now fixes the 'all my files' umbrella services as well as the 'all local files' one. it also has nicer description, does some additional file-removal cleanup, and triggers a file recount if problems are found
- moved 'clear orphan files' to the 'files' maintenance menu

next week

Next week is a medium size job week. I want to have another go at building a larger metadata import/export pipeline. I want to start unifying tags, urls, ratings, everything into one thing that can eat up or spit out XML or JSON.



Version 489 Anonymous Board owner 06/22/2022 (Wed) 21:56:54 Id: e43963 [Preview] No. 1306 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=6rboksqjPy4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Linux.-.Executable.tar.gz

I had a good week getting back into the swing of things. I fixed some important bugs and improved some UI.

highlights

All the downloader pages--gallery, watcher, urls, and simple--have a revamped status system. All the text that shows how file or gallery downloads are going is now generated in a better way, with more error states (e.g. it will tell you when your gallery stopped because it hit the file limit, or when one of the emergency pause states under the network menu has kicked in), and logic in edge cases is improved. Everything is unified now, so the texts are the same across all pages. Also, if a gallery query or watched thread is 'pending', its text now reports that it is waiting for a work slot, rather than staying blank. There _shouldn't_ be any situations now where a downloader is unpaused with work to do but has blank status.

If you use the multiple local file services system, the archive/delete filter now presents more options when you are done. If the files are in more than one local file service, you can choose where you delete them from, including all applicable. This was confusing and opaque before, so I hope this makes it more clear what is happening and gives you more choice.

Message too long. Click here to view full text.



Anonymous Board owner 06/22/2022 (Wed) 21:57:12 Id: e43963 [Preview] No.1307 del
- archive/delete:
- the final 'commit/forget/back' confirmation dialog on the archive/delete filter now lists all the possible local file domains you could delete from with separate file counts and 'commit' buttons, including 'all my files' if there are multiple, defaulting to the parent page's location at the top of the list. this let's you do a 'yes, purge all these from everywhere' delete or a 'no, just from here' delete as needed and generally makes what is going on more visible
- fixed archive/delete commit for users with the 'archived file delete lock' turned on
- .
- misc:
- fixed a bug in the parsing sanity check that makes sure bad 'last modified' timestamps are not added. some ~1970-01-01 results were slipping through. on update, all modified dates within a week of this epoch will be retroactively removed
- the 'connection' panel in the options now lets you configure how many times a network request can retry connections and requests. the logic behind these values is improved, too--network jobs now count connection and request errors separately
- optimised the master tag update routine when you petition tags
- the Client API help for /add_tags/add_tags now clarifies that deleting a tag that does not exist _will_ make a change--it makes a deletion record
- thanks to a user, the 'getting started with files' help has had a pass
- I looked into memory bloat some users are seeing after media viewer use, but I couldn't reproduce it locally. I am now making a plan to finally integrate a memory profiler and add some memory debug UI so we can better see what is going on when a couple gigs suddenly appear
- .
- important repository processing fixes:
- I've been trying to chase down a persistent processing bug some users got, where no matter what resyncs or checks they do, a content update seems to be cast as a definition update. fingers crossed, I have finally fixed it this week. it turns out there was a bug near my 'is this a definition or a content update?' check that is used for auto-repair maintenance here (long story short, ffmpeg was false-positive discovering mpegs in json). whatever the case, I have scheduled all users for a repository update file metadata check, so with luck anyone with a bad record will be fixed automatically in the background within a few hours of background work. anyone who encounters this problem in future should be fixed by the automatic repair too. thank you very much to the patient users who sent in reports about this and worked with me to figure this out. please try processing again, and let me know if you still have any issues
- I also cleaned some of the maintenance code, and made it more aggressive, so 'do a full metadata resync' is now be even more uncompromising
- also, the repository updates file service gets a bit of cleanup. it seems some ghost files have snuck in there over time, and today their records are corrected. the bug that let this happen in the first place is also fixed
- there remains an issue where some users' clients have tried to hit the PTR with 404ing update file hashes. I am still investigating this

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/29/2022 (Wed) 01:37:24 Id: 346102 [Preview] No.1311 del
I had a good week. I did a variety of small work and one important bug fix that should speed up media browsing for power users.

The release should be as normal tomorrow.



Q&A Thread Anonymous Board owner 08/08/2019 (Thu) 00:24:05 Id: 348093 [Preview] No. 10 [Reply] [Last 50 Posts]
Please feel free to ask questions about hydrus here.

As a reminder, the help and getting started guide is here:

https://hydrusnetwork.github.io/hydrus/help/index.html
198 posts and 16 images omitted.


Anonymous 05/22/2022 (Sun) 14:04:43 Id: 908cd8 [Preview] No.1295 del
i've been using the PTR for a while, and most of the tagging I've done has been done in the PTR rather than my tags. but i think the PTR is taking up too much space on my SSD

if i use "migrate tags" with the default settings, can i copy the PTR tags to the local my tags for JUST the files that I currently have, so that I can remove the PTR without losing all the tagging I've done up till now?


Anonymous 06/22/2022 (Wed) 07:02:30 Id: d26dbb [Preview] No.1305 del
How do I dl danbooru pools & groups?


Anonymous 06/23/2022 (Thu) 12:20:24 [Preview] No.1308 del
The e621 login script seems to be half-broken: when trying to login it consistently gets blocked by cloudflare, but when testing it on the "edit login script" page it works perfectly.
This happens even on the same IP, and with less than a minute between both tests.
Anyone has a clue as to what might be causing this difference?


Anonymous 06/23/2022 (Thu) 22:53:58 Id: 35424d [Preview] No.1309 del
when using stuff like "collect by title" in the search page, is it possible to sort by random for the overall results page but sort by page number within the individual collection?

e.g. say I have a few titled image sets with red_hair tag, with 2-10 images in each. when I sort by random I would like to see a random image set, but then as I scroll through the page I want to see page 1, page 2, page 3 etc from that image set. right now, if I sort by random and group by title, I will see pages in a random order like 6,10,4,1,9, etc


Anonymous 06/28/2022 (Tue) 01:53:37 Id: 6aedd7 [Preview] No.1310 del
Is there way for nitter importer to apply post date as file metadata date and if it is possible for multiple images in one tweet to have relationship between each other?



Version 488 Anonymous Board owner 06/08/2022 (Wed) 21:56:11 Id: 3589b3 [Preview] No. 1303 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=AQOfIENN2tk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Linux.-.Executable.tar.gz

I had a good simple week making a clean release before my vacation.

Everything is misc this week, nothing earth-shattering, just a bunch of cleanup and little stuff. If you have any wavpack files, try importing them!

full list

- the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
- added a new file maintenance action, 'if file is missing, note it in log', which records the metadata about missing files to the database directory but makes no other action

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/22/2022 (Wed) 05:01:40 Id: 4ff1a0 [Preview] No.1304 del
I had a good week. I did a mix of cleanup and improvements to UI and an important bug fix for users who have had trouble syncing to the PTR.

The release should be as normal tomorrow.



Version 487 Anonymous Board owner 06/01/2022 (Wed) 22:09:24 Id: 82e2be [Preview] No. 1300 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=VKuGYKkH3oA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Linux.-.Executable.tar.gz

I had an ok week. I was a unexpectedly short on time, so I couldn't get everything I wanted done, but I cleared out some small work.

highlights

The big update last week, which I recommended only for advanced users, went well. There don't seem to be any obvious problems with the logic of the new search cache, so I now recommend it for everyone. You will be presented with a popup just before the update runs, giving you an estimate of how long it thinks it will take. Most users should take 5-10 minutes, but if you have millions of files, it will be longer. Just let it run and some things will run a bit faster and neater in the background. If you have played with 'multiple local file services', then check out the new 'all my files' domain you will see--this is basically an efficient umbrella of all your local file services. It works super fast for things like the duplicates system.

I also put some time into the duplicate filter this week. The logic of the queue is improved again, so some rare errors when reaching the end of a batch should be fixed. I also integrated manual file deletes into the queue processing: now, when you manually delete a file, or both, the deletes will not happen until you commit--just like the other decisions you are making--and they are undoable if you select 'forget' or go back a pair. You also won't see a file you manually deleted again in a batch (it'll auto-skip if that file comes up again).

Message too long. Click here to view full text.



Anonymous Board owner 06/01/2022 (Wed) 22:13:09 Id: 82e2be [Preview] No.1301 del
full list

- misc:
- updated the duplicate filter 'show next pair' logic again, mostly simplification and merging of decision making. it _should_ be even more resistant to weird problems at the end of batches, particularly if you have deleted files manually
- a new button on the duplicate filter right hover window now appends the current pair to the parent duplicate media page (for if you want to do more processing to them later)
- if you manually delete a file in the duplicate filter, if that file appears again in the current batch of pairs, those will be auto-skipped
- if you manually delete a file in the duplicate filter, the actual delete is now deferred to when you commit the batch! it also undoes if you go back!
- fixed a bug when editing the external program launch paths in the options
- fixed an annoying delay-and-error-popup when clearing the separator field when editing a String Splitter. now the field just turns red and vetoes an OK with a nicer error text
- also improved how string splitters report actual split errors
- if you are in advanced mode, the _review services_ panels now have an 'id' button that lets you fetch the database service id
- wrote a new database maintenance routine under _database->check and repair->resync tag mappings cache files_, which is a lightweight way of fixing ghost files or situations where files with a tag are neither counted nor appear in file results. this fixes these problems in a couple minutes, so for this it is much better than a full regen of the cache
- .
- cleanup and other boring stuff:
- the archive/delete filter now says which file domain it will be deleting from
- if an archive/delete filter is launched on a 'multiple locations' file domain, it is now careful to only make delete records for the deleted files for the file services each one is actually in
- renamed the 'default local file search location' option to 'fallback' and updated its tooltip a bit. this was really a hacky thing I needed to fill some gaps while rewriting from 'my files' to multiple local file services. the whole thing needs more attention to become more useful. I also fixed an issue where it could become invalid 'nothing' if you deleted a file service it was referring to (issue #1155)

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/08/2022 (Wed) 01:50:49 Id: b3bad0 [Preview] No.1302 del
I had a good week. I did some simple work to make a clean release before my vacation.

The release should be as normal tomorrow.



Version 486 Anonymous Board owner 05/25/2022 (Wed) 22:20:57 Id: 63b815 [Preview] No. 1296 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=ZUrcYKghr-Y [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Linux.-.Executable.tar.gz

I had a great week working on a variety of smaller issues and some important database updates.

The release this week is only recommended for advanced users. I make an important change, and I want to make sure the update works quickly and without problems before I roll it out to everyone. If you are not an advanced user, please check back in next week!

The update will also take a few minutes this week.

all my files

Message too long. Click here to view full text.



Anonymous Board owner 05/25/2022 (Wed) 22:21:31 Id: 63b815 [Preview] No.1297 del
full list

- This week's release is for advanced users only! I make a big change, and I want to make sure the update is fast and there are no unusual problems before rolling it out to all users.
- all my files:
- the client adds a new virtual file service this week, 'all my files', which is an umbrella covering all your local file domains. if you do not engage the multiple local file services system, you won't see it much, but if you do, you'll now have a convenient tool for saying 'all my stuff' without including trash and repository updates
- it will take a minute or two to generate this new service on update. if you have a client with millions of files, it may take a while
- 'all my files' now appears in the file domain selector button on your tag entry box if you have more than one local file domain. selecting this searches the union of all your local file domains with fast and precise count (as opposed to 'multiple locations' of the full union, which will have imprecise counts and be slower). it also does duplicate file work laser-fast (again, unlike 'multiple locations', which is often slow due to UNION complexity)
- 'all my files' also appears in review and manage services, very similarly to 'all local files'
- a heap of hacks I instituted when getting multiple local file services ready are now replaced with this clean 'yeah this file is valued and worth looking at' domain. for instance, downloader pages now view files in this way.
- mr bones and the file history chart also use 'all my files', and are significantly faster to calculate. the chart also excludes repo update files and trash now
- calls to delete or undelete on 'all my files' (this is mostly Client API and some 'default' situations) will be converted to a blanket 'force send to trash' and 'force undelete all deleted records'
- the 'undelete files?' dialog is now a button selection dialog. it also now has an 'all the above' option when more than one local service may apply, which tells the client to undelete to all services the files have been deleted from
- updated multiple local file services help to talk a little about the new domain
- rearranged the sort in a couple of places where the different local file services appear. they should now be: local file domains, all my files, trash, repo updates, all local files
- ADVANCED: the 'presentation import options' under 'file import options' now allows a full-fledged location context using the new multiple local file services system rather than the previous 'in your files(and trash too)' choice. it defaults to the new 'all my files' domain
- .
- misc:

Message too long. Click here to view full text.



Anonymous Board owner 05/25/2022 (Wed) 22:33:31 Id: 63b815 [Preview] No.1298 del
- right-clicking on files when in pages with 'multiple locations' file domains is now much much faster
- the filename tagging dialog now starts with the 'tags for all' focused, and the 'press up/down on empty input' shortcuts are now plugged in, so pressing up/down will change service
- I believe I may have completely eliminated the additional superlag that sometimes occurs when adding or deleting a service. it was a database maintenance routine getting carried away with other outstanding work
- move/add actions in the new multiple local file system now operate asynchronously and politely, spreading their work time out when the client is busy, and for large jobs they will also make a cancellable progress popup
- cleaned up how the autocomplete entry sends some of its signals to other parts of the program
- did some misc help and code edits/refactoring, including brushing up the Windows install section with more advanced options
- removed the 'hydrus zooms big bad' warning from the 'media' options page. hydrus zooms big good now!
- .
- some database stuff:
- tl;dr: database cleans up after itself better now
- some users have had trouble with database journal files (the 'wal' files in your db directory) on certain clients getting huge after lots of work, multiple GB, and causing the OS a headache if the journal is doing work through a computer sleep. these journals are 'supposed' to checkpoint and clean themselves up naturally, but I think a busy database chokes them. therefore, I have improved the hydrus maintenance this week: 1) the 'journal size limit' PRAGMA, which applies softly after every 30 seconds or so, is now 128MB down from 1GB. 2) databases in PERSIST (rare) mode will now specifically zero out their journal fifteen minutes. 3) databases in WAL mode (the default), in addition to regular PASSIVE checkpointing now every five minutes, will force an additional TRUNCATE checkpoint every fifteen. this should force a regular full flush and maybe help some other problems like gigantic memory bloat the same users sometimes saw. if you are a very advanced user and do active debug on the database while hydrus is using it, please note this new TRUNCATE command is aggressive and may block itself or you inconveniently. let me know how you get on!
- moved the recent 'be careful of usb drives' section in 'installing' help to 'help my db is broke.txt'. it is very likely this problem was related to the above WAL stuff, and it was not just usb drives, I rewrote it as generalised help for anyone who gets 'delayed write failed' errors at the OS level
- massively optimised several critical duplicate files filtering methods if the current location context has more than one file domain, and I think I cleared out the basic 'get duplicate info for this file' call of all slow calls in complex location contexts
- the repair routine that regenerates mapping caches if any tables are missing on boot is now more reliable and covers the entirety of the mappings cache system using the new modules system. it also now regenerates just for the tag services with missing tables, not the whole cache
- if multiple types of mapping cache tables are missing on boot, and multiple waves of regenerations covering different areas are planned, duplicate regenerations will now be skipped

next week

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/01/2022 (Wed) 04:03:04 Id: 52974a [Preview] No.1299 del
I had an ok week. I didn't have time to get to the big things I wanted, but I cleared a variety of small bug fixes and quality of improvements.

The release should be as normal tomorrow.



Version 484 Anonymous Board owner 05/11/2022 (Wed) 22:13:39 Id: 804973 [Preview] No. 1280 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=OtPsKtUyGxg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Linux.-.Executable.tar.gz

I had an ok week. I fixed some things, improved some quality of life, and made internal file migration a bit easier.

highlights

Last week's debut of multiple local file services went well! As far as I know, no one who tried it out had any big problems, and my main concerns--mostly that it needs some better migration tools and workflows and 'this file is in here' UI--proved true. So, I know what I have to do and will keep working. Multiple local file services remains for advanced users for now, but I hope to launch it properly for everyone, with nice help, next week.

However, while doing this work, I did accidentally break the simple version of the 'delete files' dialog when files were in the trash--rather than say 'delete these permanently?', it just wouldn't appear. This was due to a logical oversight where it wasn't testing and counting up 'trash' status correctly. It is fixed now.

Message too long. Click here to view full text.

5 posts omitted.


Anonymous 05/15/2022 (Sun) 08:47:05 Id: 8000ff [Preview] No.1287 del
>>1286
To add more information, I run Hydrus on an external HDD. I used crystaldisk to check the state of the drive and everything seems ok, no bad signs. It just seems like a file went corrupt and went missing from my database in the folder. The file was imported 19 days ago from now but not archived, I'm not sure if being archived does anything but I'll be on the lookout for any more issues.


Release Tomorrow! Anonymous Board owner 05/18/2022 (Wed) 03:03:51 Id: 43ad88 [Preview] No.1288 del
I had a good simple week. I cleaned some code, improved some quality of life, and made multiple local file services ready for all users.

The release should be as normal tomorrow.

>>1286
>>1287
Hmm, this is a shame. I am not sure what happened here. The most likely culprit is a hard drive fault, although those tend to make a file 0KB more often than simply disappearing a file. Sometimes you can have anti-virus go bananas and remove an mp3 or something, but this seems to be just a jpeg.

If you are worried this may have happened to more files, the place to check is database->file maintenance->manage scheduled jobs. Under 'add new work', set up a job for 'all media files' for an appropriate job type. The default, 'if file missing, try to redownload, else remove record' is good.

If it is any help, I don't have that file, but I think I found it through its PTR tags here:

https://danbooru.donmai.us/posts/5084454

You can probably see that on the borked thumbnail's right-click menu too. Just enter that URL into a new URL downloader page and fingers crossed hydrus should be able to automatically fix the file for you. The above maintenance job will do this automatically.

Let me know if you have any more trouble, and make sure you have a good backup!

Message too long. Click here to view full text.



Anonymous 05/18/2022 (Wed) 04:37:57 Id: 8000ff [Preview] No.1289 del
(175.59 KB 1124x690 57457547.jpg)
>>1288
Thanks, I encountered a few more. Same day from that same session. I'm not sure what happen that day, I don't think I badly closed out of the program or improperly dismounted the drive. It might be that I haven't been doing regular maintenance other than shutting down and I definitely haven't been defraging the drive like I probably should so its probably that and might be bad sectors or something.

Also the other few I mentioned I encountered gave me a slightly different message. I can view the thumbnail but whenever I tried to open it it gave me this message. It would then redownloaded the file but still wouldn't load it. I forgot to check the actual folder where the file is located and just set my HDD aside until I got a proper solution.


Anonymous Board owner 05/18/2022 (Wed) 18:42:51 Id: a1bf88 [Preview] No.1290 del
>>1289

Ah, for files that seem to be malformed, there are two answers:

1) The file might have been corrupted. In the 'file maintenance' system, there's a set of jobs called 'missing/incorrect' that scan for this. These jobs are expensive though (they have to load the whole file from disk), so don't add them for all files. Try to target with a tag search or system:hash search just to hit the file(s) you think were affected.

2) The file might have once been supported but now is not. This is rare, but a handful of truncated jpegs and other borked files used to load in hydrus until I turned some safety rules on (the lack of safety was causing crashes in some systems). Best answer here is to just delete the file, or export it, run it through a load/save cycle in Gimp or Photoshop, and then reimport the fixed file. Some boorus just have jank files unfortunately.

If you like, you can post that image here (if the site can accept it), or zip it up and email it to me, and I can have a look. But if it is private, no worries.

These bad files being from the same session makes sense that this was a one-time event. I would still suspect it is a hard drive issue, but it could be something else, especially since crystaldiskinfo was good for you. There's a very small chance that if you have a program crash or other hard turn-off that the client can become unsynced, but that typically ends up being the other way around: the file gets added to your hydrus file storage, but the client isn't aware of it. The client can lose up to 30 seconds of database work if it ends non-cleanly.

Let me know how you get on!


Anonymous 05/19/2022 (Thu) 04:54:04 Id: 8000ff [Preview] No.1293 del
>>1290
I ran some checks and found no more issues. I did the "if the file is missing/incorrect" option. It seems like it was just 2 sessions from that day acting up. 1 tag session entry gave me 2 missing file errors and the other entry session gave me 1 corrupt file error. Maybe it was my anti-virus acting up and doing scans over the files I was just viewing. Those were just normal jpgs and png images, nothing really out of the ordinary.


I'll probably be more cautious and do backups just in case. I already have a NAS but never used it and don't know how I should set it up with hydrus but I'll probably make a new thread here(or somewhere more active) to discuss storage backups and safety. Thanks again.



Version 485 Anonymous 05/18/2022 (Wed) 20:51:51 Id: ea442c [Preview] No. 1291 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=AKgjOCuW_MU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Linux.-.Executable.tar.gz

I had a good week. The multiple local file services system is now ready for all users.

multiple local file services

I have written some proper help for this new system to talk about what it is and how to use it. The basic idea is you can now have more than one 'my files', which lets you compartmentalise things for privacy or workflow reasons. The help is here:

https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html

Message too long. Click here to view full text.



Anonymous 05/18/2022 (Wed) 20:52:11 Id: ea442c [Preview] No.1292 del
full list

- multiple local file services:
- multiple local file services are now available for everyone! you no longer need to be in advanced mode to create them. all are welcome, but in terms of skill level, I most recommend it for users who are comfortable with tag siblings and parents
- the tl;dr: you can now have more than one 'my files', which lets you put things in isolated locations
- I wrote a proper help document on multiple local file services--what they are, how they work, my recommendations, and a bit of extra info about hydrus file search in general, right here: https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html
- file searches in 'multiple locations' on large clients are now massively faster in almost all situations. the only place multiple location searches are still slow is whenever the duplicates system (system:file relationships) comes into play
- .
- misc:
- in the page tab menu, you can now sort pages by total file size
- the 'force system:limit for all searches' option is moved from the 'speed and memory' to 'search' panel
- when files download from sites, if the raw file is served by cloudflare and has a timestamp radically different to a parsed source time, that CF timestamp is saved under a different domain rather than overwriting the original domain timestamp. this seemed to affect danbooru on about 1 in 10-20 files. note this does not change much at the moment, but when you can see and sort on individual domain modified dates, this should improve the sort
- updated the 'installing' help to talk about bad install locations for the database. network locations are bad, and thanks to user reports, we now know USB drives can be bad if the database is busy when the OS goes to sleep
- if a 'database is malformed' error occurs on boot, the client now recognises it and points the user to 'install_dir/db/help my db is broke.txt' for the next steps
- .
- boring code cleanup:
- another 60KB or so of code pulled out of ClientDB.py:

Message too long. Click here to view full text.




Version 483 Anonymous Board owner 05/04/2022 (Wed) 23:00:51 Id: cb293b [Preview] No. 1275 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=ymI1g2VjyCY [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Linux.-.Executable.tar.gz

I had a good couple of weeks doing some regular work and getting 'multiple local file services' ready for testing.

multiple local file services

This is not ready for everyone yet! Advanced users only for now please.

I turned multiple local file services on in debug mode last week, just to see how things were looking, and it turned out suprisingly great, no big problems. For several months now I have been doing prep work for it, and that seems to have paid off. I decided to finish the last important things and get a v1.0 out.

Message too long. Click here to view full text.



Anonymous Board owner 05/04/2022 (Wed) 23:01:23 Id: cb293b [Preview] No.1276 del
the rest

This is a two week release because I was ill earlier on and it cut into my work time. So, there is a mix of different small work. Updated downloaders, reworked sibling&parent help with some neat new charts, fixes and improvements to the duplicate filter, some quality of life in UI labelling and texts. Nothing super important, but some things should be a bit smoother!

full list

- multiple local file services:
- the multiple local file services feature is ready for advanced users to test out! it lets you have more than one 'my files' service to store things, which will give us some neat privacy and management tools in future. there is no nice help for this feature yet, and the UI is still a little non-user-friendly, so please do not try it out unless you have been following it. and, while this has worked great in all testing, I generally do not recommend it for heavy use on a real client either, just in case something does go wrong. with those caveats, hit up _manage services_ in advanced mode, and you can now add new 'local file domain' services. it is possible to search, import to, and migrate files between these and everything basically works. I need to do more UI work to make it clear what is going on (for instance, I think we'll figure out custom icons or similar to show where files are), and some more search tech, and write up proper help, and figure out easy client merging so users can combine legacy clients, but please feel free to experiment wildly on a fresh client or carefully on your existing one
- if you have more than one local file service, a new 'files' or 'local services' menu on thumbnail right-click handles duplicating and moving across local services. these actions will preserve original import times (e.g. if you move from A to B and then back to A), so they should be generally non-destructive, but we may want to add some advanced tools in future. let me know how this part goes--I think we'll probably want a different status than 'deleted from A' when you just move A->B, so as not to interfere with some advanced queries, but only IRL testing will show it
- if you have a 'file import options' that imports files to multiple local services but the file import is 'already in db', the file import job will now examine if and where the file is still needed and send content update calls to fill in the gaps
- the advanced delete files dialog now gives a new 'delete from all and send to trash' option if the file is in multiple local file domains
- the advanced delete files dialog now fully supports file repositories
- cleaned up some logic on the 'remember action' option of the advanced file deletion dialog. it also supports remembering specific file domains, not just the clever commands like 'delete and leave no record'. also this dialog no longer places the 'suggested' file service at the top of the radio button list--instead it selects that 'suggested' if there is no 'remember action' initial selection applicable. the suggested file service is now also set by the underlying thumbnail grid or media canvas if it has a simple one-service location context
- the normal 'non-advanced' delete files dialog now supports files that are in multiple local file services. it will show a part of the advanced dialog to let you choose where to delete from


Anonymous Board owner 05/04/2022 (Wed) 23:02:59 Id: cb293b [Preview] No.1277 del
- misc:
- thanks to user submissions, there is a bit more help docs work--for file search, and for some neat new 'mermaid' svg diagrams in siblings/parents, which are automatically generated from a markup and easy to edit
- with the new easy-to-edit mermaid diagrams, I updated the unhelpful and honestly cringe examples in the siblings and parents help to reflect real world PTR data and brushed up all the text in the top sections
- just a small thing--the 'pages' menu and the page picker dialog now both say 'file search' to refer to a page that searches files. previously, 'search' or 'files' was used in different places
- completely rewrote the queue code behind the duplicate filter. an ancient bad idea is now replaced with something that will be easier to work with in future
- you can now go 'back' in the duplicate filter even when you have only done skips so far
- the 'index string' of duplicate filters, where it says 53/100, now also says the number of decisions made
- fixed some small edge case bugs in duplicate filter forward/backward move logic, and fixed the recent problem with going back after certain decisions
- updated the default nijie.info parser to grab video (issue #1113)
- added in a user fix to the deviant art parser
- added user-made Mega URL Classes. hydrus won't support Mega for a long while, but it can recognise and categorise these URLs now, presenting them in the media viewer if you want to open them externally
- fixed Exif image rotation for images that also have ICC Profiles. thanks to the user who provided great test images here (issue #1124)
- hitting F5 or otherwise saying 'refresh' explicitly will now turn a search page that is currently in 'searching paused' to 'searching immediately'. previously it silently did nothing
- the 'current file info' in the media window's top hover and the status bar of the main window now ignores Deletion reason, and also file modified date if it is not substantially different from another timestamp already stated. this data can still be seen on the file's right-click menu's expanded info lines off the top entry. also, as a small cleanup, it now says 'modified' and 'archived' instead of 'file modified/archived', just to save some more space
- like the above 'show if interesting' check for modified date, that list of file info texts now includes the actual import time if it is different than other timestamps. (for instance, if you migrate it from one service to another some time after import)
- fixed a sort error notification in the edit parser dialog when you have two duplicate subsidiary parsers that both have vetoes
- fixed the new media viewer note display for PyQt5

Message too long. Click here to view full text.



Anonymous Board owner 05/04/2022 (Wed) 23:03:30 Id: cb293b [Preview] No.1278 del
- boring code cleanup:
- cleaned up search signalling UI code, a couple of minor bugs with 'searching immediately' sometimes not saving right should be fixed
- the 'repository updates' domain now has a different service type. it is now a 'local update file domain' rather than a 'local file domain', which is just an enum change but marks it as different to the regular media domains. some code is cleaned up as a result
- renamed the terms in some old media filtering code to make it more compatible with multiple local file services
- brushed up some delete code to handle multiple local file services better
- cleaned up more behind the scenes of the delete files dialog
- refactored ClientGUIApplicationCommand to the widgets module
- wrote a new ApplicationCommandProcessor Mixin class for all UI elements that process commands. it is now used across the program and will grow in responsibility in future to unify some things here
- the media viewer hover windows now send their application commands through Qt signals rather than the old pubsub system
- in a bunch of places across the program, renamed 'remote' to 'not local' in file status contexts--this tends to make more sense to people at out the gate
- misc little syntax cleanup

next week

Some small misc jobs and user-friendly-isation of multiple local file services.


Release Tomorrow! Anonymous Board owner 05/11/2022 (Wed) 04:38:18 Id: c23322 [Preview] No.1279 del
I had an ok week. I fixed some bugs (including some bad trash delete, and an issue with the new duplicate filter queue skipping badly), improved some quality of life, and integrated the new multi-service 'add/move file' commands into the shortcuts system and the media viewer.

The release should be as normal tomorrow.



Version 482 Anonymous Board owner 04/20/2022 (Wed) 21:07:40 Id: 32cdef [Preview] No. 1267 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=PGEZutQ-tCM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Linux.-.Executable.tar.gz

I had a great week doing cleanup and other simple work.

highlights

I fixed the problem where clicks on a scrolled taglist were going to the wrong location. I was cleaning up some ancient wx->Qt code hacks and it seems I rarely scroll and click when working, so I never noticed the problem. I have a new test to make sure this does not happen again. Sorry for the trouble!

The URLs in the top-right hover menu are now styled better. No longer underlined, and now colourable by QSS. I have updated all the default stylesheets that come with the client (you can set these under options->style) to have some decent colours. If you have your own custom QSS, check my default to see how to set it yourself.

Message too long. Click here to view full text.



Anonymous Board owner 04/20/2022 (Wed) 21:08:11 Id: 32cdef [Preview] No.1268 del
- code cleanup:
- about 80KB of code moved out of the main ClientDB.py file:
- refactored all combined files display mappings cache code from the code database to a new database module
- refactored all combined files storage mappings cache code from the code database to a new database module
- refactored all specific storage mappings cache code from the code database to a new database module
- more misc refactoring of tag count estimate, tag search, and other code down to modules
- hooked up specific display mappings cache to the repair system correctly--it had been left unregistered by accident
- some misc duplicate action options code cleanup
- migrated some ancient pause states--repository, subscriptions, import&export folders--to the newer options structure
- migrated the image and thumbnail cache sizes to the newer options structure
- removed some ancient db and dialog code from the retired dumper system

next week

I want to catch up on some github issues and do a little more multiple local file services work.


No Release Tomorrow! Anonymous Board owner 04/27/2022 (Wed) 02:08:11 Id: e85311 [Preview] No.1273 del
I was ill for the start of the week and am short on work time. Rather than put out a slim release, I will spend tomorrow doing some more normal work and put the release off a week.

483 should be on the 4th of May. Thanks everyone!


Release Tomorrow! Anonymous Board owner 05/04/2022 (Wed) 03:52:22 Id: 4b549e [Preview] No.1274 del
I had a good couple of weeks. There are a variety of small fixes and quality of life improvements and the first version of 'multiple local file services' is ready for advanced users to test.

The release should be as normal tomorrow.