/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Board Rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Magrathea | Catalog | Bottom

Expand All Images


Version 477 Anonymous Board owner 03/16/2022 (Wed) 21:04:34 Id: 42fe11 [Preview] No. 1248
https://youtube.com/watch?v=mVG77xTPH6E [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Linux.-.Executable.tar.gz

I had a good week. There is a mix of small work, an expansion to the Client API, and a bit of fun.

misc

The network engine now pulls source time directly from file downloads if the server provides a date. This means a whole bunch of sites that haven't provided a good source time until now suddenly do, which improves the new aggregate modified time and also subscription and watcher check timings.

With our new apng parsing tech, I fixed up apng duration parsing, which was until now relying on a fallback default of 24 fps if ffmpeg couldn't figure it out. All apngs are scheduled for another scan.

I fixed an important precedence bug in the network engine that matches URLs to URL Classes. If you have been making a downloader and had Gallery URLs matching as Post URLs, please give it another go. Sorry for the trouble!

Client API

A user has written a cool expansion to the Client API, which I appreciate. You can now fetch, set, and delete file notes! If you are an API dev, check out the documentation for the new calls (fetching notes is now a parameter on file_metadata).

He also made technical improvements. The Client API now supports far longer GET requests, up to 2MB of URL if needed, and the whole API has tentative and experimental support for CBOR instead of JSON if you wish.

file history chart

Another user has for some time been playing around with drawing charts of a client's file history in matplotlib using raw database data. You may have run his script yourself. We have been talking for a while about integrating this into hydrus, and this week I finally got around to implementing it in QtCharts.

Please hit 'view file history' on the help menu to see the new chart. This is a simple, first attempt on my end, but it should show you a cool history of how many files you have had. If you have been using the client for any time, the lines for deleted files and inbox will be very incomplete, but this data will fill out in time.

This was fun to do, and I learned a bit more about QtCharts. I fixed a couple of ugly things in the bandwidth bar chart I made before, and I think I'll do some more here too. I have a thought to start drawing some of our other data, let's say file size or number of file views, and seeing if pareto or normal distributions pop out. Anyway, let me know what you think, and feel free to share your file history chart!


Anonymous Board owner 03/16/2022 (Wed) 21:05:15 Id: 42fe11 [Preview] No.1249 del
full list

- misc:
- the network engine now parses the 'last-modified' response header for raw files. if this time is earlier than any parsed source time, it is used as the source time and saved to the new 'domain modified time' system. this provides decent post time parsing for a bunch of sites by default, which will also help for subscription timing and similar
- to get better apng duration, updated the apng parser to count up every frame duration separately. previously, if ffmpeg couldn't figure it out, I was just defaulting to 24 fps and estimating. now it is calculated properly, and for variable framerate apngs too. all apngs are scheduled for a metadata regen this week. thanks to the user who submitted some long apngs where this problem was apparent
- fixed a bug in the network engine filter that figures out url class precedence. url classes with more parameters were being accidentally sorted above those with more path components, which was messing with some url class matching and automatic parser linking
- improved the message when an url class fails to match because the given url has too few path components
- fixed a time delta display bug where it could say '2 years, 12 months' and similar, which was due to a rounding issue on 30 day months and the, for example, 362nd day of the year
- fixed a little bug where if you forced an archive action on an already archived file, that file would appear to get a fake newer archived timestamp in UI until you restarted
- updated the default nitter parsers to pull a creator tag. this seemed to not have been actually done when previously thought
- the image renderer now handles certain broken files better, including files truncated to 0 size by disk problem. a proper error popup is made, and file integrity and rescan jobs are scheduled
- .
- file history chart:
- for a long time, a user has been generating some cool charts on file history (how many files you've had in your db over time, how many were deleted, etc...) in matplotlib. you may have run his script before on your own database. we've been talking a while about integrating it into the client, and this week I finally got around to it and implemented it in QtCharts. please check out the new 'view file history' underneath Mr Bones's entry in the help menu. I would like to do more in this area, and now I have learned a little more about QtCharts I'd like to revisit and polish up my old bandwidth charts and think more about drawing some normal curves and so on of other interesting data. let me know what you think!
- I did brush up a couple things with the bandwidth bar chart already, improving date display and the y axis label format
- .
- client api:
- a user has written several expansions for the client api. I really appreciate the work
- the client api now has note support! there is a new 'add notes' permission, 'include_notes' parameter in 'file_metadata' to fetch notes, and 'set_notes' and 'delete_notes' POST commands
- the system predicate parser now supports note system preds
- hydrus now supports bigger GET requests, up to 2 megabytes total length (which will help if you are sending a big json search object via GET)
- and the client api now supports CBOR as an alternate to JSON, if requested (via content-type header for POST, 'cbor' arg for GET). CBOR is basically a compressed byte-friendly version of JSON that works a bit faster and is more accessible in some lower level languages
- cbor2 is now in the requirements.txt(s), and about->help shows it too
- I added a little api help on CBOR
- I integrated the guy's unit tests for the new notes support into the main hydrus test suite
- the client api version is now 27
- I added links to the client api help to this new list of hydrus-related projects on github, which was helpfully compiled by another user: https://github.com/stars/hydrusnetwork/lists/hydrus-related-projects

next week

Next week is cleanup. I will focus on clearing out old code, particularly in file handling for multiple local file services.


Anonymous Board owner 03/16/2022 (Wed) 21:08:51 Id: 42fe11 [Preview] No.1250 del
Here's my chart! Two notes:

1) My inbox is a little higher than it should be because I rescheduled everything for a second scan some years ago.
2) Guess when 'many queries on one subscription' was added. :^)
Edited last time by hydrus_dev on 03/16/2022 (Wed) 21:14:17.


Release Tomorrow! Anonymous Board owner 03/23/2022 (Wed) 03:49:30 Id: 2ed77f [Preview] No.1251 del
I had a good week. I cleaned a heap of code, fixed some bugs, brushed up the new file history graph, and wrote some small extensions to the Client API.

The release should be as normal tomorrow.



Top | Catalog | Post a reply | Magrathea | Return