Document at what rate the public data sets are published

[ncouture]

Hello MaginaliaSearch!

Upon finding this interesting open data sets, I noticed that their size is not very large and they already pre-date 2024 by a month or so.

Would you consider automating these exports? If not or if it requires a new issue in another project (MaginaliaSearch, for instance), could one document this in this repo's README.md?

Thank you and keep being awesome!

P.S. search.marginalia.nu is the search engine I've been wanting for years before it was created <3

[vlofgren]

Hey. These exports are currently a bit ad-hoc, I have some automation but it's not fully automated. They'll probably be updated at most one a quarter in the future, as the dataset isn't changing much faster than that.

The data is a bit stale though, so I triggerd a new export. I just want to verify the data integrity before I upload the new set. Will be a few days. I'll close the issue when it's done, and maybe make a note in the readme about the state of things.

[ChillarAnand]

blogs.txt has only ~4k blogs. Is the dataset complete?

I am building https://slashindex.fyi and if you can clarify if all blogs are part of blogs.txt, that would be helpful.