Description
[Booru imageboards](https://www.kaggle.com/printcraft/eshuushuu-tags-v1) contains huge count of images and are replenished constantly by end-users and batch crosspostings.
There is no strict quality control but general (booru dependent) moderational rules and community self-cleanup.
As a result raw imageboards content and datasets over it [such as Danbooru 2020](https://www.gwern.net/Danbooru2020) are somewhat "dirty"
and a lot of preprocessing required to exclude some evident "outliers".
The [BC2021 torrent](https://nyaa.si/view/1384820) aimed to be comparable size but much cleaner from esthetical and technical points of view with the same
amount of community metadata (tags, user ratings etc) and also with technical metadata about images and image content.
**THIS IS THE ADDON to BOORU CHARS 2021** with no duplicates and minumum similarities for pre-2016 art .
The practical task to solve was an auto-cleanup ~510k images of my minimally prepared "cluttered basement" :
- to throw out ~10% "worst outliers" not to store and distribute garbage
- to rank the rest pictures with "attractiveness score function"
Both tasks are generally subjective, but there are some esthetical "rules of thumb" projected on numeric criterias
which can be estimated with image processing workflow, described in the README included.
**This release contains:**
- **463.873 sampled images** (max size 1280px, 1024 for 1x1) with metadata
* clustered by aspect ratio and also number of heads (0,1,2,3+) detected
* ordered and grouped into 1000-th zip/folders by "attractiveness score function"
- detailed results for detection algorythms and also full tags list
- python script and data to visualize findings
- several XLS with summary queries results
- sample code (commandline, python, PL/SQL) not "ready to use" but key building blocks