FineWeb2-HQ

Multilingual data filtering tools for LLM pretraining using FastText and Transformer-MLP classifiers.

Toolset for multilingual data quality filtering using FastText language identification and Transformer-MLP quality classifiers. Supports embedding generation, model training, and automated dataset curation for the FineWeb2-HQ corpus covering 20 languages. Accompanies an arXiv preprint on model-based data selection.