High-quality corpora for under-resourced languages

GET CORPORA
EVALUATION REPORT
First Evaluation Report Second Evaluation Report
Girl tagging sentences illustration

What is MaCoCu?

INTERNATIONAL SOURCES

We collect data across the internet in order to retrieve sentences in any of the under-resourced languages of the European Union

TOP-TIER CORPORA

We use advanced filtering techniques to obtain the finest results among all the crawled data

EXTRA KNOWLEDGE

Our corpora is shipped with additional information for each sentence, like language variants or domain identification

MaCoCu's Corpora at a glance

MaCoCu corpora links below will get you to single pages where each corpus is explained in detail. A brief explantation of the main steps to build them is given here.
1) MaCoCu corpora, monolingual and bilingual, are obtained by crawling extended internet top-level domains, eg. the .hr or .sl domains, in 2021/2022.
2) Texts from crawled websites are extracted and processed to provide high-quality web corpora. We take care removing boilerplate, duplicates, very short texts, text in non-targeted languages, very noisy texts, bad translations, etc.
3) Texts are, then, enriched with metadata and delivered in convenient formats which allow filtering them based on text quality and other criteria.
This makes MaCoCu corpora highly useful for corpus linguistics as well as for training language models, machine translation models and other language technologies.
Monolingual corpora are delivered in XML (and sometimes TSV) formats while bilingual corpora are released in TXT and TMX formats.
Metadata covers a wide range of information that may appear at document, paragraph or sentence level
At document level: title, crawl date, url, web domain, file type of the original document, distribution of languages inside the document, and a fluency score based on a language model.
At paragraph level: ID, type (heading/not heading), quality and fluency scores, language, translation direction and human vs machine translation in the case of bilingual corpora, language variant for some languages and sensitive information.
At sentence level: URL, ID, simhash + quality score for (near) duplicates, language, fluency score, sensitive information.

Corpora

There have been 2 different releases of MaCoCu corpora. Some languages were released in both and some just in the second one.
The processing pipeline changed among versions to bring fixes, improved filtering and richer metadata.

Select a language

Albanian
Bosnian
Bulgarian
Catalan
Croatian
Dutch
English
Greek
Icelandic
Macedonian
Maltese
Montenegrin
Serbian
Slovene
Spanish
Turkish
Ukrainian

Feel free to explore, review and contribute to our code

Bitextor Monotextor MaCoCu
Explore MaCoCu's Corpora →
Girl coding illustration

What is the corpora creation process?

It depends. The corpora creation process will differ depending on whether we wish to obtain monolingual or parallel corpora and on the corpora version. A detailed description is provided in the Clarin repo where corpora are linked in the previous section. We provide here an overview of the process:

monolingual corpora:

01

DATA CRAWLING

Data gets collected in the targeted languages from top level domains and related websites. Large quantities of raw data are gathered and stored in a prevertical format.

02

PREPROCCESSING

Relevant data is extracted from prevertical files following a standardization process: plain text extraction, boilerplate, near-duplicated paragraphs and very short texts removal, language identification, encoding normalization, document and paragraph identification. Duplicate documents are deleted. Manual checks over the largest domains are applied to spot and delete very bad domains (e.g. machine translated).

03

SCORING FOR FILTERING

Monocleaner is then applied to score texts for fluency (score between 0 and 1) using a language model, double check language, and to further remove duplicates at the desired level (document, paragraph, or sentences).

04

ENRICHMENT AND FORMATTING

Data is enriched with further info such as language variety, personal/sensitive information, domain (DSIs), paragraph quality (labels, such as “short” or “good”, assigned based on paragraph length, URL and stopword density via the jusText tool). Data is converted to XML (and sometimes TSV) formats.

parallel corpora:

01

DATA CRAWLING

Data gets collected in the targeted languages from top level domains and related websites. Large quantities of raw data are gathered and stored in a prevertical format.

02

PRE-PROCESSING

Relevant data is extracted from prevertical files following a standardization process: plain text extraction, boilerplate, near-duplicated paragraphs and very short texts removal, language identification, encoding normalization, document and paragraph identification and sentence spliting. Duplicate documents are deleted.

03

DOCUMENT ALIGNMENT

First step for the identification of parallel data. TF/IDF + machine translation is used to identify parallel documents.

04

SENTENCE ALIGNMENT

The Bleualing technique, using neural machine translation, is applied to identify parallel sentences.

05

FIXING, FILTERING AND SCORING

Noisy bilingual sentences are automatically fixed or filtered. Repeated segments are deduplicated.

06

ENRICHMENT AND FORMATTING

Data is enriched with further info such as language variety, personal/sensitive information, domain (DSIs), translationese or human/machine translation identification. Other metadata might be included. Data is converted to TXT and TMX formats.