Cite MaCoCu

MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages

Marta Bañón, Miquel Esplà-Gomis, Mikel L. Forcada, Cristian García-Romero, Taja Kuzman, Nikola Ljubešić, Rik van Noord, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Peter Rupnik, Vít Suchomel, Antonio Toral, Tobias van der Werff, and Jaume Zaragoza. 2022. MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 303–304, Ghent, Belgium. European Association for Machine Translation

View BibTeX citation

Select the text to copy the citation

@inproceedings{non-etal-2022-macocu,
    title = "{M}a{C}o{C}u: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages",
    author = "Ba{\~n}{\'o}n, Marta  and
      Espl{\`a}-Gomis, Miquel  and
      Forcada, Mikel L.  and
      Garc{\'\i}a-Romero, Cristian  and
      Kuzman, Taja  and
      Ljube{\v{s}}i{\'c}, Nikola  and
      van Noord, Rik  and
      Sempere, Leopoldo Pla  and
      Ram{\'\i}rez-S{\'a}nchez, Gema  and
      Rupnik, Peter  and
      Suchomel, V{\'\i}t  and
      Toral, Antonio  and
      van der Werff, Tobias  and
      Zaragoza, Jaume",
    booktitle = "Proceedings of the 23rd Annual Conference of the European Association for Machine Translation",
    month = jun,
    year = "2022",
    address = "Ghent, Belgium",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2022.eamt-1.41",
    pages = "303--304",
    abstract = "We introduce the project {``}MaCoCu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages{''}, funded by the Connecting Europe Facility, which is aimed at building monolingual and parallel corpora for under-resourced European languages. The approach followed consists of crawling large amounts of textual data from carefully selected top-level domains of the Internet, and then applying a curation and enrichment pipeline. In addition to corpora, the project will release successive versions of the free/open-source web crawling and curation software used.",
}

Publications

2023

Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models

Investigation of Classification Methods in the Era of Large Language Models Machine Learning and Knowledge Extraction 5, no. 3: 1149-1175.

Kuzman, Taja, Igor Mozetič, and Nikola Ljubešić.

PDF

View BibTeX

Select the text to copy the citation

@Article{make5030059,
AUTHOR = {Kuzman, Taja and MozetiÄ, Igor and LjubeÅ¡iÄ, Nikola},
TITLE = {Automatic Genre Identification for Robust Enrichment of Massive Text Collections: Investigation of Classification Methods in the Era of Large Language Models},
JOURNAL = {Machine Learning and Knowledge Extraction},
VOLUME = {5},
YEAR = {2023},
NUMBER = {3},
PAGES = {1149--1175},
URL = {https://www.mdpi.com/2504-4990/5/3/59},
ISSN = {2504-4990},
ABSTRACT = {Massive text collections are the backbone of large language models, the main ingredient of the current significant progress in artificial intelligence. However, as these collections are mostly collected using automatic methods, researchers have few insights into what types of texts they consist of. Automatic genre identification is a text classification task that enriches texts with genre labels, such as promotional and legal, providing meaningful insights into the composition of these large text collections. In this paper, we evaluate machine learning approaches for the genre identification task based on their generalizability across different datasets to assess which model is the most suitable for the downstream task of enriching large web corpora with genre information. We train and test multiple fine-tuned BERT-like Transformer-based models and show that merging different genre-annotated datasets yields superior results. Moreover, we explore the zero-shot capabilities of large GPT Transformer models in this task and discuss the advantages and disadvantages of the zero-shot approach. We also publish the best-performing fine-tuned model that enables automatic genre annotation in multiple languages. In addition, to promote further research in this area, we plan to share, upon request, a new benchmark for automatic genre annotation, ensuring the non-exposure of the latest large language models.},
DOI = {10.3390/make5030059}
}

VarDial 2023

2023

Get to Know Your Parallel Data: Performing English Variety and Genre Classification over MaCoCu Corpora

Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023). 2023.

Rupnik, Peter, Taja Kuzman, and Nikola Ljubešić

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{kuzman2023get,
  title={Get to Know Your Parallel Data: Performing English Variety and Genre Classification over MaCoCu Corpora},
  author={Kuzman, Taja and Rupnik, Peter and Ljube{\v{s}}i{\'c}, Nikola},
  booktitle={Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)},
  pages={91--103},
  year={2023}
}

VarDial 2023

2023

BENCHić-lang: A Benchmark for Discriminating between Bosnian, Croatian, Montenegrin and Serbian

Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023). 2023.

Rupnik, Peter, Taja Kuzman, and Nikola Ljubešić

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{rupnik2023benchic,
  title={BENCHi{\'c}-lang: A Benchmark for Discriminating between Bosnian, Croatian, Montenegrin and Serbian},
  author={Rupnik, Peter and Kuzman, Taja and Ljube{\v{s}}i{\'c}, Nikola},
  booktitle={Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)},
  pages={113--120},
  year={2023}
}

IS 2022

2022

Exploring the impact of lexical and grammatical features on automatic genre identification

Data Mining and Data Warehouses - SiKDD: Information Society - IS 2022, pages 17-20

Taja Kuzman and Nikola Ljubešić

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{Kuzman_Ljubesic_2022,
 series={Data Mining and Data Warehouses - SiKDD: Information Society - IS 2022},
 title={Exploring the impact of lexical and grammatical features on automatic genre identification},
 ISBN={2630-371X},
 ISSN={2630-371X},
 url={http://library.ijs.si/Stacks/Proceedings/InformationSociety/2022/IS2022_Volume-C - SiKDD.pdf},
 booktitle={Data Mining and Data Warehouses - SiKDD: Information Society - IS 2022},
 publisher={Jo\v{z}ef Stefan Institute},
 author={Kuzman, Taja and Ljube{\v{s}}i{\'c}, Nikola},
 editor={Mladeni{\'c}, Dunja and Grobelnik, Marko},
 year={2022},
 pages={17–20},
 collection={Data Mining and Data Warehouses - SiKDD: Information Society - IS 2022} }

JTDH 2022

2022

Assessing Comparability of Genre Datasets via Cross-Lingual and Cross-Dataset Experiments

Proceedings of the Conference on Language Technologies and Digital Humanities, pages 100-107

Taja Kuzman, Nikola Ljubešić and Senja Pollak

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{Kuzman_Ljubesic_Pollak_2022,
 series={Proceedings of the Conference on Language Technologies and Digital Humanities},
 title={{Assessing Comparability of Genre Datasets via Cross-Lingual and Cross-Dataset Experiments}},
 booktitle={Proceedings of the Conference on Language Technologies and Digital Humanities},
 publisher={Institute of Contemporary History},
 author={Kuzman, Taja and Ljube{\v s}i{\'c}, Nikola and Pollak, Senja},
 editor={Fi{\v s}er, Darja and Erjavec, Toma{\v{z}}},
 year={2022},
 pages={100–107},
 collection={Proceedings of the Conference on Language Technologies and Digital Humanities} }

GhentEAMT2022

June, 2022

Automatic Discrimination of Human and Neural Machine Translation: A Study with Multiple Pre-Trained Models and Longer Context

In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 161-170

Tobias van der Werff, Rik van Noord, Antonio Toral

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{van-der-werff-etal-2022-automatic,
    title = "Automatic Discrimination of Human and Neural Machine Translation: A Study with Multiple Pre-Trained Models and Longer Context",
    author = "van der Werff, Tobias  and
      van Noord, Rik  and
      Toral, Antonio",
    booktitle = "Proceedings of the 23rd Annual Conference of the European Association for Machine Translation",
    month = jun,
    year = "2022",
    address = "Ghent, Belgium",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2022.eamt-1.19",
    pages = "161--170",
    abstract = "We address the task of automatically distinguishing between human-translated (HT) and machine translated (MT) texts. Following recent work, we fine-tune pre-trained language models (LMs) to perform this task. Our work differs in that we use state-of-the-art pre-trained LMs, as well as the test sets of the WMT news shared tasks as training data, to ensure the sentences were not seen during training of the MT system itself. Moreover, we analyse performance for a number of different experimental setups, such as adding translationese data, going beyond the sentence-level and normalizing punctuation. We show that (i) choosing a state-of-the-art LM can make quite a difference: our best baseline system (DeBERTa) outperforms both BERT and RoBERTa by over 3{\%} accuracy, (ii) adding translationese data is only beneficial if there is not much data available, (iii) considerable improvements can be obtained by classifying at the document-level and (iv) normalizing punctuation and thus avoiding (some) shortcuts has no impact on model performance.",
}

GroningenNITS2022

May 13th, 2022

Creating, Evaluating and Enriching Parallel Corpora in the MaCoCu Project.

In Groningen conference NITS2022, May 2022

Rik van Noord and Antonio Toral

PDF BibTex

DublinACL2022

May 27, 2022

Human evaluation of web-crawled parallel corpora for machine translation

Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 32 - 41

Gema Ramírez-Sánchez, Marta Bañón, Jaume Zaragoza-Bernabeu, Sergio Ortiz-Rojas

PDF

View BibTeX

Select the text to copy the citation

@inproceedings{ramirez-sanchez-etal-2022-human,
    title = "Human evaluation of web-crawled parallel corpora for machine translation",
    author = "Ram{\'\i}rez-S{\'a}nchez, Gema  and
      Ba{\~n}{\'o}n, Marta  and
      Zaragoza-Bernabeu, Jaume  and
      Ortiz Rojas, Sergio",
    booktitle = "Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.humeval-1.4",
    doi = "10.18653/v1/2022.humeval-1.4",
    pages = "32--41",
    abstract = "Quality assessment has been an ongoing activity of the series of ParaCrawl efforts to crawl massive amounts of parallel data from multilingual websites for 29 languages. The goal of ParaCrawl is to get parallel data that is good for machine translation. To prove so, both, automatic (extrinsic) and human (intrinsic and extrinsic) evaluation tasks have been included as part of the quality assessment activity of the project. We sum up the various methods followed to address these evaluation tasks for the web-crawled corpora produced and their results. We review their advantages and disadvantages for the final goal of the ParaCrawl project and the related ongoing project MaCoCu.",
}

MarseilleLREC2022

20-25 June 2022

Bicleaner AI: Bicleaner Goes Neural

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 824-831

Jaume Zaragoza-Bernabeu, Marta Bañón, Gema Ramírez-Sanchez, Sergio Ortiz-Rojas

PDF

View BibTeX

Select the text to copy the citation

@InProceedings{zaragozabernabeu-EtAl:2022:LREC,
  author    = {Zaragoza-Bernabeu, Jaume  and  RamÃÂrez-SÃÂ¡nchez, Gema  and  BaÃÂ±ÃÂ³n, Marta  and  Ortiz Rojas, Sergio},
  title     = {Bicleaner AI: Bicleaner Goes Neural},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {824--831},
  abstract  = {This paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The tool, which now implements a new neural classifier, uses state-of-the-art techniques based on pre-trained transformer-based language models fine-tuned on a binary classification task. After that, parallel corpus filtering is performed, discarding the sentences that have lower probability of being mutual translations. Our experiments, based on the training of neural machine translation (NMT) with corpora filtered using Bicleaner AI for two different scenarios, show significant improvements in translation quality compared to the previous version of the tool which implemented a classifier based on Extremely Randomized Trees.},
  url       = {https://aclanthology.org/2022.lrec-1.87}
}

MarseilleLREC2022

20-25 June 2022

The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1584-1594

Taja Kuzman, Peter Rupnik, Nikola Ljubešić

PDF

View BibTeX

Select the text to copy the citation

@InProceedings{kuzman-rupnik-ljubei:2022:LREC,
  author    = {Kuzman, Taja  and  Rupnik, Peter  and  LjubeÃÂ¡iÃâ¡, Nikola},
  title     = {The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {1584--1594},
  abstract  = {This paper presents a new training dataset for automatic genre identification GINCO, which is based on 1,125 crawled Slovenian web documents that consist of 650,000 words. Each document was manually annotated for genre with a new annotation schema that builds upon existing schemata, having primarily clarity of labels and inter-annotator agreement in mind. The dataset consists of various challenges related to web-based data, such as machine translated content, encoding errors, multiple contents presented in one document etc., enabling evaluation of classifiers in realistic conditions. The initial machine learning experiments on the dataset show that (1) pre-Transformer models are drastically less able to model the phenomena, with macro F1 metrics ranging around 0.22, while Transformer-based models achieve scores of around 0.58, and (2) multilingual Transformer models work as well on the task as the monolingual models that were previously proven to be superior to multilingual models on standard NLP tasks.},
  url       = {https://aclanthology.org/2022.lrec-1.170}
}

MarseilleLREC2022

June 25, 2022

Building Domain-specific Corpora from the Web: the Case of European Digital Service Infrastructures

Proceedings of the 15th Workshop on Building and Using Comparable Corpora (BUCC 2022) @LREC2022, pages 23-32

Rik van Noord, Cristian García-Romero, Miquel Espla-Gomis, Leopoldo Pla, Antonio Toral

PDF

View BibTeX

Select the text to copy the citation

@InProceedings{vannoord-EtAl:2022:BUCC,
  author    = {van Noord, Rik  and  GarcÃa-Romero, Cristian  and  EsplÃ -Gomis, Miquel  and  Pla Sempere, Leopoldo  and  Toral, Antonio},
  title     = {Building Domain-specific Corpora from the Web: the Case of European Digital Service Infrastructures},
  booktitle      = {Proceedings of the BUCC Workshop within LREC 2022},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {23--32},
  abstract  = {An important goal of the MaCoCu project is to improve EU-specific NLP systems that concern their Digital Service Infrastructures (DSIs). In this paper we aim at boosting the creation of such domain-specific NLP systems. To do so, we explore the feasibility of building an automatic classifier that allows to identify which segments in a generic (potentially parallel) corpus are relevant for a particular DSI. We create an evaluation data set by crawling DSI-specific web domains and then compare different strategies to build our DSI classifier for text in three languages: English, Spanish and Dutch. We use pre-trained (multilingual) language models to perform the classification, with zero-shot classification for Spanish and Dutch. The results are promising, as we are able to classify DSIs with between 70 and 80\% accuracy, even without in-language training data. A manual annotation of the data revealed that we can also find DSI-specific data on crawled texts from general web domains with reasonable accuracy. We publicly release all data, predictions and code, as to allow future investigations in whether exploiting this DSI-specific data actually leads to improved performance on particular applications, such as machine translation.},
  url       = {https://aclanthology.org/2022.bucc-1.4}
}

TilburgCLIN32

June 25, 2022

The MaCoCu project: Massive Collection, Curation and Evaluation of Monolingual and Bilingual Data

Computational Linguistics in the Netherlands

Rik van Noord, Antonio Toral

PDF BibTex

Presentations

MarseilleDockerCon 2022

June 17, 2022

The Bitextor pipeline

Online event

Leopoldo Pla Sempere

Watch the presentation