Accelerating researchers and developers building multilingual AI with a new open dataset (opens in new tab)
GitHub has released the GitHub Multilingual Repositories Dataset, an open metadata resource for finding public repositories containing non-English developer content. Covering more than 40 million repositories, it classifies language use in READMEs, issues, and pull requests, helping researchers build multilingual AI tools and study representation in open source. GitHub emphasizes that the dataset is a discovery tool—not definitive language ground truth—and releases it under CC0-1.0. ## Dataset Scope and Contents - Contains over 80 million classification rows across more than 40 million public repositories. - Classifies: - READMEs - The most-commented issue - The most-commented pull request - Uses the first 150 characters of each text source, excluding samples shorter than 20 characters. - Includes classifications and confidence scores from: - fastText - Google’s CLD3 - lingua-py - Only classifications with confidence above 0.5 are included. - Repository metadata includes creation date, disk usage, stars, forks, primary language, SPDX license, issue and pull request counts, and snapshot date. ## Why Multiple Classifiers Are Exposed - GitHub does not combine the three classifiers into one definitive language label. - Classifiers differ in language coverage and confidence calibration, particularly for lower-resource languages. - Users can choose their own precision and recall strategy: - Require agreement among all classifiers for high-precision research. - Use a single classifier for broader exploratory studies. ## Language Patterns in Developer Content - Language distribution varies by repository content type. - Korean is the most common non-English language in issue text but ranks only fifth in README classifications. - Portuguese is the leading non-English README language, appearing in more than 3 million repositories. - These differences show that developer communities may use different languages for documentation, issue discussions, and code collaboration. ## Potential Applications - Find repositories with documentation or collaboration in particular languages. - Study how multilingual communities use READMEs, issues, and pull requests. - Create evaluation datasets for coding assistants, documentation generators, and code review tools. - Measure representation of European and other underrepresented languages in open source. - Provide evidence for expanding language support in developer tools and AI systems. ## Limitations and Responsible Use - Repository text is often short and may contain badges, commands, code, usernames, templates, or multiple languages. - A 150-character sample may not represent the language of an entire repository. - Classifier performance varies, especially for lower-resource languages. - The dataset should not be treated as a ground-truth language-identification benchmark. - It provides repository-level signals and should not be used to infer sensitive characteristics of repository owners, contributors, or communities. ## Importance for Multilingual AI - Many European languages are underrepresented in the data used to train and evaluate AI systems. - Developer content provides domain-specific examples of software collaboration, including installation guidance, bug reports, feature requests, and code reviews. - The dataset can help identify language gaps, improve evaluation, and support more inclusive AI tools for developers worldwide. GitHub recommends using the CC0-licensed dataset to conduct research, build evaluation sets, develop tools, and improve its classifications. Its transparency and multiple confidence signals allow users to tailor the data to their own research needs while accounting for its limitations.