github·Jun 15, 2026·3 min readCurated summary

Accelerating researchers and developers building multilingual AI with a new open dataset

GitHub has released the GitHub Multilingual Repositories Dataset, an open metadata resource for finding public repositories containing non-English developer content. Covering more than 40 million repositories, it classifies language use in READMEs, issues, and pull requests, helping researchers build multilingual AI tools and study representation in open source. GitHub emphasizes that the dataset is a discovery tool—not definitive language ground truth—and releases it under CC0-1.0. ## Dataset Scope and Contents - Contains over 80 million classification rows across more than 40 million public repositories. - Classifies: - READMEs - The most-commented issue - The most-commented pull request - Uses the first 150 characters of each text source, excluding samples shorter than 20 characters. - Includes classifications and confidence scores from: - fastText - Google’s CLD3 - lingua-py - Only classifications with confidence above 0.5 are included. - Repository metadata includes creation date, disk usage, stars, forks, primary language, SPDX license, issue and pull request counts, and snapshot date. ## Why Multiple Classifiers Are Exposed - GitHub does not combine the three classifiers into one definitive language label. - Classifiers differ in language coverage and confidence calibration, particularly for lower-resource languages. - Users can choose their own precision and recall strategy: - Require agreement among all classifiers for high-precision research. - Use a single classifier for broader exploratory studies. ## Language Patterns in Developer Content - Language distribution varies by repository content type. - Korean is the most common non-English language in issue text but ranks only fifth in README classifications. - Portuguese is the leading non-English README language, appearing in more than 3 million repositories. - These differences show that developer communities may use different languages for documentation, issue discussions, and code collaboration. ## Potential Applications - Find repositories with documentation or collaboration in particular languages. - Study how multilingual communities use READMEs, issues, and pull requests. - Create evaluation datasets for coding assistants, documentation generators, and code review tools. - Measure representation of European and other underrepresented languages in open source. - Provide evidence for expanding language support in developer tools and AI systems. ## Limitations and Responsible Use - Repository text is often short and may contain badges, commands, code, usernames, templates, or multiple languages. - A 150-character sample may not represent the language of an entire repository. - Classifier performance varies, especially for lower-resource languages. - The dataset should not be treated as a ground-truth language-identification benchmark. - It provides repository-level signals and should not be used to infer sensitive characteristics of repository owners, contributors, or communities. ## Importance for Multilingual AI - Many European languages are underrepresented in the data used to train and evaluate AI systems. - Developer content provides domain-specific examples of software collaboration, including installation guidance, bug reports, feature requests, and code reviews. - The dataset can help identify language gaps, improve evaluation, and support more inclusive AI tools for developers worldwide. GitHub recommends using the CC0-licensed dataset to conduct research, build evaluation sets, develop tools, and improve its classifications. Its transparency and multiple confidence signals allow users to tailor the data to their own research needs while accounting for its limitations.

Read original(opens in new tab)

Multilingual AiGithubNatural Language ProcessingOpen Dataset+4

google·May 1, 2025Original article

Amplify Initiative: Localized data for globalized AI (opens in new tab)

The Amplify Initiative by Google Research addresses the critical lack of linguistic and cultural diversity in generative AI training data by establishing an open, community-based platform for localized data collection. By partnering with regional experts to co-create structured, high-quality datasets, the initiative aims to ensure AI models are both representative and effective in solving local challenges across health, finance, and education. This approach shifts data collection from a top-down model to a participatory framework that prioritizes responsible, locally respectful practices in the Global South. ## The Amplify Platform Framework The initiative is designed to bridge the gap between global AI capabilities and local needs through three core pillars: * **Participatory Co-creation:** Researchers and local communities collaborate to define specific data needs, ensuring the resulting datasets address region-specific problems like financial literacy or localized health misinformation. * **Open Access for Innovation:** The platform provides high-quality, multilingual datasets suitable for fine-tuning and evaluating models, specifically empowering developers in the Global South to build tools for their own communities. * **Author Recognition:** Contributors receive tangible rewards, including professional certificates, research acknowledgments, and data authorship attribution, creating a sustainable ecosystem for expert participation. ## Pilot Implementation in Sub-Saharan Africa To test the methodology, Google Research partnered with Makerere University’s AI Lab in Uganda to conduct an on-the-ground pilot program. * **Expert Onboarding:** The program trained 259 experts across Ghana, Kenya, Malawi, Nigeria, and Uganda through a combination of in-person workshops and app-based modules. * **Dataset Composition:** The pilot resulted in 8,091 annotated adversarial queries across seven languages, covering salient domains such as education and finance. * **Adversarial Focus:** By focusing on adversarial queries, the team captured localized nuances of potential AI harms, including regional stereotypes and specialized advice that generic models often miss. ## Technical Workflow and App-Based Methodology The initiative utilizes a structured technical pipeline to scale data collection while maintaining high quality and privacy. * **Privacy-Preserving Android App:** A dedicated app serves as the primary interface for training, data creation, and annotation, allowing experts to contribute from their own environments. * **Automated Validation:** The app includes built-in feedback loops that use automated checks to ensure queries are relevant and to prevent the submission of semantically similar or duplicate entries. * **Domain-Specific Annotation:** Experts are provided with specialized annotation topics tailored to their professional backgrounds, ensuring that the metadata for each query is technically accurate and contextually relevant. The Amplify Initiative provides a scalable blueprint for building inclusive AI by empowering experts in the Global South to define their own data needs. As the project expands to India and Brazil, it offers a vital resource for developers seeking to fine-tune models for local contexts and improve the safety and relevance of AI on a global scale.

Multilingual AiAILarge Language ModelsGenerative AI+5