techlist.io
KR EN
discord

Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data (opens in new tab)

Discord scaled its data infrastructure to manage petabytes of data and over 2,500 models by moving beyond a standard dbt implementation. While the tool initially provided a modular and developer-friendly framework, the sheer volume of data and a high headcount of over 100 concurrent developers led to critical performance bottlenecks. To resolve these issues, Discord developed custom extensions to dbt’s core functionality, successfully reducing compilation times and automating complex data transformations. ### Strategic Adoption of dbt * Discord integrated dbt into its stack to leverage software engineering principles like modular design and code reusability for SQL transformations. * The tool’s open-source nature allowed the team to align with Discord’s internal philosophy of community-driven engineering. * The framework offered seamless integration with other internal tools, such as the Dagster orchestrator, and provided a robust testing environment to ensure data quality. ### Scaling Bottlenecks and Performance Issues * The project grew to a size where recompiling the entire dbt project took upwards of 20 minutes, severely hindering developer velocity. * Standard incremental materialization strategies provided by dbt proved inefficient for the petabyte-scale data volumes generated by millions of concurrent users. * Developer workflows often collided, resulting in teams inadvertently overwriting each other’s test tables and creating data silos or inconsistencies. * The lack of specialized handling for complex backfills threatened the organization’s ability to deliver timely and accurate insights. ### Engineering Custom Extensions for Growth * The team built a provider-agnostic layer over Google BigQuery to streamline complex calculations and automate massive data backfills. * Custom optimizations were implemented to prevent breaking changes during the development cycle, ensuring that 100+ developers could work simultaneously without friction. * By extending dbt’s core, Discord transformed slow development cycles into a rapid, automated system capable of serving as the backbone for their global analytics infrastructure. For organizations operating at massive scale, standard open-source tools often require custom-built orchestration and optimization layers to remain viable. Prioritizing the automation of backfills and optimizing compilation logic is essential to maintaining developer productivity and data integrity when dealing with thousands of models and petabytes of information.

scalabilitydata-engineeringdbtdagster+3
discord

Announcing Discord’s Social SDK, Helping Power Your Game’s Social Experiences (opens in new tab)

Discord has launched the Discord Social SDK, a free toolkit that allows game developers to integrate Discord's social infrastructure directly into their titles to drive player engagement and discovery. The SDK enables features like friends lists and messaging for all players, regardless of whether they have a Discord account, while offering deeper integration for those who choose to link their profiles. By tapping into Discord’s ecosystem of over 200 million monthly active users, the tool aims to help developers overcome discovery challenges in a market where a small number of franchises dominate total playtime. **Solving Discovery Through Social Connectivity** * With 20,000 games launching annually, Discord aims to leverage the fact that 50% of its users discover a new game on the platform every month. * Internal data suggests that playing with at least one friend increases gameplay session lengths sevenfold. * The SDK facilitates "friend-to-friend influence," where 28% of users launch a game within an hour of watching a friend stream it on the platform. **Core SDK Integration Features** * **Unified Friends List:** Synchronizes in-game and Discord friends lists, allowing players to maintain connections both inside and outside the game environment. * **Deep-linked Game Invites:** Enables players to send invites from their in-game list that allow Discord friends to join a specific party, lobby, or session with a single click. * **Rich Presence:** Displays real-time gaming activity on Discord profiles across PC, console, and mobile, serving as a passive discovery tool for a player's social circle. * **Flexible Account Requirements:** Developers can provide a unified social experience to all players without requiring a Discord login, though account linking unlocks more persistent social features. **Advanced Communication Tools in Beta** * **Cross-Platform Messaging:** A closed beta feature that allows players to continue game-related conversations across desktop, console, and mobile devices. * **Linked Channels:** Enables developers to sync in-game chat with specific Discord server channels, providing persistent messaging for guilds and squads. * **Integrated Voice Chat:** Grants developers access to Discord’s proprietary high-quality audio technology to power in-game voice communications. **Technical Specifications and Partnerships** * The SDK is currently available for C++, Unreal Engine, and Unity developers. * Initial support covers Windows 11+ and macOS, with console and mobile support listed as coming soon. * Early adoption partners include major studios such as Theorycraft Games, Facepunch Studios, 1047 Games, and Scopely. By providing these social tools for free, Discord is positioning itself as the foundational social layer for the gaming industry. Developers looking to capitalize on existing social graphs and improve player retention should consider integrating the SDK to bridge the gap between their game and the "digital living rooms" where players already spend their time.

voice-chatdiscord-social-sdkgame-discoveryrich-presence+4
gitlab

Announcing general availability for GitLab Duo Agent Platform (opens in new tab)

The GitLab Duo Agent Platform has reached general availability, marking a shift from basic AI code assistance to comprehensive agentic automation across the entire software development lifecycle. By orchestrating intelligent agents to handle complex tasks like security analysis and planning, the platform aims to resolve the "AI paradox" where faster code generation often creates downstream bottlenecks in review and deployment. ### Usage-Based Economy via GitLab Credits * GitLab is introducing "GitLab Credits," a virtual currency used to power the platform’s usage-based AI features. * Premium and Ultimate subscribers receive monthly credits ($12 and $24 respectively) at no additional cost to facilitate immediate adoption. * Organizations can manage a shared pool of credits or opt for on-demand monthly billing, with existing Duo Enterprise contracts eligible for conversion into credits. ### Agentic Chat and Contextual Orchestration * The Duo Agentic Chat provides a unified experience across the GitLab Web UI and various IDEs, including VS Code, JetBrains, Cursor, and Windsurf. * The chat utilizes multi-step reasoning to perform actions autonomously, drawing from the context of issues, merge requests, pipelines, and security findings. * Capabilities extend beyond code generation to include infrastructure-as-code (IaC) creation, pipeline troubleshooting, and explaining vulnerability reachability. ### Specialized Foundational and Custom Agents * **Foundational Agents:** Pre-built specialists designed for specific roles, such as the Planner Agent for breaking down work and the Security Analyst Agent for triaging vulnerabilities. * **Custom Agents:** Developed through a central AI Catalog, these allow teams to build and share agents that adhere to organization-specific engineering standards and guardrails. * **External Agents:** Native integration of third-party AI tools, such as Anthropic’s Claude Code and OpenAI’s Codex CLI, provides access to external LLM capabilities within the governed GitLab environment. ### Automated End-to-End Flows * The platform introduces "Flows," which are multi-step agentic sequences designed to automate repeatable transitions in the development cycle. * The "Issue to Merge Request" flow builds structured code changes directly from defined requirements to jumpstart development. * Specialized CI/CD flows help teams modernize pipeline configurations and automatically analyze and suggest fixes for failed pipeline runs. * The Code Review flow streamlines the feedback loop by providing AI-native analysis of merge request comments and code changes. To maximize the impact of agentic AI, organizations should move beyond basic chat interactions and begin integrating these specialized agents into their broader orchestration workflows to eliminate manual handoffs between planning, coding, and security.

database-designgitlabci-cdagentic-ai+4
datadog

Robust statistical distances for machine learning | Datadog (opens in new tab)

Datadog has introduced Toto, a new open-weights foundation model specifically designed for time-series forecasting and anomaly detection within observability contexts. While general-purpose time-series models often struggle with the unique volatility and high-frequency patterns of IT telemetry, Toto is pre-trained on a massive dataset of 500 billion observations to provide superior zero-shot performance. This release, accompanied by the BOOM benchmark, addresses the critical need for specialized AI tools capable of handling the complexity of modern cloud infrastructure. ### Toto Model Architecture and Training * Toto utilizes a decoder-only transformer architecture, adapting large language model (LLM) principles to the domain of continuous numerical data. * The model employs a "patching" mechanism, which groups multiple time-series data points into single tokens to improve computational efficiency and allow the model to capture longer historical dependencies. * It incorporates Rotary Positional Embeddings (RoPE) to better handle sequences of varying lengths and maintain temporal relationships across different frequencies. * Training was conducted on a curated dataset of 500 billion anonymized data points from real-world observability metrics, including CPU usage, memory consumption, and network traffic. ### Specialized Observability Features * Unlike existing models like TimesFM or Chronos, which are trained on diverse but general datasets like weather or retail trends, Toto is optimized for the specific "spikiness" and abrupt level shifts common in IT environments. * The model supports zero-shot forecasting, allowing users to generate predictions for new metrics immediately without the need for expensive or time-consuming fine-tuning. * Toto is designed to handle varying sampling rates, from one-second intervals to hourly aggregations, making it versatile across different infrastructure layers. * The open-weights release on Hugging Face allows researchers and engineers to integrate the model into their own AIOps workflows or private cloud environments. ### The BOOM Evaluation Framework * Datadog released the Benchmarking Observability Models (BOOM) framework to provide a standardized method for evaluating time-series models on infrastructure-specific tasks. * BOOM focuses on metrics that represent real-world operational challenges, such as seasonal traffic patterns and sudden system failures. * Comparative testing shows that Toto consistently outperforms general-purpose models in Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) when applied to observability datasets. * The benchmark provides a transparent way for the industry to measure progress in time-series foundation models, moving beyond generic datasets that do not reflect the realities of microservices and distributed systems. Organizations looking to automate capacity planning, optimize cloud spend, or implement intelligent alerting should consider adopting Toto for their time-series analysis. By utilizing the open-weights model alongside the BOOM benchmark, teams can achieve high-accuracy forecasting and objective performance validation without the overhead of building specialized models from scratch.

machine-learningopen-sourcedatadogobservability+1
discord

Osprey: Open Sourcing our Rule Engine (opens in new tab)

Osprey is an open-source safety rules engine designed to help platforms address emerging threats without the need to build custom security infrastructure from scratch. Developed through a collaboration between industry experts and the internet.dev team, it provides a standardized framework for investigating real-time activities and deploying dynamic safety measures. By using this engine, companies can significantly reduce engineering overhead while maintaining a proactive stance against platform abuse. **The Osprey Safety Rules Engine** * Functions as an open-source toolset aimed at replacing the fragmented, "reinvent the wheel" approach many companies take toward platform safety. * Enables security and moderation teams to monitor live activity streams to identify patterns of misuse as they occur. * Allows for the rapid creation and implementation of dynamic rules, ensuring that safety protocols can evolve alongside shifting threat landscapes. **Streamlining Engineering and Investigation** * Reduces the technical debt typically associated with building and maintaining internal safety tools by providing a pre-built, scalable engine. * Facilitates deep-dive investigations into platform events, allowing teams to react to threats with minimal intervention from core engineering staff. * Promotes a collaborative approach to safety by making the underlying technology accessible to the broader developer community. For organizations looking to strengthen their security posture, Osprey offers a practical alternative to bespoke safety systems. Teams should consider integrating the engine to automate their threat responses and leverage its real-time investigation capabilities to protect their users more efficiently.

open-sourcetrust-and-safetyreal-time-monitoringosprey+3