Scaling self-serve analytics: The tools empowering 5,000 employees
Datadog scaled self-serve analytics from 200 to 5,000 employees by building an open-source-based platform around three pillars: trusted data, accessible tools, and organizational knowledge. The goal is to let employees answer routine questions and make informed decisions without relying on a centralized Data & Analytics team. This approach combines a single source of truth, self-service data pipelines and transformations, data discovery, quality monitoring, and training. ## The Purpose of Self-Serve Analytics - Datadog’s mission is to “empower everyone at Datadog to make data-informed decisions on their own.” - Self-service allows Data & Analytics teams to focus on higher-value initiatives instead of handling every request. - The organization identified three primary user profiles: - **Analytics Explorers:** Need discoverable data and ready-made reports. - **Analytics Builders:** Create reports and run advanced queries. - **Analytics Experts:** Expose new data, maintain business logic, and manage quality. ## Data as a Single Source of Truth - Datadog centralizes product, operational, and business data so consumers work from the same version of reality. - Its “Bring Your Own Data” (BYOD) tool lets teams expose their own data for analytics. - The shared data layer supports BI tools, notebooks, data discovery, programmatic access, and machine-learning models. - Trust depends on: - Consistent naming and modeling conventions. - Comprehensive documentation. - Continuous data-quality monitoring. ## Self-Serve Data Intake - Teams can connect internal and third-party data sources through integrations and BYOD. - The platform provides scheduling and a user interface for exposing or requesting datasets. - Pipeline observability covers: - Pipeline execution. - Data quality. - Actionable alerts when failures occur. ## Self-Serve Transformation - Analysts manage their departments’ business logic using SQL and dbt. - The development environment integrates with workflow management, metadata, and pipeline-run systems. - Enforced conventions keep the shared modeling layer consistent and understandable as more analysts contribute. - Analysts can inspect lineage, pipeline runs, quality checks, and alerts. ## Data Discovery and Metadata - Every employee can browse datasets and fields in the central data platform. - Search capabilities help users identify which data can answer a particular question. - Metadata explains: - The dataset’s origin and owner. - Definitions and intended meaning. - Where the data is used. - Sensitivity and reliability. - This context helps employees determine whether data is both relevant and trustworthy. ## Supporting Adoption - Tools alone are insufficient; Datadog also provides data knowledge, support, and training. - The Data & Analytics organization acknowledges that self-service has limits and works to mitigate risks such as misunderstanding data or applying incorrect business logic. - Success is tracked through adoption and the effectiveness of the overall self-service strategy. Datadog’s experience suggests that self-serve analytics scales best when data is treated as a product: centralized, documented, observable, and accessible through tools designed for users with different levels of expertise.
Read original(opens in new tab)