Data Projects: Managing Data Assets at Netflix Scale
Data Projects address Netflix’s difficulty managing millions of data assets and tens of thousands of workloads as teams and employees change. They replace asset-level permissions and human-owned workload identities with project-level grants and durable, synthetic identities. This makes access easier to maintain, workflows more resilient, and newly created assets easier to organize automatically. ## The Limits of Asset-Level Permissions - Netflix historically managed access through individual ACLs on each table. - Organizational changes required updating hundreds or thousands of permissions manually. - This overwhelmed support teams and encouraged overly broad access, such as granting access to the entire company. - The model did not scale with frequent reorganizations, team changes, and ownership transfers. ## The Limits of Human-Owned Workloads - Scheduled jobs and asynchronous workloads traditionally ran under the identity of their author. - When that person changed roles or left Netflix, the workload’s permissions changed or disappeared. - Reassigning the job to another employee often introduced new permission gaps. - This created a recurring “permissions whack-a-mole” across tens of thousands of business-critical workflows. ## Data Projects as a Management Container - A Data Project groups related tables, workflows, secrets, and other assets under one logical umbrella. - Teams manage permissions for the project instead of maintaining ACLs across every individual asset. - Grants can be assigned to users, groups, applications, and CI jobs. - Roles such as Contributor and Viewer define read/write or read-only access at the project level. ## Durable Project Identities - Each project receives a Netflix application identity and, optionally, an AWS IAM role. - Scheduled workloads execute as the project rather than as an individual employee. - The IAM role supports AWS use cases such as Spark jobs on Amazon EMR. - Privileged project members can assume the project identity from laptops or notebooks for testing and troubleshooting. - This provides a development context that matches the identity used in production. ## Gravity and Automatic Asset Organization - Assets created by workloads running under a project identity are automatically added to that project. - For example, tables created by a Maestro workflow become project assets without extra configuration. - This “gravity” keeps related outputs organized and makes future access and discovery easier. - Newly created assets inherit the project’s access model rather than requiring separate permissions. ## Securing Maestro Workflows - Maestro runs ETL pipelines, data movement jobs, machine-learning training, and other batch workloads. - As a Trusted Workload Manager, Maestro can mint identity tokens for scheduled executions. - A single workflow may be checked against table ACLs, Netflix resource policies, and AWS IAM policies. - Using a durable project identity prevents failures caused by changes to the original author’s account. - Project-scoped secrets also remain available when ownership changes. Data Projects provide Netflix with a scalable foundation for access control, workload execution, and asset ownership. Moving management from individual assets and employees to durable, team-owned projects makes the platform more stable, auditable, and resilient to organizational change.
Read original(opens in new tab)