3 posts
pinterest How Pinterest Built a Real‑Time Radar for Violative Content using AI -- 2 Listen Share Faisal Farooq | Sr. Director Trust Engineering; Aravindh Manickavasagam | Staff Technical Program Manager; Attila Dobi | Sr. Staff Data Scientist People come to Pinterest to find ideas they fe…
discord ROOST, a non-profit dedicated to digital safety, has launched two open-source tools, Coop and Osprey, to provide enterprise-grade content moderation and threat investigation capabilities to organizations of all sizes. By open-sourcing technology previously developed by industry leaders like Discord and Cove, ROOST aims to democratize access to the infrastructure required to detect, triage, and respond to online harms. This initiative shifts Trust and Safety from a proprietary competitive advantage to a shared public resource, enabling platforms to prioritize user protection without the burden of expensive enterprise software. ### Content Review and Compliance with Coop Built on technology acquired from Cove and utilized by platforms like Notion, Coop focuses on the human-in-the-loop aspect of content moderation. * The platform provides robust tools for content review, allowing teams to route specific cases to subject-matter experts for deeper analysis. * It includes built-in integration with the National Center for Missing & Exploited Children’s (NCMEC) API, automating the mandatory reporting process for child sexual abuse material (CSAM). * The interface is designed to surface relevant context and metadata, ensuring moderators can make informed decisions and take immediate action against policy violations. ### Incident Response and Investigation with Osprey Osprey is a lightweight investigation tool originally developed by Discord to manage large-scale safety incidents and platform-wide threats. * It serves as a foundation for incident response, helping safety teams understand platform trends and investigate coordinated threats like phishing or harassment campaigns. * The tool is designed to be user-friendly and accessible for grassroots communities while remaining powerful enough for established platforms. * Early adopters, including the decentralized social network Bluesky, are implementing Osprey to demonstrate that effective safety infrastructure can be scalable and resource-efficient. ### A Collaborative Model for Safety Infrastructure The launch of these tools represents a strategic shift toward a collaborative "public-interest" model for digital defense. * ROOST acquired the intellectual property of Cove and received the donation of Osprey from Discord to ensure these tools remain available as a public good. * The initiative is backed by philanthropic funding and legal support from Perkins Coie, removing the financial barriers that often prevent smaller platforms from implementing high-level safety measures. * Major industry players like Notion and Bluesky are championing the move, signaling an industry-wide push to share safety innovations rather than silo them. Platforms and developers should prepare to integrate these tools into their safety stacks as they become publicly available in the coming months. By adopting open-source infrastructure for routine tasks like NCMEC reporting and incident triage, organizations can focus their internal resources on platform-specific innovations while maintaining a high standard of digital safety.
discord Osprey is an open-source safety rules engine designed to help platforms address emerging threats without the need to build custom security infrastructure from scratch. Developed through a collaboration between industry experts and the internet.dev team, it provides a standardized framework for investigating real-time activities and deploying dynamic safety measures. By using this engine, companies can significantly reduce engineering overhead while maintaining a proactive stance against platform abuse. **The Osprey Safety Rules Engine** * Functions as an open-source toolset aimed at replacing the fragmented, "reinvent the wheel" approach many companies take toward platform safety. * Enables security and moderation teams to monitor live activity streams to identify patterns of misuse as they occur. * Allows for the rapid creation and implementation of dynamic rules, ensuring that safety protocols can evolve alongside shifting threat landscapes. **Streamlining Engineering and Investigation** * Reduces the technical debt typically associated with building and maintaining internal safety tools by providing a pre-built, scalable engine. * Facilitates deep-dive investigations into platform events, allowing teams to react to threats with minimal intervention from core engineering staff. * Promotes a collaborative approach to safety by making the underlying technology accessible to the broader developer community. For organizations looking to strengthen their security posture, Osprey offers a practical alternative to bespoke safety systems. Teams should consider integrating the engine to automate their threat responses and leverage its real-time investigation capabilities to protect their users more efficiently.