The Infrastructure Behind AI Search in Figma | Figma Blog
Figma’s AI search lets users find designs and components through text, screenshots, or layer selections. It relies on multimodal embeddings, vector nearest-neighbor search, and large-scale indexing of frames and components. The main infrastructure challenge was generating and indexing billions of embeddings efficiently while controlling costs. ## AI-Powered Search Flows - **Search for designs** indexes frames across users’ files, including unlabeled frames buried in complex files. - Users can search designs: - Lexically with a text description - Visually with a screenshot - By selecting similar Figma layers - **Search for components** enhances the Assets panel with semantic matching. - A component representing 😀 can be found with terms such as “smiley,” “happy,” “face,” or “grin.” - Designers no longer need to manually add every possible keyword to component descriptions. - Components can also be found using visual queries. ## Multimodal Embeddings - An embedding model converts text or images into numerical vectors that represent their meaning. - Figma uses the open-source **CLIP** model, which places text and images in the same embedding space. - The embedding for the word “cat” should be numerically close to an embedding generated from an image of a cat. - Figma’s models were not trained on private customer files or data. - Fine-tuning used interface images from public, free Community files. - Search works by: - Generating embeddings for indexed content - Creating an embedding for the user’s query - Finding indexed vectors that are nearest to the query vector - Figma tested embeddings based on textual representations such as JSON, but image-based embeddings produced better results and supported the same workflow as screenshot search. - Layer selections are converted into screenshots before being passed to the embedding model. ## Populating the Vector Search Index - Each searchable item requires: - A thumbnail or rendered screenshot - An embedding - Metadata stored in the search index - Figma uses DynamoDB for metadata and embeddings because the workload primarily requires high-throughput key-value reads and writes. - Identifying searchable frames is difficult because unpublished frames are not directly enumerable. - Figma runs a headless, server-side version of its C++ editor in asynchronous jobs to discover frames within files. - These jobs use server-side sandboxing techniques to safely run the editor. Figma’s approach combines CLIP-based multimodal representations, server-side rendering, asynchronous processing, and scalable vector storage to make visual and semantic search practical across large design systems.
Read original(opens in new tab)