InstructPipe: Generating Visual Blocks pipelines with human instructions and LLMs (opens in new tab)

InstructPipe is a research prototype designed to simplify machine learning prototyping by generating visual programming pipelines directly from natural language instructions. By leveraging a multi-stage large language model (LLM) framework, the system automates the selection and connection of nodes to lower the barrier for novice users. The result is a streamlined workflow that transforms abstract text commands into functional, editable node-graph diagrams within the Visual Blocks for ML environment.

Pipeline Representation and Efficiency

• Visual Blocks pipelines are structured as Directed Acyclic Graphs (DAGs) and are typically stored in a verbose JSON format.
• To improve LLM performance, InstructPipe utilizes a "pseudocode" intermediate representation that is highly token-efficient, compressing pipeline data from 2.8k tokens down to approximately 123 tokens.
• This pseudocode defines output variables, unique node IDs, and node types while specifying arguments such as input images or text prompts (e.g., pali_1_out:pali(image=input_image_1, prompt=input_text_1)).

Two-Stage LLM Refinement

• The Node Selector module acts as a high-level filter, using brief node descriptions to identify a relevant subset of tools from the library based on the user's intent.
• The Code Writer module receives the filtered list and uses detailed node configurations—including specific input/output data types and usage examples—to draft the actual pipeline logic.
• This dual-prompting strategy mimics human developer behavior by first scanning documentation categories and then focusing on specific function requirements to ensure accurate node connections.

Interpretation and Execution

• A dedicated Code Interpreter parses the generated pseudocode to reconstruct the final JSON-formatted pipeline required by the visual editor.
• The system renders the resulting graph in an interactive workspace, allowing users to immediately execute, modify, or extend the machine learning workflow.
• Technical evaluations indicate that this approach effectively supports multimodal pipelines, such as those involving the PaLI model for vision-language tasks, while significantly reducing the learning curve for new users.

InstructPipe demonstrates how LLMs can bridge the gap between high-level human intent and low-code visual programming environments. For developers and researchers, this approach mitigates the "blank canvas" problem, allowing for faster experimentation and the rapid prototyping of complex machine learning architectures through simple text-based collaboration.