AI agents and no-code pipelines: Databricks Agent Bricks and Lakeflow Designer

As part of the Data + AI Summit, Databricks announced a number of new features, including the launch of Agent Bricks, an automated method for creating customized AI agents for companies, and the preview of Lakeflow Designer, which allows data pipelines to be created with AI support using a visual drag-and-drop interface. In addition, the open source tool MLflow, designed as a platform for the lifecycle management of machine learning projects, has reached version 3.0.

Customized AI agents with domain expertise

Based on the data available in a company, the new Agent Bricks are designed to provide targeted support in providing cost-efficient and trustworthy AI agents and optimizing them automatically. In addition to the individual customer data, the agent bricks automatically generate domain-specific synthetic data and task-related benchmarks to relieve the company's teams of the necessary – and often laborious – optimization process. The core task here is to precisely understand the customer data and their domain in order to subsequently build AI agents for specific tasks. "For example, a customer might want to provide an agent system that can answer questions about the products they offer," explains Joel Minnick, Vice President Marketing at Databricks.

Based on this objective, the agent bricks then generate a series of so-called LLM judges and synthetic data matching the customer's data domain. Using a catalog of questions and an iterative question-and-answer process in which the LLM judges act as evaluators, the system then gradually approaches the customer's objective. In doing so, the agent bricks draw on all the optimization techniques and ML models available in the Databricks platform –, including those that are still reserved for the research department at Databricks.

Customers can decide the scope of the iteration process themselves and weigh up the quality of the resulting AI agent for productive use against the costs required. Just how quickly a suitable result can be achieved has been demonstrated in the trials of the first test customers, says Minnick: "A pharmaceutical company has built an agent for knowledge extraction that has examined hundreds of thousands of clinical trial documents for data to optimize clinical research. It only took 60 minutes to build this AI agent." The Agent Bricks are now available in beta.

Reliable ETL pipelines via drag-and-drop

Databricks has also announced the preview of Lakeflow Designer. This new no-code ETL function is designed to enable non-technical users such as business analysts to create production-ready data pipelines using a visual drag-and-drop interface and a GenAI wizard in natural language. As the name suggests, Lakeflow Designer is based on the Lakeflow data engineering tool, which is now also generally available. This tool helps to build reliable data pipelines – more quickly, even with business-critical data. In order to create ETL pipelines with the same scalability and the same governance and maintainability requirements without programming knowledge, Lakeflow Designer offers the support of an AI assistant. With its help, the pipelines can be easily assembled step by step on the drag-and-drop interface.

Other announcements on the Databricks blog include a new development environment (IDE) for data engineering, which offers AI-supported coding, debugging and validation, as well as new point-and-click ingestion connectors for Lakeflow Connect, which can be connected to various services such as Google Analytics and SQL Server. The MLOps tool MLflow has also received an update and in version 3.0, which is now generally available, primarily offers extended GenAI functions – for tracing, LLM judges, application versioning and prompt management, for example.

(map)