A core element of any data retrieval operation is the use of a component known as a retriever. Its job is to retrieve the relevant content for a given query. In the AI era, retrievers have been used as part of RAG pipelines. The approach is straightforward: retrieve relevant documents, feed them to an LLM, and let the model generate an answer based on that context.While retrieval might have seemed like a solved problem, it actually wasn't solved for modern agentic AI workflows.In research published this week, Databricks introduced Instructed Retriever, a new architecture that the company claims delivers up to 70% improvement over traditional RAG on complex, instruction-heavy enterprise question-answering tasks. The difference comes down to how the system understands and uses metadata."A lot of the systems that were built for retrieval before the age of large language models were really built for humans to use, not for agents to use," Michael Bendersky, a research director at Databricks, told VentureBeat. "What we found is that in a lot of cases, the errors that are coming from the agent are not because the agent is not able to reason about the data. It's because the agent is not able to retrieve the right data in the first place."What's missing from traditional RAG retrieversThe core problem stems from how traditional RAG handles what Bendersky calls "system-level specifications." These include the full context of user instructions, metadata schemas, and examples that define what a successful retrieval should look like.In a typical RAG pipeline, a user query gets converted into an embedding, similar documents are retrieved from a vector database, and those results feed into a language model for generation. The system might incorporate basic filtering, but it fundamentally treats each query as an isolated text-matching exercise.This approach breaks down with real enterprise data. Enterprise documents often include rich metadata like timestamps, author information, product ratings, document types, and domain-specific attributes. When a user asks a question that requires reasoning over these metadata fields, traditional RAG struggles.Consider this example: "Show me five-star product reviews from the past six months, but exclude anything from Brand X." Traditional RAG cannot reliably translate that natural language constraint into the appropriate database filters and structured queries."If you just use a traditional RAG system, there's no way to make use of all these different signals about the data that are encapsulated in metadata," Bendersky said. "They need to be passed on to the agent itself to do the right job in retrieval."The issue becomes more acute as enterprises move beyond simple document search to agentic workflows. A human using a search system can reformulate queries and apply filters manually when initial results miss the mark. An AI agent operating autonomously needs the retrieval system itself to understand and execute complex, multi-faceted instructions.How Instructed Retriever worksDatabricks' approach fundamentally redesigns the retrieval pipeline. The system propagates complete system specifications through every stage of both retrieval and generation. These specifications include user instructions, labeled examples and index schemas.The architecture adds three key capabilities:Query decomposition: The system breaks complex, multi-part requests into a search plan containing multiple keyword searches and filter instructions. A request for "recent FooBrand products excluding lite models" gets decomposed into structured queries with appropriate metadata filters. Traditional systems would attempt a single semantic search.Metadata reasoning: Natural language instructions get translated into database filters. "From last year" becomes a date filter, "five-star reviews" becomes a rating filter. The system understands both what metadata is available and how to match it to user intent.Contextual relevance: The reranking stage uses the full context of user instructions to boost documents that match intent, even when keywords are a weaker match. The system can prioritize recency or specific document types based on specifications rather than just text similarity."The magic is in how we construct the queries," Bendersky said. "We kind of try to use the tool as an agent would, not as a human would. It has all the intricacies of the API and uses them to the best possible ability."Contextual memory vs. retrieval architectureOver the latter half of 2025, there was an industry shift away from RAG toward agentic AI memory, sometimes referred to as contextual memory. Approaches including Hindsight and A-MEM emerged offering the promise of a RAG-free future.Bendersky argues that contextual memory and sophisticated retrieval serve different purposes. Both are necessary for enterprise AI systems."There's no way you can put everything in your enterprise into your contextual memory," Bendersky noted. "You kind of need both. You need contextual memory to provide specifications, to provide schemas, but still you need access to the data, which may be distributed across multiple tables and documents."Contextual memory excels at maintaining task specifications, user preferences, and metadata schemas within a session. It keeps the "rules of the game" readily available. But the actual enterprise data corpus exists outside this context window. Most enterprises have data volumes that exceed even generous context windows by orders of magnitude.Instructed Retriever leverages contextual memory for system-level specifications while using retrieval to access the broader data estate. The specifications in context inform how the retriever constructs queries and interprets results. The retrieval system then pulls specific documents from potentially billions of candidates.This division of labor matters for practical deployment. Loading millions of documents into context is neither feasible nor efficient. The metadata alone can be substantial when dealing with heterogeneous systems across an enterprise. Instructed Retriever solves this by making metadata immediately usable without requiring it all to fit in context.Availability and practical considerationsInstructed Retriever is available now as part of Databricks Agent Bricks; it's built into the Knowledge Assistant product. Enterprises using Knowledge Assistant to build question-answering systems over their documents automatically leverage the Instructed Retriever architecture without building custom RAG pipelines.The system is not available as open source, though Bendersky indicated Databricks is considering broader availability. For now, the company's strategy is to release benchmarks like StaRK-Instruct to the research community while keeping the implementation proprietary to its enterprise products.The technology shows particular promise for enterprises with complex, highly structured data that includes rich metadata. Bendersky cited use cases across finance, e-commerce, and healthcare. Essentially any domain where documents have meaningful attributes beyond raw text can benefit."What we've seen in some cases kind of unlocks things that the customer cannot do without it," Bendersky said.He explained that without Instructed Retriever, users have to do more data management tasks to put content into the right structure and tables in order for an LLM to properly retrieve the correct information.“Here you can just create an index with the right metadata, point your retriever to that, and it will just work out of the box,” he said.What this means for enterprise AI strategyFor enterprises building RAG-based systems today, the research surfaces a critical question: Is your retrieval pipeline actually capable of the instruction-following and metadata reasoning your use case requires?The 70% improvement Databricks demonstrates isn't achievable through incremental optimization. It represents an architectural difference in how system specifications flow through the retrieval and generation process. Organizations that have invested in carefully structuring their data with detailed metadata may find that traditional RAG is leaving much of that structure's value on the table.For enterprises looking to implement AI systems that can reliably follow complex, multi-part instructions over heterogeneous data sources, the research indicates that retrieval architecture may be the critical differentiator. Those still relying on basic RAG for production use cases involving rich metadata should evaluate whether their current approach can fundamentally meet their requirements. The performance gap Databricks demonstrates suggests that a more sophisticated retrieval architecture is now table stakes for enterprises with complex data estates. ...read more read less