This post is the seventh in a series about how to implement legal AI that knows your law firm. In the series, we cover the differences between LLMs and search, the elements that make a good search engine, the building blocks of agentic systems (e.g., RAG), and how to implement a system that is fully scalable, secure, and respects your firm’s unique policies and practices.
AI agents bridge the gaps between answers, decisions, and actions
To truly implement a legal AI platform that knows your firm, it not only needs an “always on” full scale retrieval system (as we discussed in the previous post), it also needs to be molded to your firm’s unique ways of working.
AI agents are swiftly becoming the vehicle of choice for delivering the power of large language models to legal professionals and their clients.
To fully understand how and why AI agents have become a big part of the answer, it is useful to recount the history in four parts: the birth of LLMs, the age of prompt engineering, the age of use cases, and finally the age of agents.
The birth of LLMs
When ChatGPT suddenly brought generative AI into the mainstream, the first response of many in the legal profession was to focus on simple AI prompts. These professionals learned to leverage AI by constructing effective system prompts that yield results from large language models. Many found immediate value in the conversational generative AI tools that are widely available. Users approached Generative AI with fairly simple prompts and refined them using the conversational nature of many AI tools. This early stage also had its growing pains, as lawyers learned that Gen AI could produce convincing-sounding responses but was prone to hallucinations, such as citing nonexistent sources.
The age of prompt engineering
The shortcomings of this simple approach led to the development of a new discipline called prompt engineering. Specialists started taking prompts to the next level by applying structure and discipline to the development of AI prompts. Users began to develop longer, more sophisticated prompts that more fully exploited AI’s ability to provide the correct answer in the right formats. Prompt engineering also allowed organizations to collect and re-use prompts in prompt libraries, as a way to capture and leverage the firm’s expertise and to apply it consistently.
A new class of product, called (perhaps derisively) “GPT wrappers,” emerged, offering a “prompt engineering as a service” approach. With these products, users can submit a fairly simple prompt, but in the background, that prompt is submitted to the LLM embedded in other prompt components designed to enhance the accuracy and quality of the generated output.
The disadvantages of this approach were myriad. The prompts generated had to pull a heavy load, cramming multiple instructions and reference data into a single prompt. Moreover, they were difficult to build and brittle to maintain; model changes or improvements could make a formerly reliable prompt completely unusable until it was redone.
Soon enough, people realized that, as with so many other workflow processes, breaking tasks down into multiple steps is often more efficient.
The age of use-cases (mirroring real-world workflows)
As “engineered” prompts became longer and more sophisticated, they also became unwieldy, with lengthy instructions for the specific task, context, and personas to be adopted, as well as examples, rules, and constraints, prescribed output formats, edge cases, and preprocessing steps.
It became clear that what was needed was a more modular approach, where the steps that go into completing a specific task could be broken down into separate sub-prompts. In fact, this is what is happening in the background of many AI systems, where a user’s initial prompt may generate a specific output, which is then fed into a second prompt for further analysis and production. For example, a prompt that requests a document be summarized might first generate a summary in outline format. That outline might, in turn, be sent to a second prompt with instructions to fill in the outline with substance, and then that output can be sent on to a third prompt with instructions for the final summarization. Like humans, AI systems are more effective at tackling large, complex problems when they are broken down into smaller, shorter parts.
At this stage, several additional techniques came into play. For example, the processing of other types of data, such as images and various formats, became the subject of some of this sub-processing. And it’s in this phase that we increasingly see the emergence of the retrieval augmented generation (RAG) techniques described above—in which an initial step in a chain of prompts involves leveraging search index to isolate a set of relevant data known to be authoritative and responsive to the task.
If you flip back to the “typical RAG flow” we described in Chapter 1 of this paper, you should now notice the multistep approach there: breaking down the workflow into several steps, tied to the specific use case requested.
The age of agents
Currently, the AI world is in a new age: the age of agents. Agents go well beyond just being a string of AI prompts; they accomplish a wider variety of tasks and do more than just prompt LLMs for an output.
Broadly speaking, there are three main characteristics of AI agents:
They do work. Agents take on some unit of work that humans used to do. An agent is an autonomous process that undertakes a defined part of a work process. Agents don’t take over the entire work responsibilities of humans, but they do autonomously take over well-defined units of work, freeing humans from that part of their roles.
They use tools. Agents go beyond simply interfacing with LLMs; they use other external tools. This may include user interactions (such as requesting additional input from a user). It includes reading and writing files—agents might not only access and work on an organization's work product files, but they can also output to those same databases of work product. They can execute network requests, interact with other systems (e.g., via APIs), generate communications, and perform other tasks.
Search engines are one of the most important tools agents can leverage. They empower the agent to retrieve additional information relevant to the task. This is critical in the law firm environment, where an agent can use a search engine to identify the most relevant and authoritative documents to use as references for feeding to an LLM for AI generation. A search engine can be tied to any relevant content index—external, such as case law or a web search, or internal, such as enterprise search.
They have autonomy. Agents can now perform multiple steps, make informed decisions, and plan ahead to complete a task. They can perform tasks in parallel, executing multiple tasks simultaneously, and they can repeat tasks multiple times, refining each iteration. They can make decisions, for example, by evaluating a result and deciding whether to go on with an analysis or not. They can communicate with other agents and be connected to them, sometimes on their own initiative, depending on the situation.
The critical role of enterprise search in continuously informing AI agents
Although the exact shape is not yet clear, all of the above developments, from simple prompts to the most sophisticated agents of today, are already or will become part of any fully featured AI workflow platform.
AI agents significantly enhance the utility and scope of generative AI within a law firm and bridge the gap between using AI to simply find an answer and actually taking action. An AI agent can execute an entire workflow, encompassing planning, evaluation, decision-making, and execution of the workflow's steps. They can also learn from past actions and improve decision-making over time.
For a law firm, the rise of workflow agents expands the extent to which AI captures and leverages a firm’s expertise. Agents embody not just the firm’s proprietary data, but also its established work processes.
In the world of legal AI agents, search is evolving from a tool that intermittently informs legal professionals’ work, and becoming the critical layer that continuously provides context and insight that drives AI workflows’ actions. Search will be “always on”—happening constantly behind the scenes to drive complex agent-driven processes that are mission critical to the organization.
As this illustration shows, all AI agents build on the data that enterprise search engines access, and extend its usefulness from simply retrieving information to identifying insights, making decisions, and taking action. Access to data shifts from being an interruption to the workflow towards being a continuous feed of real-time knowledge flowing from the firm’s various systems. And—as we discussed in the preceding chapters—the quality of the enterprise search system largely will determine the accuracy of and degree to which your AI agents, and therefore ultimately your entire AI platform, reflect your firm’s expertise.
This is true across the wide range of tasks that legal professionals do. Searching for specific contract terms, summarizing matters or sets of related documents, and analyzing precedents are all tasks that generative AI can perform, but these tasks are executed in specific contexts, such as drafting contracts, client communications, analyzing market terms, updating clients on changes in law, and even marketing and business development initiatives. It is now clear that what legal professionals usually need are multiple AI workflows and agents, each built around one of the many tasks that make up their daily activities. Being connected to your firm’s data ensures these processes truly reflect the unique knowledge, expertise and way of working within your firm.
Explore the blog series “Legal AI That Knows Your Firm”
Posts in this series:
- The Allure (and Danger) of Using Standalone LLMs for Search
- Why Retrieval Augmented Generation (RAG) Matters
- All Search Engines Are Not Created Equal
- Why good legal search is informed by the entire context of your institutional knowledge—not siloed or “federated”
- How can your AI securely use all of your firm’s data?
- Why an “always on” search engine is a prerequisite for scalable AI adoption
- Building AI agents that are informed by your real-world legal processes
- As the variety of tasks automated by AI agents proliferate, how does a firm manage it all? (Coming soon)
- How do I adapt workflow agents to the specific needs of my firm? (Coming soon)
- Does your AI platform set your firm apart from the competition? (Coming soon)
This post was adapted from our forthcoming 24-page white paper entitled "Implementing AI That Knows Your Firm: A Practical Guide." Sign up for our email list to be notified when the guide is available for download.