This post is the first in a series about how to implement legal AI that knows your law firm. In the series, we cover the differences between LLMs and search, the elements that make a good search engine, the building blocks of agentic systems (e.g. RAG), and how to implement a system that is fully scalable, secure, and respects your firm’s unique policies and practices.
Search has long been the primary tool for extracting meaning, patterns, and insights from a firm’s data. Today, the accessibility and ease-of-use of generative AI systems makes them tempting substitutes for search engines, but relying on them in this way introduces significant risk. Generative AI can appear to produce reliable and accurate answers and outputs—yet almost anyone who has manually double-checked generated results has found instances of hallucinations and misleading results presented as facts.
The simple truth is that LLM technology is not suitable for use as a search engine, no matter how many times people attempt to use it as one.
The reason they often appear to work is because LLMs trained on a large corpus, including the internet and various legal data, can spit out what sounds accurate based on their training data. The problem is, these answers are probabilistic—i.e., they generate a sequence of words the LLM calculated was most likely to follow from a set of words in the prompt. Such answers won’t reflect the data and knowledge embedded in a firm’s documents that were not part of the training data.
One technique sometimes used to try to improve the accuracy of an LLM is to take internal or proprietary data and “fine-tune” or “retrain” a model. In theory, making this training data a part of the model was thought to improve performance on recall-like tasks. In practice, however, fine-tuning on a firm’s documents rarely produces a meaningful improvement in accuracy. One underlying reason is that the data used in fine-tuning typically represents only a tiny fraction of the vast corpus the general model was originally trained on, so it does not fundamentally change the model’s behavior. More importantly, even after fine-tuning, the LLM remains a probabilistic system that can still hallucinate, confidently producing statements that sound plausible but are not grounded in any real evidence. As a result, fine-tuning often yields only modest gains in style or familiarity with certain terminology—while leaving the core risk of hallucinations intact.
Another common misconception is that uploading, or pasting, large volumes of documents into the model’s prompt (what some call “context window stuffing”) will reliably solve the problem of hallucinations or inaccurate answers. In reality, this approach suffers from serious limitations. Because the model still operates probabilistically, it does not inherently know which portions of the “stuffed” text are authoritative or relevant to the question asked. Instead, it tries to pattern-match sequences of words, often producing outputs that appear credible but fail to accurately reflect the embedded evidence. Moreover, context windows have finite size limits, so longer documents could be truncated or omitted entirely, leading to partial or misleading synthesis. As a result, context window stuffing is a brittle and inadequate substitute for proper retrieval-augmented methods that explicitly index, search, and cite supporting material.
So, if LLMs alone are not suitable, how is a law firm to implement AI that uses its own data and expertise?
The answer is in this blog series. The only way to ensure that an LLM can produce an accurate and firm-specific answer is to augment it with quality retrieval, i.e., with techniques like retrieval-augmented generation (RAG). Accessing the necessary data for the RAG process is the job of the enterprise search system, not the LLM itself as we discuss in the next article in this series (coming soon).
This post was adapted from our forthcoming 24-page white paper entitled "Implementing AI That Knows Your Firm: A Practical Guide." Sign up for our email list to be notified when the guide is available for download.