When we started building OnFinance AI, one of the earliest technical questions was: what can an LLM actually do reliably with Indian regulatory text, and where does it fail in ways that matter for compliance? This is not an academic question. The answer determines which parts of the workflow an LLM can own and which parts require a human reviewer before any output becomes part of a compliance record.
After processing several hundred RBI, SEBI, IRDAI, and PFRDA circulars through our pipeline, we have a reasonably clear picture of where LLMs are reliable, where they are unreliable in predictable ways, and where the failures are unpredictable enough that human review is non-negotiable. This piece is an honest account of that picture. We are not selling a fully autonomous compliance system, and writing as if LLMs are more accurate than they are would not serve the compliance officers who use tools like ours.
Where LLMs Perform Well on Regulatory Text
The tasks where LLMs are reliably accurate on Indian regulatory text are those that involve pattern recognition and structured extraction from well-formatted documents.
Circular metadata extraction is one. Identifying the issuing regulator, the circular reference number, the date of issue, the addressees (all banks, NBFCs above a threshold, specific categories), and the broad topic from the header and opening paragraphs of an RBI or SEBI circular is highly reliable. The format of Indian regulatory circulars is consistent enough that a well-tuned extraction pipeline produces near-zero error rates on structured metadata.
A second reliable task is identifying text that was explicitly added or removed when comparing two versions of a Master Direction that the system already has. When the prior version and the new version are both present in context, and the change is explicit rather than implicit, LLMs can accurately identify the changed paragraphs and produce a clean differential. This is true for changes stated as "in partial modification of Paragraph X, the following text replaces the existing text." It is less reliable for implied changes that require inferring that a later provision supersedes an earlier one without explicit supersession language.
Categorizing circulars by type is a third reliable use. Determining whether a circular is a Master Direction, a standalone notification, a press release with compliance implications, or an amendment to a specific existing document is something LLMs handle well when the corpus is Indian regulatory publications. The vocabulary and framing conventions are distinct enough that category misclassification is uncommon.
Where LLMs Are Unreliable in Predictable Ways
The more interesting category is unreliable in predictable ways. These are failure modes we understand well enough to detect and route to human review systematically.
Cross-document reference resolution is the primary one. Indian regulatory circulars frequently reference prior circulars by their reference number. An RBI circular may say "as per our circular RBI/2023-24/112 dated August 14, 2023" and proceed to modify what that circular established. An LLM operating on the new circular alone cannot resolve this reference unless the prior circular is in context. When it is not, the LLM will either omit the reference or attempt to interpret what the prior circular likely said based on the topic, which is not reliable.
This failure mode is predictable and manageable. The solution is to build a corpus of prior circulars and retrieve the referenced document when a cross-circular reference is detected. This is a retrieval problem, not a language model problem, and it is solvable. The failure becomes problematic only when the retrieval step is absent and the LLM is asked to analyze a circular in isolation without its references.
A second predictable failure mode is term disambiguation. Indian regulatory text uses defined terms precisely. "Non-Deposit taking NBFC" has a specific regulatory meaning that is distinct from an NBFC that happens not to accept deposits; it refers to a specific category in the regulatory classification framework. "Regulated entity" in an RBI circular may or may not include NBFCs depending on the specific direction being issued. An LLM that reads "regulated entity" and interprets it as applying to all RBI-regulated institutions will produce incorrect scope assessments for circulars where the term has a narrower meaning in context.
This failure is also detectable and manageable. A regulatory domain dictionary that maps Indian financial regulatory terms to their precise definitions, maintained as a grounding reference for the LLM, significantly reduces this error class. It does not eliminate it because new terms are introduced, existing terms are sometimes redefined in a specific circular, and context-dependent disambiguation can still fail at the margins.
Where Failures Are Unpredictable: The High-Stakes Cases
The category of failures that is genuinely difficult to manage is where the error is not predictable from the document structure or terminology alone, and where the error has direct compliance consequences.
The most problematic case is transitional provision interpretation. When a regulation moves from one framework to another with a phase-in period, the transitional provisions governing which entities must comply with which requirements by which dates are often the most complex and consequential text in the entire document. An LLM reading a complex transition schedule may correctly identify the existence of a transition period while incorrectly characterizing which category of institution the earlier deadline applies to versus which category gets the later deadline.
This kind of error is hard to detect automatically because the LLM's output looks plausible. The circular reference is correct. The general characterization of the transition is correct. The specific scope of who falls into which bucket is wrong. A compliance officer who reads the LLM summary without checking the source text may act on the incorrect scope assignment.
This is why our architecture treats transitional provision summaries as requiring human review before any action is taken. The LLM produces the summary. The flag is set. The compliance officer reads the original text for the flagged section. This is not a failure of the technology; it is an acknowledgment of the technology's limits in a specific high-stakes case.
A second unpredictable failure case is circular-to-circular contradiction. The RBI regulatory corpus has instances where a later circular creates requirements that, when read against an older Master Direction on the same topic, appear to be in tension. The conflict is often resolved by the implicit supersession principle, but the LLM may not handle this correctly when processing each document separately. The risk is that a summary of the current compliance obligation in a given area draws from a circular that has been partially or fully superseded without flagging the potential conflict.
The Architecture Implication: Structured Human Review Gates
The practical implication of this failure taxonomy is that the architecture of a reliable compliance LLM system needs to be structured around human review gates positioned at the points where failure probability or consequence is highest, not at every step uniformly.
A system that requires human review for all LLM output is not useful. The volume of circular processing makes uniform human review impractical. A system that requires no human review is not safe for compliance purposes. The design space is the combination of automated confidence scoring, flagging of known failure patterns, and targeted human review for flagged outputs.
The flags we use most consistently: cross-document references where the referenced document is not in context; scope characterizations involving defined regulatory terms with multiple possible meanings; transitional provisions with phase-in schedules; any output where the LLM's confidence on the specific provision characterization is below threshold; and cases where two separate analyses of related documents produce scope characterizations that are in tension.
Flagging these cases does not require the LLM to know that it is wrong. It requires the system architecture to know which structural patterns in the input documents are associated with higher LLM error rates, and to route those cases to human review regardless of the LLM's apparent confidence.
What This Means for Compliance Teams Evaluating LLM Tools
Any compliance team evaluating tools that use LLMs on regulatory text should ask three questions. First, what is the human review gate architecture? If the answer is "our model is very accurate so human review is optional," that is not a satisfactory answer for a compliance context. Second, how are cross-document references handled? If the tool processes each circular in isolation, it will fail systematically on the circulars that rely most heavily on prior publications. Third, how are errors surfaced when they occur? LLMs will produce errors; the question is whether the tool's design ensures that errors in high-consequence outputs are caught before they enter a compliance record.
We are not saying that general-purpose LLMs cannot be used for regulatory text. We use them. We are saying that using them responsibly in a compliance context requires understanding where they fail and designing the workflow so that those failures are caught, not assumed away. The compliance record that incorporates LLM output should be one where the output has been verified at the points where it matters most, and the system provides an audit trail of that verification.
The tools that will prove durable in the compliance space are the ones built on an honest model of LLM capability, not an optimistic one. The compliance officer's liability does not transfer to the software vendor when a compliance failure occurs. The responsibility stays with the institution. Choosing a tool that has been designed for that reality, rather than one that markets away the limitations, is the operationally sound decision.