Document is a "Costume" a Financial Statement Wears
This article is inspired by an article by Georg Philip Krog, Why AI Can’t Read Your Contracts — And What To Do About It, where he explained why artificial intelligence cannot read legal contracts. I am applying similar thinking to why artificial intelligence cannot read financial statements. The metaphor of a "costume" is excellent.
Popular AI systems built on large language models (LLMs) face a fundamental limitation when confronted with something like a financial statement. LLMs can often extract and correctly summarize much of the information, say perhaps 87%. The key point is, it is never 100%. However, "much" or even "most" is not sufficient in a domain where precision is non negotiable. You never know which 13% is wrong, and that uncertainty makes the entire output unreliable.
LLMs excel at the capability of predicting the next plausible word. That is their core competency. But interpreting a financial statement requires logical reasoning, not linguistic prediction. There is a fundamental difference between guessing and proving.
Symbolic or rules-based artificial intelligence, by contrast, does not guess. Rules-based artificial intelligence applies explicit known logic and a specific known set of information in the form of rules and facts.
While rules-based systems cannot yet understand every nuance of a financial statement, what they do understand is provably correct within the scope of the rules they implement. Humans must still bridge the remaining gap, which makes it essential to clearly delineate which tasks are performed by deterministic logic, which require human judgment, and which are merely assisted by probabilistic artificial intelligence such as an LLM.
A financial statement is not natural language. A financial statement is encoded financial logic, expressed through the vocabulary of accounting which follow the known mechanics and dynamics of accounting machinery and wrapped in a document like form for human consumption. This is even more true in the XBRL era, where the underlying representation is explicitly structured using a global open industry standard technical format.
But beneath the surface of what looks like a document lies a rigorously designed logical substrate. That is why rules-based artificial intelligence excels at validating consistency, coherence, and provability; because the content is fundamentally logical, not linguistic.
Relying entirely on statistical guesses by a machine in a high stakes environment is dangerous. While spectacular failures are easy to see, it is the subtle failures that tend to be the most problematic. Misinterpreting a single term or relationship can cascade into million dollar errors. LLMs can be helpful collaborators, but only if we maintain bright lines between:
- conclusions reached by provable, rules-based logic,
- work performed by skilled and experienced human professionals, and
- plausible but fallible suggestions produced by an LLM.
Responsibility cannot be delegated to a machine that guesses. A single incorrect inference from an LLM can create catastrophic liability; whether during the preparation of a financial statement, validating and verifying that the financial statement was created correctly, or using that financial statement for decision making.
Remember, computers are dumb beasts. We need to give those dumb machines a chance.
We need to separate the “document” and the “information”. To enable a computer to even have a chance to succeed at working with information, we need to take a “graph first” approach as some call it or a “model-driven” approach to representing that information. We can then convert that information reliably and predictably from the machine interpretable graph of information into a human consumable version in the form of a document using reliable algorithms.
The solution is to begin with a provable, machine interpretable knowledge graph; that "graph first" or "model-driven" representation of the financial statement. That is the master copy of the information. Then, generate the human readable “document costume” from that one authoritative source.
This makes the boundaries unmistakably clear: what the rules-based system proved, what the skilled experienced human contributed, and where the LLM provided optional assistance that may or may not be correct. Provability becomes the anchor.
Financial reporting to meet a compliance requirement of some sort demands proof. Machine interpretable rules, facts, and a known common logic driven artificial intelligence can deliver guaranteed results, reduce cost, and eliminate repetitive, error prone tasks. Skilled and experienced humans can then focus on the high value analytical and judgment based work that only humans can perform.
Using this approach the all too common "bucket brigade" of spreadsheets connected together by expensive but error prone humans can be reliably automated, reducing the monotonous, repetitive janitorial work performed today by expensive skilled and experienced professionals that could be better utilized to perform more value added work; work computers cannot possibly perform.
Here is a simple, basic example of a graph first or model-driven financial statement represented using the global open industry standard XBRL. Note that the renderings were automatically generated from the raw XBRL.
Comments
Post a Comment