DataBook

As explained by Kurt Cagle and Chloe Shannon in their article,  DataBooks: Markdown as Semantic Infrastructure, a DataBook is effectively a microdatabase.  

A DataBook is a document a human can read, a data file that a computer can process, and a toolbox that caries its own instructions. A DataBook is a technique that enables data and an explanation to travel together in a data pipeline.

One important part of the magic of DataBook files to understand is that a DataBook can also easily be read and interpreted by LLMs.

Another part of the magic of the DataBook is that everything travels together within one file including: 

  • data
  • meaning
  • rules
  • queries
  • documentation

Finally, DataBook files can easily be versioned by Github and Gitlab.  Both Github and Gitlab support MD files which are both based on CommonMarkup. And so, there appear to be different "flavors" of MD files, but they are close.

There are no separate files which can be forgotten or lost.  This technique uses a markdown file (.md) as the container which holds everything.  No separate files to lose, no context to forget. A markdown file provides a powerful yet straightforward way for users, both technical and non-technical, to write plain text documents that can be rendered richly as HTML but also easily read by a computer software application.

Within the markdown file you can provide fenced blocks (a.k.a. section) of structured data formats such as YAML, RDF/Turtle, JSON-LD, SPARQL, SHACL, SQL, CSV, and other such well understood structured formats.

That’s it; one physical file that contains many formally fenced off layers.  Here are some of the different fenced blocks that can be provided within a DataBook:

  • Markdown: this is the primary format of the file; holds the human-readable text and all of the other structured information, those fenced blocks.
  • YAML frontmatter: Used at the top of the file for metadata like title, author, version, provenance. YAML is a unicode based data serialization language which is broadly useful for programming needs ranging from configuration files to internet messaging to object persistence to data auditing and visualization.
  • RDF/Turtle: Used to store a graph of data a.k.a. linked data.
  • JSON‑LD: Popular alternative or complementary linked‑data format.
  • SPARQL: Queries and updates embedded directly in the file.
  • SHACL: Validation rules for the data.
  • Other typed structured blocks: Depending on the use case, a DataBook may also include:

Here is a very simple, basic example of a DataBook. You can read the databook file here on Github and a machine can be sent this raw databook file of the file on Github.


As I understand it, work is being done to try and get DataBook to become a W3C standard. Even if DataBook is not a global standard, it is a useful convention and might even be considered a best practice.

Additional Information:


Comments

Post a Comment

Popular posts from this blog

Inhabiting Babel, A Manifesto for Responsible Meaning Engineering

Image
Inhabiting Babel, A Manifesto for Responsible Meaning Engineering is a book written by Nicolas Figay . This book is very practical and easy for a motivated business professional to read. For me, this book has both tempered my expectations as to what is possible and also reassured me of the real possibilities that exist if you do the required work. Accountants and auditors tend to be on the conservative side of the spectrum.  Me included. I am  actually rather conservative in terms of what technology might bring to the table for accounting, reporting, audit, and analysis. I have learned how to "cut the cards" when dealing with sales people and software vendors. What this book does is bring the wisdom from the experiences of an expert in system interoperability, enterprise architectures, and complex systems to helps motivated business professionals and technical professionals that truly want to understand artificial intelligence, cutting through the rhetoric and hype. This boo...
Read more

Rethinking Financial Reporting: the Model-driven Financial Statement

Image
It is time to rethink the design of financial statements. Clay tablets worked well and they had a good run which ended many years ago when the clay tablets were replaced by papyrus and then ultimately paper.  Paper had a good run also but paper financial statements were ultimately replaced by what amounted to an electronic version of that paper; PDF, HTML, and such. We can do better.  Today's demands call for more modern accounting tools . Idealized design  (a.k.a. greenfield project ) is the notion of designing a system without imposing constraints of prior work.  This technique may, or may not, work for rethinking the financial statement. Idealized design might be a harder path in the short term, but taking this approach has very positive long term ramifications. Here are a number of ideas that I would throw at the wall if I were in charge of rethinking the financial statement to make such tools modern: Digital first : The first thing that I would do is abandon the...
Read more

PLATINUM Business Use Cases, Test Cases, Conformance Suite

Image
A brick wall is made of exactly two things: bricks, mortar. But a brick wall created by a master craftsman, or mason, and a brick wall created by a “weekend warrior” with no knowledge of masonry will be very different. Master craftsmen are created and that process takes time and effort. This blog post is a first prototype that brings several things together into one useful resource for business professionals.  It pulls together the human readable information of the Showcase of Reports   and the machine readable information of the Conformance Suite of XBRL-based Financial Reports Prepared using the Seattle Method .  Then, I want to incorporate the information found in these PLATINUM XBRL-based Financial Report Examples . The three resources can be useful and used independently; but they are even more useful when used together.  Business professionals really could not care a less about the technical details of getting some result that they want.  But technical pro...
Read more