Also, if there are any Africa-based devs here (but of course, you're welcome to come from wherever you are) we're running a systems programming conference with Derek Collison of NATS, Jamie Brandon of TigerBeetle and HYTRADBOI, Andrew Kelley of Zig, and many other great folks.
Next week, Feb 9th and 10th in Cape Town. Maybe we'll see you there!
Please make sure you record the talks and put them on youtube , then repost on HN when they’re out. I’m sure lots of people here will be highly interested in what those people have to say !
I've wrangled with ledgers on and off for 10+ years now. I can some value in pushing some of the application level concerns to lower down the stack. One could call these as Domain Specific Databases. It'll be interesting to see how this evolves.
Adding a few thoughts off the top of my head.
From my experience I can say that money transfers are rarely, if ever, done atomically between two end user accounts. i.e., <Account A debit, Account B> being one operation. Instead the money first moves from Account A to a pooled account, some business checks/operations take place, and then from a pooled account (could be same or different) to account B. The reason to do so is to run a host of business checks which could take quite some time, in some cases even a day or more. Which is why the two phase account transfer semantic of "Reserve", "Settle" scales so well. One can build all kinds of higher order business flows on top of them. Subscriptions, scheduled payments, and so on.
Account transfers are subject to a bunch of velocity limits such as daily/weekly/monthly count and volume of transactions. It depends on a bunch of factors such as user's KYC levels, fraud rules, customers' age in the system etc., An account starts off at a lower limit and those limits are gradually increased as the account ages in the system. This was a big pain to pull off in scalable fashion. The running counter of limits ends up being bottle neck. For example, a popular merchant account doing a few million transactions a day. Subjecting that account to velocity limit checks million times will mean that merchant account will become a hotspot.
Maybe I should do a blog post sometime, just to give a structure to my brain dump :-P.
Tigerbeetle does support these use cases quite well. It has linked transfers that allows multiple chains of transfers to be linked and all atomically commit or not. Coupled with two-phase commits it provides for some powerful primitives.
TigerBeetle does support two phase reserve/settle semantics. I think we saw a similar problem to what you saw, and wanted to make that primitive first class.
Would love to chat sometime, joran@tigerbeetle.com
Hey Joran, thank you for responding! I now noticed that you do indeed have reserve/settle primitive. I'll surely reach out to you if I have further questions.
What I meant to say was that this is the first time I came across a ledger use case specific DB, so I just coined that phrase, Domain Specific Database (similar to Domain Specific Languages).
And yes, you could say that the industry does suffer from some version of NIH. The concept of ledger after all is thousands of years old and yet software engineers don't bother to learn end up mucking up the implementation.
I attribute it to software engineers who are not willing to step back and learn the domain of money and payments. Whenever money is involved one should only have to look how professional accountants maintain their books and handle it, learn from existing products (such as Stripe[1], Modern Treasury [2]) before running off to roll out a half baked version of double-entry book keeping. The system kinda works initially but when the requests increase the errors begin to accumulate very fast and the businesses end up unable to trace millions of $$ or worse. I've seen it happen way too many times.
Joran Greef (TigerBeetle's CEO) had a fascinating conversation with Richard Feldman on his Software Unscripted podcast. Highly recommend giving it a listen!
Thanks fbouvier! Zig has been great with how fast it moves, being able to get io_uring in like that, and other features (e.g. @prefetch). The core team are awesome.
i feel like the branding of "financial accounting database" is underselling the broad potential of Tigerbeetle. Finance feels "niche" to developers but really think about how often we use "transactions" in everyday database concepts. Double entry accounting is the obvious next step in terms of auditability of transactions, and it's thousands of years old. Now we have it as a database.
As someone who worked on a financial system where we did basically what you described in the blog post (create in house API backed by some database) this would’ve been amazing.
I've been looking through the docs, and I can't find how you're intending on supporting the metadata that would go along with a transaction. Say I want to post a journal, where would things like department, customer/vendor (entity), cost centre etc exist? Or header information? Or would they have to be linked externally? Or is big ERP software not the target market? If not, what is?
Just flipped back to the docs again, think I've found it: 'Set user_data to a "foreign key" — that is, an identifier of a corresponding object within another database.' -- This is all well and good, but if you're having to write to this other database at the same time in order to store said other data, doesn't it make your ledger a bit pointless? I'm just struggling to see the use cases. Can anyone help me "get it" ? (For the record, I work with ERP/accounting systems in my day job)
tldr; we anticipate you'll have some database with all your customer information. The metadata fields inside of TigerBeetle currently can be used to link back to rows in your existing database. Or if you don't need an external database and your metadata fits into our metadata byte fields, you can use them as you like.
Understood, thank you. How much space is available in the metadata byte fields? I can't see the likes of banks or financial institutions wanting to use it if they can't do everything in a single atomic transaction, unless you have a cunning 2 phase commit option/strategy with Oracle/DB2 etc.
Just want to say that I'm not trying to be negative - you're a long way down the dev road, and must have thought about all of this, so I want to read about it! :-)
This FAQ entry begins to describe the built-in two-phase transfer feature we support: https://docs.tigerbeetle.com/FAQ#what-is-two-phase-commit. But we will have to brush up that answer with more details and some recipes/sample code.
I was heavily influenced by the LMAX architecture, when I was CTO'ing a transactions based system. We had postgres in the backend and heavily used redis's streams for event sourcing, but we were only targeting in the region of 1000s of tps, and planned to shard heavily by client for growth. No longer doing that, but very interested to see other players in the field succeed.
Advantages for using Postgres ( assuming a double entry schema[1] ) and you're using Postgres for your main app db;
- You can do financial transactions inside business db transactions, and roll back both atomically
- Adding up numbers is one of the things computers are best at, and Postgres can easily handle a huge amount of financial transactions
- Re-use PaaS Postgres hosting/scaling/clustering/backups
- Easier integration with the rest of your app with foreign keys to relevant records relating to the financial transaction
- Easy integration with BI tools given Postgres is well connectable
It's dated, the API is a bit messy and needs work, as it was initially written 10+ years ago, but for a web based app I would choose a v2 of it over a non-postgres ( assuming you are using Postgres for your app ) solution.
I blogged about this recently - here's a DB schema that can do all of the double entry stuff with about 2 tables. I also have example queries for balance sheets and other useful bits:
I hadn't seen that gem before! My comment on most accounting tools is that they ignore the concept of "normal" account balances, which means they don't really think about "debits" and "credits". This is fine mathematically, of course, but makes it awfully difficult for accountants to understand how the software is supposed to behave!
We're running on mid-sized Postgres right now, but find that clickhouse (or duckdb) are something like 100x faster once we do GROUP BYs on 5M+ rows.
Incidentally, I just came across Journalize. But I'm struggling to understand what it's for. It might be helpful to say - either here, or on your homepage - who or what it is competing with. I'm probably not the exact target audience, but it seems to do more or less what Xero does for us (but I don't want to rule out that I just don't understand the homepage).
If your business generates a lot of transactions, to the point that you have a full time accounting or engineering staff whose job it is to wrangle 200MB excel files or complicated sql queries, then Journalize is for you.
We’re basically BI tailor made to create outputs that are relevant to accountants (we know what “depreciation” is supposed to look like) and let them take a monthly ledger entry and drill down to a specific user’s impact on the GL and tie it to the line of source data.
In this way, we’re doing the data engineering work, helping accounting teams have BI type tools to set up accounting rules, and giving transparency to “why” an aggregate number is correct.
I think they key part is "distributed financial accounting database". For anything that didn't need to be distributed to handle massive volume of transactions I'd also pick Postgres, but I can imagine there's a limit where that wouldn't work, and as soon as you need to do this in a distributed database things get exciting.
No experience with the linked gem but the description states
> While this gem acts like a double-entry bookkeeping system, as it creates two entries in the database for each transfer, it does not enforce accounting rules.
And that seems like a far cry from things like "rejecting transfers if the accounts involved would exceed their net debit or credit balances".
>> While this gem acts like a double-entry bookkeeping system, as it creates two entries in the database for each transfer, it does not enforce accounting rules.
> And that seems like a far cry from things like "rejecting transfers if the accounts involved would exceed their net debit or credit balances".
The READEME needs to be updated[1] for clarity. It doesn't enforce "accounting" rules, however The DoubleEntry gem has rules to optionally ensure a balance cannot go negative, and it also has an allowlist of allowed transfers ( which are defined by source account, dest account and code ).
> I think they key part is "distributed financial accounting database". For anything that didn't need to be distributed to handle massive volume of transactions I'd also pick Postgres, but I can imagine there's a limit where that wouldn't work, and as soon as you need to do this in a distributed database things get exciting.
That is why I qualified my statement with "you're using Postgres for your main app db".
For every financial transaction you'd likely have many more business logic level transactions, and you would want your business transactions to be as consistent as your financial transactions. No point storing a financial transfer if you can't match it up with a purchase on your business db.
The particular technical challenges tiger beetle is taking on (Protocl-aware recovery, io_uring, etc) are interesting, so my question is... why build a double entry system when you could build a new... database?
It's designed as a replicated state machine. You will one day be able to swap the state machine (double entry features) for another state machine, and leverage all the fault-tolerance and performance of TigerBeetle.
How will it be able to do protocol-aware recovery if it doesn't know the exact application? I think that's the key here, but it doesn't explain how this can extend beyond the problem it was coded for.
Also, the choice of zig is great (yay no memory allocation), but this is not the first database to do that. Many older databases did as well. There are advantages to dynamic memory management. But, moreover, postgres, et al, are written in C as well. Zig is not magic. Thus, I don't understand the differentiating factor here. Sure, zig is nicer, and sure, this choice of language makes it more likely to be performant than many new databases, but it sounds like you're trying to compete with the likes of Postgres, and this is then a less differentiating factor.
The paper “Protocol-Aware Recovery for Consensus-Based Storage” is orthogonal to choice of state machine, since it's at the consensus/storage engine layer underneath. So custom state machines benefit from this.
While I love C, and while C was the other choice for TigerBeetle, I wanted to pick a modern systems language to invest in for the next 30 years. There are several advances in Zig around safety: bounds checking, checked arithmetic, error handling, better tooling, cross-compilation, comptime, enforcing memory alignment through the type system, ability to handle memory allocation failure etc.
The big difference compared to Postgres is that TigerBeetle has better performance for the domain (TB can be far more aggressive with group commit since financial transactions in TB are only 128 bytes), a stricter storage fault model, and is a distributed database with automated leader election and fault tolerance. TigerBeetle is also designed to be deterministically simulated, like FoundationDB, so we can test to a high standard. For example, by injecting the theoretical limit (in terms of the consensus protocol) of corruption on all machines, including the leader, and then test that TigerBeetle self-heals.
Does TigerBeetle have a formal proof of serializability or has it been verified by the Jepsen tests? It's mentioned in the blog post and I'm curious how it fares in that department.
This isn’t as thorough as model-checking or proving a formalization of the algorithm, but it has the benefit of working with the real code. It is a bit less real than Jepsen (as we essentially mock all syscalls), but is quite a bit faster.
Oh wow, are you on the TigerBeetle team then? I only knew of you as the rust-analyzer guy, with awesome posts on your blog.
Do you have any thoughts on how working on TigerBeetle with Zig is? It seems like for a high performance DB rust would be a natural choice. And how's the development experience without a zig-analyzer?
Yup, I am that matklad. I have thoughts, will put then into a post some time soonish.
zig-analyzer is great — much further along than I’d expected at this point. Though, I am still curious what the end game IDE support would look. https://github.com/ziglang/zig/issues/3028 is a big component of that.
Curious, how does this compare/contrast to Formance? (I haven't used either, but looking at Formance right now for tracking "IOU points" between multiple parties
Formance is awesome. I've had a few fantastic calls with Clément. I think we see it that Formance operates at a layer above TigerBeetle, whereas TigerBeetle provides the underlying ledger infrastructure. So they are complementary.
This will be an uphill battle against established solutions like MS Dynamics.
That's man-millennia of painfully evolved application code, catering for tax legislation of many countries and The Way Accountants Want It, and database performance/serializability is a fourth-order concern.
I mean there's no reason you couldn't use it for crypto as a high speed frontend to tx on the blockchain, some sort of lightning-fast network, one might even say
Not if you want speed and you have no need to make it government resistant by going full rogue cyberpunk by having millions of copies of your data in a constantly distributed state
https://github.com/tigerbeetledb/tigerbeetle
Also, if there are any Africa-based devs here (but of course, you're welcome to come from wherever you are) we're running a systems programming conference with Derek Collison of NATS, Jamie Brandon of TigerBeetle and HYTRADBOI, Andrew Kelley of Zig, and many other great folks.
Next week, Feb 9th and 10th in Cape Town. Maybe we'll see you there!
https://systemsdistributed.com