## Blogs:

- http://muratbuffalo.blogspot.com/

- https://bartoszsypytkowski.com/

- https://decentralizedthoughts.github.io/

- https://www.the-paper-trail.org/

- https://blog.acolyer.org/

- https://pathelland.substack.com/

## Other web resources

- https://aws.amazon.com/builders-library/ - set of resources from Amazon about building distributed systems

- https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_H... - lecture series from Cambridge

## Books

- https://www.cl.cam.ac.uk/teaching/1213/PrincComm/mfcn.pdf - A great book on the maths of networking (probability, queuing theory etc...)

I wrote this post a while ago, aiming to answer a similar question about finding paper: http://brooker.co.za/blog/2020/05/25/reading.html One key point there is that there are, in my mind, multiple 'modes' of reading, and I like to use different approaches to finding material for different modes. Those blogs you list are great for curiosity mode. Another great resource there is Twitter: following distributed systems practitioners and researchers, and seeing what they tweet about. When I read a (recent) paper I really like, I often see if the authors are active on Twitter and follow them there if they are.

It's also important not to weight too much on recency. A lot of the canon is actually more approachable than newer papers. For example, Lamport's classic "Time, Clocks" (https://www.microsoft.com/en-us/research/publication/time-cl...) and distributed snapshot (https://www.microsoft.com/en-us/research/publication/distrib...) papers, and Gilbert and Lynch's CAP paper (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67....) are approachable without deep background or systems knowledge. Similarly, John Little's " A Proof for the Queuing Formula: L = λW" (https://pubsonline.informs.org/doi/abs/10.1287/opre.9.3.383) is quite approachable if you have a math background but no systems knowledge, and is one of the foundational results behind the practice of building stable systems.

I've got some longer-form dives into researcher's work here: http://brooker.co.za/blog/2014/03/30/lamport-pub.html http://brooker.co.za/blog/2014/09/21/liskov-pub.html http://brooker.co.za/blog/2014/05/10/lynch-pub.html

Finally, books. There are a couple recommendations for Martin Kleppman's Designing Data-Intensive Applications book, which I like a whole lot. Alex Petrov's "Database Internals" is also a very approachable introduction. I wish every practitioner in the field would read Harchol-Balter's Performance Modeling and Design of Computer Systems.

Material from a university master course on distributed systems engineering. We developed our own university material in .ipynb format at Github. Should give you the basics in 5 days. Applies distributed systems theory in 5 steps. Most improvements in past decade are not due to hardware improvement, but algorithm and tooling advances I believe.

- Distributed systems. Overlays and communication network. Introduction to simulation framework https://github.com/grimadas/BlockchainEngineering/blob/maste...

- Gossip. Convergence of the transactions, information https://github.com/grimadas/BlockchainEngineering/blob/maste...

- Faults in distributed systems: crashes and disruptions https://github.com/grimadas/BlockchainEngineering/blob/maste...

- Malicious nodes, adversary model https://github.com/grimadas/BlockchainEngineering/blob/maste...

- Consensus and agreement despite malicious nodes https://github.com/grimadas/BlockchainEngineering/blob/maste...

I also agree with the recommendations for "Designing Data-Intensive Applications" and "Database Internals". Though, having read the latter for a book club at $employer, I felt it served better as a sort of "index for the space" for people who already had some DB experience, rather a true introduction.

What's your twitter handle?

See https://lamport.azurewebsites.net/pubs/pubs.html#paxos-simpl... Ok, might not be fair to call it an error, but it is super confusing, and a great example of Lamport's larger point that writing prose about these things is really hard, and we need better tools (like TLA+).

I find it surprising that he didn't actually attempt to change this problematic sentence if it was clearly shown that it leads to incorrect implementations. I get his point about prose, but a tiny footnote on a webpage doesn't feel like the best way to highlight what seems to be a common misunderstanding.

Any idea what this ambiguous sentence might be, and what is this mistake that implementers tend to make?

My former manager recommended it to me when I first started working in distributed systems and I found that it unlocked a huge variety of topics despite its simplicity. (Thanks Steve!)

https://www.microsoft.com/en-us/research/publication/time-cl...

https://dl.acm.org/doi/book/10.1145/3335772#issue-downloads

All of the chapters are freely available (scroll down to "Book Downloads").

It's a great book that goes into pretty much all of the commonly used strategies to scaling data-intensive applications. It's not incredibly deep on any of them but it will allow you to get a great overview of the entire space. For each component, there's usually references to places where you can read and study more about them.

https://youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe...

And I found some MIT course as well on YouTube:

https://youtube.com/playlist?list=PLrw6a1wE39_tb2fErI4-WkMbs...

Making reliable distributed systems in the presence of software errors

https://erlang.org/download/armstrong_thesis_2003.pdf

“Why Do Computers Stop and What Can Be Done About It?”

https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf

His paper, "Time, Clocks, and the Ordering of Events in a Distributed System" is still considered a serious read after 40 years.

[0]:https://lamport.azurewebsites.net/pubs/pubs.html

https://www.youtube.com/playlist?list=PLNPUF5QyWU8O0Wd8QDh9K...

- Concise

- Approachable

- Entertaining

- Insightful

- Timeless

http://homepage.divms.uiowa.edu/~ghosh/ssDijkstra.pdf

Enjoy :)

> For brevity's sake most of the heuristics that led me to find them, together with the proofs that they satisfy the requirements, have been omitted and--to quote Douglas T. Ross's comment on an earlier draft, "the appreciation is left as an exercise for the reader."

this helps keep it concise ;)

Many seem to have had the same reaction as you! ;)

Communicating Sequential Processes "CSP" by Tony Hoare[0] has a strong influence on Go and Clojure. He also published/contributed to other interesting and influential books and papers.

Making reliable distributed systems in the presence of software errors by Joe Armstrong[1] (Erlang, BEAM). An implementation of the actor model and functional programming to optimize for reliability.

Conflict-free Replicated Data Types by Marc Shapiro, Nuno Preguiça, Carlos Baquero, Marek Zawirsk, "CRDTs" [2]. Enable strong eventual consistency, which is typically useful (and implemented) for databases, p2p (chat) applications and other distributed systems.

[0] https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf

[1] https://www.cs.otago.ac.nz/coursework/cosc461/armstrong_thes...

[2] https://hal.inria.fr/hal-00932836/file/CRDTs_SSS-2011.pdf

I have a tangential question, in case anybody here has the answer. Have there been any attempts to prove (either formally or informally) that CSP either:

- Can model any computation?

- Is suitable for modelling business processes?

I've heard both of these claims quite often, but haven't been able to find any primary source on the matter.

https://github.com/theanalyst/awesome-distributed-systems

https://dancres.github.io/Pages/

Introduction to Reliable and Secure Distributed Programming (https://www.amazon.de/-/en/Christian-Cachin/dp/3642152597).

I took a class with Luis Rodrigues (one of the authors), the book introduces the fundamentals of distributed systems. For example, you would build leader election from first principles.

This was the first article that really made it all click for me

Distributed systems fans of HN, why are you reading about distributed systems?

[0]: https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Erlang isn't theoretical. It's practical engineering. It works because message passing is what distributed systems have to do and at scale portions of a distributed system will become unavailable.

There are very specific problems that require more detailed engineering like Lamport Clocks and Raft Consensus Protocol. But not the general case. The general case is "being good enough" as is the nature of engineering.

In Search of an Understandable Consensus Algorithm

https://raft.github.io/raft.pdf

I think raft has stood the test of time so far. A very popular implementation of raft is etcd, which is used as Kubernetes' backing store for all cluster data.[0]

[0] https://kubernetes.io/docs/concepts/overview/components/#etc...

Paul Baran's research on Distributed Communications that led to the Internet:

"Paul Baran and the Origins of the Internet" - https://www.rand.org/about/history/baran.html

"On Distributed Communications" - 1964 https://www.rand.org/pubs/research_memoranda/RM3767.html

imho the most useful book you can read.

https://github.com/donnemartin/system-design-primer

[0] https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39...

https://archive.is/5m3s4

https://understandingdistributed.systems/

Distributed Operating Systems Distributed Systems: Principles and Paradigms

[1] https://en.wikipedia.org/wiki/Andrew_S._Tanenbaum#Books

https://vadosware.io/post/paxosmon-gotta-concensus-them-all/

https://gist.github.com/macintux/6227368

https://news.ycombinator.com/item?id=13311124