It was an in-engine opening cutscene which was very nearly final; the file we shipped was about two or three weeks out of date compared against the version that should have gone on the disc (It had one missing shape key on a character's face at the end of a shot, and a couple other missing elements). My code was wrangling the whole animation; doing all the stuff which our at-the-time-primitive animation system couldn't do itself (animating texture coordinates and etc). And my code was just silently handling all the errors it ran into so that we never even noticed that anything was wrong.

The difference was subtle enough that in the twelve years since the game was released, nobody but the original animator has ever noticed and mentioned it to me (and that, years after release). But that one experience and knowing how much worse it could have been was enough to convince me that "crashes early and crashes loudly with as much detail as possible" is by far the better strategy. At least for entertainment products. And doubly so for entertainment products which can't be patched after release.

(for clarity, this screw-up was 100% my fault. The animators had made the final changes to the cutscene data files in plenty of time for inclusion in the final build, I just somehow didn't import the changed data files into the game when I made the matching changes to the code side, and then my code didn't throw any errors to tell me or anyone else on the project that anything was wrong.)

You don't want your released game to crash in level 11 if the player happens to look behind the wrong lightpole because a texture is missing, but you do want to notice that in development.

But with that said, we didn’t generally crash the game due to a missing texture, even during development, as that’s a super common problem which would have impacted development too much; instead, we just drew anything which used a missing texture in max-saturation pink and green instead, alternating between the two colors once per second, to make sure it’d be super visible to anyone looking at the game during development.

We did usually change that behaviour to instead render missing tetxures in alternating black/near-black instead of pink/green for our final releases, as that was deemed a relatively safe change.

The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric.

The system fails all the time at this scale, but heals itself pretty quickly. In over 1 year of operation it hasn't caused a single incident, and it has survived all outages.

The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the error. If you don't, then exceptions are the best approximation of this pattern - much better than result types or error codes, which litter your code with irrelevant error handling details.

Originally we had a cleanup operation in case of errors. But then we found those could fail as well. As it turns out, we need a catch-all way to clean up resources if anything else fails.

The solution is simple. The main service never cleans up, and a secondary service (I like to call it The Reaper) just cleans up orphaned resources. This keeps both services simpler and more resilient.

Of course this pattern works in our particular circumstances. In other domains it might lead to resource leaks and such, so apply your best judgement.

There’s certainly a minimum of error handling that has to be done in order for code to be considered generally correct, but a lot also depends on the reliability requirements.

Sometimes it’s just inappropriate to abort and this may deeply change the architecture of a program, including by making hard demands on the toolchain, hardware and OS.

- an isolated process which failure doesn’t cascade other part’s failure

- as parent mentionned, where and what failed needs to be super clear

- people are available to timely react to the error, so a rerun will succeed

Failing any of the above, and you’ll need extensive and probably complex error handling that can at least help the system work in a degraded state until the error source is handled.

In my experience, it’s really the last part that motivates engineers to try to deal with the maximum error cases automatically instead of having to deal with them weekends after weekends.

In k8s, if I remember well, the default retry policy for failures will involve incremental back offs, which means if your transient error lasts 20 min your next retry might be in a few hours. It’s fine if your system is ok with that, otherwise jobs will need to “succeed” even when they fail, and so handle errors as gracefully as possible.

Same actually for the user input one: you need to tell your user it’s a recoverable error and not just throw a random “oopsy” message, which means at least some handling of the error to come clean at the end of the tunnel.

My take is, errors are complicated. It’s nice when a script can just die at the first error and not care about how or what happens from there, but that’s such a niche case.

Edit to add: I still think exceptions are better in practice, as you also get a stack trace when the failure happens, whereas Either and ? don't really help track down the error unless you add code to create a manual "stack trace".

Explicit or implicit panic of course also attaches a backtrace. It can also be caught, although that is a can of worms. So panicing is the closest thing rust has to exceptions - somewhat similar to java.lang.Error on the jvm. https://docs.oracle.com/javase/7/docs/api/java/lang/Error.ht...

With anyhow, error handling in rust really is quite pleasant.

exceptions usually trigger cleanup code.

If your cleanup code is mostly closing files and clearing memory, then it's useless because the OS will do that for your crashed program anyway.

Our use case was to build a service to manage the dynamic part of our infrastructure. These are infra pieces that are created/deleted/modified on the fly according to some policies, instead of defining them statically as code. The implementation is simply a lambda function that runs every minute, loads a policy, compares the current state of the system, and then creates/deletes/modify resources as needed.

I am currently in the process of writing a talk that I will deliver to the rest of my org. This will help me crystalize my thoughts, but here are some pointer on why I think it worked in this case:

* The service is stateless. On each run it just loads a policy, compares it to the current state of the system, and acts accordingly. This avoids handling complicated state or coordinating executions. In theory two policies could contradict each other, but in practice we partition our policies in such a way overlap is not possible.

* Operations are idempotent. This is one of the reasons the system converges to a desired state. This makes the service resilient to both failures and eventual consistency.

* Deviation from policies doesn't affect correctness. We are fortunate that our system is not directly customer facing. Deviation from policies affects only performance. The system can work several minutes (if not hours) outside policy band without consequences. This probably will be a critical blocker for most production systems.

Other than control theory and let it fail, I had the chance to play with other cool concepts while implementing this service. These are some of those:

* Parse, don't validate/anti-corruption layer: The service downloads and parses a policy at the beginning of each run. If parsing fails, it errors out. Otherwise, it passes the policy object to the rest of the execution. This makes the system easy to test, and avoids the anti-pattern of peppering your code with instructions reading input, just to find mid-execution that the policy was invalid.

* Pluggable policies: The main body of the service is a very simple sense/act loop. For the actors, we use a strategy pattern, where policies can choose what strategy to use. This approach has helped us to introduce new behavior with minimal code changes.

* Typescript as configuration language: This service replaces an older, less flexible one. A major pain of the old service is that policies where defined as Jinja templates over plain text files. This became unmaintainable as the number and complexity of policies grew. Our new service defines policies in Typescript. Policies are statically typed, and we use regular programming constructs (functions, loops, variables...) to build them at compile time. The output is still a plain JSON file.

Hope that helps.

In Kubernetes you can define your own types, Custom Resources (basically JSONs with schema) and deploy "operators" - services that should handle these new types. Every time you create or modify your custom resource, the operator is triggered and it should "reconcile" your resource.

Now this reconciliation process is stateless. It doesn't know what exactly changed in your resource, so it should just go through the list of all the things that it needs to do (create or remove pods, services, configmaps, etc.) and if something is not right (e.g. a missing service), try to make it right or fail. In any case, the output should be written in the custom resource's .status section.

There's no active waiting - if the operator sees that some other resource is not ready yet (a required pod is still starting), it should just mark your resource as not ready and finish. If the pod state changes, the next reconciliation will notice it. It should do as much as it can to bring reality to the expectation, but not more.

If implemented correctly, this is surprisingly resilient. The idempotent nature of the reconciliation loop makes it perfect for errors handling. For instance, your reconciliation may fail because some pod is not running correctly. It's nothing that your operator can fix. But if the pod auto-heals (maybe the network connectivity was restored or an external service is available again), the operator will auto-heal as well, without a manual intervention. The next reconciliation loop will just see the pod is available again and carry on.

What if your logging code was written with the same philosophy?

To your point, Erlang has the concept of supervisor trees and handling of errors. Similarly, our supervisor is the Lambda runtime. If everything goes wrong, the runtime will emit a metric error.

But what if Lambda itself fails? Or Cloudwatch? or any other supporting service? of course this is possible. But probably at that point the system is so fucked up that the fate of our little service is of little relevance. At least we know from past outages that when normal operations resume, the system will correct itself.

What I mean is that once something unexpected happens your code should ideally fail in that step itself.

The simplest most common example I’ve seen with python programmers is when they pass around dicts as arguments in complex code bases. Methods expect various keys to be present, and often methods also have fail safe defaults if some keys are absent. The defaults are written for the specification, sure, but often they also tolerate unexpected exceptions that happened upstream.

Now when an unexpected exception happens, your program fails somewhere else and the stacktrace is useless. The only way to figure out what went wrong is to debug it line by line.

With python there’s still no elegant solution. I’m now trying to ensure all my methods are typed and use dataclasses and pydantic classes to type and group these parameters but there’s still opportunities for these “fail later” errors. Solutions and suggestions would be appreciated!

Ban the usage of default values or default parameters anywhere outside of top-level / public facing functions. Plus assert everything all the time.

I've gotten into arguments with other developers over it but I'll take the inconvenience in developing now over tearing hair out over bugs later, anytime.

This is where static checking comes in. Static tests should fail if it's assumed (and not asserted) that a key exists.

It took me a little too long to realize that a data class represents a contract about the structure of your data, meaning that no matter how many calls deep you are passing it around, you will always know its structure without having to trace it back to the origin, and that's a powerful thing.

Highlights included:

* DIY “json” logging function that did some obscene string concat work every time it was called and abused global vars; it also didn’t output valid JSON. Suggestions to just use a normal logging library were aggressively disregarded.

* dozens of functions, all of which indirectly mutated this extremely nested dictionary of data. They all had slightly different names, and none of them took this dictionary as a parameter, they just abused global vars. All of them would do these insane checks to ensure that the specific keys they were looking for existed

* none of it was in git properly; the 2 data engineers writing it passed the code back and forth using a google drive.

* instead of importing functions from the Python files they wrote, they’d invoke the functions by shelling out, calling Python <other file.py>, string interpolating the values and then waiting for completion by *waiting for a file of a specific name to be written into the file system.

Oh yeah and when they decided they wanted parallelism, instead of doing the sane thing and using something like joblib or multiprocessing to make stuff easy, they’d just shell out and invoke more Python processes via xargs…

My recipe:

— annotate variables, attributes, arguments and return values;

— run a good type linter (we use mypy) at all times;

— never pass around generic dictionaries: use dataclasses[0], TypedDicts, etc. instead.

That way you define a subclass inheriting from, say, TypedDict and declare that your function only takes that subclass. After that, you’ll get a loud error if you pass any dictionary that doesn’t match the spec (missing keys, wrong values, etc.)—ideally, right in your IDE.

(To reiterate, this would be a pointless exercise if you don’t lint all the time; most IDEs support this.)

[0] You can additionally use them with Pydantic, which can validate data at runtime at a cost of some performance overhead.

Use a language with strong typing?

The problems coming from dynamic typing are just unnecessary, and the only thing that made python palpable for me again was mypy, which introduces static typing into python. But it's not the default, so it still only really works for small things.

I implemented a “check_type” decorator I use in many places but am starting to think I should decorate all the methods in my code with this decorator while building it or something.

    class foo:
        pass
    
    obj = foo()
    obj.bar = "I thought Python was strongly typed?"
    print(obj.bar)

    class foo:
        a = 42
    obj = foo()
    print(obj.a)
    del foo.a
    print(obj.a)

Contrast Python (a strongly typed language):

    >>> 1 + "1"

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      TypeError: unsupported operand type(s) for +: 'int' and 'str'

    >>> [] + 1
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      TypeError: can only concatenate list (not "int") to list

    1 + "1"
    "11"

    [] + 1
    "1"

I'm always wary of these, because you can define that as strongly typed if it's the operation which is defined to perform the conversion internally, which IIRC is how it works in javascript.

For instance the first example will do the exact same thing in Java, because addition between a string and a non-string is defined as converting the non-string to a string then concatenating.

The second operation is not defined such in Java, but in theory you could have a universal toNumber protocol and define the addition of a non-integer and an integer as converting the non-number to a number then adding.

Stand upon a language and look down. You get to raw physics as you go down. All the abstractions are a useful reconception, not the reality. Stand upon the language and look up. You see all the unrealized programs that can be built atop it. Focus in on the programs of a particular type: those that implement a language within the language. Spot one in particular - the one that uses say `@property` `isinstance` and `raise` and `TypeError` to always preserve type safety in every situation a person cares about.

So what I am concerned about is the behavior of the finite set of elements provided by the language and their properties. I can make claims about these concrete things - the addition operator in one language rejects by type but in another it doesn't. But I'm quickly overwhelmed by infinities when I try to do more.

Yes, when objects are involved it's internally translated to:

    ([]).toString() + 1

    > Array.prototype.toString = function() { return 'Boo!'; }
    > [] + 1;
    "Boo!1" 

The answer is that addition first checks if either operand has a "primitive value" which is string-typed, if so it's a string concatenation, otherwise it's a numerical addition, at which point it converts both operands to numbers and adds them.

The primitive value of a `Number` is a `number`, so changing `Number.prototype.toString` has no effect (it's not even called). However if you set `Number.prototype[Symbol.toPrimitive]` then you can influence the rest of the process. Still won't affect an addition of primitive `number` values but:

    > Number.prototype[Symbol.toPrimitive] = function(hint) { return String(this.valueOf()) }
    > new Number(4) + 2
    < "42"
    > 4 + new Number(2)
    < "42"

[numeric conversion]: https://262.ecma-international.org/13.0/#sec-tonumeric

Really think about what you said and it compiles to something akin to:

   type_strength(sample(python_programs, size=1, heuristic=representative_of_claim)) < undefined

   type_strength(sample(python_program, size=2, heuristic=represenative_of_claim_simple_programs_first) > type_strength(sample(javascript_program, size=2, heuristic=represenative_of_claim_simple_programs_first))

Did someone say PHP?

A thorough test suite is also needed, of course.

Unfortunately, a lot of other devs hate to see that your code may actually crash and start asking questions about what scenario could cause it and asking if maybe there's a more gentle way to get out of the error. So, I'll often back down and start having softer error-handling, but on the whole it does complicate things further as the errors cascade and now you have to reason about handling combination of errors that have low likelihood of happening. So, to me, just having an early crash is way better.

Had to deal with the same issue as you.. other devs and managers don't like those errors.. but it makes things fragile and more difficult to troubleshoot.

Does Swift have assert statements? If so, is there a reason you chose this method instead?

To be clear, I tend to use assert statements more than fatalErrors.

Basically the happy path is that this simply never happens. You open a websocket and listen for incoming messages and process them. The actual situation is that you open a websocket and some time later it dies and then you simply attempt to reopen it until it succeeds and resume processing messages. The app has several states: connected, connecting, and not connected and should transition from one to the other depending on what happens.

Our frontend people struggled a lot with this exact issue. They only thought of the happy path and simply ignored any form of expected failure. So the first version of the app worked great for a while until it just stopped working. The fix: "just reload the app" was of course not really acceptable. All that was needed was a little defensive coding: assume this call will sometimes fail and simply try again when that happens. Then also handle the case where retrying will also fail because actually the request is wrong (input validation) and the error is the system telling you that it is wrong. If you don't have any code that handles that, you are going to have a very flaky UX.

    var o = new SomeObject();

    if (o.computeSomething != null && o.computeSomething != undefined) {
       o.computeSomething(...);
    }

    SomeObject.prototype.computeSomethinnn = function () ...

* you shouldn't check for null or undefined, but rather do `o.computeSomething instanceof Function`

* there's no need to do `!= null` and `!= undefined` because `!=` (as opposed to `!==`) actually checks for both

* you shouldn't do the check at all because if you actually mistype the function name all you're doing is hiding the error. Failing sooner is better.

* a missing method should be picked up in the unit tests (but they didn't have any tests at all because "our system is too complex to be tested automatically")

* probably some others...

That team really hated JavaScript and their code showed it.

BTW, the indentation above is not wrong... they did indent by 3 spaces. I read a story about 3 space indents on thedailywtf.com and thought that it was clearly made up... after this team I believe it.

It also helped when tutoring new python students -- when they mixed space-indented code they copied from the internet with tab-indented code they copied from the internet, they'd get all sorts of fun errors. Setting the tabwidth to an even number sometimes allows them to hide. 3, though, really makes them stick out.

As far a I know, ESlint can't detect missing methods, and they weren't even using a linter. TypeScript can, but they weren't using that.

Probably a compromise between 2 and 4?

I do stick to 4 spaces at work though.

I'd also wish for similar rigor from people developing whatever filesystens my data is on. :-)

Fail fast is generally a good idea, if you can do it safely.

Software fails, you can make failures rarer, but you can't make they go away. You have to deal with it, it's not an option.

This involves a lot of thinking and collecting information about potential risks and evaluating their probability and severity.

Then you just mitigate the worst risks, probability times severity (other factors are also possible). Some residual risk always remains.

1. Determine the last sane state of the system, and work forward from there. (Read the servo position and try to go from there)

2. Have a the "recovery" routine to reset the system. (Take all positions to "zero")

3. Just stop. (Yes, I know this can be bad). And ask a human for help.

Stable storage is a key factor in making this philosophy work. [1]

[1] https://qconlondon.com/london-2012/qconlondon.com/dl/qcon-lo...

It's extremely useful in a lot of situations: if you do work on a laptop that gets regularly unplugged, having running test services that know to reconnect makes your life easier. In production, having things automatically reconnect means a lot less restarting of services once whatever root cause problem is corrected. Just shrugging and giving up ends up being a lot more work in the end.

I like to tell junior developers to catch everything they can, and handle it or die as nicely as possible. Of course you can't plan for everything, but you can write around network and disk issues and issue warnings in a way that makes the root cause more obvious. That involves catching errors.

Definitely write code to handle network issues. Don't write code to handle random bitflips, ways to handle garbage coming back from the service you're connecting to, or try to handle OOM errors. Just let those fail.

Do not catch everything you can. That's the whole point of "let it fail". An app crashing is totally fine and expected behavior, in a lot of cases (of course if it's not fine, e.g. someone dies, don't do that but if you're working on that kind of software and taking advice from me, you're super duper screwed).

It depends what you're doing. There's no fixed threshold for "errors that you should handle" so smackeyacky is right - handle the errors you can (but don't spend an inordinate amount of time handling very unlikely errors).

Bitflips are not very unlikely on huge systems so you need to handle them.

In my experience trying to distinguish between "expected" or "normal" errors and "unexpected" or "exceptional" errors is pointless and impossible. It's better to think about the likelihood of errors.

Don’t handle errors that aren’t relevant to your specific code.

When my email went down last week, I wasn't worried. Eventually all messages would arrive.

I like that system.

It wouldn't matter whether that actor/server died from a regular error or a "let it fail" error, the retrying would still work the same.

This depends on the message. I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds.

I think like all pieces of wisdom, sometimes it's OK to let it fail, and sometimes its OK to handle the errors. If anyone ever tells me to always do X or never do X, it's typically not sound advice. The one thing we can always count on is generic advice failing sometimes :)

(And even in this article, you shouldn't always try to handle known errors and never try to handle unknown errors, there will be exceptions)

I do, but only while also associating a stable nonce to the transaction.

Letting BEAM handle that stuff like it is designed to could probably do a better job than your junior devs, and of course then free them up to be writing useful stuff instead.

If the original client is going to retry for failure, including timeout, any intermediate retries are likely to result in signficant multiplication during outages, and that makes for a more difficult recovery.

It's also easy to miss reporting on intermediate retries and your system is running poorly and you didn't know.

Having things automatically reconnect is separate from automatic retries of individual requests.

In python this is

    @retry.retry(exceptions=[RpcException], tries=5, backoff=2, jitter=1)
    def my_external_rpc_call(...):
        ....

Similarly, rpc services can abstract over this entirely, grpc (and presumably others) allow you to configure the retry policy per rpc service or method, and have it reflected everywhere that is used, without writing wrappers[0].

Which really all is to say, once you have solid libraries that handle retries of operations that are known to be error prone (file IO, network IO, things that could lock/block, etc.) you pretty quickly get into "any error implies we're totally boned".

[0]: https://github.com/grpc/grpc-go/blob/f601dfac73c9/examples/f...

In short, I've found retrying well is harder than it looks.

I have an ETL I am told (I switched jobs) that is still working, from 2008. It was built to be a tank, and I also did another forbidden thing: Pokemon Exception Handling. It's a guideline, not a law of physics, and it is fine to resort to a general error catch when you really don't know every possible error (and let's be honest, if you have enough libraries in the mix, some surprises will happen) and you want the other 99.999% of the data to go through. Yes, this one little thing didn't load, and let's log that, let's examine that and figure out how to prevent that going forward, but overall, the rest of the program must continue.

How did it get so tanklike? Every time a little bit failed and it got logged, I figured out what went wrong, fixed it, and then tried to generalize a class of similar errors. After a while, I got into Things I Was Told Would Never Happen in the data we ingested and programmed for when never happened. Reader, never came a little sooner than expected.

Anyway, I largely agree with the idea but there are places where you want the exact opposite, and I think it is important to look for those places lest this heuristic become so stiff it can lose utility.

In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?

My feeling is that, if you have a foundational business process like this, it should be designed to be maintained if there is a serious problem, and it ought to keep working. I know, haha, "the Internet is a series of tubes" but I really wanted this thing to be like a chunk of very uninteresting ductwork that just moves air from one place to another: it should just do its job with as much fanfare.

* Expected errors - Checked Exceptions

* Unexpected errors - Unchecked Exceptions

Idiomatic Java also makes heavy use of asserts, e.g. using the Guava Preconditions library.

The precondition thing, while indeed common, drives me sorta insane. I think it's a pattern java folks need to move on from. You've got this lovely type system (used loosely). If you need a precondition because you've got some fundamental invariant in the system, doing it at runtime rather than encoding it into the type system is such a missed opportunity. If I try to do something inherently wrong, I don't want the code to even compile!

This is echoed in an amazing book called Domain Design Made Functional, which radically changed how I thought about what a type system is and what it can actually do for us if we lean on it correctly (even a relatively crummy one like Java's!).

Perhaps some sort of language level solution could have been found (eg: have explicit interfaces to mark exceptions as expected or unexpected and then exceptions are assigned that using generics or something), but that ship has sailed long ago.

Just fyi, Scala predates Kotlin in not enforcing checked exceptions.

Our good friend UncheckedIOException.

In software, this is known as an assertion failure. When the assert trips, the program is, by definition, in an unknown state. A program cannot reasonably be allowed to continue in an unknown state - it may launch nuclear missiles. The only thing to be done is exit directly, do not pass Go, do not collect $200.

I wonder how easy the certification is for such software? For work I might have to write Do178 code in the future.

Often, we see an "unknown" variant in the error enum, as a catch-all for a library's unexpected errors. Then, anyone who calls them must also have an "unknown" enum. And anyone who calls them, and so on.

In the end, this "unknown" variant is similar to a panic, in that there's very few reasonable reactions to it: Log it, cancel the request, return error 500, perhaps retry.

For this reason, I often recommend people to just use assertions and panics.

Sometimes you want to return 200 even if most of the backends fail. Sometimes one part may want to retry based on any error.

Even aside from this, disallowing exceptions leads to a very predictable control flow, and makes program state able to be expressed in the type system, which is useful for many reasons on it's own.

While yes, it's often just like an exception or panic, I'll take that over exceptions in my code any day

I wouldn't recommend someone only use Result<T, E> and never panic. If we do, then anything that might indirectly be invalid, such as a map lookup or an array index, will have ? operators on it, often every line of some functions. In the end, our control flow is just as unpredictable as if we just used panics and our signal is lost in the noise.

For this reason, I think a blend of Result<T, E> and panics/assertions is really the way to go.

Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and quitting only if there's a serious reason to do so.

Maybe a more useful distinction could rather be "business logic errors" vs. everything else ("infrastructure errors", "programming errors" and "input validation errors"). Business logic should clearly define what should be done when an error happens, to avoid inconsistent state. But infrastructure-level errors or programming errors, you can't do much about them, other than log and/or retry.

That honestly depends on how the language and program are written, python is a great and horrible example of where you can handle any exception even ones that were just created by the program:

    try:
        crash_hard_here()
    except: # by default (and unfortunately) will catch *everything*
        pass # and this is one of the worst offenders inside an except, to just outright ignore the exception and continue as if nothing happened and to not even log it.

This is fine, as one has the choice of exposing error details to outside code or not.

The example of a missing file seems not very good since its a problem that is probably not solved by waiting. A better example is probably a busy DB that is temporary not reachable?

The whole “let it crash” thing comes from Erlang’s process supervision - in practice the DB API isn’t actually retrying. It’s continually failing to connect and if it can’t the process just crashes. The supervisor then notices and starts a new process in its place, this continues until either a process successfully starts or the configured retry count is hit. If the retry count is exceeded then the supervisor crashes, either taking the entire application with it, or more commonly being restarted itself by the next supervisor up the tree.

I also am delighted to see the idea of expected errors here. Another thing I've been doing for a long time is tagging erros with an expected = true property when it's something we expect to see, like, oh, we went to get this oauth token but the credentials were wrong. Expectedness shows up in thr logs now & we can see mych more clearly where there are real problems.

https://hexdocs.pm/elixir/1.12/GenServer.html#c:terminate/2

The only way I could see the VM itself crashing is if it can't allocate memory, then it all crashes.

Firstly, an Erlang VM process is not a Linux child process: “Erlang processes are lightweight, operate in (memory) isolation from other processes, and are scheduled by Erlang's Virtual Machine (VM). The creation time of process is very low, the memory footprint of a just spawned process is very small, and a single Erlang VM can have millions of processes running.”.

An Erlang program can use kernel resources. Objects related to file descriptors, filesystem data, the process namespace, and signal handlers. Kernel timers, semaphores, memory allocations (e.g. for FFI), sockets, DMA, etcetera.

I presume the Erlang library code tracks resources (such as file access function tracking file descriptors per Erlang process), so they can be cleaned up if we “let-it-fail”. I also presume there is a canonical way to register to clean up resources - the equivalent to atexit()[1].

[1] https://man7.org/linux/man-pages/man3/atexit.3.html

Many BEAM terms [1] are easy to cleanup; you don't have to do anything special to get rid of a number, or an atom, or a tuple or a list. Some binaries are heap binaries, they're stored in the process's own heap, so they're easy; other binaries are RefC (short for reference counted) binaries, a ProcBin is stored in the process's memory that references a binary in the global (per node) binary heap; when those are cleaned up, the global binary's reference count needs to be decremented and if it's now zero, it needs to be cleaned up too.

Ports are how file descriptors are generally interfaced with. I haven't done much with port drivers, but documentation for the port driver stop callback [2] says it will be called when the port is closed explicitly or if the port owner is terminated.

Another way to interface with things outside of BEAM is through NIF resources [3], a Native Implemented Function can call enif_alloc_resource to allocate memory and pass the resource back to BEAM code where it can be used as with any other term. When the last reference to a NIF resource is garbage collected (which could be at process termination, or otherwise), its destructor is called, and external resources can be cleaned up at that point. NIF resources aren't strictly owned by one process, if you send to a process on the local node, the underlying object won't be destructed until it has been garbage collected from all processes. Prior to OTP-20.0, NIF resources sent to another node or otherwise serialized would be indistinguishable from an empty binary <<>>, but since that release, serialized NIF resources can be unserialized into a reference to the same resource, but only if it hasn't been destructed already.

If you wanted to build your own cleanup action in pure BEAM code, you would need to spawn_monitor (or spawn_link, perhaps) a new process, which would trap errors and cleanup if the original process died or you otherwise got a cleanup message. Of course, if the code in that process crashed, you wouldn't get your cleanup. OTOH, if the code in your port driver or NIF crashes, that will bring down the whole BEAM node.

Unfortunately, with a quick look, I can't find an example of handling your very real, and reasonably simple use case of a temporary file, automatically deleted on process exit, but it wouldn't be too difficult to build. Of course, there's the question of what to do if the open fails, or the delete fails --- which comes back to the original topic ;)

[1] https://www.erlang.org/doc/reference_manual/data_types.html [2] https://www.erlang.org/doc/man/driver_entry.html#stop [3] https://www.erlang.org/doc/man/erl_nif.html#resource_objects

I've been working in Elixir exclusively since 2016. I do think a lot of the Let It Fail is just marketing from Elixir (and BEAM) but there is a lot of truth in it. In reality you will most definitely not write everything under an explicit supervisor. You will just see errors in function clause matches and add another arity.

Failing (as crashing is now termed ;) immediately when the data didn't match the pattern is exactly the let it fail approach. If the data doesn't meet the expectations, there's nothing to do but crash. Maybe you've got a nice supervision tree, maybe not, but crashing immediately where things didn't match expectations usually gives you the right place to start looking; maybe it was some reasonable data, so you just handle it. Maybe it is unreasonable, so you need to look at where it came from, but usually (not always, of course) you just got the data and are pattern matching it, so you know where it came from too.

That's the point of the supervisor _tree_. Certainly not every process will have its own supervisor, but all processes should be linked to other processes, which are linked to other processors, and at some point you have a process that is quite fundamental to the application and has a supervisor.

As long as there's a supervisor somewhere on that tree, the whole subtree will be restarted, hopefully in a non-erroneous state, and the application will continue on its merry way.

Wouldn't the same scheme be better suited for a procedural language, with deliberately dirty code full of gotchas?

I write a ton of hacky scripts, like last time I needed to rewrite 1000s of xml-s I wrote a crude regex replace for it. It worked 99% of the time, and I fixed up the rest manually. This sort of thinking - that a subprocess might fail due to whatever reason, including sloppy code - but the whole process will keep trucking on would be perfect for the this paradigm.

Additionally this would open the path for stuff like trivial hot code replacement - since in this system, a subprocess that crashes every time - like an invalid program would be handled by the system.

In pure code, an exception could essentially be passed up, and transformed into an error return value at the point where it's called by non-pure code.

In this case, wherever `throw_exception` is called, the programmer would have to handle the exception, and either refactor the return value to indicate an error, change the return value to an Optional (and silently fail by returning an empty Optional), or cause/trigger a side effect (like terminating the thread/program termination early, with an exit code).

How do people approach this in Python? NodeJS? Rust? .NET?

Outside of k8s I try to use the lang's preferred tool. Python -> supervisord, Node -> pm2, etc.

My favorite is hardware (or software) watchdog timers. Very simple to implement and surprisingly effective.

This is not a new concept and it seems to be one of the core components of the Erlang Weltanschauung. It can be generalized further to systems as the principle of "Crash-Only Software," as advanced in this classic paper: https://dslab.epfl.ch/pubs/crashonly.pdf

As the article says, this of course doesn't mean that you shouldn't handle user errors, or even known system errors.

Golang makes it easy to ignore errors that can be ignored and defer/recover provide a way to implement a way to “let it fail”

There’s even an implementation of supervisor trees for Go [0] :)

[0] https://github.com/thejerf/suture

If something is obviously prone to fail you should empower people to do something about it, because usually "letting it fail" will lead to your users switching to your competitor's offering.

How so? Well, the happy path passes data to the next filter. In the case of an error, simply do not do that, and the next filter will be none-the-wiser.

Optionally, log the error, possibly with enough information that the "logger" could retry the operation if appropriate. But as others have pointed out, there often isn't much that can be done.

try { ... } catch (e) { console.log(e) }

then you should probably just "let it fail" since you can't actually handle the error