Hey all! I'm one of the engineers who worked on this. If anyone has questions or wants to get involved, feel free to open an issue on the github repos.
Thank you for your work! Is this an indicator that Google is interested in or using Elixir internally or is it just a response to growing requests from the community?
The latter. But you should realize Google is pretty conservative internally in the main repository because of the amount of code/tooling/etc. While we let people experiment in the main repository, they are controlled experiments. Even if we decided, tomorrow, to move the main repositories away from our current set of programming languages, that would be a fairly long term path, not a short one.
Outside of the main repository, or on google open source projects, or etc, people can generally do what they want, they just don't get the benefits of the above if they aren't using a language we already support very well.
Google's "big" client apps (Inbox / Gmail / Calendar) are a mix of Java converted to JS (GWT) and native JS with the Closure library/compiler. I think, anyway.
It's FUD aimed at a particular language, though. If you know anyone inside Google you probably know how regularly people "experiment" without pushback; all of the internal Dart projects, for example.
I have a hard time finding information about the limits of App Engine. Eg Heroku has all kinds of stuff going on like connection limits and daily restarts and whatnot. How is that with App Engine for Elixir? Is this documented anywhere?
Also, why use App Engine over Compute Engine for Elixir? I understand "if you want more control" but can you elaborate about what kind of "more control" someone would want that App Engine doesn't offer?
A question: what are the pluses/minuses of deploying to App Engine vs Compute Engine. Specifically, what are the tradeoffs of Elixir on App Engine? Thanks in advance.
That's a good question. It comes down to how much you want to control the devops vs how much you want someone else to handle it for you.
App Engine is a platform-as-a-service (PaaS) which generally means a lot of things are done for you. You give App Engine the code, and App Engine builds it, deploys it, scales it, monitors it, applies OS patches, and generally provides a bunch of related services for you.
Compute Engine is more infrastructure-as-a-service (IaaS) which means the service gives you VMs, and you handle installation, build, deployment, and setting up load balancing, scaling, monitoring, etc. the way you want.
There are also hybrid products that fall somewhere in the middle. Kubernetes Engine is an example. Indeed, you can locate most hosting platforms somewhere on this spectrum.
Note also that there are trade-offs specific to Elixir as well. PaaSes generally have to be opinionated about application architecture. Most PaaSes assume a stateless web application well suited for container-based deployment, which runs somewhat counter to Erlang's stateful services and hot code swapping. If you're writing a typical web app (say, using Phoenix) then a PaaS like App Engine will often work very well. But if you are doing something different and managing persistent nodes and processes, you may need to control deployment yourself, which makes a lower level service like Compute Engine more useful.
Gigalixir runs on GCP and seems to fill all of the Elixir specific gaps for a PaaS offering. Just knowing that there are first class API clients for the rest of GCP’s offerings makes Google a pretty clear choice for Elixir devs at this point.
Thanks for the rich reply. So you mention PaaS may not be suited for persistent processes. Do you think it could work well with Phoenix’s channels (web sockets)? As I understand, they create a process per user web socket connection. Phoenix also allows multiple nodes to discover and share these channels’ state AFAIK — are there any tools that make this discovery easier from Google’s side for Elixir to automatically connect the nodes, say when instances are brought up or down? I think it would be good to have this as part of the documentation if there is, as it would be a key selling point if it’s relatively easy.
One question (but first congrats on releasing this!) - is there a way to achieve hot-reloading currently?
This would help handle an important use case for me: ability to keep thousands of (client) websocket connections alive across deployments, to avoid missing a data update.
I wrote a book on Akka with many years of experience with it and I have been using Elixir professionally for 7 months or so on greenfield/replatforming efforts and I have to say that the BEAM VM is really an incredible piece of technology that leaves Akka on the JVM in the dust in several aspects because the VM is actor-model native. I wish there was a statically typed language that ran on BEAM but Elixir and Erlang are great technologies to run in production if you can accept some of the associated risk of newer eco-systems in Elixir (but erlang has your back!).
I love the philosophy and robustness of the BEAM VM, but I can't go back to the stone ages of a language with a poor (or non-existent) static type system.
I don't find staticness to be the gating criterion for me. Go, for example, has a static type system that's IMHO very poorly designed - for example, if you want to multiply a time interval by some integer, you have to convert the integer to a time type and then multiply, which breaks my poor science brain (violates fundamental concepts of dimensional analysis). You also have to do things like pick between implementing something with slices versus fixed size arrays, and generic functions are not an option. I couldn't write fn dot_product{n}(v [n]int){...} for example.
What you probably want is a strong type system. Functional programming languages (I'm thinking Julia as an archetype of strongly typed systems) especially are really good at dispatching over types, and can make robust optimizing inferences using a well-designed type system.
BEAM languages are strongly typed, but there are far fewer types (and no true custom types) like julia. However, gating based on functional guards is simply amazing and more than makes up for a limited type system in terms of code clarity. As for compile-time vs runtime error trapping - part of the point of BEAM languages is "who cares?" An error doesn't bring down your VM. It gets logged, you can see it, and apply a hot patch and fix it if you need to.
I like a type system that leans closer to Haskell than not. As someone else mentioned, my users prefer me to catch errors before compile time :)
Expressive, algebraic types allow me to tell my compiler, IDE, and code heirs what I'm trying to do so they know just by reading my code if I failed. That's indispensible.
> An error doesn't bring down your VM. It gets logged, you can see it, and apply a hot patch and fix it if you need to.
It won't bring down your VM, but it will impact any end-users of your service. Not all errors can be solved by restarting a process or retrying a request (but I recognize the argument that you should aim to catch those types of errors in tests because they are part of the core business logic).
Also just wanted to note that hot-patching has serious costs. Code upgrades/downgrades can be difficult to write and the upgrades themselves also need testing (and this often doesn't get the attention it needs).
Nothing is a slam dunk. I love and hate even my favorite languages (TypeScript, F#, Elm). Hoping Rust will be the silver bullet once I have time to try it.
The problem of static types for the communicating parts of the actor model is not really solved yet. In the meantime there's Alpaca-lang and Simon Fowler did a run-time monitor for Erlang with behavioral types from Scribble. I think we'll see some nice progress in this area in a few years.
Before I send you an object, I’d like a static guarantee that you can process it. Typed channels is one approach used in Cloud Haskell - I think Akka tried and abandoned it.
The guarantee you can process it only works when your compiler has access to both sender and receiver.
When you connect to a cluster to send to something on another node, it’s no different than making an API call to a service that you don’t control. The compiler can’t make any guarantees there without contracts in hand, which can change at a whim on a deploy or when new nodes are added.
For message passing across a cluster, setting up Dialyzer is probably the most realistic approach. Function calls on the BEAM are more like request routing I think.
It is, but code updates still happen. Nodes aren't all identical.
As these things change, the contracts have to be exchanged with every member of the cluster which would not only add significant communication overhead, it would also bypass those compile time guarantees.
That's before even considering things like hot deploys where the code is updated live, while running, mid-execution.
The contract enforcement becomes akin to the goals of WSDL, which added a ton of overhead with minimal benefit aside from making consuming APIs easier for statically typed languages because of generators.
Is the overhead worth the difference from...
Client (this message will fail on this server) -> don't send
to...
Client -> (this message failed)
Especially when talking about a cluster which you own...that benefit is questionable at best.
The feature I'm talking about is a compile-time constraint. The sole purpose of it is to make development and refactoring easier by providing immediate feedback to the developer when they change code in a way incompatible with its interface. It is not really about providing global run-time guarantees. Solving evolving messages through deployment is really a separate topic, and something you have to do whether you have compile time checks or not.
I wrote and maintain Dialyxir. I'm familiar with Dialyzer. What it can't do, is tell me whether any given pid can actually accept and process a message of a particular type. You can write a module interface to your actor that does this, which is what most people do. But it is not quite the same as typed channels. And whoever started this thread was complaining about the "issue" in Akka. Messages are inherently dynamic in this style of communication. Its not the only possible style in either language, but its a valid criticism of this particular approach.
This requirement is incompatible with the actor model. Use channels (Go, Rust) if this is what you want. The lines of communication between actors are untyped because real life networks are untyped. When you receive a packet over the network you always have to look at it before you use it. Typed channels are fine for single machine use, but untyped is necessary for distributed computation.
I think a protocol type system like Scribble is a really good solution to the problem. It's very tricky to verify it statically though, so it will take time to get there.
I think if you have an evolved enough type system, then there isn't such a problem. I have written an Elixir backend for Idris and given some examples of well-typed actor communication here:
I recall someone on here (HN) saying that many of Elixir's strengths are weakened significantly because K8 does similar things (auto distribution, hot reloading, etc.)
Any truth to this? Also, can you distribute nodes on VMs and deploy them to Google Cloud?
---
As a side note, is there a front-end equivalent to Elixir? Elm is functional but is non intuitive. Elixirscript is a thing but seems a bit too early.
I've used Elixir/Phoenix on K8s and there's some truth to this. For example - kubernetes' "if the process dies, kill the whole container and restart it" model doesn't work very well with Elixir (a process within the VM can die and K8s will have no idea). To some degree, K8s is replicating some of the BEAM functionality.
But there's still a lot from Kubernetes that Elixir can take advantage of. For example - service discovery, autoscaling, clustering, secret management - all made easier with a K8s deployment and Elixir.
Unless Kubernetes is killing the container without respecting the shutdown time of the underlying processes, this should not be problem.
Once Kubernetes sends an exit signal to the VM, the VM will start the shutdown of its processes, and everything should terminate gracefully.
It is OK that Kubernetes doesn't know about Elixir processes terminating, because it is not its job. It is the job of the Elixir software to express its start-up and shutdown guarantees. If you don't want the VM to start when you don't have a database connection, then you should start a process right after your connection pool starts and before your endpoint runs that attempts to issue a query and act accordingly.
Hmm I'm not quite sure I follow the process examine. If your started process crashes (let's say supervisor fails to start due to too many failures), k8s would restart the process (your app) to try again. If a managed process in elixir dies, you could make that bring down your whole app, but generally you would want to handle it yourself.
This is similar to an app server in Ruby. A single worker might die, but it's still being managed by a central coordinator. If the coordinator dies, everything is down and the app restarts
Not trying to be flippant, but by extending your analogy: if the Kubernetes supervisor dies (which is conceivable, no technology is infallible), then you are in the same scenario as something bringing your whole Elixir VM down. You can always run your Elixir VM using something like supervisord if you need another watchdog.
BTW, it's very hard to bring BEAM down. Most exceptions will bring kill the faulty process, but leave the supervisor tree untouched. Only "unmanaged" code (for example, calling a C library using a NIF) might crash the whole VM.
>BTW, it's very hard to bring BEAM down. Most exceptions will bring kill the faulty process, but leave the supervisor tree untouched. Only "unmanaged" code (for example, calling a C library using a NIF) might crash the whole VM.
Not completely true. Try to allocate more memory than available and then see what happens.
I'm actually curious. Do you have some first-hand experience with that kind of issue? I've only played around with Elixir and my only experience deploying BEAM-based applications on production has been RabbitMQ and CouchDB and both of those were very solid.
Also, isn't ENOMEM a standard error that you can handle using the standard error handling techniques?
I have and it wasn't good.
It is impossible to handle out of memory exceptions inside the BEAM. At best, you can hope for a crash of the VM. At worse, it just hangs.
And one will most likely not find out about this until they hit this scenario.
Interesting, definitely worth knowing beforehand (I was thinking of implementing a small service that could be pretty memory-intensive using Elixir.) I wonder if there's a way to try and manage memory to avoid ENOMEM.
If BEAM is allocating memory it must either be copying a message from one process to another, allocating space in a process for its stack, or for some bookkeeping like to spawn a new process. In the first two cases why doesn't it just kill the process and send the usual link and monitor messages to let other Erlang processes handle the failure?
Haha, yeah, that's the problem with ENOMEM handling in general: unless you have a very clear path to free memory immediately and then try to do some reasonable cleanup, you are screwed. Back in the bad old days of running Windows machines with 8mb of memory, I'd religiously handle that kind of error code... but I can't recall the last time I actually bothered. Commoditized hardware makes us lazy.
True true, maybe there's just a single part that I was hung up on in parent comment
> (a process within the VM can die and K8s will have no idea)
I suppose that I see nothing wrong with this and would say that it's not weird or unusual to be the case. They're separate things, although they both achieve visually similar goals (keep things running as much as possible).
My point is less about K8s vs Elixir but rather a statement that this seems acceptable and shouldn't be a reason to rule out something like running Elixir on K8s.
BEAM VM comes up and tries to connect to the database. The database is temporarily unreachable from that machine, but only the Ecto process dies and BEAM VM doesn't die. Since the last process (BEAM) in the Dockerfile is running, Kubernetes thinks it's healthy and will try sending traffic to it.
Compare this to something like node, where the node process will die if it can't connect to the database, and Kubernetes won't try to direct traffic to it and will restart it for you. You can do this with Elixir supervisors as well - that's why I said Kubernetes replicates some of Elixir's functionality.
It's definitely possible to work around all this which is why we ended up using Kubernetes for our Elixir deployment (and I'd recommend it). It's just these little things that make it clear that Kubernetes was designed with something like node in mind and not BEAM.
The behaviour you are attributing to Node is really just the behaviour of poorly constructed software. There are at least some Node-based software products which correctly come up and wait for dependencies to become available -- and gracefully handle their subsequent transient unavailability.
There are 2 types of probes: readiness and liveness [1]. Define your readiness probe so it passes when you are ready to take the traffic (eg. connected to the db).
Elixir is great for stateless apps. So are a bunch of other platforms.
Elixir is exceptional at stateful apps. And it is true, many of those things are now handled through K8S. However, K8S doesn't handle everything and requires a lot more ceremony and infrastructure than what you can get within the Elixir/Erlang ecosystem.
I think there might be a way to work with Elixir + K8S in a way specific to the domain you are working with. Not everyone needs to be able to hot-reload code without dropping live connections, but if if you do, and you want to deploy on K8S, then you can try writing a K8S operator that encapsulates that idea. However: https://gravitational.com/blog/running-postgresql-on-kuberne... (which is more about how every distributed system requires deep domain knowledge, not just specifically about PostgreSQL)
I have not deployed Elixir on GCP. I have deployed other things though, and I can tell you the private networking is much better set up than AWS. As far as distributing nodes, you can do that with any of the cloud providers.
That’s just crazy to me. The amount of effort you have to put in to achieve this with Kubernetes is way beyond what you would have to do to write you application in Elixir. If you already have multiple services in another language, it may be worth the effort, but the BEAM gives you nearly for free what is relatively expensive to get elsewhere.
If you looking into OTP, you'll see there's quite a but more and to some extent it does things differently than K8 (e.g., Observer, fault tolerance, concurrency and vertical/horizontal scaling w/message passing, etc.). See: https://www.manning.com/books/the-little-elixir-and-otp-guid...
The two options you mention are the ones I see used most often (with React & Vue being 3rd/4th), but I wouldn't say any of those are "frontend equivalents".
> As a side note, is there a front-end equivalent to Elixir? Elm is functional but is non intuitive. Elixirscript is a thing but seems a bit too early.
Aside from both being functional languages, they have relatively little in common. Purescript has Haskell-style purity (with synchronous impurity via monads) and doesn’t have an inherent concept of processes or message passing like Elixir/Erlang do. Purescript is statically typed with no (I think?) macro system, and elixir is dynamic and makes extensive use of macros. Etc.
You’ll need to evaluate GCP’s peripheral offerings for database, APIs, pricing, etc.
The biggest thing that this does is give you first class access to the ecosystem APIs and with native setups for AppEngine (Heroku-ish), Compute Engine (do it yourself on a VM) and a provider in Gigalixir which runs on GCP that gives a Heroku-like experience but is tailored specifically to everything Elixir needs (hot loading, clustering out of the box, etc).
I was reading up on Gigalixir thanks to this thread, is it basically a more "batteries included" version of AppEngine specifically targetted for Elixir?
Seems to be. I’ve read about it for a while but still haven’t tried it out. Planning to soon though.
It’s just nice to see a hosting option that tailors itself to the more unique needs of the BEAM when the rest of the industry is focusing on stateless containers.
I hate the idea of vendor lock-in, but years now of reaping the benefits has convinced me that it's just better to accept the lock-in and take advantage of your chosen platform's features. Remaining platform agnostic is difficult, complicates your code, and forces you to miss out on great services. these days I pick a provider and stick with them. It used to be AWS, but Google has been getting really good. I've been using Google cloud a lot.
Yes we do! We’ve been able to shake out a fair amount of platform bugs that the Beam VM had on GCP too! Thanks to the google team for taking them all seriously and fixing them. Some were particularly nasty :)
Surprising development. I wonder if this will complement Gigalixir or compete with it.
The ability to connect between nodes, avoid mandatory restart, avoid connection limits and keep websocket connections alive easily gives compute engine a significant advantage over platforms like Heroku.
As far as I understand Heroku lets you connect between nodes (Dynos within a private space with dns discovery [0]) and also supports websockets [1]... for the mandatory restart, wouldn't you want to design your process as stateless anyway?
Typically I design applications to be as stateless and tolerant of restarts as possible, as they do get restarted during deploys anyhow. That is a minor issue compared to the difficulties around connection restarts and communication between nodes.
It is wonderful that Heroku now offers exec, which at least allows using observer and other remote debugging tools. The tooling keeps getting better—but there is no denying that it was designed for Ruby deployment, and working the BEAM has significant production differences.
If I recall correctly, just turning on private spaces is not within most people’s price range.
With most things you’d want stateless. Erlang/Elixir is designed with 3 built in databases, the ability to hold states on isolated processes and per process supervision and recovery. It’s a different animal than most.
> wouldn't you want to design your process as stateless anyway?
> Sorry if I'm missing something obvious!
That's exactly it. The value proposition for Erlang & Elixir is essentially that they provide great abstractions for managing state and they make great back-end languages for creating stateful applications.
The problem with private spaces is the pricing is out of reach of the vast majority of early-mid-stage startups since it is only available under the enterprise model.
I'm the founder of gigalixir.com, and I haven't dug into this very deeply yet, but I'll just say that from my perspective, this is great regardless of if it competes or complements gigalixir.com. Anything that helps grow the Elixir community and helps drive adoption of Elixir will benefit everyone in this space.
This is really cool and all, but I really wish Google Cloud would move things out of beta faster. Cloud Functions are still in beta despite them being GA at both AWS and Azure for a while now. Enterprises typically won't use beta software which makes it nearly impossible to pick Google Cloud for an enterprise.
We're not doing anything special to help set up distributed Erlang at the moment. It would probably be difficult with App Engine because it's designed to manage stateless containers. But I could imagine setting up such a network with Kubernetes Engine, or definitely with Compute Engine VMs.
It currently doesn't do automated clustering of nodes out of the box. It should be possible to build a clustering strategy for libcluster (https://github.com/bitwalker/libcluster) for App Engine. Note that there's already a kubernetes strategy which could run on the Google Kubernetes Engine.
I spent a lot of time last week trying to get elixir setup on gcloud, but variables were the one thing I could never figure out and I eventually gave up.
Is there some magical way to get variables and secrets in gcloud without having to commit them to a repo?
More like, Kubernetes (and Google Kubernetes Engine) is the right level of abstraction to actually have a first-class object-type of "secret."
Compute Engine is just raw IaaS, like EC2. You "store secrets" on GCE by writing them to a volume and attaching it across your cluster, or putting them in a common database. Or just pushing data onto your instances after they're up using Fabric or something. You know, just like if they were real computers.
App Engine is made to be a complete and self-contained PaaS (much moreso than anything like Heroku), to the point that it doesn't really use external credentials for anything. For example, you can access BigQuery tables from App Engine without supplying any "BigQuery credentials." Oddly enough, if you do need to connect to third-party services (or, say, Google Cloud SQL) from App Engine, they recommend that you store the credentials to do so in... a BigQuery table. GAE is proprietary and weird. It's basically its own "abstract machine."
Kubernetes Engine, in comparison, is exactly for the use-case of "deploying workloads" of regular potentially-stateful, potentially-long-lived POSIX apps that need things like DB connection strings passed to them. If you're a modern backend developer who burns Docker images like people 10 years ago burned CDs, Kubernetes is the service you should be using on any compute cloud you interact with. Everything else is an impedance mismatch.
That’s a really interesting addition to app engine. Great idea too - try to be first to an up and coming platform rather than try to move applications that have deployed in one way for a decade.
This seems to be using App Engine's "flexible" environment. It's less appealing to me than "real" App Engine since as far as I know there's no free tier.
I'll have to look this over, I'm running a server cluster on GCE + Appengine currently aggregating and triggering alerts against about 300k device readings per minute. Huge pain point keeping things synchronized between GCE on java and the elixir backend.
* Elixir clients for Google APIs: https://github.com/GoogleCloudPlatform/elixir-google-api
* Elixir runtime for App Engine: https://github.com/GoogleCloudPlatform/elixir-runtime
* Elixir examples: https://github.com/GoogleCloudPlatform/elixir-samples