1. 82
  1.  

    1. 17

      The trend continues! I wrote the Kubernetes version. I’m sorry to everyone who finds this annoying, I totally understand.

      I’ve played around with Elixir and I definitely agree there are many underappreciated concepts in the BEAM languages. At my first job, I was using Django and was tearing my hair out debugging Celery jobs, feeling like there had to be a better way.

      A common interpretation of these posts is that everyone should just use the thing they accidentally built (compilers, databases, Kubernetes, Erlang) but I don’t interpret it that way. You’re welcome to build a simpler version or a better version, but you’re not likely to be successful if you don’t even try to understand the existing versions.

    2. 17

      We “built an erlang” and have no regrets: https://pipe.pico.sh

      1. 3

        This is lovely.

      2. 2

        This is so cool!

    3. 31

      OT: this “You have built X” meme format makes me very happy. Kudos to those who recognize good ideas in the wild :)

      1. 6

        Most of them seem to be better-explained, usually more positive, variants of Greenspun’s 10th rule.

      2. 2

        I need more

        1. 1
      3. [Comment removed by author]

    4. 9

      Erlang has always confused me from a distributed systems perspective. I get the benefits of Erlang on a local machine, but it’s confusing to me how it can scale across machines.

      I get that “if all actors communicate via message sending, it doesn’t matter whether the destination actor is remote,” but doesn’t that mean that message passing needs to have weaker semantics than if all actors are local? OP mentions supervisor trees, but how can supervisor trees work in a distributed setting? How do you know whether there’s a network interruption between nodes vs a node actually dying?

      OP also mentions Kafka, but AFAIK kafka persists messages until they can be fully consumed, whereas I thought the semantics of erlang message passing is that messages get lost if the receiving actor dies. (OFC the sender could use monitors to figure out if the actor dies and retry sending the message; assuming that the message is idempotent or if the sender can otherwise know that the actor hadn’t processed the message. But, as I mentioned above, it’s not clear to me how you could have sound distributed monitors.) But it seems like erlang’s semantics are less well-suited to distributed applications than kafka’s.

      I get that you can build a distributed system in Erlang, and how introspection and code reloading may make working with individual computers easier, but I don’t see how Erlang makes the actual distributed computing part any easier.

      1. 5

        Generally, you don’t supervise remote processes, each node supervises its local processes. But for example if you have a sharded cache, you have a central local process that monitors the remote supervisors, so that it knows (via a VM message) when a node connection is lost. When that happens, the local process updates its list of known nodes, and could for example start a rebalance operation. To query the cache, you ask your local process which process own the desired shard, an then you call that process with a timeout, regardless of whether it’s remote or local.

        1. 3

          To go with that, I have had a machine hosting an Erlang VM go down (either a physical hardware machine, VM, or Docker container), and I’ve had Erlang processes go down (naturally, bugs that are my fault), but I don’t think I’ve ever had the Erlang VM itself actually crash other than when I was doing Really Bad Things with NIFs. I’m not saying it’s impossible, but relying on Supervisors to let you know when a remote process has died has been exceptionally reliable for me and the generally short timeouts on cross-node messages (5 sec default) means that you do quickly detect if the entire node has evaporated and can handle that appropriately.

          1. 5

            Monitoring for a remote process’s death/disappearance is reliable, but supervising it seems weird. What’s the restart strategy when the node is down ? How do you handle the node coming back up (because 90% of “node down” events are actually “network down”) ?

            1. 1

              carefully, but it’s however you choose. Generally you select a restart strategy (e.g. “try 5 times, with 10 second pauses between each try, and fail hard/report if you reached max retries”). Often, the restart strategy will be enough. For stateful things where progress might have been made on both sides of a netsplit, probably you are looking for a reconciliation activity instead.

      2. 4

        I get that “if all actors communicate via message sending, it doesn’t matter whether the destination actor is remote,” but doesn’t that mean that message passing needs to have weaker semantics than if all actors are local?

        Semantic is exactly the same. Message will be delivered if the receiving process is visible. That semantic is the same in remote and local message passes. The difference is that it is much easier to notice dead process locally than it is remotely.

        How do you know whether there’s a network interruption between nodes vs a node actually dying?

        You do not know, as from the viewpoint of the supervisor/monitoring process there is no difference. Process became unavailable at some point and you simply can react to that. Erlang do not provide all handling for you, it just provide basic building blocks and you can construct whatever you want on top of that.

        I don’t see how Erlang makes the actual distributed computing part any easier.

        In an away that it removes from the equation the basic stuff of process communication and basic monitoring, so you can focus on the hard things instead (as long as you really need these hard things, because sometimes weaker model is enough).

    5. 8

      As someone who never again wants to write a k8s yaml file, nor a yaml file to give to the cloud provider to stand up a k8s cluster to consume additional yaml files, is learning a BEAM-based language for me? Or do these BEAM VM apps just get deployed to k8s pods defined with yaml files?

      I feel since I’m a distributed systems guy I should probably just learn Erlang or Gleam for the vibes alone.

      1. 23

        Learning Erlang* / OTP is definitely worth it if you do distributed systems things. It has a lot of good ideas. Monitors, live code updates, and so on are all interesting. Even if you never use them, some of the ideas will apply in other contexts.

        If someone told me I’d built an Erlang, I’d consider it praise, so the article doesn’t quite work for me.

        *Or probably Elixir now. It didn’t exist when I learned Erlang. Also, I am the person who looks at Prolog-like syntax and is made happy.

        1. 5

          Another huge benefit is that it gives you a new way to think about and solve problems. The main application I maintain at work is a high throughput soft-real-time imaging/ML pipeline. In a previous life I worked on a pretty big Elixir-based distributed system and this imaging system is definitely highly Elixir/OTP-inspired despite being written in C++. All message passing, shared nothing except for a a few things for performance (we have shared_ptrs that point to immutable image buffers, as an example). Let It Die for all of the processing threads with supervisors that restart pipeline stages if they crash. It’s been running in production for close to 6 years now and it continually amazes me how absolutely robust it has been.

          1. 2

            All message passing, shared nothing except for a a few things for performance (we have shared_ptrs that point to immutable image buffers, as an example).

            Funny thing, Erlang does exactly the same thing with large binaries.

      2. 7

        The vibes, as you suggested, are a great reason.

        1. 3

          That was excellent, thank you!

      3. 3

        Given your interest in formal methods, I can recommend having a look at Joe Armstrong’s thesis. I think he makes a very good case for why Erlang behaviours make sense from a testing and formal verification point of view (even though the thesis itself doesn’t contain any verification).

        Erlang’s behaviours are interfaces that give you nice building blocks once you implement them, e.g. there’s a state machine behaviour, once you implement it using your sequential business logic, you get a concurrent server (all the concurrency is hidden away in the OTP implementation which is programmed against the interface).

        The thesis is quite special in that he was 50+ years old when he wrote it, with like 20 years of distributed systems programming experience already, when compared to most theses that are written by 20-something year olds.

      4. 2

        This reminds me the effort of couple projects years ago to use BEAM on the cloud like Erlang on Xen (ling) or using rumprun unikernel, there also other things like ergo (“golang otp”).

      5. 1

        Yes! I was in the exact situation as you and learned elixir. The processes API is straightforward and relatively simple. Elixir documentation on the topic is superb. On top of that elixir is a great language. Functional and very ergonomic. Erlang is fine too but elixir has that quick to write syntax sugar comparable to python or ruby.

    6. 3

      I remember in the 2000s or so, lots of people doing their own frameworks for PHP - they were the 2000s version of You have built X of the time. Building your own MVC framework, your own deployment coordinator, or even your own language runtime – these are valid exercises. The new thing of You have built X trend is the acknowledgement that a) you ended up deploying your homemade contraption into production; and b) you failed to realized what you did.

      I attribute this change between now and them to the a set of dogmas that were slowly imparted into a whole cohort of developers, usually thrown around in technical debates as piece of conventional wisdom that only the illiterate wouldn’t be aware of them, like: YAGNI, Clean Code, Design Patterns, theoretical vs in-practice Big-O notation, HMVC etc.

      In 2000, when the distance between thought leaders and practitioners was larger than in 2024, developers would rely a lot of the feedback of the current task to understand if what they were doing was in a good path or not. Of course, lot of people had the wrong intuitions, and that created the room and the void necessary for thought leaders’ ideas to percolate into the developer ecosystem as remedies to these genuine mistakes. Then this generation started propagating this knowledge, but without the prior frustrating experience, to the following generation developers who, in their mind, they were just being educated by older and wiser colleagues (and often, in aggressive and violent terms: “You moron! you f****d this code up! That’s now how you do it! DRY IT NOW!”.)

      Also in 2024, our craft got way more complex than it was in 2000s. In 2000s the world was still moving from a Desktop Application single-user centric applications to narrowband Internet; deploying a web application was mostly about uploading PHP files in a shared hosting LAMP server; nowadays you have to use containers, functions, etc.

      The point here is that we are now giving rise to a whole generation of engineers that have been constantly shooting themselves in the foot, and they now are imparting this experience and knowledge to the new arrivals.

      In 2000s, YAGNI meant that you should try as much as possible to avoid overengineering solutions; in 2024, simple and boring technologies may not compose well together, now people are making ad-hoc compositions and in this process, they end up rebuilding celebrated solutions like Kubernetes and Erlang.

      I can’t wait for the next jargon… Something like: JUTBS - “Just Use This Boring Stack”.


      By the way, that’s what’s already happening in the Cryptography space. For quite sometime, the cryptography industry has been moving from “Don’t Roll Your Own Crypto” to “Just Use This Simple Crypto”. IPSec/IKEv2 offers lots of cryptographic configuration options, Wireguard gives you zero knobs; PGP gave you a lot of knobs for file encryption, age gives you nothing; TLSv1.2 libraries often expose cryptographic preference APIs, TLSv1.3 offer fewer choices and implementers are encouraged to develop algorithms that choose them for you.

    7. 2

      If only Erlang could “notify your services when data changed—nothing fancy.” :-)

      1. 2

        It is quite easy to implement such notifications on data changes.

        1. 1

          Yes, it’s easy to cobble together something, but (a) getting a robust solution is hard and (b) there’s nothing special about Erlang that makes it any easier. Well, monitors/links are a useful primitive for proper robustness, but other than that it’s the same as everything else.

    8. 1

      So. If I have an erlang (built by me or otherwise), do I need a kubernetes? Or vice versa? If I need both, what are my options? If I don’t, what are the fundamentals I need and how do we commoditize them across many languages?

      1. 4

        Kubernetes gives you fault tolerance at the infrastructure level (a crashed pod is restarted, a node is shutdown and pods are migrated to another, etc…).

        Erlang gives you fault tolerance at the application level (this network call failed due to temporary network failure, this could cause invalid state in this tiny part of the application, let’s restart only that part).

        You might want both kinds of fault tolerance, and no, Kubernetes and Erlang are not the only options there.

        1. 4

          I’m sitting in an airport right now and can’t remember the name of the library off the top of my head, but there are Elixir libraries that do a really good job of bridging those two types of fault tolerance. When k8s pods are created they can auto-join an existing Erlang cluster and once the service is up it can join the load balancer. The messaging backend I worked on used it with auto-scaling very effectively. The frontend app communicated with the backend using Websockets (actually a Dart implementation of the Phoenix.Socket client) and would robustly detect when a connection died, reconnect, and pull down the necessary state.

          K8s was definitely complex overkill for the problem (for the load we had we only usually had a handful of pods), but given that that decision had been made way before I joined the team… I was quite happy with how well the Erlang cluster and the K8s cluster integration worked.

          1. 2

            I’ve used libcluster in the past for this, is that what you were thinking of?

            1. 1

              That’s the one, thanks!

    9. 1

      Is Erlang really that good?


      I’d like to learn Gleam but after trying my hand at Rust for web development I don’t think I have the innovation tokens left to go deep on another entirely unpractical setup.

      1. 2

        If you want to do webdev in beam, you should really look at Phoenix, it’s a great framework, Elixir’s killer app. Erlang is amazing (I actually prefer it to Elixir), but it doesn’t have a high-level web framework, keep it for lower-level or non-http stuff. Gleam is very promising if you like stronger types, but is ecosystem is still very new. You can mix and match of course, all beam languages call each other without overhead, but it makes your project a pain to maintain,

        1. 3

          My biggest pain point with Phoenix is the generators. I don’t like the opiniated templates they use to structure the code (I don’t structure my projects the way they propose), and setting up Phoenix by hand is tiresome. So what I end up doing is spending lot of time refactoring what the generators generate.

          If I’m writing an API in Elixir, I’m not gonna use Phoenix, just Plug and cowboy (or bandit).

          I’m also not a fan of Ecto for totally subjective reasons.

          And I think HTMX is more straightforward than LiveView (I worked a lot with HTMX, not a lot with LiveView, so I may be wrong).

          1. 2

            Same here, I tend to avoid generators and hand-roll a lot of stuff. But if I was churning lots of APIs each year, I’d probably embrace generators more. Opinionated frameworks are both a blessing and a curse.

          2. 1

            I share this opinion and decided to change the generators to my own personal preference. It is still a work in progress (all generators work, but need refinement), and lives here: https://github.com/Zurga/punkix.git

          3. 1

            I’ve been using Phoenix since 2015, and I don’t think I have ever used the generators. Don’t be put off by them! They are 100% optional.

            Phoenix is lightweight enough that if there’s any possilbiity that at some point I’ll want to serve HTTP from an application (even for uptime checks), I use Phoenix. It’s just a bit of glue over Plug and gets out of the way otherwise.

        2. 1

          Sure, but what I was getting at is that the likelihood of being able to use it in work/production seems relatively small. I’m currently in Nest and if I learn one more it’ll likely be Kotlin/Spring just because it’s everywhere.

          1. 3

            My point is that Phoenix is a very practical setup. It’s mature, featureful, productive, performant. No framework in Erlang, Gleam, or Rust come close to it. It has many happy production users. If you need something more popular that’s fine, but then you might as well look at Node or Go instead of Kotlin.

      2. 2

        Yes, BEAM languages are amazing, and so is Phoenix. One big plus for running a web server on a BEAM language is that the server never goes into an invalid state, so I never have to restart the server. I just remotely git pull the updates server-side and have Phoenix automagically reload it in. If one of my views ends up crashing, I just write and git pull a new update and everything is ready to go again, with all previously existing non-crashed state intact, no restarting the server ever.

        So I second the decision to use Phoenix.

        About four months ago, I was going to settle on Gleam for my web app, but I was unable to get even a simple websockets application going with their two available web libraries. I think they just simply don’t support websockets yet. If you’re still interested in Gleam, I suppose you could write the low-level application logic in Gleam and have Phoenix only handle the view/controller component, but I haven’t crossed that bridge myself so I can’t say how well that will go.

    10. 1

      I’ve had a long-time interest in the Erlang VM and languages the run on it but I think I don’t understand the durability aspect. This post makes allusions to how this works in Erlang but I haven’t been able to find more details. Anyone have links to relevant tools/parts of documentation that can explain how one prevents data loss in a distributed Erlang system?

      1. 1

        Mnesia is Erlang’s database.

        1. 1

          I’ve heard of Mnesia, but was more wondering about the patterns. Do folks persist all messages before sending for an at-least-once guarantee?