(Inspo 👉 eli.li/december-adventure 💌)
Full schedule again today and didn't have much time to adventure. I did have a great housing union meeting though ❤️🔥 As a follow up to the 01.12.24 entry, I was glad to see some movement on the issue about warawara not being able to connect to the lurk.org XMPP server using a TLS config. Much thanks for @SamWhited for picking this up, recommending a work-around and getting a patch out, super cool! The fix on our side was pretty simple. I've managed to build another static binary and let it rip on the server. I'll check in again on it in a few days 😌
No rest for the wicked, as they say. A serious amount of life-admin and comms back & forth really took me out today, but I still had a few cycles to think about what I want to focus on this month.
I should “probably” be working on abra which hasn’t seen a release in a while and is more or less stalled in development pending more spoons. It just feels a little bit too much like “work” (although at the moment, that would be unpaid work until we figure out the budget again 🙃). So, yeh, not picking that up just yet.
I was keen to revisit a galaxy brain I had about a Capsul inspired system for community-hosting. I had done some prototyping around this a few months back and it was starting to take shape.
The idea in short is to find a “3rd way” between with specialised hardware in data centers (inaccessible and expensive) and self-hosting (too much “self”). On the one hand, people who know how to interact with the bureaucracy of a data center, get a suitable server with virtualisation support, configure it and foot the bill etc. are quite rare. On the other, people who can self-host are less rare but are unlikely to let others they don’t trust a lot have shell access to their homebrew server.
Since I’d rather get out of the data centers, I’m wondering about all those single board computers, old laptops and 3rd hand machines people have set up. So many people are still renting VPSs from Big Tech and meanwhile, there’s a lot of unused compute at home. Can we have some of the good aspects of the "VPS culture": private virtual spaces and simple graphical interfaces for maintenance? Can we transform aspects of self-hosting into community-hosting: non-specialised hardware of all kinds, (invite-only?) multi-tenant and low maintenance?
I first thought “Docker”. However, there is too much sharing of the host system and the container. Breaking out of containers is sometimes trivial. I did discover Incus which is actually pretty fantastic. It has the concept of "system containers" which provide more isolation but are still lightweight.
I don’t have it all worked out yet, but I do have a mini-demo. It shows an initial prototype of a “self-service” TUI where you can create a system container and access it like a VPS (running on my homebrew machine). It can mutualise the same machine and searches for a free port to proxy from the host to the container. It isolates this container from other users on the same server using an isolated project. It also takes care to put the SSH key into the container via cloud-init. The host system uses the command=... <ssh-key> trick in the ~/.ssh/authorized_keys file to limit SSH access to run the TUI.
Incus has pretty excellent security considerations, putting measures in place to limit the damage of container break outs and using sockets and user permissions in a way that reduce the blast radius of malicious users. I’ve been quite surprised by how much care is taken by the maintainer. I’ve even got a patch into the core of Incus! The maintainer seems really cool 🎉
The only missing piece for my prototype is a way to serve HTTP/S to these containers. I can imagine a scenario where people use the “self-service” TUI to create and register domain names against the containers. How will the system pick up these new containers and route requests to them?
Since my prototype already has 3 moving parts (Incus, the TUI, the intermediate service (incus-admin proxy to avoid users having root access)), I’m not very keen on adding another one. However, it seems unavoidable. I believe I have a good idea though: I can write a Caddy plugin to automatically read the labels of Incus containers and use On-Demand TLS.
So, tonight I was reading up about Caddy plugins and I believe I just need to adjust and re-work this one to work with Incus labels. The code doesn’t look too gnarly. I could then manually test firing up a customised caddy + plugin alongside a few Incus containers running stuff on port 80 and hopefully see requests get automagically routed with proxy terminated HTTPS.
There’s a lot still up in the air but I think this could be useful at some point. It doesn’t cover every community-hosting use case. It can cover some simple ones though. I’m not really sure about the setup I have now and would like to see many things simplified.
We can imagine groups/communities who just want some “on-demand” VPS-like thing to learn, host static files, run some basic web services etc. and one of them has a SBC and is willing put it to good use. If the system can help reduce the maintenance tasks of the homebrew sysadmin and supports collective approaches to maintenance, it could be nice to see where this goes.
It's been a hectic few days! All good things, but still a lot. Today was mostly spent talking to neighbours, comrades and accomplices at Solidarity Network(ing). Great chats 💖
Once home and rested (aka "face down on couch"), the urge to kick off this adventure soon arrived! With the refactor in cerca (mentioned below) rounded off with help from cblgh (🤘) and a large piece of work underway to enable invites, I'm temporarily dropping tools on cerca hacking. There's plenty of new things to test out and I don't want to disrupt the invite hacking flow 🌪
One of the cerca-adjacent things that I wanted to take a look at is our cerca XMPP notification bot, warawara (written by ugrnm (⚡)). warawara seems to be losing its connection to the server when up and running for a while. Also, I discovered that it seems to do some extra notifications on restart. So, I sprinkled some commits and will check in on the logs tomorrow.
It's not December yet, of course. However, I was at Calm Coding this evening and I feel like the idea of December Adventure has much overlap. So, here's a little sprinkle of pre-december-adventure hype, let's say. With the permacomputing.net gang, we've recently started to test out an instance of cerca and I've been hacking on various parts of it in response to discussions and needs. It's been quite a nice dynamic so far, bringing those discussions to the cerca issue tracker and then balancing that with upstream concerns. One of the things we ran into on deployment was how the sysadmin functionality was split into separate binaries and this was a bit inconvenient when setting things up. So, here's the changeset to merge all the things together.