Blogs on Sourcehut

Proposed price increases for SourceHut

Mon, 01 Dec 2025 00:00:00 +0000

As we wrap up our end-of-year budget here at SourceHut, we find ourselves with a good opportunity to reconsider our pricing. We have always promised to evaluate any potential pricing changes in consultation with our community, which is why we’re bringing our proposal forward to you now for discussion. We’re looking forward to hearing your thoughts!

tl;dr: We want to increase the base rates to €4/€8/€12, with current users grandfathered into their current rates. Furthermore, users in the EU who pay in US dollars will soon have to pay VAT when we rehome them with our EU business. Free service is still available to users who can’t pay. Read on for more details!

New here? SourceHut is a software development platform which offers repository hosting, bug tracking, continuous integration, mailing lists, and more to projects hosted with us. SourceHut is 100% free and open source software. Welcome!

SourceHut’s pricing has been the same since we first started accepting payments in 2018: $2, $5, and $10 per month, with a “2 months free” discount for users who pay annually. There is no difference between the price tiers; we use an honor-system model of “pay what you think is fair” based on your financial means and how much you use SourceHut.

In the years since, inflation has effectively reduced the prices by 30%. All of you have been subject to the same inflation and stagnant wages in the meantime, and we have preferred to take the hit to our margins rather than pass the costs onto you in an increasingly stressed economy. But, at some point, we will have to consider raising our prices.

Before we get into the proposed changes, I’d first like to reiterate that SourceHut is committed to never pricing anyone out of our services. If you cannot afford to pay for your account for any reason, then you are eligible for free service by applying for financial aid.

Our proposal is to set the standard prices for new customers from January 2026 onwards in Euro at €4, €8, and €12, and in US dollars at $5, $10, and $15.¹ In addition to these standard rates, the billing UI will be updated to offer a reduced rate of €2/$2 (the same as the present-day minimum payment) in addition to financial aid as an option for users who cannot pay the standard rates.

A mock-up of changes to payment process shows the offer of a reduced rate hidden behind a button click. Click to view the images at full size.

Users with an existing paid subscription will be grandfathered into their current price indefinitely. If you cancel your paid subscription and sign up again later, you’ll pay the new price. Users who pay at a grandfathered price point will also get a notice in their billing profile asking them to volunteer to pay the new price:

Moving US accounts to Europe

One more change to prices coming up soon: as many of you know, we are in the process of moving our business from the US to Europe. Our US business was below the thresholds required to charge VAT, so users in Europe have not been charged VAT up until now. However, our European business is of course required to charge VAT to European customers. As we shutter the US business entity and move these customers to our European business, we will have to start charging VAT for customers in the EU.

If your account is affected, you can expect to receive an email with the details at least 3 months prior to your account being migrated. You will have an opportunity to choose a lower price tier (or a higher one, for that matter), or cancel your subscription, before VAT is applied to your renewal payments. Your payment will still be denoted in US dollars, though you will have the option to switch to paying in Euro.

Customers who are already paying in Euro are already being charged VAT, so if you’re one of them then there will be no changes for your account.

There are still a number of technical hurdles to overcome planning this rollout, so the exact timeline is uncertain, but we’re hoping to complete this by Q2 – and finally close our US business shortly thereafter! 🇪🇺🎉

Why do we need to raise prices?

As briefly explained at the top, we have not adjusted our prices for inflation since our inception, and that means that our margins have been steadily shrinking as the costs of doing business have gone up. SourceHut is still profitable and sustainable today, but for our long-term financial health it is necessary to increase our revenue.

The price increases proposed exceed the price changes you would expect if we simply adjusted for inflation. The rationale is:

These prices are future-proof, so we don’t have to change prices frequently.
This allows us to offer established users the option to remain at their current price point, making up for it with new subscriptions and users who volunteer to pay the higher rates.
The extra revenue gives us more resources to invest in SourceHut. With more revenue we are better able to invest in infrastructure upgrades, new services and features, and in our staff. Our staff makes below market salaries (standardized at €2800/month), and while we probably won’t be paying ourselves market rates any time soon, it would be nice to increase our compensation modestly, or hire additional staff, with the extra revenue.

As always, all of our profits are always invested 100% back into free and open source software.

Also: we’re planning on putting together a new financial transparency report soon, so you’ll have more insights into our finances and how the new prices are working out when the time comes.

This proposal, all of its finer details and broad strokes, are up for discussion with the SourceHut community. That includes you! Please let us know your thoughts. We’re discussing it on IRC, on the #sr.ht channel on Libera Chat, and on the sr.ht-discuss mailing list [email protected]%3E">in this thread. We’re looking forward to hearing your thoughts.

Updated 2025-12-02: Clarify the situation around who will owe VAT.

As we move our primary business into the EU, our base pricing is reckoned in Euro. Prices for other currencies like US dollars are selected by taking the current exchange rate from Euro and rounding the figures towards an attractive price gradient. Note that you can also pay in the currency of your choice and let your bank do the conversion from your native currency if your bank offers a good exchange rate. ↩︎

What's cooking on SourceHut? Q4 2025

Thu, 20 Nov 2025 00:00:00 +0000

Hi everyone! We hope you’re reading today’s update under as many blankets as possible, ideally in front of a warm fireplace, as the cold of winter starts to set in. You may have read our last update in the third quarter – it’s time again to take a look at what’s new and what’s next.

Drew’s update

I’ve been knee deep in refactoring over the past few months, and it has been very rewarding. The pile of tech debt has been steadily shrinking! Two of the goals I mentioned in my last status update were met: first, I rolled out ariadne-codegen to all of our frontends to handle their GraphQL requests to each other (and to their own backends) and I have found that it improves the frontend code substantially. And, the one we’ve been waiting years for: the legacy REST API has finally been removed (almost) entirely 🎉

There is still one internal user of the legacy API, which is a legacy webhook subscription used by all services to receive account updates from meta.sr.ht. Completely paying back all of the tech debt incurred by supporting the legacy API is blocked on the removal of this last piece of the puzzle. Conrad and I are debating a few ways to replace it – do we add a user details cache, do we have services fetch user data from meta every time they need it, or do we upgrade it to a GraphQL webhook, or do some other thing? Ideally we like to reduce the number of sources of truth so that we don’t have to worry about them disagreeing with one another – the solution we come up with in the end is something that will have to wait until the next quarter’s update to become clear.

Speaking of the next quarter, I am planning on working on more user-facing improvements. In particular, I have plans to introduce first-class support for commit trailers into the git.sr.ht API as well as into the lists.sr.ht’s API for patchsets. Based on this work I will be updating the project hub to more tightly integrate the various components of SourceHut projects together, for instance by adding ticket references when a patch mentioning them is being discussed on the mailing list. The main motivator for working on this, however, is to add a commit trailer that lets you identify the dependencies of a patch which are not yet merged upstream, and have the CI pull those patches down when testing your work.

The other user-facing changes I mentioned in the last update, such as an improved profile page on the project hub, or FIDO/U2F support for two-factor auth, are still planned for the near term and I hope to find time for them in the coming months. Multiple email addresses may end up being on hold until we figure out what to do with the internal legacy user profile update thing.

Another large change I’ve made is to our process for handling tickets related to SourceHut itself. Each of the old bug trackers, one per service, has been closed to submissions and we’ve declared a sort of bankruptcy on all of the tickets that were there – we were not keeping up with it and many of them were duplciates, obsolete, or irrelevant. Instead we now have a single “sourcehut” bug tracker. I retrieved most of the tickets worth saving from the old bug trackers, and organized them onto this new one, which we are now attempting to better maintain so that you can get a view of our priorities. If we missed any tickets you cared about, please let us know on the sr.ht-discuss mailing list.

One last treat for you before I let Conrad take the mic: just this week I put out an overhaul for our install docs, for those of you who want to run your own SourceHut instance. I won’t pretend that SourceHut is any easier to install than it has ever been, but these docs should give experienced sysadmins a better starting point for their deployments. Enjoy!

Conrad’s update

Welcome back to the department of under-the-hood improvements! I certainly have some treats for you this time around.

As hinted at last time, we’ve made great progress with our Ceph cluster. The Alpine upgrade in the previous quarter unlocked an upgrade of Ceph itself, which I performed this quarter. We are now on Ceph 18 (“Reef”). There was another round of Alpine upgrades, too, so the path to Ceph 19 (“Squid”) is clear. Though I am not sure it will still happen this year. We are prudent with the Ceph version we are running and not in a rush…

The absolute highlight for me this quarter, however, was finally dropping Alembic from our stack! It has been completely replaced by sourcehut-migrate, and we are quite happy with the result. The whole operation went smoothly, but there is one little regression: the xxx.sr.ht-initdb scripts got dropped without proper replacements. But fear not: I am almost done implementing an alternative solution in sourcehut-migrate!

As is customary, I will sprinkle in some user-facing changes: we enabled GraphQL introspection on all services. This facilitates the usage of various GraphQL tools to work with our API, without having to feed them with our schemas first.

Another one that only few users will care about, but those probably all the stronger: lists.sr.ht finally supports emails in non-UTF8 encodings. I don’t think this was ever a conscious decision, it was just a few bits and pieces missing, but now it should be fixed. While we’d certainly still appreciate everyone switching to UTF8, we did not mean to lock anyone out.

Incidentally, I think the most prominent cause of encoding issues was in fact folks importing old(ish) mailing list archives from elsewhere. So while this will hopefully improve the process, there is still plenty of other reasons why an import might fail. But I have good news about that, too: lists.sr.ht will now send you an email if an import fails, providing you with actionable details.

Going back under the hood, I played a bit with golangci-lint. While we had tons of rather uninteresting warnings, it also did uncover a few very valid issues. So I bit the bullet and cleaned up the entire Go codebase to the point that we now run golangci-lint in CI, so that hopefully no new issues will creep in.

We also moved one more of our ancillary services to Kubernetes: there is now mirror.srht.network, which will soon replace mirror.sr.ht. Right now we only use it for infrastructure-related packages, but with the next Alpine upgrade we will likely switch completely, allowing us to shut down yet another VM.

Last but not least, if you’re still reading, your attention span is certainly long enough, so why not take a look at our new [email protected]%3E">build worker RFC. I cannot take credit for all the ideas put forth in there, but as I sort of started driving this I’ll take the opportunity to solicit feedback from you, the community. Because the TL;DR is that we want to allow you, the community, to run your own build workers! Wouldn’t that be a nice user-facing feature for me to write about?

Everyone else

SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months.

Simon Martin has been hard at work on dozens of bug fixes and little improvements, and also found the time to develop some nice features, too: thanks to his efforts you’ll now find a little checkbox on your lists.sr.ht dashboard which will cause the mailing list software to copy you on your own emails, if you prefer for it to. We’re not quite done reviewing his work on this next one, but soon we’ll be rolling out another feature he developed which allows you to narrow the conditions under which build jobs are submitted, such as omitting feature branches from CI in your git repos. Simon also fixed a long-standing problem with the project hub recently, de-duplicating events so that an email sent to one mailing list shared among many projects does not fill their event logs up with dozens of events. Thanks, Simon!

It’s been a relatively slow quarter for distro releases, so the workload of our volunteer build image maintainers has been pretty light. But I’d like to thank Maxell G for handling the roll-out of Fedora 44 as the new Rawhide release, and the EOL of Fedora 40, and to welcome Alex Kramer as the new maintainer of our Arch Linux images.

There were dozens of other small patches in the past couple of months, fixing little bugs or making small improvements all over SourceHut, from many different contributors. Thank you to everyone who played a part in making SourceHut better!

Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!

What's cooking on SourceHut? Q3 2025

Mon, 01 Sep 2025 00:00:00 +0000

Hello everyone! It’s time for another quarterly update on what we’re up to at SourceHut. There’s a lot of great stuff going on since you last heard from us! Let’s get started.

Drew’s update

We finally rolled out the billing overhaul! God, that was so much work. Since the announcement in July we have successfully accepted a couple hundred payments in Euro – thank you to everyone for your support! The rollout was nearly flawless, with just a few things to sort out here and there and just few embarrassing problems to deal with (a fun one: a handful of users got invoices with a “SAMPLE” watermark).

There are still some knock-on things to deal with for billing, especially when our Q3 tax bill is due, but most of it is not user-impacting. We’ll probably roll out a couple of new payment currencies and move all users to our European entity by the end of the year (which will, unfortunately, come with an annoying price increase for some users who will start being charged VAT, you’ll get an email before that happens if you’re affected). Other than that, the billing plans are now pleasantly unremarkable.

Now that I’m free of the huge pile of billing work, I’ve been spending most of my time aggressively paying back tech debt, as promised in previous updates. The biggest change in this respect is that we’re getting serious about removing the legacy REST API in favor of GraphQL. All of the API documentation has been removed, the meta.sr.ht functionality for registering new legacy OAuth clients and personal access tokens has been removed, and I have done an analysis of contemporary legacy API usage and removed all of the API features which are not still in active use. There are about 30 users who are still relying on the legacy API – I have emailed them to inform them of the looming removal of the legacy API and offered to assist in migrating them to GraphQL. Please check your inbox if you’re one of them!

Unfortunately one of those contemporary users of the legacy API is ourselves. I have been working to reduce internal consumers of the legacy API over the past few months, upgrading bits of the project hub, the build submitter for hg.sr.ht, rewriting how man.sr.ht communicates with git.sr.ht, and so on. It’s a work in progress but I’m making good strides towards refactoring all of the remaining code – at this point we’re down to a few legacy webhooks, and those are a bit more complicated to replace.

I’ve also been looking into some more experimental changes. A big one that I’ve been researching is using ariadne-codegen to generate GraphQL clients for our Python frontends to use to talk to their own backends and the APIs of other services with which they integrate. A long term goal is to remove our frontend’s access to the database and get rid of SQLAlchemy, and having our frontends communicate with GraphQL for their work – providing a single source of truth for business logic and database access. Making this work well requires us to have a robust approach to GraphQL clients in the frontend code – and Ariadne seems to be a promising option for that. I’m getting ready to roll it out for meta.sr.ht’s frontend for a start later this week.

I’ve been up to a few other things – I’ve replaced our dependency on minio’s Go SDK to talk to radosgw with the AWS SDK instead, for all of our services that speak to S3. That includes git.sr.ht and builds.sr.ht artifacts, as well as pages.sr.ht’s basic functionality. Since we moved from minio to radosgw, it had seemed a bit odd for us to continue to depend on the minio client SDK as well. I also replaced our dependency on the now-deprecated go-yaml/yaml with goccy/go-yaml, which had some knock-on effects that were resolved with ~meroje’s help.

There’s a bunch of little things all over the place that I’ve improved in my quest to pay back our tech debt – more than I can elaborate on any further without boring you. I plan to continue my work in the coming months, but I will also be working on some new features for you. Some small, nice-to-have features that I have planned include support for multiple registered email addresses on your account, as well as webauthn (FIDO/U2F) support for 2FA. I also intend on overhauling the project hub’s profile page, to act as a sole unified page for your profile across all services – which should reduce some confusion and give you a nice page to show off your work.

Conrad’s update

For me as well, finally, some of the long-term work is coming to fruition! The first one is the site-wide rollout of sourcehut-ssh, our new unified SSH ingress. This is of course one of those under-the-hood changes, but it will make life a lot easier going forward for both admins and developers. It will also be the foundation for our planned SSH ingress handling in Kubernetes.

Another one, hinted at in the last update, is sourcehut-migrate, our anticipated Go replacement for Alembic. It has already been rolled out for pages.sr.ht (which was not actually using Alembic before), but with paste.sr.ht the first switch is in the pipeline as well.

A more user-facing highlight are some of the improvements to the GraphQL playground. My part in that was actually quite a while ago, adding support for variables and making some minor UI improvements. Conveniently, that prompted Drew to add a field providing the currently composed query as a curl request, providing a simple entrypoint for scripting. He also replaced the unannotated schema definition on that page with an integration of docs.sourcehut.org. We hope that all this will help people adopt and experiment with our GraphQL API.

Our monitoring infrastructure also got some upgrades: metrics.srht.network is now on Prometheus 3 and we added a ntfy.sh sink to alertmanager. The latter is not public (for now) and the alerts will require some tuning to the new use cases that this opens up, but it definitely already improves incident response.

However, my personal highlight for this quarter - and this may sound odd at first - was performing a full Alpine Linux upgrade and subsequent reboot on one of our two Ceph cluster nodes, without any interruption whatsoever. You see, Ceph is not really designed to be operated at this small scale. But we cannot afford a whole server zoo just for storage. Ever since its inception, SourceHut has been operating with a single storage server (initially not Ceph). The anticipated redundancy has always been disk-level, not server-level (we have off-site backups for that). Not too long ago, we added a second storage server, and with this we still don’t have full server-level redundancy. But what we are able to do now is shuffle data around so that we can temporarily take down one server in a very controlled fashion. I feel like it took me a Ph.D. in Cephology to achieve this, so I am now busy writing a lot of documentation. But since this is something that every Ceph expert will (somewhat rightfully) tell you you shouldn’t be doing, I am also very happy right now that I learned the Ceph internals well enough to get it to work at all. Because offsite backups are cool, but upgrades without downtime are even cooler… ;)

Everyone else

First of all, as always our build image maintainers have been hard at work. Taavi Väänänen has been keeping an eye on the Debian builds, fixing some issues with recent upstream Debian changes and shipping a Trixie image. ~arusekk was kind enough to send along an Alpine 3.22 patch as well. As Drew mentioned earlier, ~meroje also sent us a small improvement for our updated YAML parser, which makes round-trips a bit prettier when your build manifests are edited by e.g. git.sr.ht prior to submission.

Thomas Chauchefoin has once again generously sent us a number of security bug reports, most of which had patches attached. Thanks Thomas! If you’re running your own SourceHut instance, make sure you’re subscribed to the sr.ht-admins mailing list to get notified about these issues.

Simon Martin has also continued his work on various SourceHut improvements – chronicled in weekly updates on his blog. Simon fixed a few small bugs and added minor features all over the place, such as showing the branch name on man.sr.ht footers to help contributors figure out where to find the code for your wikis, or fixing race conditions that prevent lists.sr.ht from threading messages properly. Notably Simon also added some heuristics for auto-detecting new patch versions and superseding the old versions automatically – very convenient! In related lists.sr.ht news, Robin Jarry also stopped by this quarter to continue some maintenance work on the lists.sr.ht-ingress daemon.

Big thanks to everyone to worked to make SourceHut better this quarter!

Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!

SourceHut is now accepting payments in Euro

Thu, 10 Jul 2025 00:00:00 +0000

I’m pleased to announce that, as part of our broader plans to migrate SourceHut to Europe, and after many months of hard work, SourceHut has begun to accept subscription payments in Euro today – one of our oldest and most highly demanded feature requests. The pricing is similar to the US dollar pricing at €2, €5, and €10 per month (slightly more expensive than USD due to the Euro’s purchasing power),¹ with two months free if you pay annually. We have added support for SEPA direct debits, iDEAL, and Bancontact.

As before, payment remains optional for most SourceHut features (and many features still do not even require an account). Financial support is available for those who cannot afford the lowest subscription tier. SourceHut remains committed to never pricing anyone out: hundreds of people have applied for and received financial aid for any number of reasons. Please [email protected]">apply for support if your situation requires it.

On the other hand, all users who have the means to support SourceHut are kindly asked to do so, provided that the public alpha has reached a level of maturity equal to their needs. SourceHut depends entirely on our users for support, which keeps us accountable to them first and foremost. We have never accepted private investors and we cannot be pressured to compromise on our values, on user privacy, or on our commitment to publishing 100% open source software, thanks to the generous support of our community.

We are pleased to integrate deeper into the European community, and upgrading the billing system to support the Euro is the next big step in that direction. We are now running almost all of our infrastructure in Europe,² and moving our revenue to Euro and our business operations to the Netherlands is another step towards completing our move from the United States.

Thank you as always to everyone our community for your generous support – to those who pay for their subscription, but also to the hundreds of people who collaborate with us on our open source codebase, volunteer maintainers of many of SourceHut’s subsystems, and all of the people who host their projects here, or contribute to those projects.

FAQ

Which payment methods are available?

We accept payments in Euro via:

Credit card
SEPA direct debit (IBAN)
iDEAL
Bancontact

Will you support more currencies?

Yes. The work to make our system multi-currency generalizes to other currencies. We intend to support GBP and JPY soon and we will add others as well. If you would like to request that we prioritize your currency of choice, please email sr.ht-discuss.

Will you support more payment methods?

It depends. We have added all of the Euro-based payment methods that we care to, but as we introduce other currencies we will endeavour to support the payment methods most suitable for those currencies. If you have a payment method you prefer, please email sr.ht-discuss to propose it.

Note that we have already evaluated and elected not to support PayPal and cryptocurrency.

Will you accept cash or bank transfers again?

Once upon a time, we accepted annual payments in cash at each year’s FOSDEM, and we would accept manual bank transfers as well. Both methods have the advantage of not requiring nonfree JavaScript to support. We have not ruled out accepting payments in this manner again in the future, but for right now we have no concrete plans to do so.

I pay in USD now, can I switch to Euro?

Not yet: presently Euro is only accepted for new subscriptions. We will be making it possible to change your payment currency in the near future.

Workaround: if your subscription term ends within the next couple of months, you may cancel your paid subscription and sign up again in Euro when it runs out.

Why is SourceHut moving to Europe?

Drew, speaking for himself:

The most straightforward explanation for why SourceHut is moving to Europe is that, since I moved to the Netherlands, all of SourceHut’s staff live in Europe. And because we don’t use the cloud – we colocate servers that we assembled and own ourselves – it also just made sense to move our infrastructure to Europe, to be located close to the sysadmins responsible for it.

This simple explanation obscures the reasons why I personally moved to Europe. Indeed I moved myself, and SourceHut, to Europe in part for strategic business reasons: the European open source community is stronger than the US, and moreover I believe that we have a competitive advantage in subjecting ourselves to tough European laws and regulations regarding privacy and consumer protections – our users would demand this level of attention to their needs regardless.

There are personal reasons, as well, for moving myself to Europe, thus starting a process that would naturally lead to SourceHut’s business and operations moving with me. When I moved out of the United States, back in 2021, I was moving away from a political and social landscape that I no longer felt comfortable being a part of. I expanded on these feelings in detail on my personal blog at the time. To my dismay, the years since have shown that my foresight in this matter was prudent.

I sleep better knowing that my business isn’t being coerced into complicity with the US Gestapo, or implementing policies that would be letting down our queer and trans users, women, or people of color who use SourceHut. We feel that we can provide a safer service to our community from Europe, moving user data and SourceHut staff out of the reach of US institutions.

What are your plans for winding down the US legal entity?

The process is gradual and conservative, as our revenue and therefore our ability to provide useful services to our customers depended, until now, on our US legal entity being able to process payments. We have adopted a cautious approach to make sure that we continue to have the resources we need to provide reliable service.

All customers who signed up prior to today are still having their payments processed through the US system. Once we establish confidence in the European system we’ve built, and normalized our processes and procedures at low volume, we will prepare to migrate all existing users to the European payment processing system. This is expected to take place by the end of the year.

We have one more server in the US, which is used for off-site backups. Backup data on this server is encrypted in Europe before being sent off-site. We will likely de-provision this server in the foreseeable future, but it is not a particularly high priority.

Once these steps are complete it is our intention to dis-incorporate the US legal entity and cease operations in the United States entirely.

This is, in part, an intentional inflation adjustment given that we have not updated our prices in 7 years. ↩︎
All that remains is one server used for encrypted off-site backups. ↩︎

What's cooking on SourceHut? Q2 2025

Thu, 29 May 2025 00:00:00 +0000

Hello everyone! We’re back with another quarterly update for you on how things are going at SourceHut. It’s been a busy few months, and we’re happy to share what we’re working on with you all.

Drew’s updates

As readers of my personal blog might have noticed when I posted my LLM rant in mid-March, my assessment in the Q1 update that the LLM scrapers were “more or less under control” was a bit optimistic. Since then, I think that we have actually gotten it under control after deploying the nuclear option (Anubis). A few weeks after this blog post, I moved us from Anubis to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better). We have rolled this out on several services now, and unfortunately I think they’re going to remain necessary for a while yet – presumably until the bubble pops, I guess.

Thankfully, the success of Anubis and go-away in mitigating the LLM bots problem has finally allowed me to spend most of my time on our real priorities, which is why I’m getting ready to ship a huge batch of billing upgrades next week. I’ve been hard at work (1) upgrading our Stripe integration to the latest Stripe APIs, and (2) moving most of it into meta.sr.ht’s GraphQL API. You can expect to see major user-facing improvements in the billing UX from next week onwards.

It’s kind of difficult to overstate how much work has gone into the billing system upgrades. The billing system was very simple and unsophisticated prior to all of this work, and the new system is much more capable, flexible, and maintainable – but getting there is a lot of work. Most of it is not obvious to users, since we have to move slowly and carefully to avoid recklessly breaking our revenue sources (or nasty bugs like overcharging users), but this batch of changes will bring the first big group of obvious user-facing improvements. I also encourage you to play with the GraphQL API a bit when this ships next week – the new billing system is designed with an API-first mindset and so a lot of its knobs and dials are available to you to play with.

After this big patch series lands in production, I have some more internal (non-user-facing) work to do to move our payment processing into the EU, but it should be relatively soon that we’re able to add the single longest-awaited feature of the billing upgrades: support for more payment methods, like iDEAL and SEPA transfers. This effort saw its first signs of life recently – last week I collected a payment via iDEAL in my development environment!

Billing (and bot control) have been my two major focus areas this quarter, but I have done a little bit here and there, so I’ll briefly summarize those other accomplishments. A small improvement to the GraphQL APIs includes the addition of better standardized errors, so that you can distinguish error conditions on the client side – for example, readily distinguishing “not found” from “access denied” in your GraphQL client.

In more colorful news, I have installed a server in my bike shed at home. This may seem odd to include here, but I thought I’d mention it because I have used it to replicate an infrastructure environment comparable to SourceHut production, with ceph, libvirt, and k8s all making an appearance. I’ve moved many of the services I rely on privately in my personal infrastructure constellation from The Cloud (VPS providers) to the shed, such as NextCloud, Immich, and Miniflux. I’m using this setup mainly to learn more about ceph and k8s in an environment that has realistic production constraints (I actually need this stuff to work!) but which I can more confidently experiment on without breaking anything for SourceHut users. I hope to apply this experience to help Conrad with his efforts to roll out upgrades to the SourceHut fleet – which I’ll let him share with you now.

Conrad’s updates

This time around I’ll start with the rather tangible items: first, I am happy to announce that the pages.sr.ht API has gained basic support for ACLs. It’s not really well documented yet - and only works for custom domains - but it is possible to allow other users to publish your site. We are using this ourselves to collectively publish docs.sourcehut.org. Pending some better documentation, we hope that this feature may make it easier for teams using SourceHut until there is proper groups support.

Sort of descending into the infrastructure realm, but still tangible: we made the repository that hosts our Kubernetes infrastructure public. After the DDoS, we became a bit cautious about publishing details about our infrastructure. However, we think it’s time to be more transparent about our operations again. This repository is only the stuff that runs in Kubernetes, which isn’t that much yet. But it will become much more, and we will evaluate publishing other bits and pieces over time.

Speaking of Kubernetes, this is may be a good time to lay out the current plans. Much preparation work has happened, and there are currently two distinct milestones we hope to reach soonish.

One is to route all incoming sr.ht traffic through the Kubernetes ingress controller. In the background, much or even most of the traffic might still be handled by the service VMs. But it would provide much better visibility into sr.ht as a whole, and would give us easy knobs to turn when experimenting with running services in Kubernetes. To achieve this milestone we need to change all service’s DNS records to point to the ingress nodes, but that means they’ll also have to handle incoming SSH connections. Hence, the main thing missing is the unified SSH dispatch handler that is currently in review. For sr.ht, we will probably end up employing an SSH ingress solution that leverages the power of CephFS, but we’ll take care to keep things working for smaller instances (or those simply not using Ceph).

The second milestone would be running one actual service completely in Kubernetes. I was initially entertaining the idea that it could be meta, but for various reasons it will probably be something else. Regardless of the service, there are a few things missing. First, the service’s Redict instance will have to be migrated from the on-VM instance to the new network-wide sentinel-based cluster. This way, VMs and pods can share a cache, which is important for some services. Then, of course, there need to be containers. Those are conceptually straightforward, but some minor things will still need adjustment (like the currently chosen-at-random Prometheus port for observing the work queue flushing). Also, a first attempt to add Containerfiles sparked some ideas about our repository layouts, so we may take the occasion to clean those up a bit.

Most of my other work is also still in-flight. We’ve added another server to our Ceph cluster, but the big Ceph upgrade is still to come. And I am evaluating Go-based database migration frameworks, so we can throw out even more Python code (alembic).

Everyone else

I’d like to set out a spot here in our quarterly updates going forward to talk about what all of you have been doing for SourceHut! We’re a free software project after all, and we regularly accept contributions from our users.

First, I’d like to thank our tireless builds.sr.ht image maintainers – most of our builds.sr.ht images are maintained by members of their respective communities (i.e. Fedora images are maintained by Fedora users, not SourceHut). Haowen Liu, Maxwell G, Hugo Osvaldo Barrera, CismonX, and Cayetano Santos all shipped updates to, respectively, Ubuntu, Fedora, OpenBSD, FreeBSD, and Guix this quarter, with Cayetano stepping up as the new Guix maintainer – and we’re seeing off Unwox with a big thanks for all of their work on Guix support to date.

In addition to his work on builds.sr.ht, Hugo Osvaldo Barrera added a long-awaited feature allowing users to rename more resources on SourceHut and set up the appropriate redirects, such as projects on the project hub. Long time contributor наб also made an appearance in this quarter to land a bunch of small improvements, such as improving to diff rendering on git.sr.ht and lists.sr.ht, and building a more sophisticated git-over-HTTP implementation – nice to have you back, наб! Eli Schwartz also returned to revisit his favorite subject, namely making SourceHut more deterministic in support of downstream distributions that download and checksum resources from SourceHut.

I’d also like to thank Simon Martin for making their first contributions to SourceHut this quarter, landing numerous small improvements to lists.sr.ht, such as access control improvements which allow users to update the status of their own patchsets. Be sure to read Simon’s blog post about the experience!

There were many other small contributions from our community this month – thank you to everyone who helped to make SourceHut better!

Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q3!

You cannot have our user's data

Tue, 15 Apr 2025 00:00:00 +0000

As you may have noticed, SourceHut has deployed Anubis to parts of our services to protect ourselves from aggressive LLM crawlers.¹ Much ink has been spilled on the subject of the LLM problem elsewhere, and we needn’t revisit that here. I do want to take this opportunity, however, to clarify how SourceHut views this kind of scraper behavior more generally, and how we feel that the data entrusted to us by our users ought to be used.

Up until this point, we have offered some quiet assurances to this effect in a few places, notably our terms of service and robots.txt file. Quoting the former:

You may use automated tools to obtain public information from the services for the purposes of archival or open-access research. You may not use this data for recruiting, solicitation, or profit.

This has been part of our terms of service since they were originally written in 2018. With the benefit of hindsight, I might propose a different wording to better reflect our intentions – but we try not to update the terms of service too often because we have to send all users an email letting them know we’ve done so. I have a proposed edit pending to include in the next batch of changes to the terms which reads as follows:

You may use automated tools to access public SourceHut data in bulk (i.e. crawlers, robots, spiders, etc) provided that:

Your software obeys the rules set forth in robots.txt

Your software uses a User-Agent header which clearly identifies your software and its operators, including your contact information

Your software requests data at a rate which does not negatively affect the performance of our services for other users

You may only collect this data for one or more of the following purposes:

Search engine indexing

Open-access research

Archival

You may not use automated tools to collect SourceHut data for solicitation, profit, training machine learning models, or any other purpose not enumerated here without explicit permission from SourceHut staff.

This text, or something similar, will be included in our next update to the terms of service, which will probably ship around the time we finish setting up our new European billing system.

A careful observer can see our views on scrapers elaborated on in our robots.txt file as well. It begins as follows:

# Our policy
#
# Allowed:
# - Search engine indexers
# - Archival services (e.g. IA)
#
# Disallowed:
# - Marketing or SEO crawlers
# - Anything used to feed a machine learning model
# - Bots which are too agressive by default. This is subjective, if you annoy
#   our sysadmins you'll be blocked.
#
# If you do not respect robots.txt or you deliberately circumvent it we will
# block your subnets and leave a bag of flaming dog shit on your mother's front
# porch.

One can infer from the tone of the last sentence that attempting to enforce robots.txt is a frustrating, thankless task for our sysadmins.

To add to these resources, I’d like to elaborate a bit more on our views on scrapers for you today. Scrapers have been a thorn in the side of sysadmins for a very long time, but it’s particularly important as LLM scrapers seize the entire Internet to feed into expensive, inefficient machine learning models – ignoring the copyright (or copyleft, as it were) of the data as they go. The serious costs and widespread performance issues and outages caused by reckless scrapers has been on everyone’s mind in the sysadmin community as of late, and has been the subject of much discussion online.

Aside from the much-appreciated responses of incredulity towards LLM operators, and support and compassion for sysadmins from much of the community, a significant minority views this problem as less important than we believe it to be. Many of their arguments reduce to victim blaming – that it’s not that hard to handle this volume of traffic, that we should be optimized to better deal with it, that we need more caching or to improve our performance, or that we should pay a racketeer like CloudFlare to make the problem go away. Some suggest that sysadmins should be reaching out to LLM companies to offer them more efficient ways to access our data to address the problem.

Of course, not all software is necessarily able to be as resource efficient as Joe Naysayer’s static website. Moreover, LLM companies are not particularly interested in the more expensive route of building software integrations for each data source they scrape when they could go the cheap route of making us all pay for the overhead; nor should us sysadmins have to spend the finite time and resources at our disposal (often much more modest than the resources available to these LLM companies) negotiating with terrorists and building bespoke solutions for them.

More important than any of these concerns is to address the underlying assumption: that these companies are entitled to this data. This assumption has varied roots, as benign as misplaced Libertarian ideals and as unhinged as the Rationalism cult belief that AGI is around the corner and everyone ought to be participating as best they can for the benefit of untold numbers of unborn future-humans.

It is our view that these companies are not entitled to the data we provide, nor is anyone else. The intended audience for the publicly available data on SourceHut is users of and contributors to open source software who are accessing the data for those purposes. Indeed some profitable use of public SourceHut data is permitted, as one is entitled to by the Open Source Definition, but we do not wish to provide our data in bulk for any business, megacorp or startup alike, who wants to feed it into an LLM or do anything else with it which does not directly serve our mission, which is to improve open source software.

We would not come to a special arrangement to share this data with any of these companies, either, even in the unlikely event that they offered to pay for it. We are funded by paid subscriptions, not by selling our user’s data. It is not ours to sell – something GitHub, with its own LLM products, would do well to consider more carefully. The data we have been entrusted with belongs to our users, and is dedicated to the commons, and we take our role as the stewards of this data seriously. It is our duty to ensure that it is used in the service of our users best interests. We have always put them first.

We’re also researching go-away, a new option which may be effective with a reduced user impact (notably by not necessarily requiring JavaScript). ↩︎

Announcing docs.sourcehut.org

Thu, 27 Mar 2025 00:00:00 +0000

Today, I’d like to show off our new documentation site dedicated to SourceHut’s (and hence sr.ht’s) GraphQL APIs: docs.sourcehut.org.

Our work to move from the legacy REST API to a GraphQL API is still ongoing, and certain features we are aiming for (such as federation) are still missing. Nevertheless, it is already the best - and preferred - way to programatically interact with SourceHut instances.

If you’ve never used GraphQL before, there will be a slight learning curve. But we aim to make our documentation as helpful as possible, for beginners and experts alike.

At the core of each GraphQL API, there is a schema that defines the data types and operations that the API makes available. This schema can be considered a machine readable API documentation. The schema language also has first-class support for comments, to make it more accessible to humans as well. Yet, even with comments, the presentation of a raw schema is not exactly the most ergonomic way for a human to learn about an API.

However, given that all information is contained in the schema in a machine-readable way, it is quite easy to transform it into a form that is much more suitable for human consumption. Such tools are called GraphQL documentation generators. There are plenty of them out there, and we evaluated several. But none really checked all the boxes we were aiming for:

Entire API output on a single page, so Ctrl + f is useful
Little to no JavaScript
Good support for custom directives
Not too opinionated about the CSS/HTML structure

And so, I couldn’t resist: I wrote my own. You almost can’t call it a documentation generator, because it does nothing but throwing a specific object at a template you have to supply. But this also makes it powerful: the template has access to pretty much any aspect of the provided schema. These templates are key here, so if you are interested, you can find the templates for our documentation at git.sr.ht/~bitfehler/docs.sourcehut.org.

Given that we probably would have ended up writing a bunch of templates anyway (to properly customize the output of other tools), the result is pretty satisfying. Gockel is an extremely simple tool, and it will be easy to add features should the need arise. The underlying schema object that it exposes contains a lot of information, so there are many avenues to explore for improving the documentation.

Even though we ended up not using it, I would like to give credit to spectaql - it was the most promising candidate out of the existing tools I evaluated, and the current layout of docs.sourcehut.org is certainly inspired by its default theme.

For now, I hope docs.sourcehut.org will help you navigate SourceHut’s API. We are just getting started here, so there are likely some rough edges. If you find any bugs or have any ideas for improvement, please let us know.

What's cooking on SourceHut? Q1 2025

Fri, 07 Mar 2025 00:00:00 +0000

Hello all! We’re back with another “What’s cooking”, after another too-long hiatus since September. We did promise to resume monthly updates, but in hindsight that seems a bit ambitious given everything on our plate. For now we’re going to aim for the more modest ambition of publishing updates quarterly.

Drew’s updates

My work has been proceeding as expected since September, with significant disruptions due to the LLM bot crisis, which to my consternation has occupied a substantial amount of time I would have rather spent on more important priorities. Nonetheless, I am happy to report that we have more or less got the bot crisis under control – though it consumed nearly all of my attention for several weeks, I can now occasionally enjoy up to several days in a row without one of our services being knocked out by LLM scrapers.

In any case, I finished the git webhook upgrades I mentioned in September, though for now there is no option for a webhook recipient to intervene to prevent a push from proceeding. I’ve taken advantage of this and other GraphQL-native webhooks besides to make inroads towards updating our internal users of the legacy API, notably by moving the project hub to the new API and porting over the first webhooks from the legacy system to GraphQL webhooks.

A matter of greater interest to most users is the upgrades I’ve been gradually rolling out for the billing system. There are numerous improvements in the pipeline, including support for more payment methods (e.g. iDEAL, SEPA bank transfers, etc) and more currencies (in particular Euro). But, this is a time-consuming process of gradual improvements and migrations. The billing system has been largely untouched since it was originally written in 2018, and there are thousands of paying users whose payments we’re processing every day, and we need those payments to flow correctly to keep the lights on. Each improvement is as small as possible and is carefully reviewed and monitored as it’s rolled out to ensure that the system works correctly without interruption.

This work is part of the bigger picture work to move SourceHut entirely from the US to the EU, in particular by processing payments through our European entity and in accordance with European laws and regulations. We’re getting there! I hope to continue this work through the rest of March, with the ambitious goal of accepting our first payments in Euro in Q1 (but, more realistically, early Q2… it’s a lot of work).

In addition to billing work, my main focus for the next quarter will be paying down more tech debt, and in particular finishing the deprecation of the legacy API. In the past couple of weeks we have made more aggressive moves to get rid of it, removing the legacy API documentation from man.sr.ht and disabling some features which are not still being used by anyone. I have some flashy end-user-facing features coming up soon as well – but let’s make it a surprise.

Conrad’s updates

Before I start, I’ll let you in on a little secret. Like many infrastructure engineers, I have evolved to have two questionable superpowers. One is stumbling across very weird issues that take days to debug only to be fixed eventually by a single-line code change. The second is to perform major refactorings, often coördinated across multiple codebases, resulting in plenty of code changes but no discernable difference in functionality of any of the touched components. Both these superpowers are greatly under-appreciated by managers (“Last week I stared at packet traces, then disabled a single sysctl on all servers, that’s all, really…”). Fortunately, there are no managers at SourceHut, because I have plenty of the second category for you!

All our Python packages are finally PEP440-compliant! 🎉 This took so long and added zero value for our users, yet it was inevitable. And it did have the pleasant side effect of finally unbreaking builds for submitted patches again.

Another epic, but mostly invisible undertaking was the recent “shared asset refactoring”. This one, however, brought significant quality-of-life improvements for developers. It also drastically changed the process for setting up a dev environment. The generic instructions have already been updated, but I am also working on a more detailed write-up. Piggy-backing on this big change we also renamed all executables to have a consistent naming scheme, which should help developers and instance admins alike.

And yet more from the “big changes you didn’t notice” department: we finally consolidated all SSH key handling in meta.sr.ht. Previously, services requiring SSH access (like git and hg) would have their own copy of the SSH key table, which was prone to stale data and other inconsistencies. This unlocked the removal of various chunks of legacy API surface, with even more cleanup yet to come. Specifically, the entire SSH dispatching - which currently has very similar implementations in various services - will get a face-lift, improving code re-use and reducing complexity.

Okay, I’ll give you a break - here is something very tangible: in lists.sr.ht, we removed the instance-wide configuration that blocks emails with HTML parts, putting list owners in full control to make their own choices about blocking such parts.

As for what’s cooking right now: besides the above-mentioned SSH dispatching I am working on refactoring log file locations for all services and providing log rotation policies with our packages. We are also preparing to extend our Ceph cluster, and we’ll have to retry botched Alpine upgrades on two hosts which had an unfortunate combination of packages installed, triggering some very sneaky breakage in Alpine’s ceph packages. So, you know, lot’s of epic stuff that no one will notice. Unless it breaks something.

What's cooking on SourceHut? September 2024

Mon, 16 Sep 2024 00:00:00 +0000

Hello everyone! It has been some time since we last wrote a “What’s cooking” for you. We’d like to resume this tradition as of this September. We haven’t been totally radio silent – you can get caught up on what’s been happening over these past two years reading the blog archives – but we’ll be resuming monthly updates to keep you more informed about the minutiae of our operations. We’ll be presenting our updates from the perspective of each of our staff members, Drew and Conrad.

Drew’s updates

Hi folks! As promised in our update in June, I have mostly been working on paying down our tech debt recently, in addition to shouldering a fair bit of ongoing maintenance, minor bug fixing, and so on. At the moment I am working on the steps which will unblock our legacy REST API in favor of the GraphQL API.

The main focus of my work in this area is on refactoring internal users of our legacy API (e.g. the project hub queries git for repository details via REST). I have mostly updated the project hub accordingly, and I’m taking a small diversion to buff up our GraphQL-native webhooks so that I can migrate the project hub away from using legacy webhooks. For this purpose I am developing and rolling out a more sophisticated approach to git hooks which allows users to prepare GraphQL queries that are executed during a git push, whose payloads are delivered to an arbitrary user-defined server. For now these are done asynchronously and cannot affect the outcome of git push, but I will be expanding it to allow users to accept or reject git pushes based on their own policies, such as enforcing an OWNERS file.

Following this work I will continue refactoring internal users of the legacy API, and in this process I plan on aggressively refactoring out legacy code in our shared Python codebase, core.sr.ht, which is the most important user of the legacy API right now. Once a bow is put on all of this work, I will start planning the deprecation of the legacy API and reach out to contemporary third-party users of the legacy API to assist with a migration plan for their use-cases. The next big tech-debt priorities after this will be the removal of Celery and SQLAlchemy from our dependency tree.

Aside from tech debt, the next big user-facing features I plan to address are the billing overhaul and improvements to the lists.sr.ht code review workflows. The tech debt I have on my plate is pretty substantial though, so we may not have any updates on this front by next month.

Conrad’s updates

You came for the fun, now you get to stay for the infrastructure updates (mostly)! I’ve been hard at work following up on the plans outlined earlier.

Let’s start with the elephant in the room: we successfully migrated our DNS infrastructure back in-house. While I think this is great, it had the unfortunate side-effect of it being unavailable for a brief period last Friday, due to an upstream ISP issue. But let’s be clear: even if DNS would have continued to work, all servers were unavailable anyways. That’s a trade-off you make when you’d rather own your infrastructure than operating at “planet scale”, and we continue to be proud believers in this approach.

We have also completed the separation of the Kubernetes cluster from the VM hosts. See the previous post for details, but this finally allows us to migrate actual workloads to Kubernetes. The first one is our new Prometheus setup, which you can already see at metrics.srht.network. It has already taken over the production monitoring and alerting. There are just a few more bits and pieces that need to be moved over before I’ll be able to shut down the first VM for good: metrics.sr.ht.

I’ve also performed several upgrades of Kubernetes itself and am currently preparing an Alpine upgrade on the bare metal hosts. Once all that is completed, the fun part will start: moving actual services – or parts thereof – into the cluster.

On the application layer, I am still pursuing the busywork of making our Python packages PEP440-compliant. I have also been on a mission to address some resource leakage in soju, unfortunately with mixed success. We’re slowly getting there, but the turn-around time is rather high, as we cannot constantly annoy network operators (and users) with reboots of the bouncer.

Once the major infrastructure tasks outlined above are complete, I hope to spend more time on actual application code again. But the Alpine upgrades, especially the one involving our main storage host, might still keep me busy for a while.

The European NGI fund must be renewed

Thu, 18 Jul 2024 00:00:00 +0000

In 2022, SourceHut was selected for funding from the European Union’s NGI fund. This fund has been used to provide small grants to many free software projects since 2020, and the impact of these grants has been enormous. In SourceHut’s case, the grant is directly to thank for our comprehensive GraphQL APIs, which both underpins SourceHut’s internal communication needs and makes this platform possible, and are depended on by our users to achieve our goals of personal data ownership and platform independence.

Thanks to NGI funding, Europe has been the spearhead of innovation in free software over the past four years. There are hundreds of free software projects which can thank this fund for their success, projects which would not have been possible without it – and because of this fund, the technology they provide is available in the commons, for the benefit of all. The NGI fund has enormously strengthened Europe’s technology stance and drawn the continent out as one of the best, if not the best, places to innovate in free software and support internet and technology resilience, security, sovereignty, and innovation. These goals cannot be achieved any other way – many of the projects that benefit from NGI funding cannot reasonably find their way without a funding source like this.

Moreover, the fund has been shown to be an incredibly efficient way of spending towards innovation. The NGI grants punch well above their weight. We have seen NGI grants in the tens of thousands accomplish goals which the commercial sector demands tens of millions in investments to achieve, and the NGI-funding outcomes are placed into the commons and support the public interest – something private funding cannot do. I can think of no better example of efficient public spending on innovation than the NGI fund: the returns on a small NGI investment are huge.

So, it was to our dismay to learn that the European Commission has not included the NGI program in its Horizon Europe working draft for funding programs in 2025. SourceHut joins an open letter to the European Commission calling for them to reconsider. If anything, it is not only important but urgent that the commission expands the NGI program – not shut it down. The NGI program is a cornerstone of Europe’s future in technology and innovation, and it embodies the European commmunity’s values in a way we must honor and uphold.

We urge the commission to reconsider.

– SourceHut

Update on our infrastructure plans

Wed, 12 Jun 2024 00:00:00 +0000

Now that Drew has provided an update on the general state of SourceHut, I would like to follow up with one focusing on our infrastructure. Much has happened under the hood, a lot has not happened, and plans come and go as reality changes its mind every now and then. So, let’s take a quick look at where we are right now, what our plans are, and how we hope to get there.

First and foremost, let me state the TL;DR: our grand plan has not changed, and we are still planning on moving most services to Kubernetes. But to talk about why that’s taking so long, how we are preparing for this, and what this means for folks running their own instances, I need to set the stage a little bit.

Quick recap

As Drew already mentioned, we hastily migrated all our services to our research facilities in Amsterdam during the recent DDoS attack. This was a desperate measure, and while it actually had some benefits, it came mostly with drawbacks.

Our research facilities at the time consisted of a “grand” total of three servers. They provided two things: a Ceph cluster and a Kubernetes cluster. The Kubernetes setup was far from ready for production. But in a stroke of luck it had been just recently that I had upgraded our Ceph cluster from the few spinning disks we used for toying around to a significant number of SSDs that were intended for production use. And so, we made the decision to migrate our storage to the new Ceph cluster, but keep running the services in VMs like we did before.

The research hardware is quite powerful, but it was still clear we would need more. It is also close to impossible to provide any level of redundancy with just three servers. We decided to rent two physical servers as build runners, an intensive and at times unpredictable workload. We also decided to keep our DNS in Digital Ocean, where we had moved it to get it out of the DDoS blast radius. These setups remain in place to this day.

The good, the bad, and the ugly

The migration was done under pretty intense pressure, as all our services were unavailable. Once the dust had settled after the DDoS, it was time to inspect the result more closely.

The obviously good outcome: it worked! We had managed to bring all of SourceHut back up in Amsterdam and were able to restore service. That was quite a feat. Another outcome I was initially very excited about was data locality. All of a sudden, there was a chance to bring up a service in Kubernetes and let it handle some production traffic without worrying about transatlantic latency for internal requests. We’ll see how that turned out in a minute, though.

The bad part was - at least initially - the Ceph cluster. While we had the storage capacity we needed, there had not been any performance testing yet. So this unfortunately happened in production, with the predictable, occasional hiccup. But by now, things have settled down and it seems the cluster can handle the load just fine. The object storage still feels a little less performant than before, but bearable. We’ll continue to tweak things, and hope things will get even better once we start using the advanced features of Ceph for horizontally scaling our services.

Since most time was spent tending to the Ceph cluster, it took a while until the ugly part dawned on me. In our old setup, every service was running in a VM, and the hosts that ran the VMs did nothing but that. When we frantically moved all that over as-was, we ended up in a silly situation. All the VMs were using macvtap interfaces, which are very easy and convenient to set up, but have one very annoying quirk: they can talk to anything on the network, except the very host they run on.

As you can imagine, with just three servers running a Ceph cluster and a Kubernetes cluster, using any of those only for running VMs was not an option. And so, a service brought up in Kubernetes was unable to talk to VMs that happened to run on the same host that it was scheduled on.

I spent many days thinking about workarounds, but they would have all been extremely foolish. We had very little headroom to shuffle things around. In the end, I decided that the only sane option to run anything in Kubernetes was to get rid of the macvtap interfaces, switching to a proper bridge setup. But this would have required bringing down the entire host network, which would have had a huge impact on many services. I made plans, but hesitated with the actual execution. The situation was extremely frustrating.

The plot twist

It was just then, that we received an offer that would change everything. An anonymous saint (<3) gifted us eight servers. They are of the kind that is considered old in some places, but they are perfect for us. At this point, I decided to just postpone running anything in Kubernetes. These servers would allow us to move things around carefully and straighten out the kinks without interrupting service.

It took a few weeks of preparation and planning, but the servers are now installed and operational. So, finally, we are able to move forward. And this is how we, at last, get to the interesting part.

What’s next?

The first task will be to bring our DNS back in-house. This is needed to bring back some features we need for Kubernetes, like DNS dynamic updates for completing ACME challenges.

Then, we will physically separate the Kubernetes cluster and the VM hosts, so everything can properly talk to each other. This will finally unlock experimenting with production services in Kubernetes.

On the topic of Kubernetes, I know a lot of people are excited about that, but I feel a word of caution is in order. While it will facilitate some aspects of running an instance, I am not sure it will be an “off-the-shelf” solution anytime soon. For example, some of our services will rely on CephFS for horizontal scaling. This can probably be reproduced even at small scale with Rook, but we currently do not use it, and I am not sure if we ever will.

Another issue is that SourceHut in its entirety includes several services that are not a great match for Kubernetes in the first place. Kubernetes is first and foremost designed for HTTP requests. But we have multiple services that require for example SSH, which is notoriously hard to load-balance, and does not have any equivalent of SNI. Our candidate setup for that involves dedicated IPs (external load balancers), which are usually a platform-specific feature. What works for us, may not work for you.

That said, I think solutions will pop up, and it makes sense to cross that bridge when we get there. One reason for the lack of tangible Kubernetes services so far was that we believe in solving the hard problems first. We have had a working prototype of build workers running, and proofs of concept for SSH routing. But knowing that it is all possible, priorities have shifted slightly. We will keep the rented build workers for a little longer, until we are confident in our infrastructure setup. We’d also like to do some refactoring of the builds.sr.ht codebase before moving those to Kubernetes. So, once the above steps are completed, the time may have truly come to start testing some of the “simpler” services in Kubernetes. That is, of course, unless reality changes its mind again…

Thank you for your patience

I hope this update answers some of the questions I occasionally see on IRC. It’s always hard to give brief answers to seemingly simple questions when the reality is such a complex mess. That said, I’ll do my best to monitor both #sr.ht and the [email protected]">sr.ht-discuss list. But even if I don’t respond, I am quite hopeful you will see answers soon.

Cheers!

The state of SourceHut and our plans for the future

Tue, 04 Jun 2024 00:00:00 +0000

Good morning! It’s been a tough year for SourceHut and I know many of our users are waiting to hear from us. Our last update was the post-mortem following the DDoS attack we sustained in January, and we have some additional news following this update as well as plans for the coming months to share.

I apologise for the reduced communication as of late. We are a small team dealing with a lot of big problems, and between keeping the lights on, dealing with issues as they arise, and leaving time for us to rest and spend time with our families, there has not been as much room for keeping you up to date. Nevertheless, our ship sails on, things have quieted down now, and we have an opportunity to get you filled in today.

So far in 2024

I’ll be honest with you – it hasn’t been great. We are not facing an existential crisis – we have extensive disaster planning for many scenarios we might face, and our situation is well within the scope of our planning – but nevertheless it has been a challenge.

As you know, during the DDoS we suffered a total network outage on our primary datacenter and had to perform an emergency overnight migration of all of our infrastructure to our research installation in our datacenter lease in Amsterdam. Our infrastructure became stable following this intervention, but we have been operating at a reduced capacity since – we are running SourceHut on infrastructure provisioned for research, not production. Currently we are supplementing our owned-hardware compute with a lease from OVH for build server cycles.

Following the failure of our primary datacenter, we arranged to have our equipment shipped overseas to our research datacenter, so that we could install this equipment and restore our infrastructure to its full production capacity on SourceHut-owned hardware. At our now-deprecated site we had ten servers totalling an investment of about $50,000 USD. Some of this equipment was at the end of its operational life and was disposed of accordingly, but we arranged to have five servers shipped to Amsterdam. However, all five servers were lost in the mail. These parcels were insured with the shipping provider, but we have been unable to reach the provider for any information regarding the status of the parcels or any resources for filing an insurance claim. After several months of attempts, we have ultimately had to write these servers off.

Moreover, we are facing some financial issues. The loss of this equipment represented a significant lost investment, and the shipping costs were substantial. Additionally, we are working with a tax issue in which we were over-taxed by about $20,000; we filed an appeal months ago but that money is tied up for the time being and the tax authorities have sent us timely notices every 60 days informing us that they require an additional 60 days to process the issue. We still have an entirely sustainable financial situation, but we lack access to the capital we would ideally be using to build out our infrastructure and address other needs for growth.¹

You may have heard that we also had to part ways with one of our staff members recently. This reduces our headcount to two. For the time being we will not be hiring a replacement, but our near-future plans are achievable with our current headcount. Though we usually aim for transparency to the maximum extent possible, we will not be sharing further details about this departure, as a matter of reasonable privacy.

As you can imagine, it has been a stressful time for us. However, I wish to stress that everything we’ve been dealing with is planned for in our models, both technical and financial. There is no existential threat to SourceHut. Nevertheless, we are grateful for your patience and support.

There is some good news to share – some old colleagues of ours generously provided us with eight servers to shore up our capacity in the new datacenter and mitigate some of the losses from the bungled overseas shipments. These were installed last week, and we are provisioning them now and preparing to bring them into service. We also did a marvelous job at upgrading our research installation to production readiness, polishing it into a much more robust system over the weeks and months following the disastrous DDoS attack.

Also, as a happy side-effect of our surprise move to Amsterdam, SourceHut’s datacenter installation is now entirely powered by renewable energy sources. We have also finally rolled out IPv6 support for most SourceHut services as part of our migration!

In the months to come

It follows from our story that we have been focusing on two things this year: provisioning and managing our infrastructure and getting as much rest as possible. Our situation has calmed down, and while we still have a lot of loose ends to attend to I’m happy to say that we’re resuming a sense of normalcy here and preparing to resume our work on the features you need.

Our plan for the coming months involves the following objectives:

Implementing Kubernetes for managing our infrastructure
Paying down tech debt in the codebase
Fleshing out the remaining “alpha” feature goals

Conrad will continue leading the efforts to k8s-ize our production infrastructure. Our AMS datacenter installation was provisioned with k8s in mind, rather than with the parameters we used for our earlier libvirt-based infrastructure. As a temporary measure, we have bolted our libvirt approach onto the servers we have installed here, but we are planning on gradually migrating our deployment to a k8s-based approach which is better suited to the infrastructure we have provisioned, as well as (hopefully) being more robust and scalable, meeting the original objectives of our k8s research. We have migrated our large persistent storage system to Ceph, and are working on moving our API and web services into k8s one at a time.

Then there is the matter of “tech debt”. SourceHut’s codebase traces its lineage directly to our early prototypes, and there are many design choices and bright ideas which are not so bright in hindsight. We had initially planned to work on paying down tech debt between the “beta” and “full production” phases of SourceHut’s development lifecycle, but these areas are causing us enough headache that we have made the decision to spend some time reducing our tech debt today. In particular, we have the following goals:

Remove Celery from our architecture
Remove SQLAlchemy from our codebase (and route web frontends through the GraphQL APIs instead)
Deprecate internal users of the legacy APIs and webhooks (and redesign services with a single source of truth for each data store)
Refactor user management in core.sr.ht, in particular to get rid of the old “delegated OAuth” subsystem

Following this we intend to finish up the remaining “alpha” feature goals, which includes the following:

Overhaul our billing system and migrate to a new payment processor. This will include support for paying in more currencies, an improved invoicing system, and support for many more payment methods, such as iDEAL and European bank transfers.
Overhaul and expand the web patch contribution and code review workflows.
Implement user organizations and shared resources.

We have further plans over the long-term: reducing the use of Python in our codebase, overhauling builds.sr.ht and expanding its features, improvements to man.sr.ht, and so on, but for now the goals enumerated above are chief among our priorities.

Thanks for sticking with us

Thank you for your continued support and patience while we get our house back into order. We hope you are continuing to enjoy SourceHut for your needs and the needs of your projects.

If you have any questions about the events of this year, or our plans for the future, or anything else – feel free to reach out. You can reach all of us in private by [email protected]">emailing support, or the community in public at [email protected]">sr.ht-discuss, or you can reach me (Drew) directly via [email protected]">[email protected]. We’re also around on IRC – #sr.ht on Libera Chat.

Take care!

If you have the financial means and wish to help, this would be a great time to consider upgrading your paid subscription – though it is by no means necessary for us to keep things running. ↩︎

SourceHut network outage post-mortem

Fri, 19 Jan 2024 00:00:00 +0000

It’s been a busy couple of weeks here at SourceHut. At the time of writing, we have restored SourceHut to full service following an unprecedented 170 hour outage, and while we still have numerous kinks to sort out following an unscheduled emergency migration of all services across an ocean, all services are now fully operational.

Allow me to open this post-mortem by extending my deepest apologies to the SourceHut community for this interruption in service. This outcome was unacceptable: we failed you, and I am sorry. We know that you depend on SourceHut to be reliable, and you trust us to make sure that you can always depend on our services to be there for your projects. We value this trust profoundly, and we will strive to prevent a situation like the one we faced last week from recurring. The Internet can be a fragile place, and we will do what we can to re-enforce it.

Here’s what happened, what we did about it, how we’re making things right, and what we’re doing to ensure it does not happen again.

Background

SourceHut runs on servers owned and provisioned by SourceHut, installed in colocation facilities at three datacenters, respectively codenamed PHL, AMS, and FRE. PHL was our primary datacenter, FRE was used for off-site backups, and AMS was a research installation intended for a future migration of our production infrastructure in PHL and next-generation deployment of SourceHut intended to scale into the indefinite future.

We have been setting up an installation in AMS for researching a future infrastructure migration to the EU and means of increasing our resilience and redundancy, which we had planned to gradually roll out with minimal disruption to user service over the course of at least a year. In the end, we rolled it out with maximum disruption to user service in 7 days. The AMS datacenter installation was provisioned according to a projected research workload with an eye towards expanding it to support a full SourceHut installation in the future; at the time of the incident it was provisioned at a scale appropriate for running most of a SourceHut installation.

Our FRE datacenter is used for off-site backups. We maintain a Postgres standby which is replicated in real-time and is usually up-to-date within seconds of production, as well as daily backups of large data silos such as git.sr.ht. We have a comprehensive monitoring system in place which, among monitoring other systems, keeps track of these backups and notifies us when their metrics fall outside of our window; for instance we receive an alarm if the database replica is more than 5 minutes behind production or if large storage backups exceed 48 hours.

We maintain a comprehensive set of operational plans for responding to various incidents which may arise, ranging from hard drive failure up to and including the complete failure of a datacenter, which proved to be important when addressing this scenario. We also have standard response times for responding to various tolerances being exceeded which are designed to minimize user impact; for instance we configure the storage utilization alarms with sufficient lead time to provision additional storage.

The complete failure of a datacenter is the most challenging situation we have prepared for, and it is the situation with which we were ultimately faced.

The incident

At around 06:00 UTC on January 10th, a layer 3 distributed denial-of-service (DDoS) attack began to target SourceHut’s PHL infrastructure. We routinely deal with and mitigate application layer (layer 7) DDoS attacks, however, a layer 3 attack takes place at a lower level and is not within our ability to mitigate without the assistance of our network provider. Starting from about 06:00, our monitoring systems noticed that something was wrong and raised the alarm, and we started investigating the issue shortly thereafter. However, before we could get a handle on the situation, our access to the PHL network completely disappeared at around 09:00 UTC.

We never received any kind of ransom note or other communication from the attacker. We do not know who was behind the attack, nor their motivations, and likely never will. We know that they targeted SourceHut specifically, and that they followed us as we worked on mitigations, directing their attack at new infrastructure as it was being set up.

In this post-mortem we are going to focus on the impact on our network and the steps we took to restore service, rather than going into what we know of the attack and the details of our mitigations, both for information security reasons and to avoid lending legitimacy to a bad actor. In the end, the response to the attack from our upstream network in PHL did more damage to our infrastructure than anything else.

We rent network service in PHL from our colocation provider, who provisions a subnet out of their AS and routes it through Cogent and Level 3. In response to the attack, Cogent announced null routes for our downstream AS, causing our PHL network to become unreachable both for SourceHut staff and the general public.

We attempted to contact our colocation provider at this point, but we were unsuccessful. Our provider in PHL has been acquired twice over the past couple of years, and it seems that the ticketing portal we were used to paging them with had been deprecated and our account had not been migrated to the new system. We were unable to reach them until their normal office hours opened at 14:00 UTC. Our provider restored our access to the priority ticketing system and began to investigate the issue. Eventually they were able to convince Cogent to reduce the null route from our entire /24 to a narrower /32 focusing on the specific IP address being targeted by the DDoS. As a result, service was mostly restored by 18:00 UTC.

At about 06:30 UTC the following morning, the DDoS escalated and broadened its targets to include other parts of our PHL subnet. In response, our colocation provider null routed our subnet once again. This subnet has been unreachable ever since.

At 09:00 UTC, I made the call to perform an emergency migration to AMS and restore from backups. We began urgently planning the scope of this work, making an assessment of our backups and the Amsterdam installation’s capacity to host a fully functional SourceHut installation, and planning the work that needed to be done. We verified the health of FRE and AMS and our backup systems, finding that our standby database was less than 30 seconds out of date with production and that our large storage backups were no more than 12 hours old. We found that AMS was in good health and we touched up a few research systems (e.g. Ceph) to bring them to production readiness.

What essentially followed was creating a new installation of SourceHut from scratch, importing production data, testing and verifying the installation, and bringing it online for user service on a new DDoS-resistant network. This involved hundreds of small tasks that we planned out and distributed among ourselves and executed as urgently as possible, the full details are too involved to repeat here. However, the general plan had the following broad strokes:

Migrate to external DNS
Bring up a new primary database
Restore backups from FRE to AMS
Migrate object storage from PHL to AMS
Bring up database-only services, such as meta, todo, project hub
Bring up large storage services as backup restores finish
Manually apply the diff between production data and the last backups
Provisioning a new mail system and configuring mail services for it
Identify a suitable DDoS-resistant network to bring us into general service
Provision servers to backfill AMS capacity to production readiness
Test the new systems and bring them into general service

All of this work happened in parallel and had cross-dependencies, so from here on out we will omit some timestamps.

Network solutions

The selection of a suitable network to bring into service that would not immediately collapse as the DDoS attack followed us presented some challenges. First, this was a layer 3 attack, which can essentially only be mitigated by having more bandwidth than the attacker and/or by filtering traffic at a location suitably far upstream. Furthermore, many DDoS protection systems only operate at the application layer, namely for the web as the application in question, whereas SourceHut has user-facing services through not only HTTPS but also SSH, SMTP, and IRC; moreover we strongly prefer to utilize end-to-end encryption on all traffic and terminate it on SourceHut-operated infrastructure.

We initially researched a number of solutions, and spoke to Cloudflare in particular due to their ability to provide a rapid response to ongoing incidents. However, given our complex requirements, Cloudflare quoted us a figure which was not attainable within our financial means as a small company. Other options we researched (though we did not seek additional quotes) had similar economical constraints. However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.

We also got in contact with our network operator in AMS and explained the situation to them. We found some solutions to mitigate attacks directed at AMS but we will not disclose details regarding the AMS network for reasons of information security.

We made some mistakes throughout the operation, rsyncs done incorrectly, networks misconfigured, and so on. One particularly amusing mis-step occurred when we configured the NAT through OVH: we naively NAT’d all traffic through to AMS, and when the attack resumed targeting our OVH infrastructure, the inbound DDoS was briefly forwarded to AMS before OVH’s mitigations kicked in, which made our brand new OVH account look like the source of an outgoing DDoS, with predicable consequences that took some work to resolve with OVH.

Following our initial quote from Cloudflare, we understand that some Cloudflare employees undertook a grassroots effort internally to convince the leadership to sponsor our needs, and eventually Cloudflare came back to us with an offer to sponsor our services for us free of charge. This was a very generous offer for which we are very appreciative; in the end we did not take them up on it as we had made substantial inroads towards an alternative solution by that time. I have had my reservations about Cloudflare in the past, but they were there for us in a time of need and I am grateful for that.

On January 12th, our network provider in PHL agreed to provision a temporary subnet through which we could receive out-of-band access to our PHL servers. We were able to speed up the provisioning of replacement infrastructure thanks to this being made available.

Provisioning services

Efforts to restore individual SourceHut services came with a variety of challenges. Some of them were fairly straightforward; services like paste and the project hub depend only on a working SQL server and once we had that in place we were able to restore them to service at about 19:00 UTC on January 11th.

git.sr.ht and hg.sr.ht had special considerations due to the large amount of data they were responsible for. We restored them from backups to read-only service based on somewhat stale data by 20:00 UTC, and restored to full read/write service by 11:00 UTC on January 13th. We also did our best with hg.sr.ht, but it is community maintained and restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help; it was fully restored at 08:20 UTC on January 15th.

pages.sr.ht also presented unique challenges due to its use of an S3-compatible object storage solution, which we discovered was not considered in our off-site backups. However, it was a priority to restore service to bring online the numerous personal and project websites which depend on pages.sr.ht. Migrating object storage from PHL to AMS is a slow procedure which is still underway now, we are running pages through an out-of-band link to PHL which comes with some performance limitations but provides service while the slower migration process is underway in the background. Pages also had to be moved to a new subnet and IP address, which posed a problem for users who configured their apex records to point directly to the PHL subnet; we have now emailed affected users with instructions for manual intervention.

chat.sr.ht also poses some unique constraints due to its approach to networking; for a number of reasons we implement outbound IRC connections on a unique IPv6 address per-user. Preparing the requisite network infrastructure and bringing it online required special considerations and as a consequence chat.sr.ht was the last service to be restored to full operation at 10:30 UTC on January 17th.

builds.sr.ht also required some extra considerations due to the fact that our research installation in AMS was under-provisioned for its compute requirements. Currently we are running build workers on dedicated baremetal OVH servers and we are planning to bring build service back onto SourceHut-operated infrastructure once we are able to ship our builds-oriented compute servers from PHL to AMS in the coming weeks. We began accepting build jobs again at 18:00 UTC on January 16th.

All told, we began restoring partial service availability on the afternoon of January 12th, and gradually brought up services over the following days until full service was restored on the morning of January 17th. Following the complete restoration of service we credited all paying SourceHut users with 7 days of free service.

What’s next

We have completed all user-facing steps required to restore SourceHut to full service. However, we have numerous tasks to perform internally to put the finishing touches on the new installation and to clean up workarounds and hacks put in place to restore service as quickly as possible. We have an internal bugtracker with about 60 post-incident tasks to be completed, some of which are finished and others which are still underway: for instance, we have provisioned a new standby database, configured backups from AMS => FRE, and we are working on installing a new monitoring system.

We are also deprecating our PHL datacenter installation entirely. We still cannot reach the network there, and the operator has egregiously exceeded their SLA with us. We will pack up the servers in PHL and ship them to AMS to supplement capacity there.

Ironically, this emergency migration allowed us to quickly achieve many of the long-term goals we had in mind for migrating to the EU. We are now running almost entirely on European infrastructure and we quickly finished bringing many of our plans to ensure future scalability and reliability of SourceHut infrastructure to production readiness. We had planned to do these upgrades without user impact, but, well, alas.

Part of the work involved in setting up the AMS datacenter installation was geared towards increasing resilience, reliability, and redundancy, and reducing the impact of attacks and outages of this nature. SourceHut is still in an “alpha” state, and though we have built a platform that is sophisticated and depended on by numerous free software projects, that comes with some caveats. This incident exposed one such caveat, a shortcoming that we were aware of and actively working on improving, but which was exploited before we could complete our goals for redundancy and resilience. Our definition of production readiness, our “definition of done” for the alpha, includes building infrastructure that prevents this kind of outcome. We have made unexpectedly urgent inroads on these milestones, and we will continue to build out these solutions as part of our work to bring SourceHut from alpha to beta to production.

In the service of this work, we have been researching a solution for running SourceHut in an on-premise Kubernetes cluster (cue gasp from the audience). The migration to AMS did not involve a migration to Kubernetes, but it did put us in a better position to expedite that work. We intend to utilize this work to reduce user-impacting downtime during internal maintenance, such as when scaling up our infrastructure or upgrading software, but also to improve our resistance to events such as the one that transpired this week. Once we have completed an implementation of this solution, we will be planning out a secondary production-ready datacenter which we can load balance against and which can step up to take on full service in the event of a datacenter outage, as well as making it easier to conduct the procedures that we employed this week to stand up SourceHut from scratch in a new datacenter should the need ever arise again.

As unfortunate as these events were, we welcome opportunities to stress-test our emergency procedures; we found them to be compatible with our objectives for the alpha and we learned a lot of ways to improve our reliability further for the future. We are going to continue working on our post-incident tasks, building up our infrastructure’s resilience, reliability, and scalability as planned. Once we address the high-priority tasks, though, our first order of business in the immediate future will be to get some rest.

Acknowledgements

I’d to acknowledge those who helped us in this difficult time. Of course, I extend my thanks to the rest of the staff, Simon and Conrad, whose long hours of work ensured we could restore service for our customers as quickly as possible. I would also like to thank the staff and network operators at our various datacenter locations, for their role in facilitating our needs in the rapid bring-up of new infrastructure. Thanks especially to the community members who sent us their words of support and kindness, and offers of aid, via email, Mastodon, on IRC, and so on.

Thanks are also due to our peer in the free-software-forge space, Codeberg. They hosted our status page prior to the incident, and likely faced their own DDoS attack in retaliation for the services they provided to us. We shared information across our teams extensively and their communication and relationship with SourceHut has always been superb.

I’d also like to thank the team at Cloudflare; though we did not accept your generous offer we were appreciative that you were there for us in a time of need.

On a personal note, I would like to thank my fiancé for their patience and unwavering emotional support during a challenging time.

Thank you all for your patience and support.

– Drew DeVault

Information regarding the SourceHut outage

Tue, 16 Jan 2024 00:00:00 +0000

SourceHut is currently experiencing an extended outage due to a distributed denial-of-service (DDoS) attack. We have been hard at work restoring service to our users, and expect the remaining services to come online this week. Thank you for your patience and words of support, we appreciate it very much.

Once we have finished handling the majority of the work associated with the outage mitigations, we will be publishing a full post-mortem. For up-to-date information regarding the status of the outage and work associated with it, please consult status.sr.ht.

The billing system has been disabled for the duration of the outage: you will not be charged until we have fully restored service. Once restored, we will credit paid users with service for the duration of the outage.

The current service status is as follows:

Service	Status
sr.ht (project hub)	Full service
meta.sr.ht	Full service
git.sr.ht	Full service¹
hg.sr.ht	Full service¹
lists.sr.ht	Full service
todo.sr.ht	Full service
man.sr.ht	Full service
paste.sr.ht	Full service
builds.sr.ht	Full service
chat.sr.ht	Full service
pages.sr.ht	Degraded performance²

¹ We have restored git and hg from backups which were several hours old at the time the outage began, and manually imported changes which took place between the time of the last backup and the start of the outage.

If your repository is still out-of-date, please take the following steps:

If you have access to the up-to-date repository locally, please git push or hg push your changes to bring the online version up-to-date.
If you do not have access to your repository locally, please [email protected]">contact support for assistance.

² pages.sr.ht is online for full service, with the following caveats:

Publish operations are working with degraded performance
Manual intervention is required for users using custom domains with apex records (e.g. example.org rather than subdomain.example.org)

For users with apex records, please update your apex record as follows:

@   IN  A   46.23.81.157

This IP address is available on a temporary basis. Make sure your email address on file is up-to-date; we will email you if and when it changes to notify you of the steps required to update your website.

SourceHut's 2022 Financial report

Mon, 27 Mar 2023 00:00:00 +0000

In summary, SourceHut continues to be sustainable, profitable, and growing. We continued to grow YoY between 2021 and 2022. We were able to make some significant investments, particularly in establishing a new datacenter presence in Europe, and we intend to expand this investment over the course of 2023 and 2024, and we have plenty of capital to invest in this and other business initiatives.

Disclaimer: this report is a summarized approximation of our financials, and is not used for tax purposes.

2022 revenue sources

SourceHut has two main revenue streams:

Paid subscriptions for the forge platform, which are generally consistent
Contracts via our consulting arm, which are generally more inconsistent

We use the forge platform revenue as the basis for our financial planning and sustainability model, and treat consulting revenue as capital to fund new projects and investments.

Platform revenue

We issued 12,965 invoices in 2022. Subscriptions are available at three price points: $2, $5, and $10; billed monthly, or annually for a 2 month discount. Invoices paid in 2022 break down as follows:

6,530   $ 2   (paid monthly)
2,456   $ 5   (paid monthly)
  917   $10   (paid monthly)
2,141   $ 2   (paid yearly)
  604   $ 5   (paid yearly)
  227   $10   (paid yearly)

The total revenue from platform subscriptions in 2022 was $132,226.50, which, after payment processing fees, becomes $123,677.57; a 25% increase over 2021. In 2022 a total of 8,544 users signed up for an account, of which 1,066 have paid for their account. This is a conversion rate of about 12%, which is comparable to 2021. Among all registered users (35,877 at the time of writing), about 11% have paid accounts (5,651).

When the beta is completed, all users who own resources on SourceHut (source code repositories, mailing lists, bug trackers, etc) will be expected to pay for their account. Users who contribute to projects owned by others are not expected to pay, and users who wish to own resources but cannot afford the subscription fee will be offered financial aid. In 2022, 81 users received financial aid; 100% of requests for financial aid in 2022 were approved.

The main reasons cited in applications for financial aid remain largely the same as 2021:

Inability to pay due to financial constraints
Lack of support for their preferred payment method
Political sanctions due to the situation in Ukraine and Russia

Consulting revenue

SourceHut accepts consulting projects at the discretion of our full-time engineers, part-time. These contracts are executed in public and are required to only produce free and open source software.

In total, SourceHut received $367,810 in revenue from consulting engagements in 2022.

Summary of revenue

Our total revenue from all sources for 2022, before taxes, expenses, and fees, was $505,307.

Cash on hand

At the close of the year, we had $124,964 USD and €23,297 EUR in cash on hand.

2022 expenses

In 2022 we had the following costs, excluding payroll:

$20,103     European infrastructure investment
$12,018     Infrastructure (colocation, power, network, etc)
$ 8,548     Payment processing fees
$ 1,861     Sponsorships (OpenStreetMap, Codeberg)
$ 1,353     Business-related travel (events)
$ 1,309     Services (Migadu, third-party backups)
$ 1,153     Misc. equipment
$   816     Domain name registrations
$   209     Misc. other

Each of our three staff earns a fixed $2500 monthly base pay, and received a $5000 bonus. Staff and contractors (a total of five people in 2022) receive additional income from the consulting arm which varies depending on the contract and amount of time worked.

Base pay for 2022 works out to about $105,000. Including compensation for consulting-related work, the total spent on payroll in 2022 is about $250,000.

Figures for staff compensation are sensitive to currency-related factors due to our presence in the US, Canada, and Europe; these figures are more approximate than the rest of the report.

Our total expenses and investments for 2022 were approximately $272,000.

2022 financial summary

Revenue: $505,307
Expenses & investments: -$272,688
Taxes: -$60,000 (approx)
Profit: $172,619 (approx)

Fun fact: as of this year, the lifetime revenue for SourceHut has exceeded $1M USD.

Expectations for 2023

We have three full-time staff paid under the standard compensation model, and about $1400/mo in recurring expenses. Currently, we make about $11000/mo from the platform, putting our monthly profit at about $2000, without factoring in consulting revenue. Thus, the platform provides for a sustainable business independently of our other lines of business.

The breakdown of active subscriptions at the time of writing is as follows:

$2    monthly   595 users
$5    monthly   231 users
$10   monthly    79 users
$2    yearly  2,137 users
$5    yearly    609 users
$10   yearly    234 users

We anticipate spending more on our European datacenter investment as we continue building it out, and we hope to shut down the Philadelphia datacenter in 2023 or early 2024, saving about $600/mo in operating expenses.

We have closed out a major consulting engagement in Q1 2023 and we anticipate a significant reduction in consulting revenue as a consequence. We hope to complete the beta this year (you may have heard that before…), which would likely correspond with a significant increase in platform revenue.

We also anticipate moving some or all our platform revenue into Europe in 2023. We hope to be dealing primarily in Euro by 2024.

Conclusion

In short, SourceHut is profitable and is comfortably succeeding in its efforts to provide sustainable software development & infrastructure for the free and open source software community. We remain committed to investing 100% of our profit into the free and open source community. I want to extend my thanks to the SourceHut community for its role in achieving our goals: sincerely, thank you.

SourceHut will (not) blacklist the Go module mirror

Mon, 09 Jan 2023 00:00:00 +0000

Update 2023-01-31: Russ Cox of the Go team reached out to us to address this problem. After some discussion, an acceptable plan was worked out. The Go team is working on deploying an update to the “go” tool to add a -reuse flag, which should substantially reduce the traffic generated by this system for all users of Go.

In the meantime, the automated refresh traffic from proxy.golang.org was disabled for SourceHut, which the Go team assures us should have little-to-no impact on users and which reduces the burden on our system to a managable level. Following this change by the Go team, we have observed traffic from the Go module mirror reduced to an acceptable level. The Go team has decided that the automatic refresh behavior is their responsibility, not the responsibility of other operators, so any other small hosts will hopefully not be affected as the Go team will enable or disable the refresh behavior at their discretion with the burden on third-party operators in mind.

Consequently, we have cancelled our plans to disable Go traffic to git.sr.ht. No action is required by users to continue receiving service. Thanks Russ!

The original post can be read below.

SourceHut will disable git access for the Go Module Mirror on February 24th, 2023. This will cause a service impact for Go users. This article explains why this step is necessary and how Go users can work around the issue.

tl;dr

From February 24th, users running go get or a similar command on Go packages which import modules from SourceHut repositories will be met with an error message similar to the following:

$ go get
go: downloading git.sr.ht/~sircmpwn/foobaz v0.0.0-20230108094957-81402546c10e
go: git.sr.ht/~sircmpwn/[email protected]: verifying module: git.sr.ht/~sircmpwn/[email protected]: reading https://sum.golang.org/lookup/git.sr.ht/~sircmpwn/[email protected]: 404 Not Found
	server response:
	not found: git.sr.ht/~sircmpwn/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/568e5edafe93f7887c0b6f718b0f17ea91c63c35822fb28628535f172b5429b7: exit status 128:
		fatal: unable to access 'https://git.sr.ht/~sircmpwn/foobaz/': The requested URL returned error: 429

The following workaround will correct the issue:

$ export GOPRIVATE=git.sr.ht
$ go get # works

For more detail, read on.

Background

The Go programming language fetches modules via git, such that a user who imports “git.sr.ht/~sircmpwn/dowork” will cause the toolchain to fetch the corresponding repository via git in order to make it available in the user’s Go environment. Each of these requests is routed through a proxy service at proxy.golang.org, which provides a number of features:

Offers reliable and fast access to module downloads via Google’s cache
Stores redundant copies of Go modules to ensure availability
Records an independent source of their checksum for the checksum database
Provides an index of new Go modules

This comes with a number of downsides. For example, most Go users are unaware that every package they fetch is accompanied by a request to Google’s servers, and implies a trust relationship with Google to return authentic packages. Additionally, should the underlying source repository disappear or become out of sync with the cache, the problem is hidden from Go users, which can cause their software to become dependent on modules or module versions which no longer exist.

More importantly for SourceHut, the proxy will regularly fetch Go packages from their source repository to check for updates – independent of any user requests, such as running go get. These requests take the form of a complete git clone of the source repository, which is the most expensive kind of request for git.sr.ht to service. Additionally, these requests originate from many servers which do not coordinate with each other to reduce their workload. The frequency of these requests can be as high as ~2,500 per hour, often batched with up to a dozen clones at once, and are generally highly redundant: a single git repository can be fetched over 100 times per hour.

This traffic produces an excessive background workload which is constantly being serviced by git.sr.ht. Due to the relatively large traffic requirements of git clones, this represents about 70% of all outgoing network traffic from git.sr.ht. A single module can produce as much as 4 GiB of daily traffic from Google.

Seeking other solutions

Blacklisting GoModuleMirror is our last resort, and we wanted to avoid it if possible. We attempted to work with the Go team on a solution, but were unsuccessful.

On February 24th, 2021, we reported an issue to the Go team regarding this problem. The Go team initially helped us narrow down the cause, first by setting an appropriate User-Agent to help identify this traffic, then through discussions regarding the behavior of this system. We made recommendations to Google for how to service their requirements without generating an excessive amount of redundant traffic. However, the discussion stalled and no further changes were made by Google to address the issue, and we continued to receive an excessive amount of traffic from the module mirror.

The situation remained so for over a year. In that time, I was banned from the Go issue tracker without explanation, and was unable to continue discussing the problem with Google. With few options left, I wrote a blog post on May 25th, 2022 outlining the issue and petitioning the public for support in addressing this problem. However, the problem remains unsolved.

February 24th, 2023, the date that we plan to disable Go traffic, marks two years since the initial complaint was submitted to the Go team. The cost of bearing this traffic is no longer acceptable to us, and the Go team has made no attempts to correct the issue during this time. We want to avoid causing inconvenience for Go users, but the load and cost is too high for us to continue justifying support for this feature.

Recommendations for the Go team

From February 24th, git clone requests with a GoModuleMirror User-Agent will receive a 429 (Too Many Requests) response. To restore service, we have the following recommendations for the Go team:

Obey robots.txt, including the Crawl-Delay directive, to control the rate at which modules are fetched.
Perform a shallow git clone rather than a full git clone; or, ideally, store the last seen commit hash for each reference and only fetch if it has been updated.
Reduce redundant traffic: fetch each git repository less often. It should not be necessary to fetch the same git module up to 2,000 times per day.

If these issues are addressed, we would be pleased to re-enable GoModuleMirror’s access to our services. The Go team can reach me [email protected]">via email to discuss the matter further, if you have ideas for other solutions or require any additional details to address the problem.

Recommendations for Go users

Unfortunately, this will affect Go users and will require workarounds to be implemented in order to compile Go software which depend on modules hosted on git.sr.ht. If you encounter an error similar to the following:

$ go get
go: downloading git.sr.ht/~sircmpwn/foobaz v0.0.0-20230108094957-81402546c10e
go: git.sr.ht/~sircmpwn/[email protected]: verifying module: git.sr.ht/~sircmpwn/[email protected]: reading https://sum.golang.org/lookup/git.sr.ht/~sircmpwn/[email protected]: 404 Not Found
	server response:
	not found: git.sr.ht/~sircmpwn/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/568e5edafe93f7887c0b6f718b0f17ea91c63c35822fb28628535f172b5429b7: exit status 128:
		fatal: unable to access 'https://git.sr.ht/~sircmpwn/foobaz/': The requested URL returned error: 429

You can correct it by bypassing proxy.golang.org:

$ export GOPRIVATE=git.sr.ht
$ go get # works

We will enable these variables by default for builds.sr.ht jobs prior to disabling service for the Go module mirror.

We understand if Go users need to migrate away from SourceHut to deal with these problems. We apologise for this inconvenience, and we hope to see you return should the problem be resolved. The SourceHut team is available to assist with any necessary migration efforts via sr.ht-discuss.

NLnet funding comes to an end

Mon, 21 Nov 2022 00:00:00 +0000

Sourcehut was selected for an NLnet grant in January of this year. NLnet Foundation is a Dutch organization which funds many projects to promote an open and secure internet.

The typical NLnet grant is structured by milestones. At the beginning of the grant, a project plan is drafted containing the milestones to be completed and an agreed upon payment for each milestone. Once a milestone is completed, NLnet verifies that the milestone had been completed and approves the payment. If you think this might be a good fit for your project, consider applying for a grant!

Our grant work focused on completing the GraphQL APIs for Sourcehut services to help with Sourcehut’s API 2.0 efforts. This work included many changes and improvements to the GraphQL APIs as well as implementing support for GraphQL-native webhooks.

Implementing GraphQL-native webhooks for git.sr.ht was the first milestone on our roadmap. This work involved:

Designing and implementing the API for creating GraphQL webhooks
Updating the GraphQL mutations to trigger webhooks
Updating the Python frontends to route all necessary operations through the GraphQL APIs so that webhooks are triggered
Updates to ancillary services/daemons (gitsrht-shell and gitsrht-update-hook)

A similar process was followed for hg.sr.ht, todo.sr.ht, builds.sr.ht, lists.sr.ht, paste.sr.ht, and pages.sr.ht.

Our focus then turned towards implementing a GraphQL API for hub.sr.ht. Near the end of June I began experimenting with GraphQL Federation in the hopes that it would make implementing the hub.sr.ht GraphQL API easier. This work led to the development of api.sr.ht, a GraphQL API gateway.

Before we can adopt GraphQL Federation, we needed to unify user IDs. Previously, user IDs were not guaranteed to be the same across SourceHut services. This did not play well with GraphQL Federation, which requires some sort of unique key to identify users across services. Patches for nearly every Sourcehut service were needed to address the issue.

In order to ensure that the ID unification roll-out went as smoothly as possible, we decided that we needed better tools for testing. This resulted in the development of srht-fakedb, a tool to populate Sourcehut service databases with fake data for testing purposes. The ID unification roll-out took place on October 26. With that now out of the way, work on the hub.sr.ht GraphQL API can resume.

October 30 officially marks the end of the NLnet grant. We achieved most of what we set out to accomplish, and some more. We’ll be wrapping up this work on SourceHut’s dime in the coming weeks.

Working with NLnet has been a great experience. Many thanks to NLnet and NGI Zero for sponsoring this work! We appreciate your generosity, and are looking forward to more projects developing under your fundraising umbrella.

Four years of SourceHut

Tue, 15 Nov 2022 00:00:00 +0000

Today is the fourth anniversary of SourceHut’s public opening. In these four years, we have developed a free software platform which hosts nearly 10,000 projects, among them more than 68,000 repositories, 5,000 mailing lists with nearly a quarter-million emails, 6,500 bug trackers hosting 50,000 tickets, and 914 build-days over 880,000 jobs on builds.sr.ht, all in the service of our 33,516 users.

I’m tremendously proud of the community that has grown on this platform. I offer my sincere thanks to the thousands of people who have joined SourceHut and helped build a welcoming, productive community of free software developers. I offer my thanks to you, dear reader, for your profound role in these achievements. I struggle to find the right words to express the enormity of my gratitude.

In lieu of adequate words to describe it, we strive to express this gratitude through our actions. Our work is entirely free software, and our community are co-owners in it; our hundreds of contributors have retained the copyright for their work and our copyleft licenses enforce our promise to forever remain free software. Many of these contributors have graciously stepped into roles as maintainers of various subsystems in their lines of expertise, and we have offered them trust and responsibility equal to the degree of their contributions and investment.

We have further increased our commitment to transparency and honesty in our affairs this year, be it with our financial reporting; operations wiki, data, and monitoring; our staff wiki; or cultural artifacts such as striving to make our staff members directly accessible to users for addressing their needs. We have consulted you every step of the way, sought out and honored your insights, and carefully crafted our incentives to institutionalize a system that places your interests at the forefront.

I’m honored to be of service to the free software community, and I hope that we will continue to improve and grow to thrive in this role, and do our part to ensure the success of the free software movement.

With all of this in mind: what happened this year, and what can we look forward to for next year?

The year in review

SourceHut development in the past year has been focused on completing the roll-out of our GraphQL APIs (aka API 2.0). One of the major concerns for the alpha is the completion of a robust and stable API that we’re comfortable supporting for a long time. This effort has been quite successful; all of the major services now support GraphQL and we’re preparing to build out the stragglers (project hub and man.sr.ht) over the next few weeks. With the planned development of GraphQL federation, we will complete a system which allows the federated mini-services of SourceHut to interoperate with one another in a much more flexible and powerful manner, while still retaining their independence and flexibility for third-party installs that only need a subset of the services.

This period has not been absent of new features, however. We also completed and released chat.sr.ht, a hosted IRC bouncer service for SourceHut users. This software is built on soju and gamja, respectively a bouncer and webchat service, developed as free software by SourceHut’s own Simon Ser. This service makes IRC much more accessible and easy to use, and makes it easy for projects on SourceHut, and their contributors, to maintain and access real-time chat services for development discussions and end-user support — and for fun, plenty of us use IRC to hang out with our friends! Simon also began work on hut this year, which is a command line tool for interfacing with SourceHut that has become a staple of many of our users’ workflows.

We also expanded our roster this year. We brought on Conrad Hoffman as SourceHut’s newest full-time developer, who has been putting his talents to good use to help prepare us to move our infrastructure into the EU. We also received a generous grant from NLnet, which covered the costs to bring Adnan Maolood onto the project, whose efforts have been indispensable in helping to complete our GraphQL implementation.

Many of our services have been expanded with additional features, including new features for SourceHut pages, a greater variety of build images for builds.sr.ht, finer access controls for all services, and, recently, the long-promised self-service account deletion and data export features — important for our goals of enabling user ownership over their own data.

Coming soon

As I mentioned earlier, we’re putting a bow on GraphQL over the next several weeks, as we roll out the last couple of services and start setting up GraphQL federation. Federation will make it easier for us to implement greater inter-service connectivity, which will offer many useful improvements, including finally resolving the annoying “how do I get to a project from its git repo” problem once and for all.

We also intend to overhaul our billing system in the foreseeable future, adding support for additional currencies and payment methods, such as payments in Euro via systems like SEPA or iDEAL. This will also resolve many long-standing papercuts with the billing system and provide users with a greater degree of control over their billing details. This is also important for the business side of SourceHut, since we’ll be using this opportunity to migrate our platform revenue into Europe.

On the subject of moving into Europe, we’ve also started setting up our new European datacenter installation in Amsterdam. We have provisioned a few hosts and we’re using them to experiment with better ways to deploy SourceHut infrastructure, including, yes, the dreaded Kubernetes. If this works out, many users who wish to deploy SourceHut into a typical cloud environment will find the task more easily done. We’re also hoping to use this effort to make strides towards turning our four nines into five nines, improving on our already industry-leading stability.

We also plan on tackling another long-awaited feature soon: user groups, or organizations. This and the other features described have been blocked on the completion of our GraphQL work, and with this work out of the way, it’s time to prepare for a flurry of new feature development to round out the alpha and finally bring the SourceHut beta into life. All of these plans have one thing in common: they’ve been identified since the start as our major priorities for the completion of the alpha. It’s happening.

Coming later

This covers what you can expect over the next several months, but I also want to spend a moment discussing some of the things we’re planning to do in the longer term. Here are a few possible future services which are on our minds:

jobs.sr.ht

This service is something we drafted and mocked out several months ago, and it’s been sitting on the backburner while we address more urgent priorities. The essential idea is to provide a jobs board for hiring developers to work on free software.

Through a combination of hard work and luck, SourceHut has settled upon a financial model which allows us to sustainably work on free software projects in a manner compatible with our ethical obligations. We want to help others in the free software ecosystem achieve similar goals for themselves, so that the free software world can grow and flourish on the backs of healthy and happy developers. To this end, we conceived of jobs.sr.ht.

The plan is to allow users to list themselves on the platform so that interested clients can seek out free software developers to hire for projects. We want to accommodate a variety of approaches: a maintainer might be hired by their users to implement a desirable feature over the occasional weekend, or a group of free software developers could form a collective to take on more serious contracts to develop free software for the industry. An artist could list their services to design the mascot for new free software projects, or a maintainer could hire a talented developer to help them ship a new release.

Here’s the ticket: SourceHut will not charge any fees to either party beyond that of your existing SourceHut subscription. Users who receive financial aid will also be entitled to utilize this service. We’ll cover the expense of operating this platform out of our normal platform revenue. Additionally, we will make no efforts to insert ourselves between contractors and their clients. You will be responsible for writing and agreeing to your own contract terms, facilitating payment between yourselves, and will communicate normally through email, rather than from within a walled garden. We will never hold your means of income hostage. If you choose to leave or move elsewhere, you take all of your contracts with you, and can import your details into an instance of jobs.sr.ht hosted by anyone else.

names.sr.ht

An older idea that we’re still interested in working on is names.sr.ht, a domain registrar and DNS hosting service. This concept is born out of the frustrations of using a janky JavaScript record editor to update DNS records: what if we could just push a zone file to a git repository instead?

We also intend to act as a domain registrar (or rather as a reseller, we don’t have the resources to be a first-class registrar). We will not charge any additional margin over the price we receive, and instead will include these services as part of your normal SourceHut subscription fee. As our purchase volume grows, we will be able to negotiate bulk discounts and pass the reduced prices back to you.

Not sure when we can expect to see this service — it’ll probably be done when one of our staff members gets bored for a couple of weeks and blitzes out the initial prototype. In any case, I’m looking forward to it.

mail.sr.ht?

We’re thinking about developing a hosted email service. Email is, as you know, very important to SourceHut’s workflow. We would love to provide a killer hosted email experience for users to better facilitate this.

Such a service would include all of the trimmings that we feel are essential — standard protocols such as IMAP and SMTP, configurable sieve filters, unlimited domains, and so on. We may consider researching a model which allows for bulk mail delivery, naturally subject to the condition that you are not using it for spam — hosting something like a development mailing list or sending our confirmation emails for sign-ups to your Mastodon instance would be more appropriate. We’ll also add some features which are unique to SourceHut, such as close integration with services like lists.sr.ht — and wouldn’t it be nice to see the CI status of a patch and click “apply” right in your inbox? Once names.sr.ht is rolled out as well, we can use it to make it easy to manage the pesky DNS settings necessary to ensure smooth delivery and compatibility with things like mailing list forwarding.

In conclusion

It’s been a great four years so far, and we have more great years to look forward to. We’re well on our way to a more complete and reliable system, and we’ve been able to provide more support for the free software ecosystem than I ever imagined possible. I’ll re-iterate my thanks to you, dear reader, for reading this post, and for using SourceHut. You’re the best.

That’s all for today. Take care!

SourceHut terms of service updates, cryptocurrency-related projects to be removed

Mon, 31 Oct 2022 00:00:00 +0000

SourceHut is planning to roll out updates to our terms of service, effective in 2023. The changes most likely to impact users is the prohibition of cryptocurrency- or blockchain-related projects on SourceHut.

These domains are strongly associated with fraudulent activities and high-risk investments which take advantage of people who are suffering from economic hardship and growing global wealth inequality. Few to no legitimate use-cases for this technology have been found; instead it is mostly used for fraudulent “get rich quick” schemes and to facilitate criminal activity, such as ransomware, illicit trade, and sanctions evasion. These projects often encourage large-scale energy waste and electronics waste, which contributes to the declining health of Earth’s environment. The presence of these projects on SourceHut exposes new victims to these scams and is harmful to the reputation of SourceHut and its community.

We recognize that the basic idea of a blockchain, as it were, may be generally useful. However, most projects which market themselves with blockchain technology are subject to the same social ills as cryptocurrency. Consequently, we have chosen to include “blockchain” related projects in this ban for the time being.

We will exercise discretion when applying this rule. If you believe that your use-case for cryptocurrency or blockchain is not plagued by these social problems, you may ask for permission to host it on SourceHut, or appeal its removal, by [email protected]">contacting support.

Projects which seek out cryptocurrency donations are strongly discouraged from doing so, but will not be affected by this change.

What to do if you don’t agree to the changes

If your project is affected by these changes, or you do not agree with them, you have until January 1st, 2023 to migrate to another platform. You may use hut(1) to obtain an export of your account data using standard formats that you may import into a third-party SourceHut instance, or another host which is compatible with these formats (such as Codeberg, GNU Mailman, etc). If you wish, you may request to have your account deleted by [email protected]">contacting support.¹

Other changes to the terms of service

Changes to the terms of service require notifying all users, so we try to batch them when possible to reduce the number of annoying emails everyone has to deal with. You can review a complete diff of the changes here.

Here are some other changes we’re including in this update:

Applicability of Dutch law

We are in the process of moving SourceHut to the European Union, and have incorporated in the Netherlands. Consequently, the terms have been updated to clarify that users are required to comply with Dutch law in addition to US law. The requirement to comply with US law will be removed in a future update after we close the US entity.

“Threatening and harassing others”

We have expanded the list of prohibited behaviors to include “threatening and harassing others”. We have already banned users for this behavior (we can terminate service at our discretion), but we decided to make this explicit. The scope of this is intended to cover incidents where users harass other users or make bigoted, racist, homophobic, transphobic, etc, comments on our platform.

We take account termination seriously, and use it only as a last resort. In the history of SourceHut, only two users have been banned.² The first user was harassing maintainers and failed to respond to emails seeking to discuss their behavior. The second used SourceHut to advocate for the rights of pedophiles to sexually abuse children.

Revised content license

We have updated the license which is granted to SourceHut when you upload content to our services. The purpose of this license is to ensure that we have sufficient rights over your content to do our job, and grants us permission to do things like display your content on the website.

We have clarified how this works in situations where you do not own the copyright over the content you upload, such as when you mirror a free software project on SourceHut, or when a project’s copyright is collectively held by many contributors. We have also removed the “indefinite” license and updated the terms so that the content license is terminated automatically when you remove your content from the services.

We would like to have self-service account deletion implemented by the time these changes are implemented, but it’s a complex feature that may require more time to implement. In the meantime, manual requests for account deletion will be processed normally. ↩︎
Among accounts representing real people, not including bulk spam registrations, cryptocurrency mining abuse, etc. ↩︎

What's cooking on SourceHut? October 2022

Tue, 18 Oct 2022 00:00:00 +0000

Greetings! Today we’re joined by 623 fresh new users, bringing our total of 32,904. News is light for this month, since I’ve been taking some time off. However, things are still gradually rolling forwards, and we have some important internal changes landing soon. I’ll give you the details.

Planned maintenance next week

We are rolling out a major, high-risk software upgrade on Monday, October 24th. This will require an outage, and we have set up a maintenance window starting from 09:00 UTC. For details, see status.sr.ht.

git.sr.ht

Vlad-Stefan Harbuz’s patches adding REUSE support to git.sr.ht have landed this month. Software which is compatible with the REUSE specification will enjoy native support for license analysis in git.sr.ht. This feature will be expanded in the future. Thanks Vlad!

lists.sr.ht

Thanks to Tanguy Fardet’s efforts, mailing list owners are now allowed to configure their lists to allow multipart text/html messages. Such emails are still required to have a text/plain part. This is disabled by default for new mailing lists and is discouraged for most users — you’ll know if you need it.

meta.sr.ht

Simon has continued working on improving our OAuth 2.0 implementation, aligning us closer with the specification and making it possible to use it with more out-of-the-box OAuth 2.0 client libraries. Please file tickets if you notice any additional discrepancies.

How to help improve SourceHut's design

Thu, 13 Oct 2022 00:00:00 +0000

SourceHut is a software development forge and it is designed with the software engineer’s needs first and foremost. The design prioritizes things like page speed, minimal distractions, and information-forward layouts. It does not prioritize aesthetics, and perhaps it shows.

Like many free software projects, SourceHut encourages contributions from its community in the form of patches. Not all of these have to be changes to the software — we frequently receive patches improving the documentation, for example. There are many skillsets which are valuable to offer to a free software projects. One of these skills is design competence, and over the years a few people with this skillset have attempted to improve SourceHut’s design, but many of their changes were rejected. Why? How can a designer succeed in improving SourceHut?

SourceHut’s design space imposes constraints on the visual design which present more of a challenge to the aspiring designer than most of the projects they may have worked on. We are open to (and excited about!) improvements to the SourceHut design, but such changes must be implemented within the constraints of our ethos, and these constraints are often unusual and unfamiliar to the typical web designer. SourceHut is an engineering tool first and foremost, and all of our design decisions are made with an eye on solving problems for software engineers. This is the prime directive, and all other concerns, such as subjective aesthetic value, are secondary to the prime directive.

It is under this lens that any proposed design changes are evaluated, and for this reason many are rejected. A design change for SourceHut must be carefully thought out and must balance its aesthetic value with its effect on the utility, usability, and accessibility of the services. To offer a small example: several people have proposed making the source view centered to match the rest of the UI. However, this would introduce a horizontal scroll for files with long lines, making it more difficult to read them. The prime directive of this page is reading code, therefore this change was rejected.

Of course, looking nice does, in its own way, contribute to many of the same underlying values like usability and accessibility. A page which is pleasant to look at is more usable. The user’s enjoyment of the service is derived in no small part from its aesthetic value, and that’s important to us as well.

The challenge is thus to come up with designs which are aesthetically pleasing, but also meet our goals for broad accessibility and utilitarianism, while simultaneously communicating our values and priorities. Our values differ substantially from other platforms in this space. GitHub, GitLab, Gitea, they are all fundamentally speaking the same design language and expressing the same values. Because our values differ, our design should differ, and distinguish us from the pack in a manner which makes our principles visually evident.

A designer who hopes to rise to this challenge should understand and internalize these values, which is particularly difficult given the extent to which we reject the values of the mainstream “modern web”. Should you want to meet this challenge, the community would be thrilled to work with you. I would strongly recommend joining our IRC channels and seeking feedback early and incrementally — most design proposals for SourceHut fail due to seeking feedback too late or proposing a huge redesign all at once. So long as we work together, we will find success. Good luck!

SourceHut does not ask for IP assignment from employees or contributors

Sun, 09 Oct 2022 00:00:00 +0000

A “standard” clause you’ll see in most employment contracts is an IP assignment clause. This clause assigns the intellectual property produced by an employee in the course of their work to the business that they work for. This generally extends both to copyrights and patents, and the more egregious of these clauses include work done outside of company time and without the use of the company’s assets and equipment. Often this is accompanied with a burdensome process of identifying every copyright owned by the employee on a little sheet that accompanies the employment contract.

Free software projects overseen by businesses also often want to control the copyright of external contributors. This is usually done by imposing a Contributor License Agreement, or CLA, on the contributor before accepting their change. These either assign the contributor’s copyright outright, or does so in all but name by signing over all of the rights associated with their copyright.

In the absence of such agreements, copyright for each contribution to a free software project remains held by the contributors themselves. Thus, ownership of that project’s IP is collectively held by its contributors. Consequently, the “owners”¹ of the project cannot change the license to make it more restrictive, or more permissive, or non-free outright, without the written consent of each of the copyright holders.² The project stewards are offered the same FOSS license terms for each contribution that they offer to everyone else for their own.

SourceHut does not ask our staff to sign an IP assignment, and we do not ask our external contributors to sign a CLA. Consequently, SourceHut, the business entity, does not own its intellectual property. It is collectively owned by the individuals who have worked on it, both internal and external. When I say SourceHut belongs to its users, I’m not using flowery doublespeak — I mean it literally.

To re-enforce this, we rely on copyleft software licenses. Copyleft, to offer a simplified explanation, essentially requires that any changes to the project must also be released under the same software license. If we held the copyright for our software, we could disregard these terms, but because contributors license their copyright to us under the same copyleft licenses, we are obligated to use their contribution under the same terms and release our changes as free software as well. If we used a permissive license like MIT or BSD, we could fork the project and keep our changes closed-source, but using copyleft closes this path to us and serves as a strong promise to the community that SourceHut will always be free software.

We do things a bit differently from the industry “standards”. We honor and respect each contributor’s work, both internal and external, and their copyright forms an important piece of the safeguards we use to establish trust with our community. We honor and respect our staff members as well, and do not impose upon them IP assignments, non-disclosure agreements, or non-competes. This approach is a more just way of doing business, and is our strategy for building a long-lived and sustainable free software platform, hand-in-hand with the community, on equal terms as peers working towards a shared vision.

I’ve taken to generally referring to people and entities in this role as “stewards”, because, well, they literally do not own it. ↩︎
They can also rewrite the contributions from those who do not agree to change the license. ↩︎

What's cooking on Sourcehut? September 2022

Thu, 15 Sep 2022 00:00:00 +0000

Guten Morgen, SourceHut! Today, I count 681 new users, for a grand total of 32281 registered users. As always, a warm welcome to them and the reminder to everyone else to help them feel welcome while they get settled. Today, I am filling in for Drew on short notice, so please excuse the brevity.

chat.sr.ht

Simon has done some excellent work simplifying the authentication workflow of chat.sr.ht for users. The new approach is based on standard OAuth, which means other users of soju (the IRC bouncer running on chat.sr.ht) can benefit from this feature and use any available OAuth server for authentication. You can read Simon’s blog post for all the details.

GraphQL

The ID unification rollout is still taking its toll. We are currently building a test setup (or rather, a test setup generator) that is reasonably close to production, yet does not include any user data. This will give us the needed confidence that we can roll out this major change without breaking anything.

In the meantime, the GraphQL API is now deemed mature enough that the OAuth2 dashboard has been made the default in meta.sr.ht.

Infrastructure

SourceHut goes to Europe! Next week, the SourceHut team will meet in person to inaugurate our new data-center. We’ll be installing the first couple of servers, but it will be another while before we will serve production traffic from there. We will use this chance for some experimentation to decide on the infrastructure we want to run SourceHut on in the coming years.

builds.sr.ht

Rocky Linux 9 is available
Ubuntu Impish has been removed following its upstream deprecation
NixOS 21.11 has been removed following its upstream deprecation

How does SourceHut's free software consultancy work?

Tue, 23 Aug 2022 00:00:00 +0000

SourceHut offers a consultancy in which we offer our services as experts in free software to work exclusively on free software development. Through this work we have developed improvements to Wayland, graphics, virtual reality, email, and more, entirely through free software developed in public. Our work can take the form of developing new free software from scratch, contributing improvements to projects that our client depends on, offering what sage advice we can spare, or, often, all of the above.

How does this actually work in practice? Following a discussion in our off-topic IRC channel today, I thought that it would be nice to write about the actual logistics of our operation so that others who want to sell free software consulting services are better equipped to do so, and so that future customers know better what to expect from us.

The process begins with client on-boarding. We don’t do much in the way of marketing — customers mostly come to us. When someone new comes to us with an idea, the first thing we do is seek a basic alignment on the project goals and needs, on our process, and regarding our unique free-software requirements. We evaluate proposals based on a few criteria:

Does this project benefit the free software ecosystem?
Does our team have the necessary experience?
Are the right engineers available to work on it?

Roughly half of our proposals clear this stage. In addition to a reasonably complete project proposal, we need to establish with the client an understanding of our requirements, namely:

All work is published under a free software or open source license
All work is conducted in public — no NDAs

Though we require the results to be released to the public as free software, we’re open to assigning copyright to the client. Not all clients want this, though. If the customer has a clear idea of what’s required, a few emails back and forth are suitable to establish a consensus. If a deeper analysis is needed, we might ask for a temporary contract to cover costs of initial research and planning.

Following this, we’ll draft a contract and seek consensus on the expected pricing and effort required. Our contracts are short and simple, generally covering matters like logistics for payment and the contract period, but also explicitly addressing our requirements regarding subjects like software licenses. Sometimes the client prefers to use their own contract templates, which we’re open to, but generally this requires more negotiation to remove “standard” clauses for things like NDAs.

We bill for each of the engineers we plan to bring on board separately. Our pricing is essentially a function of two competing factors: we target free software and small- to medium-sized businesses, which pushes our prices down, and we offer high-end services from an experienced team, which pushes up. Right now this balance settles on a $250/hour and $150/hour standard base rate for our senior and junior engineers respectively.

The rates are generally only somewhat negotiable — but we may tweak our rates depending on things like our perception of the positive impact a contract might have for free software, or the means of the client. An early-stage open hardware vendor might get a reduced rate for our work developing graphics drivers for them, for example. We will generally expect a deposit from unfamiliar customers.

Once everything is signed and it’s off to the races, the execution of these contracts involves a non-traditional planning style. We push back against suggestions like waterfall or agile, and prefer instead to work like a free software project often does — with broad long-term goals, loosely defined, and narrower short-term goals, more concretely defined. We also set research and planning as an objective in its own right, as this allows us to illuminate unknowns and secure a better idea of the implementation process for each task. We prefer to express progress and planning in terms of the complexity of the work required rather than the expected time to complete it.

Facilitating this mutual understanding of the work involves monthly “status update” emails sent alongside the invoices for each month’s work. These updates cover the tasks recently completed and those planned for the near future, and explain how these tasks relate to the broader objectives.

This is a good opportunity for us to seek consensus where required. Often the client will follow up with questions and clarifications, which a brief email exchange squares up. We’ll also use this as an opportunity to seek clarifications on our end, or to communicate when a decision is required from the client. We occasionally will entertain such exchanges mid-cycle as well, but we plan to anticipate decision points well before they become blockers so that we can work autonomously mid-cycle. We seek to understand and internalize the client’s vision so that we can make the right choices autonomously, reducing the opportunities for our team to be blocked and increasing the client’s confidence in the project.

Though this approach is extraordinarily effective, it relies on a great deal of mutual trust. We aim to secure this by maintaining a reputation for careful, skillful engineering, fierce honesty and transparency, and a history of shipping reliable production software. Insisting on working on free software in public puts weight to these words: our results are independently verifiable.

The model works. Our clients have expressed unanimous satisfaction with our services, and we’ve been able to do a lot of great work for the free software community. Our successes includes VR on Wayland and many improvements (1, 2, 3) to the Monado OpenXR runtime, the Wayland software powering the Steam Deck, and a new webmail and a calendaring and contacts server.

If you’d like to hire us to work on your project, take a look at our consulting page for details. If you’d like to emulate us, I hope that you found this resource helpful. Feel free to ask for advice in #sr.ht.watercooler on Libera Chat.

We have been planning to develop a jobs board for SourceHut users which allows FOSS developers to list themselves for hire called hire.sr.ht. We intend to provide this service as part of your normal SourceHut subscription,¹ and will not take a margin on top of your contracts. You would handle your own contracts and billing, control the communication channels with your clients, have the freedom to take clients off of the platform, and enjoy full ownership over your consulting income.

We have not been able to prioritize this work, but we have developed enough of it that we can work with potential contributors. If you’re interested in helping us bring this service to life, familiarize yourself with the existing code and see what you can do. Check out the GraphQL backend first. Patches to [email protected]">~sircmpwn/[email protected]. Cheers!

Including for users who receive financial aid ↩︎

What's cooking on SourceHut? August 2022

Tue, 16 Aug 2022 00:00:00 +0000

Good morning! 785 new users are joining us on this cool August morning, for a total of 31,600 SourceHut users. Remember to be kind and patient with our new members as they learn how to use the software and work with the communities here. We only have a short update for this month, so let’s get right into it.

GraphQL

Work on GraphQL has slowed this month as we get ready to tackle a major change: ID unification. Each of our services has a separate database with their own user tables synchronized with meta.sr.ht’s authoritative data, each of which has its own set of user IDs. This has worked fine thus far, but for the coming work on GraphQL federation, we will need to unify them. This is a major effort which will require thorough testing and a careful roll-out, and progress on other goals will slow as it’s being done. Stay tuned.

Infrastructure

We have selected a new datacenter location for our European infrastructure roll-out and are working with hardware suppliers to provision an initial set of research servers to install. It will still be several months before we’re ready to completely move our servers into the EU, but things are moving along.

todo.sr.ht

A new ticket resolution, CLOSED, has been added and made the default selection. The project hub has also been updated to support using CLOSES in commit trailers when pushing to git repositories.

builds.sr.ht

OpenBSD 7.1 is now available
OpenBSD 6.9 has been removed following its upstream deprecation
NixOS 21.11 is deprecated and will be removed shortly
Ubuntu Impish is deprecated and will be removed shortly

We are also expecting Rocky Linux 9 to be available soon.

dispatch.sr.ht

Dispatch is being deprecated. See the announcement for the details.

Planned deprecation of dispatch.sr.ht

Mon, 01 Aug 2022 00:00:00 +0000

dispatch.sr.ht is a SourceHut service which provides integrations between SourceHut and third-party services like GitHub and GitLab, and it is scheduled to be deprecated and shut down. Here’s the plan:

2022-08-01: Creation of new dispatch tasks is disabled.

2022-09-01: Dispatch users will be emailed with information about migration.

2022-10-01: Dispatch is shut off.

This blog post will explain the motivations for these changes and offer solutions for users who wish to migrate.

Why is dispatch being deprecated?

There are a few reasons for deprecating dispatch.

For a start, dispatch has not lived up to its initial expectations. Initially envisioned as a kind of IFTTT-for-development, the actual implementation falls far short of that goal and there is no clear path for improvement. There is also not a clear path for upgrading dispatch to support our plans for the GraphQL rollout.

Additionally, dispatch provides integrations with non-free services, which is something that we have a stated policy against. Dispatch pre-dates this policy and was grandfathered in — it was originally written to allow me to rig up builds for sway, which is hosted on GitHub.

Users of dispatch also tend to use more resources than other kinds of users. The GitHub and GitLab workflow encourages frequent pushes to PR/MR branches, which causes many builds to be submitted. builds.sr.ht is the most expensive service SourceHut operates, so this can add up. Among the top ten users ordered by total build-hours consumed, one is a cryptocurrency miner we banned, one is an account operated by a Linux distro that builds all of their packages on SourceHut,¹ and the remainder are GitHub users. In total, GitHub users account for 51% of all build hours executed in the lifetime of builds.sr.ht.

At some point we have to question if this is in our best interests. These projects generally only use SourceHut for builds, and have no intention of migrating anything else. I would not necessarily have a problem with it if GitHub and GitLab were free software, per SourceHut’s mission to “make free software better”. But as it is, it feels like we’re propping up non-free forges and giving users an excuse not to migrate elsewhere.

So, in summary: dispatch is difficult to maintain and has no clear path for addressing its implementation problems, is exceptionally expensive to operate, violates our policies for integration with non-free software, and serves a community which is largely external to SourceHut and dependent on non-free platforms.

Options for migration

All of that said, we don’t want to leave users who are depending on dispatch out in the cold. To this end, we have been working on providing alternative means of achieving similar functionality. Today, dispatch provides four configurable “tasks”:

GitHub commits to builds.sr.ht jobs
GitHub pull requests to builds.sr.ht jobs
GitLab commits to builds.sr.ht jobs
lists.sr.ht patches to GitLab merge requests

Each of these use-cases is now available via a third-party tool. For GitHub integration, see hottub. For GitLab builds, see dalligi. Support for forwarding patches to merge requests is being implemented via hashiru, but this is not complete — it will be before dispatch is shut off.

Affected users who wish to receive a refund for the remainder of the SourceHut subscription may [email protected]">contact support to receive one.

I hope that these tools will be effective for your project, and if you have any questions about migration we would be happy to assist you on IRC or on the sr.ht-discuss mailing list. However, I have a humble request for those who choose to migrate to these tools: if you are using a non-free platform like GitHub or GitLab (.com/EE) for your project’s primary hosting, please make a plan to migrate from builds.sr.ht. GitHub provides GitHub Actions and GitLab has its own CI system as well. If you’re not comfortable moving to SourceHut or another free-software platform, that’s your choice and we respect it — but it is a bit burdensome for us to accommodate you. Thank you for your understanding.

A use-case we are proud to support. ↩︎

What's cooking on SourceHut? July 2022

Mon, 18 Jul 2022 00:00:00 +0000

Good morning, SourceHut! Today we have 30,815 users, 1,203 of which are joining us for the first time this month — a big increase! To all of our new users: welcome! To all of our more tenured users: be sure to offer our new friends a warm welcome and do your part to help them learn how to use SourceHut.

GraphQL

Check this out:

meta.sr.ht: ✓ read ✓ write ✓ webhooks
git.sr.ht: ✓ read ✓ write ✓ webhooks
hg.sr.ht: ✓ read ✓ write ✓ webhooks
todo.sr.ht: ✓ read ✓ write ✓ webhooks
builds.sr.ht: ✓ read ✓ write ✓ webhooks
lists.sr.ht: ✓ read ✓ write ✓ webhooks
hub.sr.ht: ✗ read ✗ write ✗ webhooks
paste.sr.ht: ✓ read ✓ write ✓ webhooks
pages.sr.ht: ✓ read ✓ write ✓ webhooks

Adnan has been hard at work on GraphQL this month, and we’re almost done. The last remaining API to be implemented is hub.sr.ht, but before we implement it, we’ll be experimenting with GraphQL federation. This will allow us to access every GraphQL API through a single endpoint, seamlessly querying resources like git repositories and mailing lists in a single request. Since hub ties all of the services together, this will be a very good thing to have before we proceed with its API! This will also bring many improvements we’ve had planned for a long time, but will also require many serious internal design changes to our software, so it will require some effort to thoroughly test & validate them before rolling it out. Federation will also open some doors for long-requested features, such as linking resources like bug trackers back to the project they belong to.

hut

hut is a CLI tool for interacting with SourceHut’s GraphQL APIs, and it usually doesn’t make it into these updates because it’s not considered a core service. However, I wanted to mention it today because I added support for exporting your account data with hut this month. The export uses standard formats which are as portable as possible, such as bare clones of your git repositories or mbox dumps of your mailing lists, which you can re-import into any git provider or mailing list software. Following this, I will be adding a similar “import” functionality to bring your data back into any other SourceHut instance. You can dump your account data locally, or queue up a build job to do it on our servers.

Data autonomy is an important goal of the beta, so this also moves us forward towards our long-term goals. It is also one step towards unblocking features like account removal or renaming. Exciting stuff!

meta.sr.ht

No major updates this month, but I have been working on some upcoming features to tease: an overhaul to the billing system.

Some of the improvements which will be on offer include:

New payment methods, such as iDEAL and SEPA direct deposit
The ability to enable or disable automatic renewal
More currencies to pay with
Better payment method management & invoicing

This is a very complex overhaul, so it will take some time to be completed. I’m not sure when you can expect it to be released, but I’m looking forward to shipping it to you!

builds.sr.ht

Alpine 3.16 is now available
NixOS 22.05 is now available
FreeBSD 13.1 is now available
Fedora 34 has been removed following its upstream deprecation
NixOS 21.05 has been removed following its upstream deprecation

SourceHut is committed to making IRC better

Wed, 06 Jul 2022 00:00:00 +0000

Internet Relay Chat (IRC) is a wonderful protocol with a 34-year history of helping free software, there at every step alongside the rise of the internet. Many real-time chat empires have risen and fallen during its tenure, some of them leaving behind lessons IRC might learn from. But even in its original form, IRC is a simple and beautiful protocol on top of which many independent, federated networks have been built, from simple line-oriented plain-text messages (mostly) adhering to open standards, and implemented with free software.

IRC has held a special place in my heart and in the hearts of my collaborators at SourceHut, both within the company and in the broader community. We rely on it every day, professionally and socially, and we want to help it be the best it can be. We want this because we love IRC for what it is: simple. There are many other solutions we could invest in, some of which are doing great things for free software (such as Matrix), and some of which are not (such as Discord). But, IRC is by far the simplest and most open, and these essential traits are both compatible with our culture and something we wish to preserve.

What can we do to make IRC better while honoring what makes it so good?

Our work focuses on making IRC easier to use and more accessible to a broader audience of free software contributors. While I might personally be satisfied with Weechat in a terminal on my other monitor, many new programmers have different wants. And while I was happy running my IRC bouncer for many years, I can’t exactly suggest that it is a good solution for IRC’s problems.¹

The most obvious contribution to IRC from SourceHut is chat.sr.ht, our hosted IRC bouncer which provides a “just works” IRC bouncer for SourceHut users. A pleasant webchat (gamja, GNU AGPL 3.0) offers a jump-in-and-go experience, and our bouncer backend (soju, also AGPL) provides a scrollback, automatic log keeping, multi-device synchronization, and many other useful features that close most of the usability gap for IRC. The maintainer of these projects, Simon Ser, is also developing a new Android app for IRC (goguma, (also AGPL)). Importantly, chat.sr.ht is not a new IRC network: we just facilitate access to independently operated and governed networks like Libera Chat and OFTC.

chat.sr.ht is the face of SourceHut’s IRC work, but there’s also a lot of work you don’t see. Simon is working with the IRCv3 community to improve IRC for everyone. Simon and others working on our stack have forwarded a half-dozen new IRC specifications for review and standardization by the community, including support for push notifications, bouncer auto-configuration, and multi-device synchronization improvements. We also participated in the process of getting many other extensions standardized, such as chat history, account registration, and web socket support, and have offered many improvements to the documentation at ircdocs.horse.

Our approach is to make conservative, obvious improvements to IRC that close the usability gap without sacrificing the any of the things which we love about IRC. We believe in honoring and respecting our peers in the community, focusing on making the ecosystem stronger instead of seeking to make our foothold in it stronger. The fall of Freenode reminds us all of the importance of a healthy community of equal peers working to improve standards in the interests of us all, rather than in the interests of a small few. We’re proud to do our part. #sr.ht was registered on Libera Chat only minutes after it opened and we were one of the first projects to make the move.

I’ve been using IRC for most of my life. It has played a crucial role in my social and professional life, playing its part in the success of many of my projects and in establishing many long-term friendships and professional relationships. Come add your voice with us in #sr.ht (on-topic) or in #sr.ht.watercooler (off-topic), both on Libera Chat. See you there :)

I might acknowledge, however, that many traits of the ephemeral discussions that stem from IRC’s constraints are worth keeping even in the absence of these constraints. The lack of any expectation to scroll up and read the discussions you missed, and the cultural pressure to move important information out of chat and into more permanent mediums, like mailing lists or tickets, are two positive things that IRC’s ephemeral nature encourages. As we move into a better world for IRC, let’s be mindful of these things. ↩︎

What's cooking on SourceHut? June 2022

Wed, 15 Jun 2022 00:00:00 +0000

Hello everyone! Let’s get straight into the news today. Our user count today is 29,612 users, of which 576 have joined since the last update. Remember to be patient with these new users as they learn the ropes. Welcome!

GraphQL

Adnan continues to ship GraphQL services for SourceHut, this month shipping webhooks for lists.sr.ht, and preparing to-be-reviewed patches adding webhooks to builds.sr.ht and paste.sr.ht. The latest GQL support matrix looks like this:

meta.sr.ht: ✓ read ✓ write ✓ webhooks
git.sr.ht: ✓ read ✓ write ✓ webhooks
hg.sr.ht: ✓ read ✓ write ✓ webhooks
todo.sr.ht: ✓ read ✓ write ✓ webhooks
builds.sr.ht: ✓ read ✓ write ✗ webhooks†
lists.sr.ht: ✓ read ✓ write ✓ webhooks
hub.sr.ht: ✗ read ✗ write ✗ webhooks
paste.sr.ht: ✓ read ✓ write ✗ webhooks†
pages.sr.ht: ✓ read ✓ write ✗ webhooks†

† To be added soon!

The last major effort for the initial roll-out of GraphQL support will be hub.sr.ht. Following this, we will be experimenting with GraphQL federation, and starting to rewrite our cross-service communication to use GraphQL. We’ll then make plans for sunsetting REST.

lists.sr.ht

We are preparing some changes to mailing list visibility to bring it inline with the design of other services. Permissions will be consolidated into a single default set, and a “visibility” parameter will be added which matches the behavior of other services. Sane defaults for these new configurations will be selected based on your current configuration — you will have received an email if your mailing list configuration may require manual attention during the migration.

builds.sr.ht

Image updates:

freebsd/13.1 is now available

NixOS 22.05 will be available soon, and Fedora 34 will be removed soon.

What's cooking on SourceHut? May 2022

Mon, 16 May 2022 00:00:00 +0000

Hello everyone! We’re back at it for another month of news in the SourceHut sphere. Of our now 29,036 users, 630 are new this month: please offer them a warm welcome, and your patience, as they learn about the new platform.

todo.sr.ht

Comprehensive GraphQL-native webhooks shipped for todo.sr.ht this month.

Here’s the latest GraphQL support break-down:

meta.sr.ht: ✓ read ✓ write ✓ webhooks
git.sr.ht: ✓ read ✓ write ✓ webhooks
hg.sr.ht: ✓ read ✓ write ✓ webhooks
todo.sr.ht: ✓ read ✓ write ✓ webhooks
builds.sr.ht: ✓ read ✓ write ✗ webhooks
lists.sr.ht: ✓ read ✓ write ✗ webhooks
hub.sr.ht: ✗ read ✗ write ✗ webhooks
paste.sr.ht: ✓ read ✓ write ✗ webhooks
pages.sr.ht: ✓ read ✓ write ✗ webhooks

We’re getting there! The remaining webhooks should be fairly straightforward — lists will be the most complex of these — and updating hub.sr.ht will be the last major project before we complete the initial GraphQL implementation. Following this we will be experimenting with GraphQL federation as well, and once we have an answer regarding federation, the GraphQL milestone will be complete, and we’ll have taken the largest step towards the beta.

Coming improvements to hut

The hut command-line tool provides programmatic access to the SourceHut GraphQL APIs. I have been working on support for a comprehensive data export via this tool recently, which will be followed by an equivalent data import tool. This will allow you to take your account data from sr.ht to another instance, or to an instance with compatible software — most of the exported data is in standard formats like bare git repos and mbox archives. This is also another step towards the data ownership goal of the beta, and will be followed soon by self-service account deletion and renaming — which also require the completion of the GraphQL APIs that are progressing apace.

lists.sr.ht

The new go-emailthreads library was integrated into lists.sr.ht’s GraphQL API, which is the next step for improving the heuristics used on the code review screen.

meta.sr.ht

Work is underway, some of which landed this month, to improve PGP support on SourceHut, expanding the suite of supported ciphers and making it more reliable. Additional changes are expected in the foreseeable future to handle PGP key expiration, notifying users when their key needs to be cycled.

builds.sr.ht

Image updates:

ubuntu/kinetic is now available
ubuntu/jammy is now available
fedora/36 is now available

pages.sr.ht

Users may now configure custom Cache-Control headers to tune their site performance.

What's cooking on SourceHut? April 2022

Fri, 15 Apr 2022 00:00:00 +0000

Hello! Another month of improvements rolls on by, and I’m here to tell you all about them. In terms of site growth, we saw 510 new users join us, bringing our total to 28,406. As always, please be patient with our new peers as you help them learn about the site.

General news

This month we released our annual financial report. In summary, SourceHut is financially healthy. Feel free to browse the full report to get all of the details.

We have been making excellent progress on the GraphQL initiative thanks to Adnan’s help, and I anticipate that we will be able to put a bow on this sometime in the next couple of months.

hg.sr.ht

Work is underway on integrating improvements to the hg.sr.ht GraphQL API. The API is now writable, supporting all essential resource mutations, and GQL-native webhooks should be landing shortly.

builds.sr.ht

Setting the oauth keys in your build manifest will now pre-configure your build environment for use with the hut tool.

Image updates:

Plan 9 support has been fixed and improved thanks to Benjamin Riefenstahl
FreeBSD 12.x has been updated to FreeBSD 12.3
Alpine 3.11 was removed following its upstream deprecation

lists.sr.ht

We have integrated the new go-emailthreads library with lists.sr.ht, which provides much better heuristics for parsing mail threads into trees. We will soon update the lists.sr.ht frontend to take advantage of this.

git.sr.ht

The branch name is now included in the build tags when submitting pushes to builds.sr.ht. You may want to update your badge URLs if you use them.

pages.sr.ht

It is not available yet, but in the next few days we expect to merge support for configuring per-file cache behavior for your site.

SourceHut's 2021 Financial report

Fri, 08 Apr 2022 00:00:00 +0000

In summary, SourceHut is financially healthy, profitable, and growing. We enjoyed substantial YoY growth between 2020 and 2021. SourceHut hired its third full-time software engineer, and with the help of the NGI0 Discovery fund we had another developer join us to help with the GraphQL development work. We have a sustainable business model and sufficient capital with which to execute all of our planned expansions, such as a new European datacenter presence.

We had previously published public financial reports quarterly, but preparing such reports is a time-consuming process, particularly given the growing complexity of our finances. Starting with this report, and going forward, we will publish reports annually. The exact format of these reports and the ability to compare numbers across years will be somewhat inconsistent as we gradually move our business operations into Europe.

Disclaimer: this report is a summarized approximation of our financials, and is not used for tax purposes.

2021 revenue sources

SourceHut has two main revenue streams:

Paid subscriptions for the forge platform, which are generally consistent
Contracts via our consulting arm, which are generally more inconsistent

We use the forge platform revenue as the basis for our planning and treat consulting revenue as capital to fund new projects.

Platform revenue

We issued 10,484 invoices to SourceHut subscribers in 2021. Subscriptions are available at three price points: $2, $5, and $10; billed monthly, or annually for a 2 month discount. Invoices paid in 2021 break down as follows:

5,153   $ 2   (paid monthly)
2,098   $ 5   (paid monthly)
  818   $10   (paid monthly)
1,672   $ 2   (paid yearly)
  496   $ 5   (paid yearly)
  184   $10   (paid yearly)

The total revenue from platform subscriptions in 2021 was $105,777.08, which, after credit card processing fees, was $98,741.95. This is an increase of 35% over 2020. In 2021, a total of 7,184 new users signed up for an account. Of these users, 956 signed up for a paid account. Of these 956, 45 are currently delinquent. This is a conversion rate of 12.6%, slightly improved from 2020’s 11.7%. Among all users registered in all years (a total of 28,262 at the time of writing), 11.35% (or 3,209 users) have paid accounts.

This modest increase may be at least somewhat explained by the introduction of paid-only services such as builds.sr.ht and chat.sr.ht in 2021. Note that, following the completion of the alpha, all users who own resources (such as repositories, bug trackers, mailing lists, etc) will be expected to move to a paid account. Contributors will not be expected to pay, and users who wish to pay but cannot will be offered financial aid.

On the subject of financial aid, at the time of writing, 59 users have received financial aid. 100% of users who requested financial aid received it. Reasons provided by users include, in descending order of frequency:

Inability to pay due to financial constraints
Lack of support for their preferred payment method
Political sanctions due to the situation in Ukraine and Russia

Consulting revenue

SourceHut accepts consulting projects at the discretion of our full-time engineers, part-time. These contracts are executed in public and are required to only produce free and open source software.

In total, SourceHut received $285,739 in revenue from consulting engagements in 2021.

Summary of revenue

Our total revenue from all sources for 2021, before taxes, expenses, and fees, was $391,516.

Cash on hand

At the close of the year, we had $102,402 in cash on hand.

2021 expenses

In 2021 we had the following expenses, excluding payroll:

$8,277    Colocation costs (space, power, network)
$7,035    Credit card processing fees
$2,125    Sponsorships
$1,110    Domain names
$  995    Miscellaneous
$  570    Insurance

We cancelled our GitHub sponsorships partway through the year, and began sponsoring the OpenStreetMap project as a Bronze sponsor.

From 2022 forward, we use a standard approach to compensation. Each full-time SourceHut engineer earns $2,500 per month for self-directed work furthering our mission to “make free software better” at their own discretion; working on projects like the forge but also maintaining independently directed projects and contributing to the broader ecosystem. Additionally, SourceHut engineers can elect to, at their discretion, participate part-time (generally no more than two days per week) on consulting projects. Furthermore, my compensation as CEO was dealt with separately — previously I made a bit less than everyone else, and going forward I will also be paid the standard compensation.

In 2021, our compensation scheme was not standardized, and varied throughout the year. We also had three special personnel arrangements in 2021: one paid intern, one short-term, part-time contractor to help with a specific internal project, and one medium-term, part-time contractor to help with a specific consulting engagement. For privacy reasons, and due to the complexity of the situation, we will report our compensation figures in aggregate. In 2022, we will break this figure down further for transparency.

Not including my compensation as CEO, we paid a total of $131,597 between all forms of staff compensation.

Thus, our total expenses for 2021 were $151,459.

2021 financial summary

Revenue: $391,516
Expenses: -$151,459
Taxes: -$50,000 (approx)
Profit: $189,000 (approx)

Expectations for 2022

We have three full-time staff paid under the standard compensation model, and about $700/mo in recurring expenses. Currently we make about $9200/mo from the platform, putting our monthly profit at about $1000, without factoring in consulting revenue. Thus, the platform provides for a sustainable business independently of our other lines of business. The breakdown of active subscriptions at the time of writing is as follows:

$2    monthly   518 users
$5    monthly   210 users
$10   monthly    80 users
$2    yearly  1,710 users
$5    yearly    500 users
$10   yearly    190 users

We have three ongoing consulting contracts which we expect to continue through most of the year. We have already migrated or established some of these contracts with the new Dutch business entity, and we intend to migrate more of our business over the Atlantic throughout 2022.

At the time of writing, SourceHut has $62,608 and €30,590 in cash on hand.

The only major investment we have planned is the deployment of a European datacenter presence, which is likely to begin in late 2022, but may be delayed to early 2023.

Conclusion

What's cooking on SourceHut? March 2022

Tue, 15 Mar 2022 00:00:00 +0000

Today we celebrate another 487 new members who’ve joined since our last update, bringing our community to 27,896 users in total. As always, I’m relying on you to be patient with them as they learn the ropes and to make sure they feel welcome here. This month saw many improvements, but much of them were internal changes or bug fixes, so this update will be somewhat short.

The Mumble meeting for this month is again cancelled, as the call for agenda items returned no answers. Going forward, we will set up meetings only for specific discussions, such as future discussions on billing changes for the alpha to beta transition.

General News

SourceHut for Alpine Linux 3.15 is now available, and the 3.14 repository is no longer maintained. Self hosters are encouraged to upgrade at their earliest convenience.

git.sr.ht

It is now possible to create new repositories by cloning them from an existing git URL.

builds.sr.ht

Alpine 3.11 has been removed following its deprecation upstream. Issues with 9front builds have also been corrected, and further Plan 9 improvements are underway.

What's cooking on SourceHut? February 2022

Tue, 15 Feb 2022 00:00:00 +0000

Hello again! Today is a rare sunny and beautiful day in Amsterdam, and I am spending it inside writing this post with the blinds down, as is befitting of a proper hacker. I’m pleased to welcome 529 new users to our service this month, who together bring our community to 27,409 members. I hope our new members are enjoying the service, and that our existing community does its part to make them feel welcome.

This month’s Mumble meeting will not take place, as the call for agenda items went unanswered. Please add topics to the March agenda if you wish to address any matters in next month’s meeting.

New staff v2

Last month, I introduced Adnan Maolood, who, thanks to a generous sponsorship from NLNet, will be working with us on GraphQL for a while. This month, I am pleased to introduce Conrad Hoffmann, who joins us as the latest full-time free software engineer working on SourceHut. Check out the blog post to meet him!

todo.sr.ht

After many delays, the todo.sr.ht writable GraphQL API has finally shipped! Thank you for your patience. The documentation is available here and you can play with it directly on the todo GraphQL playground.

git.sr.ht

Adnan’s first batch of GraphQL improvements landed yesterday as we shipped support for GraphQL-native webhooks in git.sr.ht alongside a number of other improvements. Nice work, Adnan! Adnan will continue with more GraphQL improvements for other services in the next few weeks.

pages.sr.ht

One of Conrad’s first contributions to SourceHut introduced Content-Encoding support and compression which, together with improvements from Umar Getagazov, should make the performance of SourceHut pages noticably better. Dhruvin Gandhi also added support for custom 404 pages. Great work, everyone!

builds.sr.ht

Image updates:

ubuntu/hiruste has been removed following its upstream deprecation
Fedora images now include dnf-plugins-core by default

Improvements are also underway for Guix and Plan 9 support. Thanks to Haowen Liu, Ryan Gonzalez, Dhruvin Gandhi, and Benjamin Riefenstahl for their work in these areas.

names.sr.ht

More community members have been expressing interest in working on names.sr.ht as a grassroots, community-led effort. Conrad has set up a project to organize these efforts, and I have posted a summary of my thoughts to offer some direction. Please get involved if this interests you!

Other news

A quick “thanks!” is owed to Ignas Kiela for his continued work on improving SourceHut’s monitoring infrastructure this month. Thanks to his work on our web services, we have sane data available for response codes like 500 errors and alarms configured to alert us when they occur in volume.

Another quick shout-out to the emacs community, who, in the process of evaluating SourceHut for their needs, sent a dozen or so patches improving lists.sr.ht, particularly with respect to importing third-party mbox archives. Many thanks!

Welcoming Conrad Hoffmann, the newest SourceHut developer

Wed, 02 Feb 2022 00:00:00 +0000

I’m pleased to introduce the SourceHut community to Conrad Hoffmann, aka bitfehler, who joins us this month as SourceHut’s third full-time FOSS developer. Like the rest of our full-time staff, Conrad’s role is to do self-directed work to “make FOSS better” per the mission of SourceHut, and in addition to working on the forge, Conrad will be working with many FOSS projects in the community, including projects on SourceHut and in the FOSS ecosystem as a whole.

Conrad is a Berliner whose introduction to FOSS came from his experience with SUSE Linux during his mandatory military service. He fell in love and started contributing to projects like KDE and heading down a career path which brought him deeper into Linux and the FOSS ecosystem, learning to use (and enjoy) the C programming language in particular. Conrad’s last job was at SoundCloud, where he contributed to HAProxy and bore witness to the birth of Prometheus, which eventually became the basis for SourceHut’s own monitoring at metrics.sr.ht.

In his role at SourceHut, Conrad will keep maintaining his Prometheus exporters and will branch out into a number of new interest areas. His experience with ops and infrastructure is something we’re hoping to utilize when setting up the European datacenter, and he’s also interested in joining the community efforts towards building a SourceHut DNS service. We can also expect some interesting things from Conrad in the Pine64 and OpenStreetMap communities, both of which have received funding and support from SourceHut in the past. He also intends to make a difference at a local level as his newborn daughter grows up in a society run by commercial proprietary software by advocating for the adoption of FOSS in schools and other places in his community.

I am very pleased to have Conrad on board at SourceHut, and I’m looking forward to working with him. I hope that you, members of the SourceHut community, will offer him your welcome as well as he begins to work on projects on SourceHut and in the broader FOSS ecosystem. He invites you to reach out and get to know him:

I am bitfehler on libera.chat, you can find me in the #sr.ht.watercooler channel or message me if you feel like chatting!

Please say hello! Welcome to SourceHut, Conrad!

What's cooking on SourceHut? January 2022

Mon, 17 Jan 2022 00:00:00 +0000

Hello and happy new year! After a bit of well-deserved rest during the holidays, our staff (and many of our contributors) have spun our work streams back up and development continues. Our userbase has grown this month by another 634 users, bringing the total to 26,880. As usual, I’m depending on you to make our new members feel welcome and to offer them your patience as they learn how we do things here.

This month’s public Mumble meeting will take place tomorrow, January 18th (later than originally announced), at 10:00 UTC in the usual place, voice.mnus.de in the sourcehut room. I called for suggestions for an agenda on sr.ht-discuss — now would be an opportune time to add your own suggestions if you haven’t already.

New staff

I announced last week that, thanks to a generous grant from NLNet, Adnan Maolood will be joining us for a while this year to help with the GraphQL implementation. He’s been hard at work on git.sr.ht’s GraphQL-native webhooks already, and I’m looking forward to having his help. Welcome! We are also bringing on a new full-time employee (at SourceHut’s expense) from February 1st, who will be introduced in a coming blog post.

hut: A SourceHut CLI tool

Simon Ser has started developing a command line tool called hut for using the GraphQL API in your terminal or in shell scripts. Check it out!

todo.sr.ht

I had hoped to ship todo.sr.ht’s GraphQL API this week, but its inherent complexity combined with time off during the holidays has pushed the schedule back a bit more once again. All that remains is the completion of the legacy webhooks for the writable API, plus acceptance testing and a security review, then we can ship. This should land sometime this week, if everything goes according to plan.

git.sr.ht

Adnan has been working on GraphQL-native webhooks for git.sr.ht, which is going well. This will also hopefully ship before the next “what’s cooking”.

lists.sr.ht

A number of long-standing bugs when importing mbox files to populate mailing lists from an external archive have been addressed thanks to contributions from Martin Vahlensieck. Thanks Martin! A number of small fixes and improvements to the GraphQL schema were also introduced this month.

meta.sr.ht

Some discrepancies between our OAuth 2.0 implementation and the RFC were discovered and corrected. Thanks to ~omz13 for the tip!

SourceHut selected for NLNet NGI Zero funding

Mon, 10 Jan 2022 00:00:00 +0000

I’m pleased to share that SourceHut has been selected by NLNet to receive funding as part of the NGI Zero: Search & Discovery fund for the purpose of completing API 2.0, our GraphQL API development. NLNet is providing monetary support to SourceHut contributor Adnan Maolood, aka ~adnano, who has agreed to work full-time to help with our API 2.0 efforts over the course of this year.

The NLNet Foundation is a Dutch organization which provides funding to various projects which promote open and secure digital infrastructure, and has provided funding for hundreds of free and open-source projects, many of which you rely on every day, and some of which rely on SourceHut! Prior recipients of NLNet funding include projects like Tor, WireGuard, postmarketOS, Jitsi, and more, and I’m proud to see SourceHut join them.

API 2.0 is one of the major outstanding tasks between us and the completion of the SourceHut beta. We can look forward to much better access to SourceHut’s internals, a more robust implementation, and a simpler deployment system for third-party SourceHut installations. We’ll be building upon this work to develop alternative frontends, such as the hut CLI tool and a new Gemini-based frontend. This also paves the way for many other features which we’re cooking up, including user groups/organizations, better data autonomy tools for import & export and account deletion & renaming, and future plans like names.sr.ht’s DNS hosting and domain registration services. Exciting stuff!

I’m excited for this initative, and I’m glad to have Adnan on board to help. Big thanks to NLNet for agreeing to sponsor his work! Let’s get this beta done.

How does SourceHut's FOSS business model work?

Sun, 09 Jan 2022 00:00:00 +0000

SourceHut makes a profit, but we are not motivated by profit. I founded SourceHut with the explicit goal of making my free software work sustainable full-time. As SourceHut grows, we aim to expand and generalize this approach. We want to make free software better, and the business is a tool we use to facilitate this.

We have two lines of business which provide us with revenue: the SourceHut software-as-a-service (SaaS) platform, and free software consulting. You can run the SourceHut platform yourself (instructions are available here), but it’s expensive and time consuming to maintain the platform, so most people pay us to do it for them via monthly or annual subscription fees. This model sets up incentives which favor users over investors,¹ which gives us an honest focus on user-motivated design (as opposed to profit-motivated design) and ensures that we are dis-incentivized to use unethical means to make our margin, such as selling your personal information. Your payments also provide for the development and maintenance of the service, which we try to honor by ensuring reliable uptime & performance, comprehensive backups, and so on, via a robust operational strategy which is not economically practical for small-scale deployments.

Our second line of business is free software consulting. Free software has eaten the world: it is ubiquitous in all software products. This has created a market for free software experts who have a broad understanding of the free software ecosystem and experience working upstream. We provide consulting services to this market, in which we are paid to write or contribute to free software. Upstream projects we’ve worked on under this model includes software like Linux, Mesa, X.org, Monado, and many others, for clients like Valve Software, Status.im, and Migadu. We have also developed greenfield software for these clients, such as alps, wxrc, and gamescope, as well as improvements to software which our team maintains independently, such as wlroots or Wayland.

We use the revenue from these lines of business to fund free software projects through the work of our full-time employees, of which there are two: myself and Simon Ser. We work at our discretion on free software projects, some of which are not obviously monetizable, for the purpose of making the free software ecosystem stronger as a whole. Sometimes this work ultimately provides revenue — for example, Simon’s soju and gamja projects were developed independently and eventually were incorporated into the paid chat.sr.ht product. We also developed wlroots together (starting before SourceHut was founded), which eventually was the basis of much of our consulting revenue. However, it’s more common that these projects don’t produce a return.

Many of our projects have no profit motive and are probably unmonetizable. Our mission is to make free software better, profitable or not. Projects like visurf or gemini, our upcoming systems programming language, upstream work in Alpine Linux or freedesktop, or sponsorship of projects like OpenStreetMap — none of these have obvious returns for us. We work on them because we believe that they are important. If we were pressed to justify this in business terms, we might say that we are growing the free software market as a whole through this work, or that they are good for marketing (both of which are true), but fundamentally it’s because we believe in free software.

We will be hiring a third full-time engineer in February, who we will introduce to you when the time comes. Following our tradition of transparency, you’re welcome to read the onboarding manual we have prepared for them, which goes into more detail on what it’s like to work for SourceHut. We also occasionally bring on temporary workers to work within specific goals. For example, thanks to an NLNet sponsorship, we will have Adnan Maolood on for the year to help with the GraphQL work. We have also hired contractors for things like consulting work, or for help with some of our independent FOSS projects or upstream needs.

This is how we do business. Does it work?

In short: yes. We will answer this question in great detail in our coming annual financial report,² but I can summarize it for you now. The books for 2021 are not quite closed yet, but it looks like we’re going to end up in the vicinity of half a million dollars in gross revenue, which I think is pretty damn good for a bootstrapped company in its third year, especially one which gives away 100% of the software it builds free of charge.

There are many ways to finance free software. This is our way, and it works. We fund improvements to hundreds of free software and free culture projects, many which would be otherwise impossible to do profitably or even sustainably. All of our profit has been or will be invested strictly in free software and free culture projects, most of which are not even maintained by us. There’s no “open core” going on here: every line of code we write is genuine free-as-in-freedom software.

I’m very proud of what we’ve built.

We built it with your help. Many SourceHut community members have taken on serious roles in our projects, contributing heaps of code to every project, even maintaining some subsystems, or occasionally entire sourcehut services like hg.sr.ht, in their spare time. We have had community contributions to thank for many improvements in functionality, reliability, documentation, and more. Even the simple choice to host your projects with us is an investment in our community, and one which can be difficult to make against the temptations of more popular nonfree or semi-nonfree platforms in the market like GitHub or GitLab. I am grateful to everyone who has had a hand in this success. Thank you.

Making free software better means making it better for everyone, not just us. We want to make free software profitable for you, too. One of our upcoming products is “hire.sr.ht”, which will provide an index of independent free software contractors to solicit clients.

(click to enlarge)

I hope to see this platform used for a variety of free software work. It could be weekend hackers who maintain some projects in their spare time and get paid by users to implement the features they want, or weekend hackers who get paid by users to send patches to other projects. Or, you could be working on a FOSS project of your own and hire someone with a relevant skillset to help you get it started. Maybe we’ll see full-time free software consultants listed here, taking on larger long-term engagements to work on free software for businesses, or even free software consultancy co-ops or businesses like SourceHut itself.

We plan on offering this service as part of your normal sourcehut subscription fee,³ without taking any commissions or trying to control the line of communication between you and your clients. You will write your own contracts, negotiate with clients directly, handle billing and payments without our involvement, and retain complete and independent ownership over your work and client relationships. We will support you in figuring these things out, and I hope to see an organic community of mutual support form between contractors (and clients) to provide these things for each other, without tying your livelihood up under SourceHut’s control.

Like the rest of SourceHut, this is an open source project, and we’re developing it with you as much as we are developing it for you. We could use your help in building this. If you’re interested in working on this project, please [email protected]">reach out. There’s also a grassroots community project to build out domain registration and DNS hosting services on SourceHut which could use your help. If you have any other ideas, I want to hear them. If you have time, passion, and ideas, we’ve got servers and a community. Let’s talk.

Which we don’t have. Investors, that is. ↩︎
If you’re wondering where these went, this is the answer. They’re too much work to do quarterly, especially as our business situation was complicated by the recent introduction of a Dutch business entity and our growing consulting revenue. Sorry that these stopped for a while! I will write up the next one soon. ↩︎
Including for users who need financial aid and rely on our subsidized or free subscriptions. ↩︎

What's cooking on SourceHut? December 2021

Wed, 15 Dec 2021 00:00:00 +0000

Hello and happy holidays! I hope that our 26,246 member userbase is enjoying the cold weather (or the warm weather, in the south). Remember to remind the people you work with on SourceHut, and others besides, that you care about them. We have many new users this month, and I hope you will extend them the same kindness and patience which you remember from your own first steps on SourceHut.

This month’s public Mumble meeting will take place tomorrow at 10:00 UTC in the usual place, voice.mnus.de in the sourcehut room. This time, I called for suggestions for an agenda on sr.ht-discuss — now would be an opportune time to add your own suggestions if you haven’t already.

chat.sr.ht

Towards the beginning of this month, we announced chat.sr.ht, a hosted IRC bouncer for SourceHut users. Please give it a shot if you haven’t yet. Simon has been hard at work improving it further still since the launch, and mainly making improvements to usability and fixing bugs. The next major development will be the introduction of the REGISTER extension, which we will follow-up on by writing patches for the rest of the IRC ecosystem to help nudge adoption along, alongside our other IRC standardization work.

todo.sr.ht

Thanks to Robin Jarry’s patches, you can now reference and close tickets on todo.sr.ht from your git.sr.ht commits using standard git trailers. Check out the docs for details. Robin was very patient with my demands for this patchset — thanks Robin!

Ignas Kiela has also taken it upon themselves to spend more time improving SourceHut monitoring and performance, landing patches throughout SourceHut this month to improve our metrics and using the information derived from these improved metrics to guide his work in todo.sr.ht patches. Thanks, Ignas, keep it up!

The planned writable GraphQL API release for todo.sr.ht is taking longer than anticipated, and will require a couple more weeks still. todo.sr.ht is one of the most complex services on SourceHut, and getting every little interaction right is a fair bit of work. It’s not helped by the fact that we’re slowing down a bit to rest at the end of the year — sorry for the wait.

git.sr.ht

You can now upload more than one artifact to a git tag at the same time. Adnan Maolood has also helped to refine the vcs-autodiscovery specification, alongside ~ancarda, and implemented it both in git.sr.ht and godocs.io, to provide a standard means of accessing forge-related metadata for git repositories.

builds.sr.ht

Image updates:

nixos/21.11 is now available, and 20.09 has been removed

An interview with 100 rabbits

Wed, 08 Dec 2021 00:00:00 +0000

Hundred Rabbits is an artistic duo hacking their way around the Pacific on their sailboat. I invited them to sit down for an interview to talk about about their lifestyle, art, philosophy, and their SourceHut projects. This interview was conducted live in the #sr.ht.watercooler IRC channel on Libera Chat.

Drew: Hi Devine! Happy birthday, Rek!

Rekka: Haha, thanks!

Devine waves

Drew: Would you two introduce yourselves?

Rekka: We are two artists who live and work on a sailboat named Pino. We traveled around the Pacific Ocean for 5 years, learning about technological resilience. I am an illustrator, but I also write, and Devine is a programmer that also makes music.

Drew: Why live at sea?

Devine: We don’t live at sea: we live on the water, near the coasts, and sometimes we traverse large spans of ocean. We chose to live on a boat so we could go where the wind would take us. We quickly look for shelter when we can and try to limit the time we spend at sea to a minimum.

Drew: That makes sense. The sea intimidates me, to be honest.

Devine: Us, too.

Drew: Have you found your boating lifestyle to be a good platform for the art projects you build?

Rekka: Yes, definitely. We find that we work really well with constraints.

Devine: A lot of our projects are advised by the extreme position in which we find ourselves, away from internet connectivity and one-day delivery networks.

Drew: Projects like?

Rekka: We’re always working on our wiki, it’s the project we update the most. We document everything we learn like food preservation, boat repairs, places we’ve been, etc.

Drew: I like that community-building mindset very much, Rek.

Devine: The energy we collect from the sun dictates the number of cycles our software can use to run, and how much time we can dedicate on working on the computer to build them. This has ruled out a lot of modern technologies, it’s also what brought us here, to be using SourceHut in the first place.

Drew: I imagine that the energy constraints are also why many of your projects involve stepping away from the computer, like food preservation and log-keeping.

Devine: If we can use less technology to solve any one task, we will. Our ideal amount of technologies is as little technology as possible.

Drew: You’ve led me to another question I wanted to ask: how does SourceHut fit into your workflow?

Devine: Our work is done almost entirely offline, but when we do have connectivity, we’re looking for building mirrors of our work for redundancy. As much as people like to throw the words “why don’t you self-host” at us, having someone making sure that our repos are available while we’re days or weeks away from shore is what keeps our projects alive and gives us peace of mind.

Drew: I cannot imagine a boat in the middle of the Pacific making for a good place to host a server. Did you try any other platforms before settling on SourceHut?

Rekka: We were on GitHub for a few years.

Drew: How does it compare?

Devine: It’s hard to put into words, there’s a general trend in software right now to compete for attention and skew people’s behavior to act in favor of large ecosystems. GitHub is heavily afflicted by that sickness. SourceHut, less so.

Drew: Sick of manipulative corporate behavior?

Devine: Yep, that’s the word I was looking for.

Drew: I admire the Rabbits for similar reasons: you have this down-to-earthness that I can connect with.

Drew: In more practical terms, do you find the lightweight approach to SourceHut’s UI design to be easier on your power and bandwidth constraints?

Devine: It’s worlds apart. Because we work entirely from donated second-hand devices, backward compatibility is more important to us at this point. In our eyes, better software is software that gets smaller over time, that sheds the superfluous, and that reaches further backward in time for that onto which it can run. SourceHut appealed to us instantly because there are so few examples of this willingness to reduce consumption in the wild.

Drew: I use a 12 year old laptop myself. I think it’s also important to recognize that the ability to recklessly consume is a privilege that not everyone has, and you’re locking out a lot of users by only designing for the latest and greatest. Even in FOSS, it can be a challenge to get people on board with that philosophy.

Devine: That’s definitely a big part of this. You can only call yourself anti-capitalistic for so long while also catering only to people with the latest gizmos.

Drew: On the subject of FOSS, why did you choose to release your software works as free (as in freedom) software, and your artistic works with Creative Commons?

Rekka: We don’t want our projects to die with us.

Devine: We can’t be there for people, we’re at sea for months at a time. The code has to speak for itself, be inspectable and repairable without our being there. We cannot sell our works as services, because we’re never there for people who might need our help fixing it.

Rekka: And things can happen at sea… as grim as it is to say that.

Devine: During every long passage we consider that we might have pushed our last commit, and is that the final state of this idea, possibly not. We like to think that someone might pick up our work where it was left.

Drew: That speaks to an admirable level of attentiveness to the needs of your users. And FOSS is a good way to cement your legacy.

Drew: May I ask what motivates your work? Why these projects? Why software as art? Why illustrations?

Rekka: The two of us like to create worlds. And art is a way is a way to do that.

Devine: Software is a way to make that world-building interactive and invite people into these worlds.

Drew: Do you view your work as having more of a performative or collaborative nature?

Devine: We obviously don’t share the same limitations as most. While I think people like our general philosophy, we will not convince Ubisoft to start making Dreamcast games. To most people, we’re exploring a fork in the road. The emulation scene constantly offers a reminder of what-if… what if Plan 9 had been picked up, what if Genera, what if… Art does this sort of exploration better than anything else. Our work is advised by our immediate limitations, but it’s visiting that fork in the road where 4K television did not actually made people happier. And watching Terry Davis work on TempleOS, that was performance to me, because there was no place in my reality to make any use of it.

Drew: Do you hope to offer the world something like that?

Devine: We work on small systems that people emulate on their M1.

Drew: For any readers who want to check out your work, or get involved themselves, where should I direct them?

Rekka: To our wiki, Dev’s website, or mine, or to git.sr.ht, where we host our projects.

Devine: We’ve been meaning to migrate to the new sourcehut pages, that’s what’s next.

Drew: Cool! Thanks for taking some time to chat with me. I’m very proud to host you on SourceHut, I’m a big fan myself.

Drew: Is there anything you wish I had asked or that we had discussed today?

Devine: Are you familiar with permacomputing? It’s a holistic approach to computing and sustainability inspired from permaculture.

Drew: No, tell me more.

Devine: Well, one thing that we’ve been thinking about is like… Fahrenheit 451 is a book about the world ending, and a handful of people take it upon themselves to preserve books they think have meaning. We often see folks echoing that computing causes more problems than it solves, and that what was once a tool of emancipation has been warped into a tool of control.

Devine: Are computers something we’d like to preserve going forward? How much is worth keeping? And so forth. This is what people consider when thinking about permacomputing. I don’t know the answer but it’s something interesting to explore. There isn’t a critical mass of folks thinking about these ideas right now, but we’re seeing more and more folks joining in the discussion.

Drew: I would like to see more thoughts on these lines, too. Hopefully we can signal boost that together and get some more brains involved.

Drew: Speaking of which: let’s open the floor to the rest of the channel?

Following the interview, we opened the IRC channel back up to general discussion for a while. Here are a few choice quotes from the ensuing chat:

alderwick: One thing I specifically wanted to say about 100r and SourceHut is that I wouldn’t have been able to contribute to Uxn if the repo was on GitHub. Like ddevault, I’m on old hardware. My laptop won’t load more than a few GitHub pages without crashing, but SourceHut is never a problem.

Devine: And without alderwick there would be no Uxn, I’m endlessly thankful that it made our meeting possible.

sigrid: Echoing alderwick here. GitHub doesn’t work on Plan 9 in Mothra. SourceHut does with no issues.

nihilazo: I like permacomputing as an idea but I struggle with it in a way because I’m required to use cloud services and such. I had to start using a more powerful laptop just to run MS Teams. Is there some way to reconcile trying to do permacomputing-y stuff with also working within a world that (currently) requires I use garbage?

Devine: There is no way to reconcile using MS Teams and permacomputing.

AWildThorp (later): What are your thoughts on cloud services with respect to permacomputing? I have mixed feelings about them in general. I kind of view them as buses vs cars, but on the other hand they can enable wastefulness

nihilazo: I understand that you can’t reconcile the two is there any way to contribute to permacomputing while still being trapped in systems antithetical to it?

Devine: Cloud services abstracts the waste from its users, I think until there’s more transparency they are antithetical to permacomputing, otherwise they could be an efficient tool to create some sort of permeance of information.

eletrotupi: nihilazo: I think you can and should contribute to permacomputing regardless of being trapped. Unless you are deliberating picking up antithetical services.

Devine agrees with eletrotupi

Drew: I think that most people aren’t trapped even if they feel so, too. You usually have a choice. For instance, you could seek out a different employer, or save up and start your own business. Positive steps in the right direction are almost always possible.

nihilazo: I am trapped by having to use Teams for college work, and I can’t really just switch colleges, given that they are the only college in the area.

Drew: You could start grassroots student organizations which try to tackle these issues. Throwing up your hands and saying it’s out of your control is not the best option. It takes people taking deliberate action to make changes in our society.

nihilazo: I agree on that point, I guess.

Anonymous: Was sailing a transformational experience led to permacomputing as a goal? And if so - can the lifestyle change required to avoid “MS Teams” be achieved without such a strong transformational experience?

Devine: We had to be put in an extreme situation to make that drastic a change. As an example, we were making iOS development when we cast off, thinking that we could do it along the way. Now it seems to naive to believe that modern technology can even survive being 15 nautical miles from the coast without it totally breaking down. But, we don’t think people have to be sailing to explore that space.

Rekka: Working from an older computer or from a shitty off-grid connection can help ;)

nihilazo: I’ve been very conscious of the potential power consumption of my computer use and trying to cut down but I am finding it difficult. I’m an internet addict, I guess. Especially things like video games, which are almost the very definition of pointless energy consumption.

Rekka: It’s difficult without actual constraints. If you are limited to the power in your batteries, say. You know your limits and it’s easy to limit usage.

kfx: I think it’s possible to separate those worlds. I use all kinds of unfortunate technology at work, and I don’t let it intrude on what I do for myself. Keeping those dividing lines sharp makes it easier to walk away from the gross stuff when the opportunity presents itself. Teams in fact was busily crashing my work computer about twenty minutes ago, but I kept reading this IRC discussion on my personal machine.

AWildThorp: On a similar note, if I go above my data limit on my smart phone, I have no bandwidth issues browsing the gemini space.

HamAdams: I’m really glad you all did this interview. I think this is putty some words to some general feelings I’ve been having recently regarding using technology more consciously.

Devine: Glad to hear :) The ComputingWithinLimit crew released a bunch of papers last year about this way of thinking about computers. The LIMITS2021 papers are excellent; I recommend having a look. They will put even more words to these thoughts.

Big thanks to the Rabbits for joining us for this interview! Please check out their cool projects. Orca is my personal favorite.

Announcing chat.sr.ht: a persistent IRC session for sourcehut users

Mon, 29 Nov 2021 00:00:00 +0000

About one month ago, we began a private beta for chat.sr.ht, the next flagship sourcehut product. Starting today, this service is now available to all paid sourcehut users.

chat.sr.ht is a hosted IRC bouncer service, which maintains a persistent IRC connection for you and extends IRC with useful features like offline messaging, log persistence, and various other improvements. In addition to accepting connections from any third-party IRC client, chat.sr.ht offers a ready-to-use web client.

Like the rest of the SourceHut services, chat.sr.ht is powered by free software. The bouncer is based on soju and the web chat is based on gamja. There are many other soju deployments in the wild — chat.sr.ht is just the newest. gamja is a standalone web client: it doesn’t depend on specific server software and several independent networks provide it to their users today.

The IRC networks you use with chat.sr.ht are independent entities, governed by their own stewards and unaffiliated with SourceHut. Thanks to the standard, open protocols that IRC is built on, chat.sr.ht users are able to participate in the established community of IRC networks like Libera Chat.

Our main goal is to make IRC easier to use for free software projects. For example, the bouncer exposes a special protocol extension to automatically connect to all of your IRC networks, which helps skip the tedious process of configuring every network you use. The web client can also be configured to open irc:// links around the web, so that getting involved in a project’s discussion can be as easy as clicking a link in the project’s README.

Our work is not limited to chat.sr.ht itself — we want to improve the IRC ecosystem as a whole. We are involved in upstream IRCv3 development: we participate in the standardization process by providing implementations and feedback for work-in-progress specifications (for instance chathistory), and we’re also pushing new specifications (for instance extended-monitor). Additionally, we’ve been contributing to modern-irc, the de-facto IRC reference document. We want chat.sr.ht to be a platform for improving the IRC community.

Moving forward, we plan to continue collaborating with the IRC community and improving the IRC standards. Our first priority is to develop the account-registration implementation in upstream ircds, which will bring a first-class network registration protocol to IRC. We’re also hoping to add anonymous access to chat.sr.ht, so non-paying users or users without a sourcehut account can participate in your project’s chat room.

In the meantime, please give it a shot! Head over to chat.sr.ht to take it a spin.

SourceHut's third year

Mon, 15 Nov 2021 00:00:00 +0000

Three years ago today, I announced that SourceHut would be making its alpha open to the general public after two years of development in private. Since then, in addition to moving across the Atlantic ocean, I have overseen the development of a service, and a business, which has grown to exceed all of my expectations. Today, SourceHut is home to 25,000 users, 5,000 projects, 42,000 git repos and 5,000 Mercurial repos. 153,000 emails have made their way to the 3,000 mailing lists hosted here, and 26,000 tickets have been filed across 5,000 bug trackers. Our CI system has completed 625,000 builds totalling 2½ years of continuous build time.

And so, despite its persistent “alpha” status, SourceHut has been a comfortable home to thousands of projects productively going about the business of building free software. I couldn’t be more proud of our work, or more thankful for the trust and support this community has offered us.

In return, what have we offered the FOSS community? Earlier this week, I asked the sr.ht-discuss mailing list to share some of their favorite projects on SourceHut. The favorites include a mobile client for itch.io, a multi-track audio editor reclaimed from the hostile takeover of Audacity, a hacker-first OpenStreetMap viewer, a user-first Medium frontend, and official mirrors of SourceWare projects like GCC and glibc, all of which are new to SourceHut this year. I also recently discovered that the French government’s new site for discovering French FOSS projects uses SourceHut as well. Check out the thread for more, or reply to add some of your own favorites!

SourceHut staff have also been hard at work contributing directly to free software again this year. As part of our free software consultancy, we have taken on projects like expanding OpenXR support on Linux, contributing to projects like the Linux kernel, Monado, Mesa, Xwayland, and more. We’re also responsible for the free software Wayland technology at the heart of Valve’s Steam Deck. We are also the release managers for Wayland and Weston, and maintainers for parts of the freedesktop and Alpine Linux infrastructure, and we became a Bronze supporter of OpenStreetMap this year.

Many new free software projects have also been developed and organized by our staff, including an IRC bouncer and webchat, a documentation site for Go, a new web browser based on NetSurf, and a project to develop a new programming language. Our forgeperf initiative has also turned our industry-leading focus on performance and accessibility into tangible improvements in GitLab and Codeberg. We will also be hiring a third engineer in February who will help us expand this work even more. In keeping with our tradition of transparency, the onboarding documents we prepared for them are available to the public.

And what of the forge itself? Our main focus this year has been on implementing our GraphQL API, which is now about halfway done. This is the main blocker for the SourceHut beta, and we have by now solved most of the unknowns and are working our way through the rote work of building out the remainder of the services. I’ll predict, for the third time, that we’ll complete the alpha and start the beta next year.

We’re comfortable taking our time to do this right, especially given that we’re quite able to provide a great deal of value to the FOSS community even during the alpha. Our definition of “alpha” has a specific meaning which differs from many other projects, specifically boiling down to meeting four important criteria:

Support for user groups/organizations
The development of a high-quality API which we can support indefinitely
Complete data autonomy, including import and export and account deletion
Comprehensive documentation for both hosted and self-hosting users

When we meet these criteria, we will have reached the necessary level of quality to ship a product we can be confident in for the long-term. However, we have already reached many important milestones which are uncharacteristic of many alpha-quality products. We’re committed to the longevity of user data, and we have many levels of redundant backups and monitoring. Our services are also extremely reliable, sporting better uptime than any of our competitors, including big fish like GitHub. Thousands of projects are already enjoying productive use of our services, and our goal is to provide them with a best-in-class product which far exceeds their expectations and industry norms.

In addition to this work in advancing the GraphQL APIs towards our ultimate ambitions of completing the alpha, we have also taken some time to develop additional products and foster a culture of shared community ownership over our services. We shipped pages.sr.ht this year, and we’re expecting to ship our hosted IRC services, chat.sr.ht, very soon. We’ve also continued to enjoy a great relationship with the Mercurial community, who are directly responsible for the maintenance of hg.sr.ht, in addition to the many volunteers representing their operating systems on builds.sr.ht. Members of our community are also working independently to develop names.sr.ht, which will eventually become a domain name registrar which uses git to store zone files and supports cool ideas like OpenNIC. We’re looking forward to involving the community even more in the coming years, as this is a key advantage of SourceHut’s FOSS design that many of our competitors lack — and works to the mutual benefit of SourceHut and the communities which rely on us. If you have a cool idea for SourceHut and you’re willing to write the code, we’re willing to provide you with support and resources to deploy it to.

With that, another year goes by, and the year ahead is full of work to do. I hope you’ll join our public Mumble meeting tomorrow to look back on our accomplishments, and look forward to the future. We’re meeting in the usual place, voice.mnus.de in the SourceHut room, at 10:00 UTC tomorrow, November 16th. Thank you for using SourceHut, and I hope we continue to serve you into the future.

What’s cooking on SourceHut?

Let’s quickly address the usual “what’s cooking” items before parting ways.

todo.sr.ht

Write support for the todo.sr.ht GraphQL API was my main focus this month, but it has turned out to be pretty complicated. todo.sr.ht is one of the most complex sourcehut services — submitting a ticket requires parsing user mentions, fetching subscribers, creating ticket and event rows, fetching or referencing new participants, sending email notifications, and more. I hope to complete this next month.

builds.sr.ht

Legacy Fedora images are being deprecated and removed next week. You will receive an email if this affects you. Thanks to Haowen Liu for taking over maintenance of the Fedora images, and to Timothée Floure for all of their hard work up to now.

apt-key has been deprecated in Debian upstream, and builds.sr.ht has been updated according to upstream recommendations. This is not expected to impact users.

We have stopped allocating a PTY for build logs. This may cause your build results to look less colorful, but should cause few problems otherwise.

Additionally:

openbsd/7.0 is now available
ubuntu/jammy is now available

pages.sr.ht

pages.sr.ht now supports partially updating only a subdirectory of your sites, which makes it easier to do things like manage different subdirectories for different projects without having all of them necessarily being aware of the content of the entire site.

What's cooking on SourceHut? October 2021

Fri, 15 Oct 2021 00:00:00 +0000

Welcome back for another month’s status update! As of today, our community numbers at 25,074 — the first time we’ve had over 25,000 users — after another 522 users joined our ranks. Please show them the courtesey and patience you’ve always done as they learn the ropes, and welcome these new users to our community with open arms.

This month’s Mumble meeting will take place on Monday, October 18th, at 10:00 UTC. We’ll be on voice.mnus.de, port 64738, in the sourcehut room. See you there!

chat.sr.ht

The first item on today’s agenda is a tiny preview of a new flagship SourceHut product: chat.sr.ht. More details will come in a dedicated announcement in a couple of weeks, but in short, chat.sr.ht is a hosted IRC bouncer based on soju which will offer all paying SourceHut users a persistent IRC session, and a webchat based on gamja, which will also provide a non-persistent session for non-paying and anonymous users.

Short answers:

We aren’t running an IRC network ourselves. We’re going to encourage users to use Libera Chat for their projects, though you can configure it to use any network (or networks) you want.
You can connect to the bouncer directly using your own IRC client, or use the webchat at chat.sr.ht.
irc:// links in your README and throughout SourceHut will be rigged up to auto-join your channel in our webchat when clicked.

The service is up and running, but currently set up as a private beta so that we can get everything tested and validated on a small scale. If you want to try the private beta, reach out to emersion on libera.chat.

GraphQL

This month, I implemented both read & write support for paste.sr.ht’s new GraphQL API, which you can try out here:

No native webhooks, but that’ll come later. I have also started working on write support for todo.sr.ht, which is actually quite complex and challenging — todo.sr.ht is one of the most complex services on SourceHut. I’m hoping to finish that over the course of the next month, and hopefully I can tick off another check-mark on the list as well.

meta.sr.ht: ✓ read ✓ write ✓ webhooks
git.sr.ht: ✓ read ✓ write ✗ webhooks
hg.sr.ht: ✗ read ✗ write ✗ webhooks
todo.sr.ht: ✓ read ✗ write ✗ webhooks
builds.sr.ht: ✓ read ✓ write ✗ webhooks
lists.sr.ht: ✓ read ✓ write ✗ webhooks
hub.sr.ht: ✗ read ✗ write ✗ webhooks
paste.sr.ht: ✓ read ✓ write ✗ webhooks
pages.sr.ht: ✓ read ✓ write ✗ webhooks

todo.sr.ht

In addition to the ongoing GraphQL work, todo.sr.ht has received some major updates with how it handles permissions and access lists. If you were affected by the change, you should have received an email explaining how to address it for your trackers. In short, trackers now have a “visibility” parameter which behaves the same as for git repos (public, private, or unlisted), and the default permissions have been consolodated into one field which applies to any users who do not have a more specific ACL entry. A similar change will soon come for lists.sr.ht.

git.sr.ht

Eli Schwartz has implemented a novel system based on git notes for attaching cryptographic signatures to git tags. He’ll be writing a guest post for this blog sometime soon which goes over this feature in detail.

builds.sr.ht

Image updates:

freebsd/11.x has been removed following its upstream deprecation.
GNU Guix is now available.

What's cooking on SourceHut? September 2021

Wed, 15 Sep 2021 00:00:00 +0000

Another month passes us by, recording further progress on the road to the sr.ht beta. Joining us on this journey are another 448 new souls, bringing our total number to 24,552. As always, I’m depending on you to show them kindness and patience as they learn how to use SourceHut.

This month’s Mumble meeting will take place on the tomorrow, September 16th, at 10:00 UTC. We’ll be on voice.mnus.de, port 64738, in the sourcehut room.

General news

SourceHut put out a job posting (now closed) a few weeks ago, and we have received many excellent applications. We are planning on making an offer to one of these candidates, and the details will be announced in the next status update. We have also spoken with NLnet, who may be interested in funding some additional contributors to work on SourceHut. Exciting stuff!

Also, as of this month, SourceHut is a bronze sponsor of the OpenStreetMap Foundation. We are proud to support a critical resource for free software and open data.

GraphQL

Following the deployment of the first implementation of GraphQL-native webhooks for meta.sr.ht, our GraphQL completion chart has a new column:

meta.sr.ht: ✓ read ✓ write ✓ webhooks
git.sr.ht: ✓ read ✓ write ✗ webhooks
hg.sr.ht: ✗ read ✗ write ✗ webhooks
todo.sr.ht: ✓ read ✗ write ✗ webhooks
builds.sr.ht: ✓ read ✓ write ✗ webhooks
lists.sr.ht: ✓ read ✓ write ✗ webhooks
hub.sr.ht: ✗ read ✗ write ✗ webhooks
paste.sr.ht: ✗ read ✗ write ✗ webhooks
pages.sr.ht: ✓ read ✓ write ✗ webhooks

Additionally, we deployed write support for lists.sr.ht this week. You can find the updated schema here: lists.sr.ht schema.graphqls.

I have written up a draft for the paste.sr.ht GraphQL API design here: paste.sr.ht schema.graphqls, which I intend to flesh out with an implemenation in the coming weeks (and also add GQL webhooks — it’s simple enough that we should be able to ship a complete GQL implementation in the first iteration). Ludovic and co. have also been hard at work on the first round of GraphQL support for hg.sr.ht, which is likely to ship in the foreseeable future. Following these, I would like to either implement a read-only hub.sr.ht GQL API, or add write support to todo.sr.ht, either of which is a considerable effort in their own right.

todo.sr.ht

Thanks to Ivan Habunek’s work, you can now specify the sort order of results on todo.sr.ht searches. Thanks Ivan!

builds.sr.ht

Image updates:

Rocky Linux is now available (thanks Haowen Liu!)
debian/bookworm is now available, and stretch has been removed
ubuntu/xenial has been removed

Tanguy Fardet also sent in their first patch this month, adding a “go to top” button on the build logs page. Thanks Tanguy!

Introducing a GraphQL-native approach to webhooks

Wed, 25 Aug 2021 00:00:00 +0000

Today, we are shipping a new system for webhooks for use with our suite of GraphQL APIs, which are under development as part of our larger beta plans. We’re not the first to use this design for webhooks, but it is somewhat uncommon, so I’ll take this opportunity to explain it to those who may be unfamiliar with it.

Let’s first establish, for contrast, how traditional webhooks work. Here’s an example of a webhook payload from GitHub:

Click to expand (large)

{
  "action": "opened",
  "number": 2,
  "pull_request": {
    "url": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2",
    "id": 279147437,
    "node_id": "MDExOlB1bGxSZXF1ZXN0Mjc5MTQ3NDM3",
    "html_url": "https://github.com/Codertocat/Hello-World/pull/2",
    "diff_url": "https://github.com/Codertocat/Hello-World/pull/2.diff",
    "patch_url": "https://github.com/Codertocat/Hello-World/pull/2.patch",
    "issue_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/2",
    "number": 2,
    "state": "open",
    "locked": false,
    "title": "Update the README with new information.",
    "user": {
      "login": "Codertocat",
      "id": 21031067,
      "node_id": "MDQ6VXNlcjIxMDMxMDY3",
      "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/Codertocat",
      "html_url": "https://github.com/Codertocat",
      "followers_url": "https://api.github.com/users/Codertocat/followers",
      "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
      "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
      "organizations_url": "https://api.github.com/users/Codertocat/orgs",
      "repos_url": "https://api.github.com/users/Codertocat/repos",
      "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
      "received_events_url": "https://api.github.com/users/Codertocat/received_events",
      "type": "User",
      "site_admin": false
    },
    "body": "This is a pretty simple change that we need to pull into master.",
    "created_at": "2019-05-15T15:20:33Z",
    "updated_at": "2019-05-15T15:20:33Z",
    "closed_at": null,
    "merged_at": null,
    "merge_commit_sha": null,
    "assignee": null,
    "assignees": [],
    "requested_reviewers": [],
    "requested_teams": [],
    "labels": [],
    "milestone": null,
    "commits_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2/commits",
    "review_comments_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2/comments",
    "review_comment_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls/comments{/number}",
    "comments_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/2/comments",
    "statuses_url": "https://api.github.com/repos/Codertocat/Hello-World/statuses/ec26c3e57ca3a959ca5aad62de7213c562f8c821",
    "head": {
      "label": "Codertocat:changes",
      "ref": "changes",
      "sha": "ec26c3e57ca3a959ca5aad62de7213c562f8c821",
      "user": {
        "login": "Codertocat",
        "id": 21031067,
        "node_id": "MDQ6VXNlcjIxMDMxMDY3",
        "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/Codertocat",
        "html_url": "https://github.com/Codertocat",
        "followers_url": "https://api.github.com/users/Codertocat/followers",
        "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
        "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
        "organizations_url": "https://api.github.com/users/Codertocat/orgs",
        "repos_url": "https://api.github.com/users/Codertocat/repos",
        "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
        "received_events_url": "https://api.github.com/users/Codertocat/received_events",
        "type": "User",
        "site_admin": false
      },
      "repo": {
        "id": 186853002,
        "node_id": "MDEwOlJlcG9zaXRvcnkxODY4NTMwMDI=",
        "name": "Hello-World",
        "full_name": "Codertocat/Hello-World",
        "private": false,
        "owner": {
          "login": "Codertocat",
          "id": 21031067,
          "node_id": "MDQ6VXNlcjIxMDMxMDY3",
          "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/Codertocat",
          "html_url": "https://github.com/Codertocat",
          "followers_url": "https://api.github.com/users/Codertocat/followers",
          "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
          "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
          "organizations_url": "https://api.github.com/users/Codertocat/orgs",
          "repos_url": "https://api.github.com/users/Codertocat/repos",
          "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
          "received_events_url": "https://api.github.com/users/Codertocat/received_events",
          "type": "User",
          "site_admin": false
        },
        "html_url": "https://github.com/Codertocat/Hello-World",
        "description": null,
        "fork": false,
        "url": "https://api.github.com/repos/Codertocat/Hello-World",
        "forks_url": "https://api.github.com/repos/Codertocat/Hello-World/forks",
        "keys_url": "https://api.github.com/repos/Codertocat/Hello-World/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/Codertocat/Hello-World/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/Codertocat/Hello-World/teams",
        "hooks_url": "https://api.github.com/repos/Codertocat/Hello-World/hooks",
        "issue_events_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/events{/number}",
        "events_url": "https://api.github.com/repos/Codertocat/Hello-World/events",
        "assignees_url": "https://api.github.com/repos/Codertocat/Hello-World/assignees{/user}",
        "branches_url": "https://api.github.com/repos/Codertocat/Hello-World/branches{/branch}",
        "tags_url": "https://api.github.com/repos/Codertocat/Hello-World/tags",
        "blobs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/Codertocat/Hello-World/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/Codertocat/Hello-World/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/Codertocat/Hello-World/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/Codertocat/Hello-World/languages",
        "stargazers_url": "https://api.github.com/repos/Codertocat/Hello-World/stargazers",
        "contributors_url": "https://api.github.com/repos/Codertocat/Hello-World/contributors",
        "subscribers_url": "https://api.github.com/repos/Codertocat/Hello-World/subscribers",
        "subscription_url": "https://api.github.com/repos/Codertocat/Hello-World/subscription",
        "commits_url": "https://api.github.com/repos/Codertocat/Hello-World/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/Codertocat/Hello-World/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/Codertocat/Hello-World/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/Codertocat/Hello-World/contents/{+path}",
        "compare_url": "https://api.github.com/repos/Codertocat/Hello-World/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/Codertocat/Hello-World/merges",
        "archive_url": "https://api.github.com/repos/Codertocat/Hello-World/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/Codertocat/Hello-World/downloads",
        "issues_url": "https://api.github.com/repos/Codertocat/Hello-World/issues{/number}",
        "pulls_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/Codertocat/Hello-World/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/Codertocat/Hello-World/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/Codertocat/Hello-World/labels{/name}",
        "releases_url": "https://api.github.com/repos/Codertocat/Hello-World/releases{/id}",
        "deployments_url": "https://api.github.com/repos/Codertocat/Hello-World/deployments",
        "created_at": "2019-05-15T15:19:25Z",
        "updated_at": "2019-05-15T15:19:27Z",
        "pushed_at": "2019-05-15T15:20:32Z",
        "git_url": "git://github.com/Codertocat/Hello-World.git",
        "ssh_url": "[email protected]:Codertocat/Hello-World.git",
        "clone_url": "https://github.com/Codertocat/Hello-World.git",
        "svn_url": "https://github.com/Codertocat/Hello-World",
        "homepage": null,
        "size": 0,
        "stargazers_count": 0,
        "watchers_count": 0,
        "language": null,
        "has_issues": true,
        "has_projects": true,
        "has_downloads": true,
        "has_wiki": true,
        "has_pages": true,
        "forks_count": 0,
        "mirror_url": null,
        "archived": false,
        "disabled": false,
        "open_issues_count": 2,
        "license": null,
        "forks": 0,
        "open_issues": 2,
        "watchers": 0,
        "default_branch": "master",
        "allow_squash_merge": true,
        "allow_merge_commit": true,
        "allow_rebase_merge": true,
        "delete_branch_on_merge": false
      }
    },
    "base": {
      "label": "Codertocat:master",
      "ref": "master",
      "sha": "f95f852bd8fca8fcc58a9a2d6c842781e32a215e",
      "user": {
        "login": "Codertocat",
        "id": 21031067,
        "node_id": "MDQ6VXNlcjIxMDMxMDY3",
        "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/Codertocat",
        "html_url": "https://github.com/Codertocat",
        "followers_url": "https://api.github.com/users/Codertocat/followers",
        "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
        "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
        "organizations_url": "https://api.github.com/users/Codertocat/orgs",
        "repos_url": "https://api.github.com/users/Codertocat/repos",
        "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
        "received_events_url": "https://api.github.com/users/Codertocat/received_events",
        "type": "User",
        "site_admin": false
      },
      "repo": {
        "id": 186853002,
        "node_id": "MDEwOlJlcG9zaXRvcnkxODY4NTMwMDI=",
        "name": "Hello-World",
        "full_name": "Codertocat/Hello-World",
        "private": false,
        "owner": {
          "login": "Codertocat",
          "id": 21031067,
          "node_id": "MDQ6VXNlcjIxMDMxMDY3",
          "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/Codertocat",
          "html_url": "https://github.com/Codertocat",
          "followers_url": "https://api.github.com/users/Codertocat/followers",
          "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
          "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
          "organizations_url": "https://api.github.com/users/Codertocat/orgs",
          "repos_url": "https://api.github.com/users/Codertocat/repos",
          "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
          "received_events_url": "https://api.github.com/users/Codertocat/received_events",
          "type": "User",
          "site_admin": false
        },
        "html_url": "https://github.com/Codertocat/Hello-World",
        "description": null,
        "fork": false,
        "url": "https://api.github.com/repos/Codertocat/Hello-World",
        "forks_url": "https://api.github.com/repos/Codertocat/Hello-World/forks",
        "keys_url": "https://api.github.com/repos/Codertocat/Hello-World/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/Codertocat/Hello-World/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/Codertocat/Hello-World/teams",
        "hooks_url": "https://api.github.com/repos/Codertocat/Hello-World/hooks",
        "issue_events_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/events{/number}",
        "events_url": "https://api.github.com/repos/Codertocat/Hello-World/events",
        "assignees_url": "https://api.github.com/repos/Codertocat/Hello-World/assignees{/user}",
        "branches_url": "https://api.github.com/repos/Codertocat/Hello-World/branches{/branch}",
        "tags_url": "https://api.github.com/repos/Codertocat/Hello-World/tags",
        "blobs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/Codertocat/Hello-World/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/Codertocat/Hello-World/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/Codertocat/Hello-World/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/Codertocat/Hello-World/languages",
        "stargazers_url": "https://api.github.com/repos/Codertocat/Hello-World/stargazers",
        "contributors_url": "https://api.github.com/repos/Codertocat/Hello-World/contributors",
        "subscribers_url": "https://api.github.com/repos/Codertocat/Hello-World/subscribers",
        "subscription_url": "https://api.github.com/repos/Codertocat/Hello-World/subscription",
        "commits_url": "https://api.github.com/repos/Codertocat/Hello-World/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/Codertocat/Hello-World/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/Codertocat/Hello-World/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/Codertocat/Hello-World/contents/{+path}",
        "compare_url": "https://api.github.com/repos/Codertocat/Hello-World/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/Codertocat/Hello-World/merges",
        "archive_url": "https://api.github.com/repos/Codertocat/Hello-World/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/Codertocat/Hello-World/downloads",
        "issues_url": "https://api.github.com/repos/Codertocat/Hello-World/issues{/number}",
        "pulls_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/Codertocat/Hello-World/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/Codertocat/Hello-World/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/Codertocat/Hello-World/labels{/name}",
        "releases_url": "https://api.github.com/repos/Codertocat/Hello-World/releases{/id}",
        "deployments_url": "https://api.github.com/repos/Codertocat/Hello-World/deployments",
        "created_at": "2019-05-15T15:19:25Z",
        "updated_at": "2019-05-15T15:19:27Z",
        "pushed_at": "2019-05-15T15:20:32Z",
        "git_url": "git://github.com/Codertocat/Hello-World.git",
        "ssh_url": "[email protected]:Codertocat/Hello-World.git",
        "clone_url": "https://github.com/Codertocat/Hello-World.git",
        "svn_url": "https://github.com/Codertocat/Hello-World",
        "homepage": null,
        "size": 0,
        "stargazers_count": 0,
        "watchers_count": 0,
        "language": null,
        "has_issues": true,
        "has_projects": true,
        "has_downloads": true,
        "has_wiki": true,
        "has_pages": true,
        "forks_count": 0,
        "mirror_url": null,
        "archived": false,
        "disabled": false,
        "open_issues_count": 2,
        "license": null,
        "forks": 0,
        "open_issues": 2,
        "watchers": 0,
        "default_branch": "master",
        "allow_squash_merge": true,
        "allow_merge_commit": true,
        "allow_rebase_merge": true,
        "delete_branch_on_merge": false
      }
    },
    "_links": {
      "self": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2"
      },
      "html": {
        "href": "https://github.com/Codertocat/Hello-World/pull/2"
      },
      "issue": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/issues/2"
      },
      "comments": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/issues/2/comments"
      },
      "review_comments": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2/comments"
      },
      "review_comment": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/pulls/comments{/number}"
      },
      "commits": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/pulls/2/commits"
      },
      "statuses": {
        "href": "https://api.github.com/repos/Codertocat/Hello-World/statuses/ec26c3e57ca3a959ca5aad62de7213c562f8c821"
      }
    },
    "author_association": "OWNER",
    "draft": false,
    "merged": false,
    "mergeable": null,
    "rebaseable": null,
    "mergeable_state": "unknown",
    "merged_by": null,
    "comments": 0,
    "review_comments": 0,
    "maintainer_can_modify": false,
    "commits": 1,
    "additions": 1,
    "deletions": 1,
    "changed_files": 1
  },
  "repository": {
    "id": 186853002,
    "node_id": "MDEwOlJlcG9zaXRvcnkxODY4NTMwMDI=",
    "name": "Hello-World",
    "full_name": "Codertocat/Hello-World",
    "private": false,
    "owner": {
      "login": "Codertocat",
      "id": 21031067,
      "node_id": "MDQ6VXNlcjIxMDMxMDY3",
      "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/Codertocat",
      "html_url": "https://github.com/Codertocat",
      "followers_url": "https://api.github.com/users/Codertocat/followers",
      "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
      "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
      "organizations_url": "https://api.github.com/users/Codertocat/orgs",
      "repos_url": "https://api.github.com/users/Codertocat/repos",
      "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
      "received_events_url": "https://api.github.com/users/Codertocat/received_events",
      "type": "User",
      "site_admin": false
    },
    "html_url": "https://github.com/Codertocat/Hello-World",
    "description": null,
    "fork": false,
    "url": "https://api.github.com/repos/Codertocat/Hello-World",
    "forks_url": "https://api.github.com/repos/Codertocat/Hello-World/forks",
    "keys_url": "https://api.github.com/repos/Codertocat/Hello-World/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/Codertocat/Hello-World/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/Codertocat/Hello-World/teams",
    "hooks_url": "https://api.github.com/repos/Codertocat/Hello-World/hooks",
    "issue_events_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/events{/number}",
    "events_url": "https://api.github.com/repos/Codertocat/Hello-World/events",
    "assignees_url": "https://api.github.com/repos/Codertocat/Hello-World/assignees{/user}",
    "branches_url": "https://api.github.com/repos/Codertocat/Hello-World/branches{/branch}",
    "tags_url": "https://api.github.com/repos/Codertocat/Hello-World/tags",
    "blobs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/Codertocat/Hello-World/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/Codertocat/Hello-World/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/Codertocat/Hello-World/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/Codertocat/Hello-World/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/Codertocat/Hello-World/languages",
    "stargazers_url": "https://api.github.com/repos/Codertocat/Hello-World/stargazers",
    "contributors_url": "https://api.github.com/repos/Codertocat/Hello-World/contributors",
    "subscribers_url": "https://api.github.com/repos/Codertocat/Hello-World/subscribers",
    "subscription_url": "https://api.github.com/repos/Codertocat/Hello-World/subscription",
    "commits_url": "https://api.github.com/repos/Codertocat/Hello-World/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/Codertocat/Hello-World/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/Codertocat/Hello-World/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/Codertocat/Hello-World/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/Codertocat/Hello-World/contents/{+path}",
    "compare_url": "https://api.github.com/repos/Codertocat/Hello-World/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/Codertocat/Hello-World/merges",
    "archive_url": "https://api.github.com/repos/Codertocat/Hello-World/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/Codertocat/Hello-World/downloads",
    "issues_url": "https://api.github.com/repos/Codertocat/Hello-World/issues{/number}",
    "pulls_url": "https://api.github.com/repos/Codertocat/Hello-World/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/Codertocat/Hello-World/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/Codertocat/Hello-World/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/Codertocat/Hello-World/labels{/name}",
    "releases_url": "https://api.github.com/repos/Codertocat/Hello-World/releases{/id}",
    "deployments_url": "https://api.github.com/repos/Codertocat/Hello-World/deployments",
    "created_at": "2019-05-15T15:19:25Z",
    "updated_at": "2019-05-15T15:19:27Z",
    "pushed_at": "2019-05-15T15:20:32Z",
    "git_url": "git://github.com/Codertocat/Hello-World.git",
    "ssh_url": "[email protected]:Codertocat/Hello-World.git",
    "clone_url": "https://github.com/Codertocat/Hello-World.git",
    "svn_url": "https://github.com/Codertocat/Hello-World",
    "homepage": null,
    "size": 0,
    "stargazers_count": 0,
    "watchers_count": 0,
    "language": null,
    "has_issues": true,
    "has_projects": true,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": true,
    "forks_count": 0,
    "mirror_url": null,
    "archived": false,
    "disabled": false,
    "open_issues_count": 2,
    "license": null,
    "forks": 0,
    "open_issues": 2,
    "watchers": 0,
    "default_branch": "master"
  },
  "sender": {
    "login": "Codertocat",
    "id": 21031067,
    "node_id": "MDQ6VXNlcjIxMDMxMDY3",
    "avatar_url": "https://avatars1.githubusercontent.com/u/21031067?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/Codertocat",
    "html_url": "https://github.com/Codertocat",
    "followers_url": "https://api.github.com/users/Codertocat/followers",
    "following_url": "https://api.github.com/users/Codertocat/following{/other_user}",
    "gists_url": "https://api.github.com/users/Codertocat/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/Codertocat/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/Codertocat/subscriptions",
    "organizations_url": "https://api.github.com/users/Codertocat/orgs",
    "repos_url": "https://api.github.com/users/Codertocat/repos",
    "events_url": "https://api.github.com/users/Codertocat/events{/privacy}",
    "received_events_url": "https://api.github.com/users/Codertocat/received_events",
    "type": "User",
    "site_admin": false
  }
}

SourceHut offers a similar approach with our legacy API. The problem here is information overload. This payload contains a lot of information, almost none of which you actually need, and may omit information you do want. One of our services, dispatch.sr.ht, uses this particular GitHub payload to configure build submissions for projects who want to use GitHub for git hosting and builds.sr.ht for CI. When we receive this webhook, we need to make several follow-up API requests to collect the necessary information for build submission.

With the new GraphQL design, the user submits not only the URL and events they want to receive, but also a GraphQL query which will be executed to customize the request payload that will be sent to them. In this custom payload, you are able to collect not only information about the event which occured, but also any other information you would find in a typical GraphQL request. So, to accomplish similar functionality for submitting a build manifest on a git push, we can configure a webhook with the following query:

query {
  # Collect details about the event which triggered the webhook
  webhook {
    uuid
    date
    profile { username }
    ... on GitPushEvent {
      refs {
        old { sha }
        new {
          sha
          author { name, email }
          message
        }
      }
    }
  }
  # Collect build manifests
  repository(id: 1234) {
    # Case 1: several manifests under .builds/*.yml
    multiple: path(path:".builds") {
      object {
        ... on Tree {
          entries {
            results {
              name
              object { ... on TextBlob { text } }
            }
          }
        }
      }
    }
    # Case 2: one manifest at .build.yml
    single: path(path:".build.yml") {
      object {
        ... on TextBlob { text }
      }
    }
  }
}

This is executed on the server before sending the webhook to our URL, and collects all of the information we need, and none of the information we don’t. We are then able to act on this webhook without any additional API requests. The payload we ultimately receive looks something like this:

{
  "data": {
    "webhook": {
      "uuid":
      "date":
      "profile": {
        "username": "sircmpwn"
      },
      "refs": [
        "old": {
          "sha": "81e98daa8341b1b5319645fecc81c7ba051c04d6"
        },
        "new": {
          "sha": "64aff2197e08a687fca5779057bd0ac3341fea8f",
          "author": {
            "name": "Drew DeVault",
            "email": "[email protected]"
          },
          "message": "This is a commit message"
        }
      ]
    },
    "repository": {
      "multiple": {
        "object": {
          "entries": {
            "results": [
              {
                "name": "alpine.yml",
                "object": {
                  "text": "image: alpine/3.14\nrepositories: ..."
                }
              },
              {
                "name": "archlinux.yml",
                "object": {
                  "text": "image: archlinux\nrepositories: ..."
                }
              },
              {
                "name": "debian.yml",
                "object": {
                  "text": "image: debian/sid\nrepositories: ..."
                }
              }
            ]
          }
        }
      },
      "single": null
    }
  }
}

We only have the information we need, and nothing we don’t, and we can fetch arbitrary information from the entire system to build our webhook payload without making any further API requests. Nice!

The GraphQL schema has been expanded with tools for managing webhooks, such as browsing past deliveries. There is also a resolver which generates a sample webhook event, which you can use for testing your GraphQL query. We’ll be rolling this out for meta.sr.ht’s API first, and expanding it to support more services in the future. You can learn more in our documentation:

Swing by our IRC channel (irc://) or fire off an email to sr.ht-discuss if you have questions or feedback!

SourceHut is hiring FOSS software engineers

Mon, 23 Aug 2021 00:00:00 +0000

This opening is now closed. Please check back again later for future openings. Thanks!

SourceHut is hiring paid software engineers to work exclusively on free and open source software. This is a full-time remote position. Our mission is simple:

We are here to make free software better. We will be honest, transparent, and empathetic. We care for our users, and we will not exploit them, and we hope that they will reward our care and diligence with success.

SourceHut is a ragtag group of hacker’s hackers, but note that we will have to be decidedly un-hackish when explaining things in business terms. Bear with us.

SourceHut is an early-stage bootstrapped company, which means that we are a young company that hasn’t taken any outside investments. This gives us greater autonomy and allows us to maintain our principles and ethics without deference to financial concerns. However, we have much less capital to work with, and stringent margins, which may affect you.

We have two main revenue streams: the forge product, and FOSS consulting. The forge product provides a reliable, but small, supply of income. We use this to pay out base salaries which are, frankly, well below market rates. We each earn a standard rate of $2,500 USD/mo, which may be supplemented with consulting work. Under the base arrangement, you are free to decide your own working schedule and choose any FOSS projects to work on.

Per the missive of “making free software better”, those projects can include projects which are not stewarded by SourceHut or directly relevant to our financial interests. We make regular contributions to a wide variety of projects in the FOSS ecosystem which have little to no direct relevance to our bottom line. We believe that by growing the FOSS ecosystem, we are making a larger market, and are making ourselves an essential part of it, which will play out in our favor (and in society’s favor) over the long term. You can work on your personal projects at SourceHut, and we encourage you to take bold risks on ambitious projects.

Our secondary income stream is through free software consulting, which provides us much more revenue than the forge, but less reliably. 100% of this work is free and open source software, written in the open. We don’t sign NDAs or non-competes. We take on 2-5 contracts per year, each lasting anywhere from 3-12 months, and you may choose which consulting projects you wish to work on. We charge high-end consulting rates, take a modest margin, and pay the rest to you. Our margin provides more capital for things like hiring new employees, expanding the forge infrastructure, long-term business planning, or simply paying out bonuses at the end of an engagement.

You have the level of autonomy and independence which best supports your needs. We have, at a minimum, bi-weekly open-ended 1-on-1 meetings, and a monthly all-hands meeting. Beyond this, the level of autonomy you enjoy is based on your need: our senior engineers get by with a vague missive of “make FOSS better” or “can you help with this one thing if you have time”, while more junior engineers may receive more structured work to provide better learning and experience.

In short, at SourceHut, you work on free and open source software, on the projects you want, on the schedule you want, and with the independence of judgement that respects you as an experienced, knowsome engineer. You work with smart and supportive peers on meaningful projects for an ethical business and a principled team. However, you receive a meager base salary, which, when combined with optional consulting work, can raise your annual pay to market, but with an inconsistency which may impact your personal financial planning. This is an arrangement which cannot suit everyone, but if it suits you, we’ll be pleased to have you along.

If you are interested in working with us, please send a plain-text email summarizing your interest, with your resume attached as a PDF file, to [email protected]">~sircmpwn/[email protected].

What's cooking on Sourcehut? August 2021

Sun, 15 Aug 2021 00:00:00 +0000

Hello again! Today we’re joined by 434 fresh faces, bringing us to a total of 24,104 users. As always, I’m relying on you to give a warm welcome to our new colleagues and to exercise patience as they learn the ropes.

This month’s Mumble meeting will take place on the normal date tomorrow, August 16th, at 10:00 UTC. We’ll be on voice.mnus.de, port 64738, in the sourcehut room.

After another month of development, how much closer are we to beta?

Operations

sr.ht for Alpine 3.14 is now available. Thank you for bearing with us during our planned maintenance window. I’ll ask for your patience again tomorrow, as we have a second maintenance window scheduled to finish our 3.14 rollout.

GraphQL

The current major milestone for the beta is the development and rollout of our GraphQL APIs, which form a more robust foundation for us to build upon.

This month, we shipped a read-only version of the lists.sr.ht GraphQL API. You can check it out here:

The latest GQL rollout progress is now:

meta.sr.ht: ✓ read ✓ write
git.sr.ht: ✓ read ✓ write
hg.sr.ht: ✗ read ✗ write
todo.sr.ht: ✓ read ✗ write
builds.sr.ht: ✓ read ✓ write
lists.sr.ht: ✓ read ✗ write
hub.sr.ht: ✗ read ✗ write
paste.sr.ht: ✗ read ✗ write
pages.sr.ht: ✓ read ✓ write

I have been prototyping GraphQL-native webhooks, which is the only outstanding major design concern for the GraphQL rollout. I expect to roll these out for meta.sr.ht in the next month. I will also start turning another one of these ✗ into ✓, but I’m not sure which yet.

Ludovic Chabant has also prepared a prototype for a read-only hg.sr.ht GraphQL API, which I will find some time to review this month.

builds.sr.ht

The following image updates have been shipped:

Ubuntu Impish and Hirsute are now available
Ubuntu Groovy was removed following its upstream deprecation

It is also expected that Rocky Linux will soon be available thanks to the efforts of Haowen Liu and Timothée Floure.

Miscellaneous

Thanks are due to several community members for their contributions this month. Sol Fisher Romanoff did the long-stale job of auditing and unifying the naming conventions for resources like git repos or bug trackers across all services, along with some unrelated minor improvements. Thanks also to Nguyễn Gia Phong, LordNature, and Juan Picca for their various contributions.

The mythical 10× programmer is just a good leader

Sat, 17 Jul 2021 00:00:00 +0000

There is some truth to the idea that some programmers are more productive than others. In practice, this is mainly a function of the breadth and depth of their experience, rather than an expression of innate talent. Under the right circumstances, the difference between two programmers can be significant, though a single programmer who is 10× more productive than the average is quite rare.

There does exist, however, a means of substantially improving productivity without relying on superstars: good leadership.

The difference between a one-person project which takes 5 years to complete, and a 10-person project which takes 2 years, is nine people. The ability for a maintainer to sell their project to potential contributors and inspire people to get involved is a major productivity hack. With 10 more years of experience, you may find yourself a 10× better programmer, but with 10 more programmers, you can get results today. This is especially true in FOSS, where attracting volunteers is often a key competency of successful projects.

Organizing the work is the most important task in this role. You need to hold a vision of the project in your head, and an actionable plan for executing it. This need only have enough detail to ensure that (1) you know what needs to be done, and in what order, and (2) you have enough work to keep everyone busy.

In my workflow, I aim to identify the most difficult problems first (those with the most unknowns), then have senior engineers address them upfront. This tends to create a lot of ancillary tasks which can be taken on by contributors at a broad range of skill levels, keeping a healthy pool of tasks ready for everyone who wants something to work on. It also makes the difficulty of the work trend towards zero, which helps a lot with planning, since you get a firmer and firmer picture of the remaining work over time. The goal is to quickly create a thin skeleton which can support work on the project’s largest challenges, allowing senior engineers to focus on the core problems while others put meat on those bones, planning out junior- to mid-level problems as they become apparent.

Working with the contributors on an individual basis is the next matter of concern. It’s up to the leaders to understand the interests and capabilities of each contributor, and help line up impactful work which they will find rewarding. Furthermore, you need to help them lay down a path of experience in which they can grow to understand more of the project’s scope, approach it with greater skill and confidence, and fill an empty niche in the team. This extends more to simple coding skills also: contributors can also be involved in planning, design, testing, research, code review, mentorship, and even leadership, taking some of the burden from your own shoulders. A good leader sees the potential in each contributor to grow in these domains, and figures out how to help them achieve that growth.

As software engineers, we’re used to applying our skills to technical problems, writing code, designing systems, and so on; but applying our problem-solving mindset to other domains as well will often yield surprisingly powerful results. When applied to the project’s social problems, leadership emerges. Success in this respect allows you to scale your engineering output exponentially.

Advertisement: SourceHut’s free and open-source software consultancy specializes in this kind of work. We provide not only talented engineers, but talented leaders. As the maintainers of dozens of FOSS projects, we are used to bringing out the best in ad-hoc teams of mixed experience, and our skills in mentorship and project management leaves that team stronger than it was when we started. Our work with upstream FOSS projects also has us constantly introducing ourselves to new projects and teams, and that experience has enabled us to execute on unfamiliar projects faster than anyone else. If that sounds like something that you need, [email protected]">get in touch.

What's cooking on Sourcehut? July 2021

Thu, 15 Jul 2021 00:00:00 +0000

Hallo uit Nederland! Today’s “What’s cooking” is brought to you from Amsterdam, where I moved to at the start of the week. Due to this, I have had less time available for work, but there has been some new progress regardless. User signups haven’t slowed, for one: we now have 23,670 users, some 442 greater than June. Please show them your hospitality and help them learn about sourcehut!

The usual Mumble meeting for this month is, sadly, cancelled. I do not have the necessary equipment to participate yet. I’ll see you for the August meeting instead.

The only substantial improvement this month has been in lists.sr.ht, though it is not yet visible: the GraphQL API. I had hoped to ship it earlier, but it will be delayed for perhaps another week or two due to issues moving abroad. Progress has been significant in any case, and it is almost complete.

Outside of this, various small bug fixes and maintenance work has landed throughout the services. Once I’m settled into my new home, I expect progress to resume at a normal pace.

What's cooking on Sourcehut? June 2021

Tue, 15 Jun 2021 00:00:00 +0000

Halfway through 2021! Things are on-track for my hopes to have the beta ship this year. Joining us on the way are 23,228 users, of which 640 are new this month. Remember to be patient with them as they learn the ins and outs of using sr.ht!

This month’s Mumble meeting will take place on the normal date tomorrow, June 16th, at 16:00 UTC. We’ll be on voice.mnus.de, port 64738, in the sourcehut room.

meta.sr.ht

I have overhauled the registration workflow in preparation for the future billing changes for the beta, more clearly distinguishing between the “contributor” approach and the “maintainer” approach — only the latter ultimately being expected to pay for their account. I’ve also made some improvements for admins of third-party instances to make it easier to provide billing support following the changes to builds.sr.ht payment.

builds.sr.ht

Mandatory payment is now in effect for all builds.sr.ht usage, for all users. If you require financial assistance, please [email protected]">email me explaining your circumstances and you will be issued free service credits.

As promised, the writable GraphQL API has shipped, allowing you to use GraphQL to submit jobs, manage job groups, and work with secrets.

Additionally, thanks to Michael Forney’s finishing touches on the old patchset, the NetBSD image is now available as “netbsd/9.x” (aka “netbsd/latest”) and “netbsd/8.x”. Furthermore:

Fedora 35 is now available
NixOS 21.05 is now available, and 20.03 has been removed
OpenBSD 6.9 is now available, and 6.7 has been removed
Minor improvements have been made to Arch Linux

lists.sr.ht

The initial draft of the planned lists.sr.ht GraphQL API has been written, and you can read it here (send feedback to IRC). Additionally, Simon Ser has broken ground on go-emailthreads, which we hope to use to provide more robust email discussion parsing for future improvements to code review tooling. Developing this GraphQL API is our focus for the coming weeks.

SourceHut is leaving Freenode

Wed, 19 May 2021 00:00:00 +0000

SourceHut has been a proud user of the Freenode IRC network since its inception. Today we have five sr.ht-related IRC channels for end-user support, operational monitoring, staff coordination, and more.

We will be moving our channels to Libera Chat, effective today. You can connect at irc.libera.chat on port 6697 (with SSL) and join us in #sr.ht.

The Freenode network we once loved is the victim of a hostile takeover by corporate interests. We entirely reject the illegitimate new leaders who have used legal threats and back-room deals to steal the network. The dedicated volunteers at the heart of Freenode’s success — the staff — have left for Libera Chat. We are sad to hear news of Freenode’s fall, but proud to be following them, our friends and colleagues, to this new network.

I’ll see you there!

Update 2021-05-25: The new staff at Freenode have been re-opening channels which moved to Libera.chat, including #sr.ht. I have re-closed this channel and will re-iterate in no uncertain terms that sourcehut has left Freenode for good. Any channels you see on Freenode which proport to represent sourcehut are not endorsed by this organization.

What's cooking on Sourcehut? May 2021

Sun, 16 May 2021 00:00:00 +0000

Running a day late on this one — was travelling all day yesterday. We’ll also be moving the Mumble meeting to tomorrow, May 17th, this month. Thanks for your patience! Today, SourceHut has 22,588 users, 698 more since the last status update. Please offer our new colleagues a warm welcome, and be patient with them as they learn how to use the software.

The monthly Mumble meeting, one day later than usual, will take place at 16:00 UTC on May 17th, in the SourceHut room on voice.mnus.de, port 64738.

builds.sr.ht

In addition to the recent billing changes, builds.sr.ht saw the initial release of its GraphQL API this month! The current version is read-only, but I expect to ship the read/write API within the next few days — it’s just finishing up the code review process now. You can take the builds.sr.ht GraphQL API for a test drive on the GraphQL playground.

GraphQL update

Today, we have the following GraphQL APIs working in production:

meta.sr.ht: ✓ read ✓ write
git.sr.ht: ✓ read ✓ write
hg.sr.ht: ✗ read ✗ write
todo.sr.ht: ✓ read ✗ write
builds.sr.ht: ✓ read ✗ write (soon)
lists.sr.ht: ✗ read ✗ write
hub.sr.ht: ✗ read ✗ write
dispatch.sr.ht: ✗ read ✗ write
paste.sr.ht: ✗ read ✗ write
pages.sr.ht: ✓ read ✓ write

The next service to receive its GraphQL API will be lists.sr.ht. Ludovic has also been working on an hg.sr.ht GraphQL API, which I’ll be helping out with soon. The project hub and write support for todo will be the last major projects for the GraphQL effort. After these, the remaining services — paste and dispatch — will be pretty straightforward.

Following the completion of the initial APIs work, the plan is to design and implement a few other features which will be shared between all of these services, most notably GraphQL-native webhooks.

I appreciate your patience while this work is ongoing. I know that there has not been much work in the realm of user-facing features over the past several months while we’ve been focusing on getting this done. This work, when completed, will unblock a lot of the workstreams you’ve been waiting for — organizations, improvements to the project hub and service interconnectivity, better ACLs, and so on. The GraphQL services are much more robust and sustainable for sr.ht, and will provide a necessary foundation for future features like these. It’s not flashy, but paying down our tech debt is a necessary pain before we can start the beta.

Sourcehut is the fastest. So what?

Sat, 08 May 2021 00:00:00 +0000

I’m writing this from 10,000 meters above the Pacific Ocean¹, typing it into a $200 ARM laptop, and I have had a pretty productive workday so far. In fact, with 600-1200ms latency, frequent network drop-outs, and bandwidth which would make a circa-2008 2G network hang its head in shame, I have had endured hardly any discernible difference from my workflow at home.

By objective measures, sourcehut is the fastest and most lightweight software forge² by a wide margin. I am able to load a git repository on git.sr.ht in about 3 seconds (DOM ready in 1.8s), while GitHub took 38 seconds (DOM ready in 20s)³. Working with any of the sourcehut services — browsing git repos, reading and filing tickets, reviewing build logs — is hardly any different in this situation than it is at my workstation at home. Browsing any other forge, on the other hand, is miserable. In the same amount of time I can load 3 pages on GitHub (one full minute!), I can load more than 30 on sourcehut.

That’s just the network conditions, too. Recall that I mentioned I’m on a cheap ARM laptop, basically a mid-tier chromebook. SourceHut has no mandatory JavaScript, and the pages are very light — almost all of them load in two requests or less, and usually transfer less than 10 KiB, even on a cold cache. It’s easy to render and use sr.ht on any class of hardware. It works half decently in Lynx!

All of that being said, I have only actually used the web services for three or four quick tasks today. SourceHut embraces open standards like email for collaboration. There is no suffering through the pull request review web UI on a craptop with a terrible connection. Patches arrive via email, and I have my mailbox stored locally in a standardized format, which I can read with simple text-based tools, then queue up my review as an emailed reply to be sent asynchronously whenever the network is up for it.

For my part, this is a story that comes from privilege. I am in a self-imposed network drought on an expensive airplane fare, and sourcehut’s performance is simply temporarily convenient. But there are a lot of people for whom this performance is an essential necessity in their daily life. This $200 craptop represents a substantial investment for a lot of people. The luxurious network conditions the developed world enjoys are just that: a developed luxury. I don’t find these users less important simply because their constraints are harder to work in. We pride ourselves on thriving under those constraints.

That ethos pays returns in many ways, like when I’m sitting here on an airplane conducting my work without interruption, or when our simpler HTML is easier to use for the new developer on your team who has to rely on a screen reader. Though at first it may seem that our performance metrics are a flashy footnote, they represent something of real importance to us and to our users, and I hope that the rest of the web catches up with us soon.

Fully vaccinated, and wearing a mask, and with a negative COVID-19 test result taken two days ago, for the record. ↩︎
These tests are conducted by sourcehut, but are based on independent measurement factors, have held up under independent scrutiny, and are open to improvements from the rest of the community. ↩︎
I can get on with my work after the DOM loads, but it’s still asking my browser to suck down images, leeching what precious little bandwidth I have, taking it away from anything else I’m trying to do. ↩︎

Bug trackers are for tracking bugs

Thu, 29 Apr 2021 00:00:00 +0000

There’s a reason that we call our bug tracking software “todo”: it designed to track things that need to be done. It’s not designed for end-user support, handling feature requests, and so on. This is a departure from the approach of some other popular forges.

We use our bug trackers for confirmed bugs and confirmed feature requests only. Due to learned behaviors from other platforms, many users do not understand this. For this reason, many project admins actually go so far as to set up read-only trackers, so that only maintainers can file tickets. This approach improves the quality of bug reports and sharply reduces duplicates, turning the tracker into a narrowly-focused tool for organizing the work of the contributors.

But what happens to end-users in this arrangement? They’re not forgotten: they are given a better venue for their problems. Most projects of this sort provide a foo-users or foo-discuss mailing list, which encourages submissions from end-users in a more forum-oriented style. Sometimes these discussions become actionable, in which case a ticket is filed by a maintainer or a janitor (the latter opening up a new contributor role for non-programmers), but more often than not the user just has to have a few questions answered. Often the answers come from other users, saving the maintainers time and encouraging a community of mutual support to develop.

Many projects also set up an IRC channel for short questions, discussions, and hands-on troubleshooting in a real-time format. Many of the discussions on IRC can also result in tickets, or help collect information to start a good mailing list thread, and provide another role still for non-programming contributors to help others out. We’ll soon be offering tools on sourcehut to help you maintain your project’s IRC channel.

In addition to this -users mailing list, most projects have a -devel mailing list for development discussions and patch reviews, and often a developer IRC channel as well. This segregation of development resources and end-user support resources helps developers stay focused and on-topic in their work, without interruption, and provides a better places for end-users to get help.

What's cooking on SourceHut? April 2021

Thu, 15 Apr 2021 00:00:00 +0000

Hello again! As another month rolls on by, sr.ht saw 849 new users join our ranks, bringing our total to 21,890. As always, please treat our new colleagues with patience while they learn the ropes.

Tomorrow’s monthly Mumble meeting is planned for 16:00 UTC, in the usual place: voice.mnus.de, port 64738. An important question will be discussed: should builds.sr.ht be made available only to paid sr.ht users? For details, see the builds.sr.ht update further on.

General News

First, and most importantly, I have some urgent news.

After ~nelson had some issues using Stripe from the Falkland Islands, I offered to accept payment for his sr.ht account in the form of cute pictures of the locals. He agreed, and here they are 🐧 He sent four pictures ([1] [2] [3] [4]), and three videos ([1] [2] [3]). Thanks, Nelson! Naturally, he will not be expected to pay for his account.

Otherwise, things have been somewhat quiet, as (1) I’ve taken a little bit of time off this month, and (2) we’ve been busy warding off attackers at the gates. Otherwise, we have been working quietly on builds.sr.ht’s GraphQL API.

git.sr.ht

This is not a new feature, but a couple of weeks ago, I made a short video demonstrating how to use git.sr.ht’s web patchset tools.

builds.sr.ht

The GraphQL API is making good progress, and I hope to have the initial version online this month — only a few features left. Ignas Kiela has been working on some improvements for performance, and a few other minor improvements have shipped alongside those.

Image updates:

FreeBSD 13 is now available

We’ve been dealing with a rash of cryptocurrency mining attacks on builds.sr.ht this month, as you may have seen on status.sr.ht. If you’ve experienced long queue times, this is why: we are playing a cat & mouse game trying to teach our systems to detect crypto miners, then updating them again after the attackers figure out how to work around them and saturate our build queue with cryptocurrency miners.

We seem to have things under control for now, but it raises a more interesting question: should we just make builds.sr.ht a paid service? Ultimately, all sr.ht services are going to require payment, but builds is both the most expensive service for us to host, and the most ripe for abuse, so it may make sense to make it paid in ahead of the beta plans. We’ll be discussing this idea with the community during the Mumble call tomorrow, feel free to join if you have some thoughts around this.

What's cooking on SourceHut? March 2021

Mon, 15 Mar 2021 00:00:00 +0000

Hi! Another month of development has passed, and I’m here to fill you in on what’s new. Another 686 signups this month has brought us to 21,041 users. As always, I’ll be counting on you to make the new users feel at home, please be patient with them and help them learn the ropes.

Last month, something suddenly came up and I had to skip the monthly Mumble call unannounced — my apologies! We’ll be holding the meeting as scheduled tomorrow at 16:00 UTC at voice.mnus.de, port 64738, in the sourcehut room.

pages.sr.ht

The big news this month is that we shipped sourcehut pages, which provides static hosting for websites and Gemini capsules. 282 users have created a site with it so far — you should check it out, too! Getting a website published only takes a couple of minutes.

todo.sr.ht

todo.sr.ht’s initial GraphQL API is now available. Check out the playground here and our GraphQL docs if you’re not already familiar with how it works.

Next up is…

builds.sr.ht

The GraphQL API for builds.sr.ht is now underway. This one is quite welcome, as builds.sr.ht is our oldest sr.ht API and the most in need of an upgrade. It will also give us some opportunities to improve security for builds.

Project hub

A minor update for the project hub this month: popular tags are now shown on the public index.

Sourcehut pages

Thu, 18 Feb 2021 00:00:00 +0000

I’m happy to announce that a long-awaited feature is available today: sourcehut pages. SourceHut users can use this service to host static websites for any of their own domains, and every user is also being given “username.srht.site”.

Update: gemini support is now available as well!

You can use any static site generator: Jekyll, Hugo, Doxygen, or your own cool new thing. You can publish from git.sr.ht or hg.sr.ht, or you can set up any publishing workflow you prefer — you just need to upload a tarball. Every domain has TLS automatically configured for you. You get your personal subdomain on srht.site, unlimited bring-your-own-domains, and up to 1G of storage per site.

Check out srht.site for the full details, but here are some shell commands you can run right now to be live in 3 minutes.

First, generate a personal access key and add it to your environment, something like this:

bearer_token=9SKUndgx8Fx55xYGNAxs1Lal8YQAh29/90m+HjsMVsA=

Then write a simple “index.html” file:

<!doctype html>
<html lang="en">
<meta charset="utf-8" />
<title>My sourcehut page</title>
<h1>My sourcehut page</h1>
<p>Welcome to my cool sourcehut page!

Put it in a tarball:

tar -cvz index.html > site.tar.gz

And publish it, being sure to replace “username” with your sr.ht username:

curl --oauth2-bearer "$bearer_token" \
    [email protected] \
    https://pages.sr.ht/publish/username.srht.site

Hey presto, your new website is live! Check out the rest of the documentation for tips on automating this process, setting up a static site generator like Hugo, and how to use your own domain name.

All sr.ht pages sites have some limitations to make them good internet citizens:

HTTPS is required
All third party resources, including Google Analytics, are blocked
CloudFlare’s reverse proxy is blocked

So everyone who visits a sr.ht page can be confident that everything is above board. Enjoy!

What's cooking on Sourcehut? February 2021

Mon, 15 Feb 2021 00:00:00 +0000

Greetings! I’m happy to share that our community has surpassed 20,000 users this month. 708 new faces have joined our ranks, bringing the total to 20,355. We’ve become a pretty big community! Please be friendly and patient with them as they learn the ropes. This month’s “what’s cooking” is somewhat deficit in interesting user-facing developments, but I will dutifully report our progress nonetheless.

Our next public Mumble conference will be held tomorrow at 16:00 UTC, on voice.mnus.de, port 64738, in the sourcehut room. Feel free to join us to share your thoughts and ask your questions. I’ll see you hear you there!

General news

We’ve upgraded (most of) our infrastructure to Alpine Linux 3.13, and made sr.ht packages available to any admins who are running third-party instances. A quick reminder that any such admins should be subscribed to the sr.ht-admins mailing list to receive this and other important announcements that affect your instance.

We will have to schedule a second round of upgrades for our hardware hosts sometime in the next few weeks. It’ll be announced in advance on sr.ht-announce and on status.sr.ht when the time comes.

todo.sr.ht

Most of our progress this month is in the new todo.sr.ht GraphQL API, which is going well. I hope to ship an initial read-only version to production before the next status update.

builds.sr.ht

Not technically an update for builds.sr.ht, but rather for the software which submits build manifests: we’ve started developing some standard environment variables which are set by git.sr.ht, hub.sr.ht, and so on, when submitting builds. You can use these to programmatically find out why your build was submitted, and access other context-specific details. Check out the docs here.

What's cooking on Sourcehut? January 2021

Fri, 15 Jan 2021 00:00:00 +0000

Another year begins, and hopefully with better prospects for us all. SourceHut has emerged from 2020 relatively unscathed, thankfully, and I hope the same is true of most of our users. A body which, by the way, today numbers 19,647 strong, up 623 from December. Please warmly welcome our new members, and be sure to lend them your expertise as they learn how to use our tools.

Tomorrow’s public Mumble conference will be held at 16:00 UTC, on voice.mnus.de, port 64738, in the sourcehut room. Feel free to join us to share your thoughts and ask your questions. See you there!

General news

First, a much asked-for feature is now available: a dark theme! I actually rolled out the first roll on this feature a few hours after publishing last month’s status update. It is enabled automatically when your user-agent requests it via prefers-color-scheme: dark. A few iterations of improvements have already been performed, and we are keeping a list of matters for attention in future patches, but please feel free to share your feedback on sr.ht-discuss as you encounter any issues.

todo.sr.ht

As promised, we broke ground on API 2.0 for todo.sr.ht this month. Feel free to check out the draft GraphQL schema if you are curious to know more.

hg.sr.ht

Following the roll-out of git.sr.ht’s API 2.0 implementation, Ludovic Chabant has been hard at work on developing a prototype of an API 2.0 branch for hg.sr.ht as well. I intend to review this and help form it into a production-ready implementation in the coming weeks.

git.sr.ht

Small news: you can now update the patchset revision in the patch preparation UI. Nolan Prescott also contributed some fixes to this workflow for some missed edge cases.

dispatch.sr.ht

An experimental feature is now available which allows dispatch.sr.ht users to forward patches sent to a lists.sr.ht mailing list to GitLab as merge requests, and convey the enusing discussions between both mediums. YMMV.

builds.sr.ht

There have been some minor fixes and improvements to the following images:

Debian
Fedora
NixOS

Additional Fedora improvements are in the pipeline, as well as a new Void Linux image.

On the subject of ethics in our industry

Wed, 13 Jan 2021 00:00:00 +0000

I am disappointed by our peers in the industry in terms of their failure to uphold a reasonable moral standard, and I am sorry that it is necessary for me to write about it today. I would much prefer that we compete in terms of our product’s utility and our technological expertise, and it shames me that we must also compete in terms of our respective levels of command over our moral compasses.

I learned today of an incident at GitHub, one of our competitors.¹ A Jewish employee was fired in apparent retaliation for commenting “stay safe homies, Nazis are about” in one of the company’s Slack channels. This occurred during an insurrection at the US Capitol by radical right-wing, racially-motivated terrorists, at least some of whom were wearing Nazi propaganda, for the purpose of uprooting the democratic process to install a fascist demagogue. The seditionist riot claimed lives.

This news comes months after GitHub made the news for accepting contracts² with US Immigration and Customs Enforcement, an organization which has infamously committed large-scale human rights violations at the behest of that same fascist demagogue, problems which were generally known at the time and which were brought to GitHub’s attention without effect. Around the same time, another company in the same space, GitLab, loosened their internal policies³ to make it easier for them to make similar agreements without regard to ethics, stating that they will, quote, do business with “customers with values that are incompatible with our own values.”

This level of moral absenteeism within the industry SourceHut shares astonishes me. I wish that there was no need for me to make the following statements explicitly.

These are the official policies of SourceHut: we condemn white nationalism, Donald Trump, and the supporters of both. We affirm that we value human rights, the democratic process, and the judicial process. We assert the rights and value of all people, of all nationalities, races, ethnicities, sexual preferences, gender identities, disabilities, religions, and economic classes. We will defend anyone who would speak up in support of these ideals. We will not do business with anyone who does not recognize and embody these truths.

We think it’s repugnant that our peers don’t demonstrate the same values. We think that you, the customer, should reward businesses that commit to upholding these values with your patronage, and withhold it from those which will not.

Also, in case the employee that GitHub terminated is reading this: [email protected]">feel free to get in touch if you’re still open to a job in the industry. We’d be happy to talk to you.

Source: Business Insider (alternate without paywall: Gizmodo) ↩︎
Source: Bloomberg ↩︎
Source: The Register ↩︎

What's cooking on Sourcehut? December 2020

Tue, 15 Dec 2020 00:00:00 +0000

A brisk wind of winter chill sets a stir down my spine, as I sit down with a fresh cup of coffee to yarn a story of careful engineering and passionate spirit that took place over the course of 30 days. The last 30 days. Cause this is the monthly “what’s cooking” post. Welcome! This month we’re joined by 815 new users, bringing the total to 19,024. Please give them your warmest welcome and help them learn the ropes.

Our next public Mumble conference will be held tomorrow at 16:00 UTC, on voice.mnus.de, port 64738, in the sourcehut room. Please join us to share your feedback, ask questions, and so on.

General news

The big news this month is the rollout of API 2.0 support for meta.sr.ht and git.sr.ht, both of which have reached feature parity with their REST counterparts. For meta.sr.ht as well, our new OAuth 2.0 backend has shipped and is fully compliant with RFC 6749.

todo.sr.ht is up next, and work for API 2.0-native webhooks is going to begin soon as well.

meta.sr.ht

Here are the resources for meta.sr.ht’s GraphQL support:

There were also some improvements to our abuse prevention capabilities after a wave of spam registrations — if any third-party sr.ht admins require similar interventions for their instances, please reach out in private.

git.sr.ht

git.sr.ht’s GraphQL API grew support for mutations, like repository creation and artifact uploads. Resources:

lists.sr.ht

Email replies to patchset threads can now include a custom header, X-Sourcehut-Patchset-Update to update the status of the patchset to ACCEPTED, REJECTED, NEEDS_REVISION, and so on. Additionally, a couple of buttons have been added to all mailing lists which allow any user — list admin or not — to download an archive of either the last 30 days, or the entire list’s history, as an mbox for importing into their local mail spool.

builds.sr.ht

NixOS 19.09 has been removed following its upstream deprecation.

SourceHut's second year in alpha

Sun, 15 Nov 2020 00:00:00 +0000

Today is the second anniversary of SourceHut being made available to the general public during its alpha phase of development, and the end of the fourth year of development. What a year it’s been! If only I could send a warning to the younger me who wrote last year’s article. As the second anniversary comes around, despite the trials we’ve endured this year, I feel positive about our accomplishments.

One year ago, I wrote that I expected SourceHut to enter its beta phase in 2020, and I was wrong. However, we do have a well-defined, finite list of action items to be completed before we can kick off the beta. We hit a lot of important development milestones this year, and we’ll put the cherry on top in 2021.

This month’s public Mumble conference will take place tomorrow (the 16th) at 17:00 UTC, in the sourcehut room on voice.mnus.de, port 64738. Please join us to celebrate our second year, and ask any questions or provide any feedback that you might have about sourcehut.

Let’s take a moment celebrate our accomplishments in 2020, and lay out our plans for 2021.

2020 in summary

Here are some of the more important things we got done:

The project hub
Initial GraphQL APIs for meta.sr.ht, git.sr.ht
git.sr.ht release artifacts, git blame & path logs
builds.sr.ht job artifacts, RSS feeds
lists.sr.ht CI integration
todo.sr.ht tracker data import/export
Public access to operations data & documentation

And some of the numbers for 2020:

7,344 users signed up
1,447 mailing lists established; 37,164 emails sent
2,024 bug trackers and 9,210 tickets filed
15,754 git repositories and 2,662 hg repositories made
233,000 build jobs completed
2× increase in revenue

And, 2,000 projects on the brand-new project hub! You can browse the 723 public projects right here. Some notable projects which started using SourceHut this year include:

Dozens of Gemini projects started on sr.ht this year
sr.ht is the home of the Plan 9 renaissance
The Zrythm Digital Audio Workstation moved in
vis and mutt set up sr.ht mirrors and CI
alacritty and Nim rigged up our CI
More in this thread and this thread

We’ve also contributed to many FOSS projects upstream, including:

git send-email improvements
Mercurial patches
Linux documentation patches
New pygit2 and libgit2 features, and financial support
musl financial support
Redis vulnerability reported & fixed
PeerTube financial support

And our engineers, under the open-ended directive of “make FOSS better”, have contributed to Wayland, Mesa, Vulkan, Go, the Linux kernel, IRCv3, Gemini, and more. Our FOSS consulting arm has produced 100% free-software output in Mesa, Xorg, and more, and built a brand-new FOSS webmail, for clients like Valve Software and Migadu.

Expectations for 2021

Well, I’ll go ahead and make the same ill-fated prediction this year: the beta will begin sometime in 2021. One of the blockers I mentioned last year still remains to be completed — user groups/organizations — but the main blocker is the completion of API 2.0. I’ve also added “beta” labels to all of our bug trackers, if you want to explore the fine details of what features and bugs are considered blockers.

The business and operational goals laid out in last year’s post have been met. The only business-related blocker is determining how the long-promised pricing changes will take shape when the beta begins, which will be heavily influenced by the ultimate implementation of user groups. Of course, the community will be involved in every part of this discussion. And, as always, I’ll re-iterate that the plan is not to price users out of the service — those who are unable to afford payments, as always, will be issued free service.

The API 2.0 efforts are of particular importance for the beta. Their completion will be necessary to establish an API that we can be confident in supporting for many years into the future, and enable us to expand the services and their awareness of one another to any degree we please. The performance, scalability, and reliability of our services is also expected to increase substantially with the completion of API 2.0.

We also made substantial progress on data ownership — you can now export and import almost all of your data on sourcehut, much of it in standard formats — but we also want to expand on this by implementing self-service account deletion and renames, and a unified solution for downloading all of your data at once.

So, in short, we have this to look forward to for the beta:

User groups/organizations
API 2.0 completion
Improvements to the cross-service awareness
Completion of data ownership goals

We still have a lot of work on the horizon, but we have a well-laid plan towards the finish line and the start of the beta phase. Once the beta begins, we’ll focus on refining our model, flushing out bugs, improving our already best-in-class performance and reliability, and establishing enough confidence in the system to declare it production ready.

SourceHut has much higher standards for production readiness than most software. Many users are already enjoying SourceHut as the platform of choice for their own production-ready projects, and we already make stability and security guarantees which many would consider necessary as such. But we’re not going to declare our system production ready until we’ve well exceeded the standards of the industry — we want the best performance, the best reliability, and we want to leave a wide margin for everyone else to catch up. We’re only going to declare it production ready when we’re far ahead of the pack and prepared to support all of our features and users for decades to come. SourceHut will be “production ready” when it’s prepared to become the bedrock of free software, and not before.

What’s cooking on SourceHut? November 2020

Now, for your regularly scheduled status update, I’ll keep it brief. Most of the work has been behind the scenes, on API 2.0.

meta.sr.ht

gemini:// and gopher:// URLs are now permitted for profile URLs

builds.sr.ht

nixos/20.09 is now available
openbsd/6.8 is now available
ubuntu/20.10 is now available
freebsd/12.x has been updated to FreeBSD 12.2

SourceHut Q3 2020 Financial report

Wed, 11 Nov 2020 00:00:00 +0000

In summary, SourceHut is financially healthy. Our revenue is still growing, and we are still profitable and becoming more so. However, growth has slowed significantly compared to Q2. In internal terms, Q3 has been characterized by an increased emphasis on planning, less visible development efforts, and a decreased emphasis in marketing — not to mention taking place during a series of global and national crises — some or all of which may provide an explanation. Regardless, this may not be indicative of a trend, and we have still experienced positive growth by all accounts.

Disclaimer: this report is a summarized approximation of our financials, and is not used for tax purposes.

Revenue sources

Sourcehut receives revenue from paid user subscriptions. During Q3, we processed 1,855 invoices. The invoices paid break down as:

1,007   $2  (paid monthly)
464     $5  (paid monthly)
155     $10 (paid monthly)
246     $2  (paid yearly)
57      $5  (paid yearly)
28      $10 (paid yearly)

The total gross revenue during Q3 was $16,134, which after transaction fees is $14,972. This is very similar to our Q2 revenue, and a 213% increase over Q2 2019.

In Q3, 1,850 new users registered for an account. Of these users, 199 are paid; 10.8%. These numbers are both somewhat lower than expected. Q2 saw 2,551 new registrations, and the usual conversion rate is 11.7%. It remains to be seen if this is a trend. As of November 11th, the breakdown is as follows:

  344   $2  (paid monthly)
  163   $5  (paid monthly)
   58   $10 (paid monthly)
1,054   $2  (paid yearly)
  337   $5  (paid yearly)
  132   $10 (paid yearly)

The monthly revenue from all subscriptions is, accounting for credit card fees and amortizing yearly payments, is $5,952. This is an increase of $408 in monthly revenue compared to Q2.

SourceHut has $13,858 in cash at the time of writing.

Expenses

Breakdown of Q3 expenses:

$4660    Servers and equipment
$1950    Philadelphia datacenter lease (inc. network, power, etc)
$300     pygit2 sponsorship
$130     Domain name registration & renewals
$100     Regulatory fees
$96      musl libc sponsorship

Our main expense this quarter was for putting together a new server, sakuya1, which is our first second-generation VM hosting server. The costs fell within the expectations explained in the Q2 financial report. Presently this server is only used for hg.sr.ht, but its role will expand in the future.

We do not have any unusual expenses planned for Q4.

Mailing lists are resistant to censorship

Thu, 29 Oct 2020 00:00:00 +0000

As a US entity, SourceHut is obliged to comply with DMCA notices. In this event, our next step would likely be to coach the affected project through the counter-notice process, and contribute to their legal costs if we believe that they’re in the right. We know that the DMCA is a constantly abused force for censorship, and there are no friends of the RIAA here. They represent much that our mission statement — to support and improve the free- and open-source software ecosystem — stands in opposition to.

Even beyond our principles, however, the mailing list based workflow we espouse is resistant to this kind of censorship. Git repositories are easily re-hosted, of course. However, if a lot of your project’s value requires rapid updates and the ongoing support of its development community, a centralized, pull-request-style system is vulnerable to censorship.

If you use mailing lists, you might not even immediately notice that something was wrong. That contributions pass through a centralized mailing list is often only a formality — the project maintainers and others likely to have comments will usually be Cc’d on the emails, and they’ll be delivered directly to them without the list’s help. Recipients can provide feedback by replying to the email, sending their comments directly to the contributor, and bypassing the mailing list entirely. The list faithfully records these emails and forwards them to interested parties, but it’s not actually required for this to work.

A mailing list also provides every maintainer, contributor, and onlooker a copy of everything which has happened on it. Like git distributes repositories to anyone working on them, mailing lists distribute a complete archive of the list to everyone who is subscribed to them. Taking this archive somewhere else and carrying on is effortless.

Moreover, you would be hard pressed to make a strong argument that new patches could be covered by the DMCA, as they represent original additions to the codebase. We’d consult our lawyers to consider the specific circumstances, but it’s entirely possible that we wouldn’t even have to shut off the mailing list when dealing with this situation. SourceHut’s tools — bug trackers, mailing lists, git repos, and so on — can work independently of one other.

Embracing decentralized, open standards like email and mailing lists is a good tool for censorship resistance. Putting all of your eggs into a basket controlled by a single corporate entity, who will be coerced by bullies and bad laws, is not a strategy for success. Times like these remind us why open source projects need to use open source infrastructure, built on open standards.

Haven’t used email with git before? Here’s what it’s like:

Check out git-send-email.io to try an interactive tutorial.

What's cooking on Sourcehut? October 2020

Thu, 15 Oct 2020 00:00:00 +0000

Once again we meet to discuss the status of the ongoing SourceHut alpha, which has made progress by leaps and bounds this month. We’re joined by 521 new users this month, bringing our total up to 17,715. As always, please welcome them warmly and show our new friends the ropes.

Tomorrow, we’ll be holding the second public Mumble conference, where you’re welcome to join us to ask questions, share feedback, or just to listen in. We’ll be meeting tomorrow, October 16th, at 16:00 UTC, in the SourceHut room on voice.mnus.de, port 64738. I’ll see (or hear) you there!

I mentioned last month that today would mark the second year of the sr.ht alpha — my mistake! It’s actually next month. I’ll put together a more detailed status update then which summarizes our progress this year and outlines the plans for next year. I should also have the Q3 financial report done by then.

General improvements

A few weeks ago I wrote a detailed log going over the progress and planning for the API 2.0 efforts. OAuth 2.0 support has been completed, and most of the new database abstractions are done. Since that post, I’ve also completed the work on asyncronous task management, and based on that also the necessary bits for legacy webhook delivery. This is the last blocker which prevents shipping the writable API into production even in an early form, as it would have caused inconsistency between services without. Accordingly, I expect to ship an early version of the writable meta.sr.ht GraphQL API into production in the next few weeks.

We’ve built a new ops tool this month called chartsrv, which generates SVG plots from Prometheus data, like this live chart of the load average across builds.sr.ht workers:

Thanks are also due to gildarts and Ignas Kiela this month, for their work adding links to Markdown headings throughout SourceHut, to allow users to easily obtain a link to a specific heading. Thanks folks!

builds.sr.ht

Debian users should be advised that an update to the Debian riggings is going to affect how packages are installed in a way which may break your builds. An email was sent to all users who have submitted Debian builds in the last 30 days with details, and another email will be sent out before the change is deployed, 15 days from now.

Thorben Günther has also improved the Arch Linux images to make them faster and more reliable. Thanks!

todo.sr.ht

наб added a user preference to todo.sr.ht this month which allows users to choose to have email notifications sent to them for their own activity on todo.sr.ht. Thanks наб! I have also fixed a minor issue with tracker import/export which may have affected users with older bug trackers.

meta.sr.ht

Thanks to Timothée Floure’s work, meta.sr.ht now includes an LDAP authentication backend, which third-party installations of sr.ht may find useful for integrating with their existing authentication system. Thanks Timothée!

git.sr.ht

Thanks due to наб and Thorben Günther for small patches this month, respectively for improving symlinks and observability, and for fixing an issue with build submissions.

Post-mortem: git.sr.ht's almost-outage today

Thu, 08 Oct 2020 00:00:00 +0000

git.sr.ht almost suffered an outage today, when available disk space for repository storage got as low as 2.8 GiB. Thankfully, the problem was identified and a solution applied before an outage was incurred. This ended up being a good case study in the value of good monitoring and a solid understanding of the performance characteristics of your system. So, what caused this to happen, why didn’t it cause an outage, and what have we learned?

A plot showing free disk space on git storage over 5 days. Higher is better.
We generate these graphs with chartsrv, by the way.

During a routine audit of our systems on September 18th, I discovered that git.sr.ht disk space was growing at a higher than expected rate. At the growth rate observed on the 18th, I estimated that we had 9 weeks before we’d run out of disk space. Having other things to deal with at the time, I wrote up my initial findings on the sr.ht-dev mailing list and planned to re-visit the problem 3 weeks later.

Run the clocks forward to last night, and a disk space usage alarm goes off. These fire when we reach 80% storage utilization on any host. Normally this incurs an immediate investigation, but I initially presumed that it was the natural progression of the growth trends we had observed a few weeks prior. I downgraded the urgency of the alarm and made a note to follow up this morning.

So, this morning rolls around, and I log into the machine and check the disk usage… and there’s only 4.6 GiB left, and falling by 2 gigs per hour! This trend is obviously heading towards disaster, so I quickly truncate some large log files to buy time, file a public incident report, and loop #sr.ht in on the issue.

The game plan is:

Establish a lower bound on disk space at which an emergency is declared and git.sr.ht is put into read-only mode. This is set at 2 GiB.
Spin up a new server with sufficient storage space to accomodate the growth for long enough to re-evaluate our plans.
Start transferring data over and get as much of the migration done as possible before hitting that 2 GiB floor and declaring an outage.
Investigate: why is this happening?

Steps 2 and 3 have some idle time baked in, so I’m doing the investigation in parallel. It seems strange, there’s no one adding especially large repos and the git storage growth rate is consistent with our normal projections. I ruminate over it while working on provisioning a replacement server. Once I’m prepared to start the zfs receive to pull the dataset over to the new host… I notice that the snapshots are unusually large.

I delete six old snapshots and immediately freed up 500 GiB of space. Crisis over.

An update is issued for the incident report: the issue was resolved without incurring an outage. Now comes the important questions:

Why did this happen?
How can we prevent it from happening again?

This should not have happened. We are aware that snapshots occupy disk space and we had measured their growth rate and factored it into our planning for git.sr.ht disk space utilization. So why were they so large?

The culprit ended up being that, on August 12th, we deployed a cronjob which runs git-gc every 20 minutes on a random subset of git repositories. Prior to this change, our git storage was basically append-only: outside of users deleting their repositories, objects were only ever added, not removed. After this change, we started to delete objects as well, which caused the rate at which differences between snapshots and the present filesystem accumulated to change, causing our snapshots to display a more pronounced increase in size as they aged. Our snapshot retention policy did not account for this different growth model.

Our Prometheus retention policy was only set to 15 days (it has since been increased to 60 days), so we can’t directly observe a change in the growth rate at the time this commit was deployed. However, we can observe that disk usage fluctuates periodically:

In this chart, a higher number means more free disk space.

The GC script runs every 20 minutes, and we take a snapshot every 15 minutes, hourly, and daily. With each GC, disk space frees up briefly, and is reduced again when a snapshot is taken.

Now that we know the cause, we can plan to avoid this issue in the future. It’s less urgent now that the emergency is resolved, but the future plans are to:

Create fine-grained storage utilization metrics which correlates utilization with purpose, so we can see separate utilization figures for git storage, ZFS snapshots, logs, etc.
Re-reason about our retention policy and storage utilization growth rate armed with new knowledge about the effect of git GC on storage utilization.
Move forward with planning for normal git.sr.ht storage upgrades. We’re planning on expanding storage 4x-8x in the foreseeable future.
Add additional instrumentation to our cronjobs to better understand the role they play in system performance.

I’ve also increased the metrics retention from 15 to 60 days to allow us to better observe long-term trends and have a larger window for forensics. This shouldn’t present a problem for storage on our monitoring system, but it monitors itself and will let us know if it starts to run out of space.

So, crisis averted for today. Let’s take this opportunity to improve our planning and observation skills to make sure that we’re better prepared for tomorrow’s crisis.

In-process work queueing for Go

Tue, 06 Oct 2020 00:00:00 +0000

In the course of our API 2.0 efforts, it is necessary for us to implement some kind of mechanism for queueing and retrying work in our Go platform. The most obvious examples of this work is sending emails and delivering webhooks, but there are more subtle examples, too — just about anything which can be moved out of the request → response hot path would improve performance for the end-user. As such, some kind of task queue is called for.

Our Python codebase uses Celery, and we have already had some experience at building Celery workers in Go. However, on the whole we’re pretty dissatisfied with Celery. It’s too complicated and inflexible for our needs. Something lighter weight and more toolkit-oriented (as opposed to midlayer-oriented) would address our use-case better.

So, I set out to design us a bespoke task queueing system for our Go daemons. Some of the goals were:

Ability to manage work in Goroutines instead of separate daemons¹
Easy handling of re-attempts with an exponential backoff
Graceful termination (stop accepting new tasks and flush the queue)
Observability with Prometheus

The solution came in the form of our new “dowork” Go library. Here’s the basic usage:

import "git.sr.ht/~sircmpwn/dowork"

queue := work.NewQueue()
queue.Start(context.Background()) // Does not block
queue.Submit(func(ctx context.Context) error {
    // Do work...
    return nil
})
queue.Shutdown() // Blocks until all pending tasks are complete

If you want to handle retries, create and enqueue your task in two steps:

task := work.NewTask(func(ctx context.Context) error {
    // Do work...
    return nil
}).Retries(5)
queue.Enqueue(task)

This will automatically retry your task with an exponential backoff if it returns an error.

Let’s take a look at this in action! Consider our email handling module.

// Returns a task which will send this email for the work queue. If the caller
// does not need to customize the task parameters, the Enqueue function may be
// more desirable.
func NewTask(ctx context.Context, m *gomail.Message) *work.Task {
	conf := config.ForContext(ctx)
	return work.NewTask(func(ctx context.Context) error {
		return Send(config.Context(ctx, conf), m)
	}).Retries(10).After(func(ctx context.Context, task *work.Task) {
		if task.Result() == nil {
			log.Printf("MAIL TO %s: '%s' sent after %d attempts",
				strings.Join(m.GetHeader("To"), ";"),
				strings.Join(m.GetHeader("Subject"), ";"),
				task.Attempts())
		} else {
			log.Printf("MAIL TO %s: '%s' failed after %d attempts: %v",
				strings.Join(m.GetHeader("To"), ";"),
				strings.Join(m.GetHeader("Subject"), ";"),
				task.Attempts(), task.Result())
		}
	})
}

// Enqueues an email for sending with the default parameters.
func Enqueue(ctx context.Context, m *gomail.Message) {
	ForContext(ctx).Enqueue(NewTask(ctx, m))
}

// Creates a new email processing queue.
func NewQueue() *work.Queue {
	return work.NewQueue("email")
}

Some code for handling contexts is omitted for brevity. Here we use a closure to enclose the message to be sent, and some extra details like the config file (which includes our SMTP details). We configure it for up to 10 retries, and log the result after the task is complete. Sending an email with this is pretty straightforward:

m := gomail.NewMessage()
m.SetAddressHeader("From", "[email protected]", "Jane Doe")
m.SetAddressHeader("To", "[email protected]", "John Smith")
m.SetHeader("Subject", "An email subject")
m.SetBody("text/plain", "An email body")
email.Enqueue(ctx, m) // Doesn't block!

The next interesting component comes when it’s time to terminate the process. We want to do the following things:

Stop accepting new connections and free up the HTTP port for the new daemon
Finish servicing existing requests, up to a timeout
Finish running any already-queued tasks
Terminate the process

So the process looks like this:

mail := email.NewQueue()
mail.Start(context.Background())

// ...

go qserver.Serve(qlistener) // Asynchronously start the main HTTP server

sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt)
<-sig
signal.Reset(os.Interrupt)
log.Println("SIGINT caught, initiating warm shutdown")
log.Println("SIGINT again to terminate immediately and drop pending requests & tasks")

log.Println("Terminating server...")
ctx, cancel := context.WithDeadline(context.Background(),
  time.Now().Add(30 * time.Second))
qserver.Shutdown(ctx)
cancel()

log.Println("Terminating work queues...")
work.Join(mail)
qserver.Close()
log.Println("Terminating process.")

As a bonus for observability, we also set up a secondary HTTP server on a kernel-assigned TCP port, which we can use to monitor the shutdown process:

mux := &http.ServeMux{}
mux.Handle("/metrics", promhttp.Handler())
pserver := &http.Server{Handler: mux}
plistener, err := net.Listen("tcp", ":0")
if err != nil {
  panic(err)
}
log.Printf("Prometheus listening on :%d", plistener.Addr().(*net.TCPAddr).Port)
go pserver.Serve(plistener)

I also added a little extra log during the shutdown process:

log.Printf("Progress available via Prometheus stats on port %d",
  plistener.Addr().(*net.TCPAddr).Port)

This is just to print the Prometheus port closer to the shutdown event in the logs, for easy reference. curl http://[::1]:$port/metrics will provide Prometheus metrics, including the queue drain progress for the sysadmin to monitor.

That’s it! Some future improvements along these lines will include:

Rigging this up with OpenRC so that our service manager can oversee these kinds of restarts
More sophisticated coordination between the old and new server processes to reduce the window during which connections might be dropped²
Dumping the queue state on the second SIGINT so that the admin can review and possibly re-queue some of the tasks later
Moving the queue into a secondary process or a remote machine, through some kind of remote submission mechanism
Applying this work further towards the upcoming builds.sr.ht work distribution overhaul

Stay tuned.

View the “dowork” project on SourceHut →

Without necessarily making it difficult to move executors to separate processes or machines later on ↩︎
Update 2020-10-07: We’ve successfully tested using SO_REUSEPORT to allow the replacement daemon to start up before the defunct daemon starts shutting down, which completely eliminates the window during which connections could be dropped. We still need to work on coaxing OpenRC into handling this service lifecycle; we’ll likely write a follow-up post about this. ↩︎

SourceHut API 2.0 dev log

Fri, 25 Sep 2020 00:00:00 +0000

Completing the project now known as “API 2.0” is one of the most important steps in finalizing the sr.ht beta, and I’ve been trying to keep the community abreast of developments, especially on the sr.ht-dev mailing list. I’m summarizing for the blog as well today, some of our recent developments in this respect and planned work to come.

Read more: why is API 2.0 is important to the beta?

SourceHut is a distributed system of "mini-services", each fulfilling its role in a particular domain — git.sr.ht handles git, builds.sr.ht handles CI, lists.sr.ht handles mailing lists, and so on. In order for these services to communicate effectively, good API design is critical. A good API is also necessary for SourceHut users to extend sr.ht with their own tools.

The legacy API was designed within the context of our Flask applications for the purpose of quickly meeting these needs during the design & development of the sr.ht alpha. It's... not great. The RESTful design implies a tree-like structure, which does not map as well onto our data model — the "graph" of GraphQL does much better in this regard. Additionally, the implementation is somewhat inconsistent and leaves a lot to be desired in terms of robustness. The stronger type system of GraphQL enforces a baseline of consistency which, while achievable with the legacy approach, is much easier with the 2.0 approach.

The API design is an essential participant in proving the architecture design of sr.ht. In order to meet the confidence level we need to start the beta, we need a greater degree of confidence than the legacy approach offered. API 2.0 meets this requirement. Additionally, stability is going to be important post-beta: this is our last opportunity to make a clean break with the legacy API and ship a better, more stable design into production which we can comfortably support for longer.

Progress updates

Some of the progress which has been completed recently includes:

OAuth 2.0 support

The new meta.sr.ht API supports OAuth 2.0 for authentication. We previously advertised “OAuth” support, but the new implementation is actually conformant with the RFC. A draft of our OAuth 2.0 documentation can be read here.

The design makes meta.sr.ht-api the source of truth for all things OAuth, and the meta.sr.ht-web (Python) frontend just issues GraphQL queries to manage it. meta.sr.ht-api is responsible for issuing personal access tokens, authorization tokens, and access tokens, and possibly refresh tokens in the future.

The relevant GraphQL resolvers are @internal, so meta.sr.ht-web is the only client which is allowed to use them. However, this is a good proof-of-concept for later work which will expand this design to other resolvers, which third-party clients are permitted to use, to relocate more sources of truth from -web into -api. The ultimate intention is to remove SQLAlchemy from the -web services entirely and just have them execute GraphQL queries to fetch the necessary information to present the web UI.

The access tokens themselves (both personal access tokens and bearer tokens) take the form of a BARE-encoded payload specifying the username, client ID, authorized scopes, and token expiration, authorized with HMAC, and base64 encoded. Resolvers which can be accessed with these tokens, and the required scopes to access them, are controlled by the new @access directive in the GraphQL schema.

To revoke a token, its SHA is computed and a revocation entry is added to Redis. Additionally, the token record in the database has its expiration set to the current time. When token authorization is checked, its hash is computed and the revocation status is quickly queried from Redis. In general, Redis is treated as an ephemeral cache on sr.ht — in the event that the data is lost, a new revocation list can be constructed by simply querying the database for expired tokens and issuing new revocations for them.

These changes brings the total number of authorization methods up to four:

@internal authentication, based on a single-use token encrypted with SourceHut internal private keys, and used for service-to-service authorization. It has access to resolvers which are not normally available to the public.
OAuth 2.0 authentication, via either bearer tokens or personal access tokens, which are limited by the scopes laid down on each resolver with @access.
SourceHut web login cookie authorization, intended for use with the web-based GraphQL playgrounds, which have an access level equivalent to personal access tokens.
Legacy OAuth tokens, which must have * permissions, and are only granted read-only access.

The fourth method, legacy OAuth tokens, is planned to be hastily removed. In order to facilitate this, I intend to start recording when a legacy token is used to access the GraphQL API. This information will be used to create a list of users who would be affected by the removal of this feature; they’ll be sent an email with another 30 days of notice (and instructions on switching to API 2.0 tokens) before it’s removed.

A side note: the new personal access token design allows you to create personal access tokens with limited scopes, and removes the ability for third-parties to create bearer tokens with unlimited account access.

These systems are security-critical, so some eyes on the code from the community would be appreciated. The bulk of the code can be reviewed in the writable-api branch of meta.sr.ht, and at the tip of gql.sr.ht.

Database abstraction

SQLAlchemy has been a major pain point for the web applications, and the database abstraction to be used with GraphQL has been carefully designed to avoid the same pitfalls. Specifically, it has been made as thin as possible. The abstraction is now mostly done, as support was recently added for GraphQL mutations. The only outstanding improvement might be to introduce better support code for cursors, which are used to access lists of things (e.g. PGP keys for an account).

Another place where there might be improvement is in reducing the number of total round-trips. It would be feasible to defer loading detailed user information until it’s actually necessary, for example, and also possible to cache some basic relationships (e.g. username to user ID) in memory, though such a cache would only affect corner cases. Additionally, if we can update the mutation support code to use PostgreSQL’s RETURNING feature, we can avoid a round-trip to fetch the record we intend to mutate. This may require forking squirrel. In theory, it should be possible to reduce the number of round-trips for any given request to be equal to the number of distinct resource types it implicates. Further consolidation may be possible with more elaborate queries in the future.

Initial performance testing

Initial performance tests on routine read-only queries suggests that we can expect to handle 1,000 reqs/sec/thread with a target service time of 20ms per request. Write queries are about half as efficient, and we could expect perhaps 100 reqs/sec/thread at 200ms each. The bottleneck on write queries seems to be in PostgreSQL, rather than in our code. More research will be necessary if it becomes an issue. In order to scale up parallelism, we’ll have to plan out the number of connections per PSQL server; I expect that we have plenty of room with our current deployment to tune this upwards.

For scaling in the future, read-only queries can be distributed over horizontally scalable PSQL secondaries. The API code is already set up to use read-only transactions when appropriate, so redirecting these should be easy. Write access will still have to be limited to a single master server, which will need to scale vertically. If this becomes a bottleneck in the future, we may research sharding options. However, writes are much less common than reads, and if we direct the read load off of the master server we should be able to scale quite far without much cost.

Some initial testing with pgbouncer raised some issues with feature compatibility between pgbouncer, the PSQL write protocol, lib/pq, and psycopg2. We plan on using pgbouncer more to meet availability requirements than scalability requirements, however, so an in-depth answer to these issues will be explored separately.

Future work

Some things which are next up in the pipeline:

Async dispatch

The next major problem which needs to be addressed is the handling of asynchronous work in the GraphQL backends. Some tasks will need to be done async, including webhook and email delivery. Additionally, a lightweight means of firing off small tasks to be completed async whenever needful would be helpful to have, for small optimizations which move more logic out of codepaths which would block delivering responses to clients.

To this end, I intend to build a general-use system for handling queuing, executing, and re-attempting async tasks in Go applications. This will be able to live in any Go process, and my expectation is that some of them will live inside of the GraphQL backends themselves, and some will live in dedicated daemons which will have work assigned to them by the GraphQL backends remotely. Whether or not a task will be handled in- or out-process is a function of the tasks predicted reliability and importance, and thus the need for external management of that task queue.

For example, email delivery is likely to be handled in-process. We deliver mail through a dedicated mail server, which has its own queuing and re-delivery mechanisms built-in. Mail delivery is likely to succeed, and relatively soon after it enters our task queue. The in-process queues are unlikely to get very full. Webhook delivery, on the other hand, is important for preserving a consistent data model across our services, and does not have a secondary queue or redelivery mechanism like email. Therefore, managing it through a separate daemon would be ideal. However, both of these can use the same code - the only difference would be where the tasks come from, be it somewhere up the call stack or a remote request.

Some things that this code will need to support are:

Warm process shutdown (including shutting down the HTTP listener in the GraphQL backends to free up the port for the next daemon, or to stop accepting new tasks in the remote daemons)
Queueing, executing, and re-delivery of tasks, with exponential backoff
Observability via Prometheus (and the relevant alarms)
The ability to serialize state to disk

This design will likely have implications for the upcoming builds.sr.ht work distribution overhaul as well.

GraphQL-native webhooks

To continue supporting legacy webhooks, we will first need to build an implementation of the legacy system for the GraphQL backends. Naturally, this is blocked by the async solution above, but it should provide a good proving grounds for that work.

We have some basic sketches for handling webhook registration and delivery with GraphQL, but a proof-of-concept needs to be built. The basic idea is that, when registering a new webhook, you provide a GraphQL query which we execute before delivering your webhook and use as the request payload for the webhook. This can be used to eliminate any further API requests your webhook endpoint might have to perform to gather any additional information it needs to complete its work.

A plan will need to be made for transitioning our existing code from legacy webhooks to API 2.0 webhooks.

Packaging

We’ll need to change how we handle the distribution packages to support these updates. It’s likely that, for example, the “meta.sr.ht” package will become a meta-package which depends jointly on meta.sr.ht-web (where the Python web application will be re-homed to), meta.sr.ht-api (the GraphQL backend), and perhaps meta.sr.ht-async. If we grow any additional frontends in the future (Gemini, for instance), this will make it easy to add them. It will also make it easy to offer GraphQL APIs without the web application, should any third-party installations desire this.

Other future work

gql.sr.ht is going to be merged into core-go soon.
A lot of refactoring has gone into the GQL support code in the course of developing OAuth 2.0 and the database abstraction for meta.sr.ht; git.sr.ht’s GraphQL proof-of-concept will have to be updated to follow these changes.
A transition plan for deprecating the legacy API entirely will have to be prepared prior to finalizing the beta.

Please make your way to sr.ht-dev to keep up with and participate in the discussion.

What's cooking on Sourcehut? September 2020

Tue, 15 Sep 2020 00:00:00 +0000

SourceHut development continues its constant march through the alpha this month, and I’m here to tell you all about it. Our users now number 17,194, thanks to 511 new members joining us since the last update. To our new users: welcome! And to our older members: thanks for being here! I hope everyone new finds a warm welcome in the rest of the community.

Earlier this month, I held a public meeting to discuss the plans for the beta, and we felt that it was a pretty good way to connect with the community. Going forward, the day following each status update (so, tomorrow), I’m going to hold a public meeting where you’re welcome to join up and ask any questions or provide any feedback. In turn, I’ll have questions of my own for the community, and will be looking forward to hearing your thoughts.

The next meeting will be held tomorrow, September 16th, at 15:00 UTC, on Mumble. We’ll be meeting in the SourceHut room on the voice.mnus.de server, on port 64738. Please join us!

And following that earlier meeting, I’d like to share with you the thoughts on the beta progress. The path towards completing the alpha is quite clear now, and as such, I’ve updated all of the bug trackers to include a “beta” label, and marked all of the tickets which block the completion of the alpha as such. The big-ticket blockers are:

Finalize pricing (a survey will be sent out - can you recommend some software for this?) and billing system overhaul
GraphQL APIs
Work distribution overhaul for builds.sr.ht
Web-based code review for lists.sr.ht
More hub.sr.ht cross-service integrations

Next month marks the end of the second year of the alpha, and I’ll prepare an extended status update which goes into detail on what remains and how we’re going to complete the beta.

General improvements

Headers in Markdown across sourcehut now include a link to that header which appears on hover, which should be useful for linking to a specific header anywhere on the services. Thanks to gildarts for adding that!

Admins of third-party instances will also be pleased at some recent improvements, including better exception emails and user administration screens, as well as fixes for long put-off exceptions in various edge cases throughout and improvements to the soundness of the database schema.

Project hub

Project #tags are now supported on the project hub, with the goal of making it easier to discover projects and for communities to establish themselves around a topic. This was only added a few days ago, so most projects don’t have tags yet — please take a moment to add some for your projects if you can! Thanks to наб again for implementing this feature.

meta.sr.ht

meta.sr.ht is the primary focus for the GraphQL expansion, and work is underway adding a new OAuth 2.0-compliant authorization process and introducing writable GraphQL API access. See the writable-api branch if you’re curious.

Some user-facing improvements have also been made this month, such as support for TOTP recovery codes. Disable and re-enable TOTP on your account to generate these. Additionally, I’ve added security emails which are sent to you whenever anything security-relevant occurs on your account, such as a password change or new SSH key.

наб also added a feature which allows new users to enter their PGP key during the registration process, so that the registration email is encrypted when sent. This makes it possible to never receive any unencrypted emails from sr.ht.

git.sr.ht

наб has introduced many improvements to git.sr.ht this month, including a blame view, path-specific logs, and two features for push options: updating the repo description or visibility, and submitting arbitrary build manifests other than .build.yml or .builds/*.yml. Docs here. Thanks наб!

todo.sr.ht

You can now edit comments on tickets.

man.sr.ht

Adding a sorely overdue feature, gildarts implemented a user index page for man.sr.ht, where you can list the wikis on your account. Thanks gildarts!

SourceHut's PeerTube bootstrap fund: first batch

Wed, 09 Sep 2020 00:00:00 +0000

In May we announced our initiative to fund video creators who wanted to get started on PeerTube. PeerTube is a free software project which creates a federated network of video sharing platforms, each of which owned and operated by independent, small groups of creators or users or both. This decentralized, democratized approach to video publishing, on an open source platform, is compelling to us, and we wanted to help bootstrap it with new content for users to enjoy on the platform.

Today we’re happy to announce that we’ve selected our first two video creators to receive funding and equipment: Matthew Bryant and Tom Cooks. We’ve brought these creators on board with our PeerTube instance at spacepub.space, which aims to be a private instance focused on video publishers. It’s federated with many other PeerTube instances which would be happy to have your user account, from where you can subscribe to enjoy these videos — or you can subscribe with each channel’s RSS feed.

Let’s introduce our first two creators:

Matthew is an FPGA (field programmable gate array) enthusiast who is making use of the manim open-source framework for educational video creation designed and made famous by YouTube channel 3blue1brown. Matthew’s channel will become a great resource on learning about FPGA’s from the ground up to the expert level.

Subscribe to Matthew’s channel ➔

Tom is a passionate cook who wants to use the platform to share videos teaching viewers how to cook traditional meals with cheap and accessible ingredients. Check out his first video on flatbread here:

Subscribe to Tom’s channel ➔

We’re working with a third creator as well (or creators — it’s a two-person team), who should have their first video prepared in the foreseeable future. We’re planning to start accepting applications for the second batch of creators soon; keep an eye on this blog for updates. Enjoy the videos, and subscribe to these creators!

SourceHut contributor spotlight

Wed, 26 Aug 2020 00:00:00 +0000

SourceHut is 100% free and open source software, and we accept contributions from any of our users. Dozens of your peers have contributed improvements to the services! I wanted to share a “thank you” with everyone who’s helped out, and draw attention to some of my favorite contributions.

From feature development, to bug fixes, to third-party packaging, to operations improvements and more, external contributors have left their impact in all facets of the software. Tools sr.ht users rely on every day were implemented by their peers, and the reliability and performance of the service has been guaranteed thanks in no small part to their hard work.

Ivan Habunek’s improvements todo.sr.ht

First, I’d like to give a shoutout to Ivan, who has the longest history of consistent high-quality contributions to the project, dating to the pre-alpha phase. We can thank Ivan for a lot of todo.sr.ht features, including:

Ticket labels
Ticket assignees
Mentioning & linking to users and other tickets
Improved search on todo.sr.ht and across the services

Ivan is also one of the “bus factor” delegates, and has access to production todo.sr.ht systems in order to mitigate the risk a single point of failure. Thank you for all of your help, Ivan!

наб’s improvements to everything

наб is a recent contributor to SourceHut, but over the past month she has become extremely prolific, submitting 73 patches and counting. In fact, that very link showing all of наб’s patches relies on a search feature submitted by none other than наб herself. Some other nice improvements include:

Blame and per-file log views on git.sr.ht
Default branches other than “master” on git.sr.ht
README files in markup languages other than Markdown on hg, git, and hub

Not to mention dozens of bugfixes, refactorings, refinements, and small improvements implemented for every sr.ht service. Thanks a lot, наб!

Distribution packagers Eli and Denis

sr.ht upstream runs on Alpine Linux, and we maintain an Alpine package repository for this purpose, and encourage third party installations to utilize it. However, thanks to the hard work of Eli Schwartz and Denis Laxalde, there are official Arch Linux and Debian repositories available as well. These are automatically kept up-to-date in stride with our upstream developments through some integrations with builds.sr.ht, all of which is maintained by volunteers. Thanks for your hard work, fellas!

Build image maintainers

builds.sr.ht supports a large variety of build images, from Ubuntu to Debian, Arch Linux to NixOS, FreeBSD and OpenBSD, and even Plan 9. Maintaining these is a lot of work and requires expertise with the system in question - and for most of the images, this expertise and maintenance is provided by community volunteers. Big thanks to Timothée Floure, Simon Ser, Francesco Gazzetta, and Jarkko Oranen for working hard to make sure that their platforms are well-represented on SourceHut.

Operations contributors Phillip and Ignas

Our monitoring software is available to the public, and we keep our alarms in a git repository. You’d think that our internal service monitoring would be an unlikely candidate for third-party contributions, but Phillip Riegger and Ignas Kiela both took it upon themselves to contribute improvements to our monitoring system. In fact, after sr.ht’s only unplanned outage in 2020, Ignas was the one who wrote the patch adding an alarm which would have caught the problem earlier. Thanks for your help, both of you!

Improvements to the broader FOSS community

SourceHut users, contributors, and developers have collectively made a great habit of reaching out to the rest of the FOSS ecosystem, writing patches and getting improvements merged which helps tie together the whole ecosystem into a more robust and cohesive whole. SourceHut users have driven improvements to git & Mercurial, pygit2 & libgit2, Go, Rust, and Python, emacs, vim, and VSCode, a half-dozen Linux distributions, FreeBSD and OpenBSD, and even competing services including GitHub, GitLab, Gitea, and Pagure, and dozens more projects still. I hope we will continue to boldly go forth and enrich the broader community!

And all the little ones…

There are dozens more contributors who didn’t make the article. There’s a consistent trickle of one- or two-off contributions, fixing small bugs, improving docs, and scratching itches. I’m grateful for these as well! Thank you to everyone who contributes. Large or small, your hard work is appreciated.

What's cooking on Sourcehut? August 2020

Sun, 16 Aug 2020 00:00:00 +0000

Another month passes and we find ourselves writing (or reading) this status update on a quiet, rainy Sunday morning. Today our userbase numbers 16,683 members strong, up 580 from last month. Please extend a kind welcome to our new colleagues! Thanks for reading these posts and keeping up with what’s new on the platform.

First, I’d like to extend a thanks to contributor наб, who has been sending an impressive number of patches lately, and is responsible for most of the features detailed in this update. Other contributors who deserve thanks this month include 2xsaiko, Amin Bandali, Chris Vittal, gildarts, and Peter Sanchez. Thanks, everyone!

For my part, I’ve been focused on operational work this month. I oversaw the installation of and migration to the new hg.sr.ht server, which should enjoy improved performance and support a larger number of repositories going forward. The same server is also going to be useful as a testbed for our new standard VM server setup, which will scale better to long-term high-availability plans. Additionally, our PostgreSQL robustness deployment has been overhauled, which should lend improvements both to integrity and availability.

git.sr.ht

git.sr.ht now supports default branches other than master. For details on changing your default branch name, consult the docs.

Many additional minor bug fixes and performance improvements have been made throughout. For example, non-UTF-8 branch & tag names are handled better now, with additional improvements heading to pygit2 upstream.

builds.sr.ht

The Ubuntu images have been overhauled to more closely map to the upstream release schedule. Consult the compatibility page for details. Affected users were emailed before the update rolled out; if you did not receive an email then it is unlikely that you will have to change your build manifests.

todo.sr.ht

A number of improvements have been made for logged-out/anonymous users, mostly minor changes. One larger change is that users without accounts may now subscribe to specific tickets by email.

lists.sr.ht

Searching by username is now supported in the From, To, and Cc fields, via the from:~username search key format. Additionally, git patches prepared with git send-email --rfc are now correctly recognized as such.

project hub

When the hub submits CI jobs to builds.sr.ht for patches in mailing lists, a builds.sr.ht job group is now utilized. This reduces the number of emails sent in response to one, even for repos with many build manifests.

Additionally, a number of small improvements have been made to increase the visibility of featured projects.

meta.sr.ht

For users who run their own instance of meta.sr.ht, PAM support has been added for authentication, allowing you to use the Unix account database for usernames and passwords rather than storing them in SQL.

Sourcehut Q2 2020 Financial report

Fri, 17 Jul 2020 00:00:00 +0000

In summary, SourceHut is financially healthy. Our revenue continues to grow and we are profitable and becoming more so. We have also been able to re-invest some of our profits into the broader free software and free culture community.

Disclaimer: this report is a summarized approximation of our financials, and is not used for tax purposes.

Revenue sources

Sourcehut receives revenue from paid user subscriptions. During Q1, we processed 1,855 invoices. The invoices paid break down as:

930     $2  (paid monthly)
433     $5  (paid monthly)
148     $10 (paid monthly)
246     $2  (paid yearly)
74      $5  (paid yearly)
24      $10 (paid yearly)

The total gross revenue during Q2 was $16,335, which after transaction fees is $15,204. This is basically identical to the revenue from Q1, however, the breakdown in revenue sources has changed. We saw growing subscriptions in almost every category — except for $10/year, which shrunk by 30%. This was unexpected, but we are not especially concerned, as it was made up for with growth in other areas and is unlikely to indicate a long-term trend. In any case, this is a 289% increase from Q2 2019, which does continue to support a trend of strong YoY growth.

In Q2 2,551 users registered for an account, of which 302 have paid (11.8%). This matches closely with our overall conversion rate of 11.7% (1,888 of 16,140 users at the time of writing). As of July 2020, the breakdown in subscription levels is the following:

342   $2  (paid monthly)
156   $5  (paid monthly)
 53   $10 (paid monthly)
916   $2  (paid yearly)
307   $5  (paid yearly)
114   $10 (paid yearly)

The monthly revenue from these subscriptions (with annual payments amortized) is approximately $5,544 after transaction fees, an increase in $895 over Q1. This is consistent with our historical rate of growth.

At the end of Q2, SourceHut had $17,383 in the bank.

Expenses

Breakdown of Q2 expenses:

$5000    Set aside for PeerTube initiative
$1950    Philadelphia datacenter lease (inc. network, power, etc)
€1000    Donation to Framasoft
$1253    Misc. server equipment
$600     LWN advertisement
$300     GitHub sponsorship of pygit2
$80      Domain name registrations and renewals

Our expenses were unusually high during this period, as we decided to re-invest in the community in several respects. The PeerTube bootstrap fund is a clear outlier, and we also donated to the PeerTube developers. Additionally, we’ve set up a recurring donation with the maintainer of pygit2, an important library that we depend on to provide git.sr.ht service. We may be interested in sponsorships for other projects in our dependency tree; please reach out if you are interested.

As an experiment, we also spent $600 on an LWN advertisement campaign during this quarter. The following advertisement was shown on LWN.net:

We received 50,000 impressions and 113 click-throughs (0.22%). This is considered fairly typical for this kind of advertisement. The results were not strong enough to consider a second campaign for the time being; our other marketing approaches are more effective.

In Q3, we plan on spending around $4K-$5K on the installation of a new server to handle hg.sr.ht growth, as well as filling a few other roles, mainly for high availability. Otherwise we have no unusual expenses planned.

What's cooking on Sourcehut? July 2020

Wed, 15 Jul 2020 00:00:00 +0000

As the unfeeling hand of time drags us relentlessly through its cold maw, we occasionally spend some of our precious moments writing (or in your case, reading), monthly status updates on the progress of the SourceHut alpha. Today is such a day - welcome back! Our audience has grown by 645 members this month, bringing our total userbase to 16,103. Please give a hearty welcome to our new peers!

General news

A short outage for hg.sr.ht is planned to take place sometime in the next few weeks, as we migrate to a new box. The plan is outlined on sr.ht-dev, and we expect to experience anywhere from 5 to 30 minutes where hg.sr.ht is read-only. The final dates will be posted on status.sr.ht and announced on sr.ht-announce when the plan is finalized.

We have also migrated all services to a new markdown engine based on Mistletoe. If you encounter any strange Markdown-related problems, please file a ticket and mention ~araspik. Thank you for taking on this work, ~araspik! It took several months of effort to carefully test everything.

I have also completed a lot of system operations work in this past month. We have planned out improvements to our PostgreSQL strategy, which will enable us to have real-time backups and failover capability. This is mostly useful for limiting downtime during planned maintanenace, but is also useful for the long-term goal of high availability.

lists.sr.ht

The most exciting development in this round of updates is likely to be the integration between lists.sr.ht and builds.sr.ht to bring continuous integration to mailing lists on SourceHut. You may have seen the announcement on the blog yesterday — give it a read. The tl;dr is that if you create a project on the project hub, and add a mailing list and git repositories, it’ll automatically submit any patches sent to your mailing list using the .build.yml (or .build/*.yml) from the repository. The subject prefix (for example, “[PATCH my-project]”) is used to identify the git repository that the patch is intended for (in this case, “my-project”).

Additionally, at the request of Elias Naur, I have added a button to each patchset and thread in the mailing list archives which will forward the whole thread to you, so you can reply and comment on patches from the comfort of your inbox even if you were not subscribed to the mailing list at the time.

Finally, mailing list deletion has been implemented at long last.

Project hub

Following the addition of mailing list deletion to lists.sr.ht, the project hub was updated to support removing mailing lists and, if you so desire, deleting entire projects.

builds.sr.ht

Routine build image updates:

NixOS images have received robustness updates (thanks Francesco Gazzetta!)
Ubuntu 18.10 and 19.04 has been removed, following their deprecation upstream
Ubuntu 20.10 (Groovy) has been added as ubuntu/next
Alpine 3.8 has been removed, following its deprecation upstream
FreeBSD 11.x has been upgraded to FreeBSD 11.4

SourceHut adds continuous integration for mailing lists

Tue, 14 Jul 2020 00:00:00 +0000

SourceHut offers a continuous integration platform called builds.sr.ht, which will boot up a virtual machine from any of nine operating systems¹, including various Linux, BSD, and Plan 9 distributions, using KVM, then clone your source code repository (or repositories) and run user-defined tasks to validate the code, automate deployment, run linters, and so on. A simple example manifest from one of my projects is the following:

image: alpine/edge
sources:
- https://git.sr.ht/~sircmpwn/scdoc
tasks:
- build: |
    cd scdoc
    make
- check: |
    cd scdoc
    make check

For some time now, we’ve automatically submitted these build manifests on your behalf when you push to your Git or Mercurial repository. To set this up for your repos, consult the tutorial and the documentation.

SourceHut also famously endorses the email-driven workflow that git was designed for, with tools like git send-email. This allows contributors to prepare patches locally, then email them to your project’s mailing list. Now, after the introduction of the project hub, we are well positioned to connect the disparate resources of your project — repositories, mailing lists, bug trackers, and so on — to bring out more functionality, and as of this week, that includes the automated testing of patches sent to your mailing lists.

This is already working now for all projects on the project hub — to configure yours, just create a project and add both your source code repository and your mailing list to the project, and it’ll take care of the rest automatically.

There are some limitations, however:

Patches have to have their subject prefix configured to match the name of the repository being tested. For example, “[PATCH example v2]” will be matched to a repository named “example”. Senders can configure this like so:
```
git config format.subjectPrefix 'PATCH example'
```
This is done automatically for patches submitted using the git.sr.ht patchset preparation interface. We plan on improving this later by testing the patch for applicability against each of the repositories on your project.
Job groups remain unimplemented on builds.sr.ht, which means that projects with many build manifests may see a lot of noise from the CI.
Mercurial repositories are not yet supported.

Leveraging the project hub to connect your resources together is something which we intend to do in a few other domains as well. For example:

Marking patchsets as integrated when they’re applied upstream
Linking git commits to tickets, and vice versa
Adding the patch submitter to the build job access list so that they can SSH into failed VMs to diagnose the problem without the project admin’s intervention

And of course, this will play nicely into our plans of a more sophisticated maintainer-side workflow for handling patch submission on the web. Enjoy!

At the time of writing, at least. We’re working on adding more. ↩︎

How we monitor our services

Fri, 03 Jul 2020 00:00:00 +0000

Monitoring all of our services makes sure we’re aware of problems when they occur, but most importantly, it helps us detect problems in advance — before they become outages. Our main tool for this task is Prometheus, an open source time-series database. It takes a snapshot of various metrics across all of our services every few seconds, then allows you to write queries which model trends in that data. Our instance is publicly available for you to explore at metrics.sr.ht.

The Prometheus docs have some good tips for writing queries. Here’s an example:

rate(node_cpu_seconds_total{mode="user",instance="node.git.sr.ht:80"}[2m])

This returns a value between 0 and 1 which represents the current CPU usage across all nodes on git.sr.ht. You can check out live graph here (note that the live graph automatically selects the scale appropriate for the data — our server is probably not as overloaded as it looks here), or take a look at this graph I generated in advance (with the correct scale):

We can easily turn this into an alarm by rephrasing it as a boolean expression and writing a little bit of YAML:

- alert: High CPU usage
  expr: &cpu_gt_75pct rate(node_cpu_seconds_total{mode="user"}[2m]) > 0.75
  << : &brief
     for: 5m
     labels:
       severity: interesting
  annotations:
    summary: "Instance {{ $labels.instance }} is under high CPU usage"

This is one of the few times where I take advantage of some of YAML’s more advanced features: &cpu_gt_75pct and &brief are a kind of macro, which allows me to easily create derivative alarms, like these ones for network activity:

- alert: High network activity
  expr: &net_gt_10mibsec >
    (rate(node_network_receive_bytes_total{device=~"eth0|ens3|enp.*"}[5m]) / 1024^2
    > 10) or (
    rate(node_network_transmit_bytes_total{device=~"eth0|ens3|enp.*"}[5m]) / 1024^2
    > 10)
  << : *brief
  annotations:
    summary: "Instance {{ $labels.instance }} >10 MiB/s network use"
- alert: Sustained high network activity
  expr: *net_gt_10mibsec
  << : *sustained
  annotations:
    summary: "Instance {{ $labels.instance }} sustained >10 MiB/s network use"
- alert: Prolonged high network activity
  expr: *net_gt_10mibsec
  << : *prolonged
  annotations:
    summary: "Instance {{ $labels.instance }} prolonged >10 MiB/s network use"

You can see all of our alarms online on metrics.sr.ht, and you can see our YAML files in our git repository. Some of our users have helpfully sent us patches to add new alarms and refine our existing ones - thanks folks! The advantages of being an open source business are numerous.

When these alarms go off, they’re routed into Alertmanager. I found this particular part of the Prometheus stack a bit more annoying than the others, but eventually we rigged it up the way I like. We combine sachet with Twilio to route urgent alarms to my cell phone, and there I have some rules set up with Automate¹ to make my phone go nuts when a matching SMS message arrives - it starts beeping and vibrating until I acknowledge the message.

“Urgent” and “important” alarms are routed into our operational mailing list, sr.ht-ops, which is also public. Browsing this list, you’ll also see some automated emails which are not related to alarms — I’ll get to these in a moment. We also have “interesting” alarms, which are routed to IRC, along with the urgent and important alarms, using alertmanager-irc-relay. This IRC channel is also public: #sr.ht.ops on irc.freenode.net.

Prometheus normally works by pulling stats off of daemons, but we also want to monitor ephemeral or one-off tasks. For this purpose, we have also set up Pushgateway. We mainly use this to keep track of timestamps: the timestamp of the last backup, the last ZFS snapshot, or the expiration dates of our SSL certificates. For example, we can get the age of our ZFS snapshots in seconds with this query:

time() - zfs_last_snapshot

This is trivially populated with a simple cronjob:

#!/bin/sh -eu
host="$(hostname -f)"
stats() {
	printf '# TYPE zfs_last_snapshot gauge\n'
	printf '# HELP zfs_last_snapshot Unix timestamp of last ZFS snapshot\n'
	ts=$(zfs list -HS creation -o creation -t snapshot | head -n1)
	printf 'zfs_last_snapshot{instance="%s"} %d\n' "$host" "$ts"
}

stats | curl --data-binary @- https://push.metrics.sr.ht/metrics/job/"$host"

Our backup age is filled in at the time the backup is taken with something like this:

#!/bin/sh -eu
export BORG_REPO=#...
export BORG_PASSPHRASE=#...

backup_start="$(date -u +'%s')"

echo "borg create"
borg create \
	::git.sr.ht-"$(date +"%Y-%m-%d_%H:%M")" \
	/var/lib/git \
	#...

echo "borg prune"
borg prune \
	--keep-hourly 48 \
	--keep-daily 60 \
	--keep-weekly 24 \
	--info --stats

stats() {
	backup_end="$(date -u +'%s')"
	printf '# TYPE last_backup gauge\n'
	printf '# HELP last_backup Unix timestamp of last backup\n'
	printf 'last_backup{instance="git.sr.ht"} %d\n' "$backup_end"
	printf '# TYPE backup_duration gauge\n'
	printf '# HELP backup_duration Number of seconds most recent backup took to complete\n'
	printf 'backup_duration{instance="git.sr.ht"} %d\n' "$((backup_end-backup_start))"
}

stats | curl --data-binary @- https://push.metrics.sr.ht/metrics/job/git.sr.ht

In addition to alarming if the age gets too large, the duration allows us to graph trends in the backup cost so that we can make informed judgements about the backup schedule:

Backups are of critical importance in our ops plan, so we have redundant monitoring for them. This is where the other emails you saw on sr.ht-ops come into play. We configured ZFS zed to send emails to the mailing list on ZFS events, and left debug mode on so that it’s emailing us even when there are no errors. We set up a cronjob to run a scrub on the 1st of the month, and the resulting zed emails are posted to the ops list. We also do weekly borg backup checks, which are posted to the mailing list every Sunday night.

Reliability is important to us. SourceHut is still an alpha, but a robust approach to reliability is considered a blocker for the production phase, and taking the development of this approach seriously during the alpha and beta periods. You can read up more on our ops work at our public operations manual. And it’s working — we have 99.99% uptime for 2020, and have only suffered one unplanned partial outage this year.² Let’s aim for 99.999% in 2021!

Unfortunately, this and Twilio are the only parts of the stack which are not open source. Replacing Twilio would be hard but I bet Automate is low-hanging fruit - [email protected]">suggestions would be welcome! ↩︎
Two pages were affected, and it lasted 46 minutes. We fixed the problem, prevented future problems of the sort, and added new alarms which would catch a broader net of problems in the future. Read the details here. ↩︎

What's cooking on Sourcehut? June 2020

Mon, 15 Jun 2020 00:00:00 +0000

Hello! With May behind us and July looming, we have another month of progress on the alpha in our cap. Today we’re joined by 695 new users, bringing our total strength to 15,458 brave hackers. Give our new peers a warm welcome, please!

General news

Please be advised that today, June 15th, starting at 18:00 UTC, we have a planned outage. Services are expected to be intermittently available for a period of up to an hour. Each outage is expected to be limited to one service at a time, and should last no more than 15 to 30 minutes each. Consult the status page for more information.

I’d also like to thank Eli Schwartz once again, for stepping up as the new official maintainer of the SourceHut Arch Linux package distribution. He will be handling the maintenance of the Arch Linux repository going forward; please direct your inquiries and patches to him. Thanks Eli!

meta.sr.ht

A new GraphQL API for meta.sr.ht is now available. Play with it here! As promised in the last update, the GraphQL playground is now JavaScript-free. I am also working on an improved means of API authentication, with finer-grained access controls and more compatibility with industry standards like OAuth2.

If you’re curious to learn more about the application of GraphQL to SourceHut, I wrote a blog post on the subject last week.

builds.sr.ht

The manifest submission page has replaced ace.js with CodeMirror for the JavaScript-enabled build manifest editor. This should work better on mobile devices and feels a bit more native to use (because it is a bit more native). As before, users with JavaScript disabled will still be able to use a plain textarea to submit build manifests.

alpine/3.12 is now available
openbsd/6.7 is now available
openbsd/6.5 has been removed

dispatch.sr.ht

Builds submitted for GitHub pull requests now include the GITHUB_PR_BODY in the environment, thanks to Paul Spooren’s patches for OpenWRT. Thanks Paul!

todo.sr.ht

Ivan Habunek has implemented support for previewing the rendered Markdown for ticket descriptions and comments, and it was further refined by René Wagner. Gabriel Augendre also rigged it up so that you’re copied on tickets submitted by email, so you get the ticket number that was assigned sent back to you, and Benjamin Lowry submitted several bug fixes. Thanks, everyone!

git.sr.ht

The GraphQL API has grown support tag artifacts, and learned many bug fixes. Thanks for helping to test this!

How and why GraphQL will influence the SourceHut alpha

Wed, 10 Jun 2020 00:00:00 +0000

Often I will remind users that SourceHut is an “alpha-quality product”, even though many people are already using it today and find it well-suited to their present needs. What I mean by this is to remind you that, while many things already work, I am still working on refining the implementation before we commit to the design going forward. This foundational work is more important than many high-level features, because it will give us the basis from which the broader SourceHut platform is built and extended upon well into the future.

Almost all of SourceHut today is built with Python, Flask, and SQLAlchemy, which is great for quickly building a working prototype. This has been an effective approach to building a “good” service and understanding the constraints of the problem space. However, it’s become clear to me that this approach isn’t going to cut it in the long term, where the goal is not just “good”, but “excellent”. The system would become more stable with the benefit of static typing, and more scalable with a faster and lighter-weight implementation. My work on SourceHut has, on the whole, really soured my opinion of Python as a serious language for large projects.

One of the core requirements for the alpha to graduate to “beta” is the implementation of complete APIs throughout the services, both for users to access and for the services to use to communicate with each other. The lackluster state of these APIs was the main motivator for some of the large-scale changes I have planned. Originally I choose to use REST for the API, because REST is boring and well-understood. Boring, well-understood approaches are the bread and butter of SourceHut. However, I have been very unsatisfied with the results, and have been unwilling to take this design forward into the beta. During the alpha is the best time to reconsider this approach and build a more sustainable API design.

Enter GraphQL. I have considered GraphQL a few times in the past, but choose not to research it further for a number of reasons:

The quality of server implementations has been rather poor on each of my research attempts, especially outside of JavaScript implementations.
The ecosystem is heavily influenced by a web development culture which is flatly rejected by SourceHut’s ethos. This leaks into the documentation and resources available for GraphQL, which makes it more difficult to evaluate through the lens of SourceHut’s more conservative values.
GraphQL does not solve many of the problems I would have hoped it would solve. It does not represent, in my opinion, the ultimate answer to the question of how we build a good web API. Therefore, someday there may come a better technology to supplant it, which makes me hesitate to base a long-term system design on it.
The uptake of GraphQL in the general web ecosystem has been somewhat slow. The value-add is difficult to understand and even more difficult to implement well, making it difficult for all but senior engineering teams to understand.

However, with my dissatisfaction with the REST approach reaching its peak in the past few months, I gave it a more serious and in-depth review. I built a research GraphQL API for git.sr.ht, trudging through the docs and specs with the aim of understanding it from the SourceHut perspective and how it could apply to our outlook on software design. I also found (or designed) solutions to some of the limitations of GraphQL which make me more confident that we can build a future-proof solution which is not going to feel outdated or bad when The Next Thing comes along. And, importantly, a new server-side implementation called gqlgen has appeared since my last attempt, which, although imperfect, is good enough for me to consider it seriously as the basis for a large system.

Another (potential) advantage of GraphQL is the ability to compose many different APIs into a single, federated GraphQL schema. If this pans out, it would be very convenient to access all of the distributed SourceHut services as a single cohesive system, trivially composing each service or subset of services as appropriate to the particular SourceHut deployment in question. Another area of research which may be improved by GraphQL are webhooks: I want to experiment with registering your webhooks with a GraphQL query, so that the webhook payload can contain all of the information you need to process the event without any additional API requests.

In short, I’ve concluded that GraphQL, while imperfect, is a significant advance in the state of the art from REST, and is mature and sophisticated enough to be useful as the basis of an API and an implementation that I can have long-term confidence in. This is important to get right, because the distributed design of SourceHut’s mini-services demands high-quality communication between each component. I cannot feel comfortable moving forward from the alpha without this goal being met, and GraphQL makes me more confident in achieving it well.

I also mentioned earlier that I am unsatisfied with the Python/Flask/SQLAlchemy design that underlies most of SourceHut’s implementation. The performance characteristics of this design are rather poor, and I have limited options for improvement. The reliability is also not something I am especially confident in. The GraphQL work was not expected to solve this problem, but it may offer an unexpected solution.

The GraphQL services are completely standalone, and it is possible to deploy them independently of the web application. With these, you can deploy a SourceHut instance with no frontend at all, using the GraphQL APIs exclusively. Today, the Python backends to the web services communicate directly with PostgreSQL via SQLAlchemy, but it is my intention to build out experimental replacement backends which are routed through GraphQL instead. This way, the much more performant and robust GraphQL backends become the single source of truth for all information in SourceHut. This could substantially increase my confidence in the system if it pans out — the Python footprint would become much smaller and simpler, and bugs in it would have less risk of creating inconsistent states in the system.

These changes can be and will be made incrementally, and as a user, you should notice no changes other than improved performance and reliability, and access to better APIs. It is my intention to carry forward user data and continue to support existing projects throughout with no significant service disruptions. The performance and reliability of SourceHut today is already best in class, but I see no reason why we can’t push the envelope even further. I also think that five nines is an achievable goal.

At the time of writing, there are two GraphQL APIs available for you to play with: meta.sr.ht and git.sr.ht. These are still considered experimental, and some larger concerns such as write access and improvements to authentication remain to be implemented. Give it a shot and please share your feedback!

Achieving accessibility through simplicity

Wed, 27 May 2020 00:00:00 +0000

I have received many emails complimenting SourceHut’s simple design and lightweight pages¹, but I have received a surprising amount of positive feedback from a particular group of users: the blind community.

For many software teams, especially web developers, accessibility is an extremely burdensome task. Many companies have written checks with an uncomfortable number of zeroes on them to get the job done. Sprinkling ARIA tags all over your DOM to annotate elements with their purpose, and updating these as your DOM changes over time, is no small task, and makes your code more difficult to maintain. It’s not especially surprising that many blind users are constantly frustrated when trying to use the web.

How did SourceHut make such an accessible website, even in the face of these trials? In truth, even though accessibility is important to me, I haven’t worked especially hard to make SourceHut accessible — I have devoted no more than three or four hours, in total, explicitly towards accessibility. Instead, SourceHut is accessible because the core principles of simplicity upon which it is built naturally lead to an accessible design. We’re here today to share our thoughts so that you might apply the same principles to your own website.

First, consider the audience you’re designing for. Your users have a diverse set of abilities, and a varied approach to accessibility is necessarily required to accommodate for them. Different users will have a different experience with your website. There are different levels of visual impairment — near- and far-sighted users, colorblind users, and many degrees of vision impairment ahead of totally blind. Talking about accessibility might bring forth thoughts of the latter group, but a lot of users with partial sight would also benefit if you put care into your visual design. There’s also other considerations than vision — for example, avoid high precision mouse actions for users with limited mobility, and add subtitles to videos for the deaf and hard of hearing.

For general advice on web accessibility, I would start with the following: “trust the web browser”. Leave the page at its default font size and avoid using custom fonts, preferring to use vague selections like “sans-serif” and “monospace”. Don’t mess about with the scroll wheel, don’t override default behaviors on the right click and text selection. Don’t use JavaScript to create custom input elements like text boxes, combo boxes, or scrollbars. The web browser will do all of these things for you — trust it to do a good job.

Remember that the browser is the user agent, not the developer agent. By trusting its defaults, you leave room for users to customize them, choosing a larger text size, a different font, and so on. The user is already comfortable with the way their browser works, and you will fail to capture the subtle nuances of their user agent with your pretty imitations. When accurately using the mouse is a struggle, a user who has gotten used to using the arrow keys in a <select> box is faced with an unreasonable challenge when they encounter a custom drop-down that hasn’t implemented this behavior quite right.

Another case to be careful of is the use of color, contrast, and images. Always make sure that text the user is expected to read is sufficiently distinguishable from its background.² Also avoid putting text over a variable background, such as a gradient or tiled background. Be conservative with your use of color — limit bright, visually attractive colors to one or two actionable items on the page, or use them to draw special attention to timely notices, or to warn the user of potentially dangerous actions like deleting data.

Try not to use a purely visual representation of information, such as an icon: these should always be paired with text. Such explanatory text shouldn’t require, for example, hovering to see it — a task which requires high mobility to hit a small target with the mouse, and holding it steady for long enough to make a tooltip appear. Also avoid moving information around — animations and visually complex state transitions. When adding images, always include an “alt” tag with a plain-English description of the image.

Do this exercise with me: cross your eyes, then close one eye. Through this blurry view, can you still identify the major elements and action items on your page? Does anything demand too much or too little attention?

The “alt” tag reference a moment ago is the first time we’ve touched upon many of the conventionally repeated wisdoms about “accessibility” on the web, often included because its value is intuitive and easy to implement. It’s no mistake that we’ve covered little of such conventional wisdom so far: the typical user who benefits from an accessible website is not completely blind, using a screen reader and taking advantage of your ARIA tags. The hard truth is that you just have to make your website simpler and easier to use — for everyone.

But, it wouldn’t do to forget the users with screen readers in any case. Some general advice for such users would be to make good use of semantic HTML, such as <main>, <article>, <section>, <nav>, and so on; along with other elements like <p> to mark paragraphs, <ul> and <ol> as appropriate to mark lists of things, <blockquote> for, well, quotes, and so on. Screen readers can interpret these to understand the layout of your page better and provide contextual clues to the user.

Prefer to organize your site’s information logically, rather than spatially. A logical hierarchy of information is more intuitive if you have less visual awareness. Make sure your page is organized so that if read linearly, from start to finish, without CSS, it still makes sense. Avoid littering marketing garbage, superlatives, and ads for other parts of the site (or even ads outright) throughout your page, as skipping these is more difficult for a screen reader than for the typical visitor. A good way of simulating the screen reader experience is to view your page with Lynx. Conveniently, Lynx does not support JavaScript. 😉

This seems like a lot of effort to go to for accessibility! But, if you read closely, this is mostly a list of things to avoid doing. The SourceHut approach side-steps many of these problems by focusing on simplicity. The JavaScript-free interface is more accessible right out of the gate, and by not cramming too much into a single page it’s much easier to flow and organize in an accessible way. This advice also leaks quite generally into the broader subject of usability, which is no mistake: investing in accessibility makes your service better for everyone.

SourceHut also benefits from some design choices which extend beyond the web. For example, by focusing on email for discussions, patches, and code review, and insisting on plain text, many blind users can choose to completely side-step the web interface and interact with the services and users on it from the comfort of their mail client, using only Plain Jane Text emails. Then, when they have questions, they can join our IRC chat room, a medium used exclusively to exchange short, plain-text messages, without the nightmare of navigating the Slack interface with a screen reader.

In summary, if you want to get accessible quick, a good start for your new website might eschew npm install in favor of this:

<!doctype html>
<html lang="en">
<meta charset="utf-8" />
<title>My cool website!</title>
<main>
  <h1>My cool website</h1>
  <p>Welcome to my cool website!
</main>

Which, in case you were unaware, is a completely valid HTML 5 document!

And, to be fair, a handful of emails to the contrary, too. ↩︎
The Lighthouse tests built into Chrome-derived web browsers can be used to help identify UI elements with too-low contrast. ↩︎

The PeerTube content bootstrap fund

Fri, 15 May 2020 00:00:00 +0000

PeerTube is an excellent project which aims to create an open and decentralized video “tube” for hosting video content with distributed governance and community-driven development. Chocobozzz, with the help of dozens of contributors, has put in the hard engineering work to build a great platform, and now we want to see it filled with content.

Notice: the first round of applicants has been processed. Check back in later - we may ask for more applications in the future.

To help make this happen, SourceHut is committing $5,000 to bootstrap content on the PeerTube network. This money will be used to help new content creators start making videos for PeerTube. We’ll pay for some equipment to get you started: cameras, microphones, video capture cards, and so on, whatever is appropriate for the kind of videos you intend to make, plus a $200 budget to spend at your discretion on the first few videos. Need a nice camera? A couple of nice microphones for conducting interviews? A drone for capturing footage? Let’s make it happen. You’ll also be set up on Liberapay so that your audience can help you build a sustainable lifestyle.

If you have an interesting hobby to share, in-depth knowledge on a subject to teach, or a talent you’d like to show off, or a great idea we haven’t thought of, please send an email to [email protected]">[email protected] with the subject line “PeerTube bootstrap application: <your name>”. Include a short summary of who you are and why you want to apply, and details on what kinds of videos you’d like to make. Tell us why open culture is important to you.

The following conditions apply:

You must not already be regularly publishing content on another platform. We’re looking to fund new content creators, not existing ones.¹
You can only upload videos to PeerTube - not to YouTube or anywhere else.
Your videos must use a Creative Commons license, like CC-BY-NC-SA.
We expect you to make at least 5 videos. If you decide video creation isn’t for you after that, we’ll take the equipment back and you’ll be off the hook.
Once you start earning at least $20/month on Liberapay, we’ll split the ongoing cost of hosting evenly among all creators: currently $45/mo. You will never be expected to pay more than 25% of your Liberapay income; SourceHut will cover the rest.

Our instance is at spacepub.space. We’ll handle the technical administration of the server and set you up with the resources you need to get started, and together we’ll create a community of open-culture creators who support one another to produce a vibrant commons. Let’s help make PeerTube a great platform!

If you’ve already made a few videos on the side, that’s fine - be sure to let us know in your application, and we’ll exercise discretion. ↩︎

What's cooking on Sourcehut? May 2020

Fri, 15 May 2020 00:00:00 +0000

Greetings! Another month of alpha development progresses and brings us that much closer to the SourceHut beta. We reached a few important milestones this month! Today we are 14,763 users strong, with 1,104 new users joining us this month. Please give them a warm welcome!

General news

Quick thanks to Eli Schwartz for patches across the board which simplify the installation process and improve support for a broader variety of distributions. Thanks also to Daniel Gröber and Thorben Günther for additional improvements in the installation process and self-hosted SourceHut experience.

hub.sr.ht

As you likely are already aware, the SourceHut project hub was opened this month. Since its opening, users have added 215 of their projects to the public index. If you want to add your project, create a project and complete the new project checklist to be added to the index. And, be sure to check out (and contribute to!) the cool projects other users have made! If you want to know more about the project hub and the future plans for it, read up on the original announcement.

git.sr.ht

The experimental GraphQL API I’ve alluded to in previous “what’s cooking” posts is now available for users to play around with. It’s still subject to change, but please take a look and share any feedback you have. I have deployed the GraphQL Playground at git.sr.ht/graphql for you to play with. Unfortunately, this is a big JavaScript SPA, entirely atypical of the SourceHut web design ethos — but rest assured that before the GraphQL workstream is done, this will have been replaced with an in-house UI which better meets your expectations.

The other major improvements I have planned for GraphQL include an overhaul to authentication, and a gateway through which all sr.ht APIs can queried through a single GraphQL request. On the whole I’m pretty satisfied with the implementation, and I’m optimistic that it will provide a much better foundation for the API going forward into the beta. It should open the doors to a lot of interesting future features as well. I will be doing a write-up on GraphQL for the sourcehut.org blog in the foreseeable future, so keep your eyes peeled for that if you’re curious to learn more.

builds.sr.ht

The 9front distribution of Plan 9 is now available for user builds. This is an unusual build target; check out the release announcement to learn how it works. Other build image updates:

nixos/20.03 has been added and nixos/stable has been updated to it; nixos/19.03 has been removed
ubuntu/lts now refers to Ubuntu Focal
dockerd now starts up correctly in Alpine Linux and Debian guests; Alpine is now the recommended guest for builds utilizing Docker

Thanks to Francesco Gazzetta, Denis Laxalde, and Ace Eldeib for their patches this month.

hg.sr.ht

Ludovic Chabant has been hard at work on bugfixes and performance improvements for hg.sr.ht. The changes are subtle, but users should notice faster pushes at least.

SourceHut + Plan 9 = ❤

Mon, 11 May 2020 00:00:00 +0000

My favorite operating system is Plan 9 from Bell Labs. The simplicity and cohesive design throughout really stands out. In my opinion, Plan 9 is much, much better than any other operating system I’ve used, and I’ve used a lot of operating systems. Plan 9 is not perfect, but it’s a substantial advance to the state of the art.

Bell Labs has moved on, but 9front keeps the dream alive as the most active (and sarcastic) fork of Plan 9. In the vain hopes that more people will become interested in Plan 9, and to provide better support to the existing community, and because I was bored, I spent some time this weekend adding 9front support to builds.sr.ht, providing the world’s first continuous integration service for Plan 9.

Here’s an example build manifest for Plan 9:

image: 9front
tasks:
- build: |
    cd /
    . /sys/lib/rootstub
    cd /sys/src
    mk nuke
    mk all

The source code for 9front is installed on every system at /sys/src, so, as you may have guessed, this build manifest compiles the operating system from source. This includes the C compiler and linker, assembler, kernel, and userspace; including Python 2, several game console emulators, and of course DOOM and Quake — in 115 seconds.

Plan 9 is very different. It’s not POSIX, and not even Unix, and you can’t approach it expecting to use or even port any of the software you’re used to. Plan 9 is best appreciated on its own merits, and you should think in terms of what new things you can do with it, rather than what old things you can force into it. If you want to check it out, go into it expecting to learn — not to mold it into something familiar.

Many staples of the modern software industry trace their lineage to Plan 9. The Go programming language is a direct descendant of Plan 9, along with Linux and BSD’s /proc filesystems, user namespaces, and more. Even if you’ve never heard of Plan 9, you’ve been affected by it. Consider submitting a few builds to play around with it!

Announcing the SourceHut project hub

Wed, 29 Apr 2020 00:00:00 +0000

I’m happy to announce that the SourceHut project hub is now available for general use! This is one of the most important developments in the progress of the SourceHut alpha thus far. If you want to see how it works interactively, try checking out the SourceHut project, add your own projects, or browse the public project index. For a more wordy introduction, read on.

So why is the project hub necessary and/or interesting? SourceHut is composed of many small Unix-style tools for software development, which each “do one thing and do it well”. These tools are things like git hosting, bug tracking, mailing lists, CI, and so on. This allows you to use them independently and compose them however you please, which maps well onto the reality of how many projects are organized. Compare this to platforms like GitHub, GitLab, Gitea, and so on, where resources like bug trackers and pull requests map 1:1 to a git repository, even when it doesn’t make sense.

In reality, many projects have dozens of git repos but would prefer to receive bug reports in one place, for example. There are many ways projects are organized, and with a one-size-fits-all approach, many of them are forced to resort to meta-repositories, bots to chase down lost users posting things in the wrong place, and so on. Additionally, this insistence on a single git repo being the face of your project is very developer-centric. Only a fraction of visitors to your project actually want to browse the source code — many others are looking for documentation, bug reports, support channels, and so on; and frontloading the source code can be intimidating.

The ability to freely compose resources on SourceHut to suit your project’s needs addresses this. The primary disadvantage of our approach has been difficulty in finding out what resources belong to which projects, and getting around between them. The new project hub addresses this with a single “hub” for your project, with a more general introduction to the project and links to the each of the tools your project is using. You can have zero, one, or any number of git or Mercurial repos, mailing lists, and bug trackers, in any combination which is useful for your needs, and the project hub will organize them for you. It streamlines the workflow for new projects and will make it easier for new users to figure out how to get started on SourceHut.

The project hub also solves a problem which has long been a matter of interest to SourceHut users: discoverability. How do you find what projects are hosted on the platform? The hub solves this with the public project index. Each public project which you create on the hub is listed here, and you can search through all of them to discover interesting projects to use or contribute to. I’ve also added a list of featured projects, where I will be hand-picking interesting projects to share.

This version of the hub is the MVP, and there’s more development I want to do in the future. Some of this includes:

Adding project tags, like #python or #linux.
Linking back to the parent project from resource pages.
Handling more project resources, like wikis.
RSS feeds everywhere!

The hub also comes with smaller changes to the rest of the site — the “sourcehut” link on every page’s navigation will now take you back to the hub, and hg.sr.ht has been made a first-class target in the nav.

Please give the new service a shot and send along any feedback that comes to mind, via sr.ht-discuss or by [email protected]">emailing me directly. Enjoy!

Prioritizing simplicity improves performance and reliability

Mon, 20 Apr 2020 00:00:00 +0000

We recently put together some automated testing which evaluates the performance of various software forges without bias¹: the Software Forge Performance Index. The goal is to push our community of software forges to strive for higher performance, to tighten your workflow loop and to better service communities without access to high-speed internet or newer computer hardware. I also included results for accessibility testing, hoping that we can push each other to improve on that front as well.

These tests show that SourceHut has room for improvement. For one, some important features are conspicuously absent, like a git blame view. GitHub beats out our performance at rendering commits and tickets, and on several pages GitHub and Bitbucket both have better accessibility scores than the equivalent sr.ht page. But the overall results are clear: SourceHut is the best performing code forge, and by a wide margin. The data also shows that our approach delivers the best reliability: our friends at GitHub and GitLab have struggled with outages 15 of the last 90 days and 18 of the last 90 days, respectively. SourceHut has had two outages in 2020, both of which were planned and announced a week in advance, and only one of which affected more than one subsystem.

The secret is, as you may have guessed, to prioritize simplicity. It’s all too easy for a development team to chase after shiny features — I know because it tempts me, too. But sooner or later the chickens come home to roost, and you have a giant single-page application downloading a megabyte of JavaScript and pinning a CPU for several seconds before the page is useful (if scrolling at 3 FPS can be considered useful).

Every SourceHut page, on a cold cache, requires two HTTP requests to complete. On a warm cache, it requires only one. The typical sr.ht page is about 30 KiB. Both of these figures are ten times better (or, in some cases, one hundred times better) than the competition. Even with a cache, your web browser has to parse your JavaScript and CSS, warm up the JIT, run all of your initialization code… this stuff doesn’t come for free, especially on anything but top-of-the-line hardware — which I know many of my peers are using. There is a lot of developer attention on improving the performance of the status quo, but to earn the order-of-magnitude performance improvements SourceHut has, you need to make order-of-magnitude reductions in complexity.

This culture of simplicity extends to our backend servers, too, which is how we deliver on reliability. We have no Kubernetes here, no Docker, no CloudFormation, no monorepos. SourceHut has 10 dedicated servers and about 30 virtual machines, all of which were provisioned by hand. We prefer Alpine Linux because it’s lightweight and simple — our average production server has only 20 to 30 packages installed, which means no moving parts that we don’t understand. Our application design helps as well: many pages are designed to make only one or two round-trips to the database, the queries for which can be carefully optimized with joins and indices built to-purpose.

Our distributed “miniservice” application design also allows any of our services to fail independently without causing a total outage. And, because we embrace email for collaboration, our entire service could be offline without affecting your workflow — the email system has 50 years of battle-tested fault tolerance and redelivery mechanisms, you can keep working normally and our servers automatically get caught up when they’re back online. We achieve reliability on several levels: simple servers which don’t break unexpectedly, monitoring which alerts us to problems before they’re serious, and a design which works even if we do experience an outage.

Microsoft bought GitHub for 7 billion dollars. GitLab is valued at 3 billion dollars. Together, the two companies have over 2,000 employees. SourceHut made 4 thousand dollars in Q1 with two employees and an intern. How do we deliver on this level of reliability and performance compared to these giants? The answer is a fundamental difference in our approach and engineering ethos. I hope that forgeperf.org is a good start for pushing our industry to mount a serious attack on the eternal problems of high performance and reliability once more.

If you want to learn more about our approach to reliability, check out our public operations manual, join our #sr.ht.ops channel on irc.freenode.net to watch our operations in real-time, or query the raw monitoring data yourself from our public Prometheus instance.

I am confident that these results are accurate and without bias towards SourceHut. We run only standard benchmarks, maintained by a neutral third-party (Google Lighthouse), and the results are corrected for latency and bandwidth differences. This was verified by having the same test suite run in several locations across America and Europe, showing consistent results even with more generous routing to our competitors. You can run it, too, and verify the results for yourself — it’s pretty easy to set up the software on Linux and it only takes about an hour to produce the full report. I’m also sure that the results are not cherry-picking pages for which SourceHut has an advantage — it’s easy to contribute new pages if you think we missed a spot. ↩︎

What's cooking on Sourcehut? April 2020

Wed, 15 Apr 2020 00:00:00 +0000

Greetings! Another month passes and we progress closer still to the beta. We are joined by 464 new users, bringing our new total to 13,659. Please provide them with a warm welcome!

General News

In case you missed it, be sure to check out the Q1 Financial Report, published earlier this week. In other general news, I recently overhauled our installation instructions, bringing them up to date with changes since their original time of writing, and completing the documentation overhaul I had planned. I hope you find these useful! We were getting tired of correcting the docs all the time in IRC as people were setting up their sr.ht instances.

Additionally, throughout the site small improvements have been made: tables in Markdown have been improved, as well as relative links and embedded images from git repositories. The OAuth implementation has been brought more into compliance with RFC 6750, and there have been minor improvements for logged-out users. Small responsiveness improvements have also shipped throughout, improving the UI for mobile users.

hub.sr.ht

hub.sr.ht is going quite well, and I hope to have an initial version online for you to start using by the time I write the next edition of “What’s cooking”. I’ve summarized the current state of the project in this video:

Your web browser does not support the webm video codec. Please consider using web browsers that support free and open standards.

There still remains a significant amount of work to do, but for the most part it’s clearly defined and just needs to be executed.

git.sr.ht

After 6 months of work, five patches across 3 different projects, and several thousand lines of code, I have finished adding custom backend support to pygit2, which unblocks a number of workstreams. Accordingly, I’ve completed the first of these new workstreams by introducing a new plumbing API to git.sr.ht, which offers low-level access to git resources. The rub is that you can create a custom pygit2 backend implementation which allows you to read and manipulate git repositories like normal, but rather than storing the repository on your local filesystem, it reaches out to the git.sr.ht API to complete I/O operations remotely. This plumbing API remains undocumented for the time being, but I hope to eventually stabilize it and make it available to you for your own needs. I’m also happy to share that SourceHut has become the first sponsor of the pygit2 project

I have also begun work on a second API-related project, with git.sr.ht as the testbed. The current set of APIs offered on SourceHut have long since left their proof-of-concept stage and are now implemented broadly enough to pull their drawbacks into relief. There’s no time which will be more perfect to revisit the API design than during the alpha, when breaking changes are more tolerable. To this end, I’ve been writing an experimental GraphQL-based API for git.sr.ht in Go. My initial impressions are very optimistic — I’ll write up my thoughts on GraphQL in more detail later — and I think this would be a marked improvement over our REST-based approach today, and maps very well onto our distributed system design. If you’re curious about this work, you can see the current status on the graphql branch of git.sr.ht. The schema itself can be found here.

The approach isn’t fully defined yet — there’s a lot of experimental code throughout, and it’s changing rapidly. Writing this backend in Go also gives me the opportunity to revisit a lot of drawbacks in the Python codebase, and the result is a lot leaner and meaner than the earlier implementation. If this ends up working out, we would be wise to invest more in this architecture.

builds.sr.ht

Image updates:

freebsd/current has been removed. If you had used this image in the 30 days preceding its deprecation, you should have received an email letting you know. If you run into any additional issues, be sure to reach out on sr.ht-discuss

Some minor performance improvements to the web UI have also shipped this month.

Sourcehut Q1 2020 Financial report

Mon, 13 Apr 2020 00:00:00 +0000

In summary, SourceHut is financially healthy.

Before we get into the stuffy business stuff, let’s get the inevitable out of the way. I hope that your family and friends are safe as you read this. I’m sure that many people reading today’s financial report are facing uncertain financial times for themselves. If this is the case for you, don’t hesitate to cancel your SourceHut subscription. We will be alright — take care of yourself and your loved ones first. On the other hand, if you are secure and want to help, we would appreciate it if you bought a paid subscription.

Eventually, payment to SourceHut will become mandatory. This was expected to occur when the alpha graduated to the beta. However, if we find ourselves amend a continued global crisis as the software matures to beta, we will postpone making payments mandatory. And, as always, anyone with extenuating circumstances is not expected to pay, even when it becomes “mandatory”.

Stay safe, and thank you for your continued support.

Disclaimer: this report is a summarized approximation of our financials, and is not used for tax purposes.

Revenue sources

Sourcehut receives revenue from paid user subscriptions. During Q1, we processed 1,673 invoices. The invoices paid break down as:

823     $2  (paid monthly)
382     $5  (paid monthly)
135     $10 (paid monthly)
228     $2  (paid yearly)
71      $5  (paid yearly)
34      $10 (paid yearly)

The total gross revenue during this period was $16,486.00, which after transaction fees comes out to $15,108.98. This is a 30% decrease from last quarter, and a 163% increase compared to Q1 last year. This is accounted for by our annual subscriptions - our yearly invoices due each quarter in 2020 break down as follows:

Yearly payments have an outsized effect on the total cash flow, and this accounts for the reduction in income for Q4 2019 → Q1 2020. These figures are in line with our expectations for Q1. 1,377 new users registered accounts in Q1, of which 156 (11%) have paid for their account. As of the end of Q1, there were 13,418 registered users in total, of which 1,553 (12%) have paid subscriptions.

At time of writing (2 weeks into Q2 2020), the breakdown of subscription types is as follows:

 $2 monthly  282 users
 $2 yearly   751 users
 $5 yearly   254 users
 $5 monthly  136 users
$10 monthly   45 users
$10 yearly    98 users

The monthly revenue from these subscriptions (with annual subscriptions realized over 12 months) is approximately $4,649, after transaction fees. This is a $534 increase in monthly revenue compared to Q4.

Sourcehut has $13,792 in the bank at the time of writing (2020-04-13).

Expenses

Breakdown of Q4 expenses:

$3303    New server equipment (git.sr.ht)
$1950    Philadelphia datacenter lease (inc. network, power, etc)
$ 981    San Francisco datacenter lease (inc. network, power, etc)
$ 490    Misc. hardware
$ 334    Travel (FOSDEM)

Note that we paid the 2020 lease for the San Francisco datacenter in full in Q1, as opposed to our typical quarterly payments.

Payroll:

 $5,500.00   Payroll
+$  179.07   Employer taxes & processing fees

Going into Q2, our monthly expenses are expected to be:

$3500/mo    Payroll (approximate)
$640.00/mo  Philadelphia datacenter lease

We have postponed our plans to hire another engineer. Our financial projections have not yet predicted a downturn in SourceHut’s economic viability, but we are a risk-averse business. We are moving onto a more conservative financial footing in preparation for whatever economic consequences of the COVID-19 pandemic may come to pass.

What's cooking on Sourcehut? March 2020

Sun, 15 Mar 2020 00:00:00 +0000

Hello! This has been a very busy month for SourceHut. Thank you for supporting it during the alpha! Today our users are 13,195 in number, with 438 new users joining us since our last status update. Please give them a warm welcome!

General News

I’ve spent a lot of time this month working on a bunch of operations-related tasks, such as monitoring and backups improvements. Our backup system is now triple-redundant and kept in check by three separate monitoring systems. You’ll see some of these reports on our public operations mailing list, or can dig into the raw data on our public Prometheus instance. I’ve also written up an operations manual to keep track of our approach and future plans for operations, so that you can be aware of what steps we’re taking to keep things up and running, and perhaps apply our techniques to your own self-hosted instances of SourceHut.

This month also saw design discussions on sr.ht-discuss regarding the eventual design and implementation of user groups (or “organizations”), and how they’ll affect billing, access control lists, reserved CI slots, and so on. Check out the archives for more: the initial proposal, and version 2.

We also deployed some improvements to responsiveness throughout the site, which you’ll probably have already noticed if you’re on mobile. Mobile users aren’t the only group to benefit, however: I took extra time to make sure the interface looks good at any subset of a 4K monitor, so our tiling window manager users can have a comfortable interface with a web browser at half-width or less.

Finally, we have a new mail server in place, which should be a little bit more maintainable going forward, and our legacy infrastructure has finally been overhauled. These were our last two servers which were not running on the standard Alpine-based load-out¹. This unblocks…

sr.ht

Finally, work has commenced on the central project hub. This will solve the problems of discoverability of projects hosted on SourceHut, searching for new projects site-wide, linking together bug trackers and git repos, and more. This project is ostensibly called “hub.sr.ht” internally, but the intention is to deploy it at the top-level domain: sr.ht.

The hub will let you create a central hub for your project, with any combination of zero, one, ten, or more git and hg repos, bug trackers, mailing lists, and so on. You can add exactly what you need, as much as you need, and nothing you don’t need. This is why SourceHut is designed the way it is today. Many large projects have dozens of source repos, and giving each one their own bug tracker, pull requests, documentation, website, and so on, does not map to the actual project structure and frequently leads to clunky workflows on other hosts. The SourceHut project hub solves this problem better.

builds.sr.ht

Following up on our recent git.sr.ht change to allow users to attach binary releases to git tags, I’ve implemented build artifacts for our CI as well. By adding a list of artifacts to your build manifest, you can have the files extracted from the build VM after successful builds and made available for download on the web or inspection via the API. Artifacts are kept for 90 days, but in the future I would like to let you indicate that certain artifacts should be kept permanently. Docs are available here.

I also provisioned a new build server this month, and as part of our ops work I’ve documented its specs here. This will increase our build throughput quite a bit, to better support our growing base of high-demand CI projects, including Nim, NeoVim, and OpenSMTPD.

With the exception of ns1.sr.ht and ns2.sr.ht, which are still running Debian. These are slated for upgrade, but are not urgent. ↩︎

What do we do when you lose your 2FA codes?

Wed, 04 Mar 2020 00:00:00 +0000

By far the most common sort of support request I receive from SourceHut users on a day-to-day basis is from users who have lost access to their TOTP (Time-based One-time Password algorithm) codes. Losing your phone, getting a new one and forgetting to migrate the keys, or wiping it to install a new OS are common reasons to accidentally lose access to your two-factor authentication.

Naturally, we cannot just disable 2FA on your account, no questions asked. The purpose of 2FA is to increase the level of scrutiny that’s placed on attempts to make authorized requests for your account. Therefore, we seek some alternative method of authenticating that you are who you say you are.

The easiest way is PGP: about 10% of sr.ht users have added a PGP key to their account. If your support email is signed with the PGP key we have on file for you, then we can assume it’s you with no further questioning. If not, we can ask you to send a follow-up email which is signed. Even more users have an SSH key added to their account, about 30%. For them, I asked my friend minus to write a small tool, sshign, which can cryptographically sign messages with your SSH key.

You can add PGP and SSH keys to your account on the key management page.

Those strategies are my preference, but there are still a fair number of users who need 2FA turned off but haven’t added any keys to their account. I have to get more creative with these. One way I’ll often choose is looking at the website added to their profile page. If they can add a file to the website or update a DNS record in response to a challenge, then that’ll often be sufficient.

One thing we used to do, but no longer, is to ask you for the last four digits of the credit card number on file for your account. I have known other services to use a similar approach. Eventually I decided to stop using this, because it’s fairly easy to get the last 4 of your CC# from anywhere you’ve used it. This information has been leaked from many services after many security incidents. I will, however, use this much information to cancel your subscription payment upon request.

There are a small number of users who ask to have 2FA reset, but have little-to-no secondary information to their account. For these few, there is no recourse — I have to tell them that I cannot help them regain access to their account. I doubt any of these folks have actually not been the authentic owner of their respective accounts, but the security of 2FA rests on this extra level of additional scrutiny.

What's cooking on Sourcehut? February 2020

Sat, 15 Feb 2020 00:00:00 +0000

Hello again! Another month into 2020, and SourceHut development continues at a strong pace - a pace I expect to pick up a bit in the coming weeks. We’re joined by another 443 users this month, bringing our total to 12,757. Welcome to SourceHut, everyone! And to our older members, thanks for your continued support.

General News

Our meetup at FOSDEM went well - thanks to everyone who attended! I’m glad I got to meet more of you in person, and we had a great chat going over the plans for the future and getting your feedback on what’s important to you.

In terms of general development, there are a few news items. First, you will probably have noticed that subtle design changes landed throughout the services early this month. These changes brought us into 100% compliance with the WCAG accessibility standards, which is a feat that no other code forge can claim. We score higher now on standard tests of performance, accessibility, and web standards than any other code hosting service. SourceHut is now objectively the fastest, lightest, and most accessible service in the biz. I even fielded a bug report from a Lynx user last week!

Search support has also been improved throughout the site thanks to Ivan Habunek’s hard work, and special thanks to Denis Laxalde as well for his improvements to builds.sr.ht search. Documentation for the new features is coming soon.

I’ve also been working with some community contributors whose aim is to improve Markdown support across the site, bringing support for CommonMark and aligning our implementation more closely with what you may have come to expect from your experiences on the rest of the web. These require changes careful testing, so please be patient while we work on them.

git.sr.ht

A cool new git.sr.ht feature landed this week: attaching files to annotated git tags. This allows you to attach compiled executables, PGP signatures, and so on to your releases. Check out the user documentation here, and the API docs here. This is the first of several planned features which utilize the new blob storage infrastructure - additional features coming soon include git-lfs support, build artifacts, build caches, and a “SourceHut Pages”-like service.

More subtle changes to git.sr.ht include detecting if your repository has a license, which leads to our wiki page with suggestions on how to choose one. I also added support for LibreJS for the few (optional) bits of JavaScript on git.sr.ht, which along with the other changes brings SourceHut’s score for the GNU ethical repository criteria into pretty good standing.

builds.sr.ht

Denis Laxalde, in addition to his work on improving builds.sr.ht search functionality, has also implemented RSS feeds for each page. You can obtain RSS feeds for your user page, any combination of build tags (useful for per-project RSS feeds), and for arbitrary search queries. Thorben Günther also implemented a small bugfix which resolved a long puzzling issue with build status emails - related to UTF-8 email addresses.

Image-specific updates:

Debian builds with custom repositories now also automatically include the source repos.
FreeBSD builds no longer use -cpu qemu64, following upstream fixes to AMD EPYC support.
OpenBSD images now include X11 headers.

What's cooking on Sourcehut? January 2020

Wed, 15 Jan 2020 00:00:00 +0000

Welcome to 2020! We kick off our first status update of the year with a community 12,314 users strong, up 387 from our last update of 2019. A warm welcome to all of our new members, and a big thanks to everyone for continuing to use and support SourceHut. This month was heavy on operations work, so this update is light on new features - but we’re turning our attention back to feature development in the coming weeks.

General News

We’re going to FOSDEM again this year! We’ll be in the J.1.106 room on February 1st at 12:00 PM. Come join us, we’ll be giving away stickers, answering questions, and showing off some of our upcoming plans.

Last week, I finished overhauling our monitoring & alarming infrastructure, and the new setup is available to the public for inspection. I’m also working on opening up more ops work to improve transparency in this respect: the #sr.ht.ops channel on Freenode is now open to the public, and I’ve also established the public sr.ht-ops mailing list. I’ve updated our alarms to send notifications to #sr.ht.ops and to the sr.ht-ops mailing list, and directed some of our cronjobs to the public mailing list as well, so you can keep abreast of things like backup integrity and get greater insights into outages and load. Our alertmanager rules are also public, and you can drop these into your own infrastructure to add similar monitoring to your third-party instance.

Work is ongoing to add more instrumentation throughout SourceHut, stay tuned. As part of this work, we also upstreamed several monitoring-related packages to Alpine Linux, and upgraded all of our machines to Alpine 3.11 (and Python 3.8).

git.sr.ht

We finally were able to source the remaining parts for the new server hardware we needed to completely resolve the performance issues you were likely experiencing with git.sr.ht. The new server is blazing fast! Thanks for your patience, I know it was frustrating before. It should be plenty fast now. Aside from some minor bugfixes and upgrades to support pygit2 1.x and Alpine 3.11, there’s not much else to discuss for git.sr.ht. Some of these improvements have been making their way into hg.sr.ht, so expect performance to gradually improve there as well.

todo.sr.ht

Some great news here: you can now re-import the tracker exports which were implemented last month, to recreate deleted trackers, migrate trackers between accounts, or even between SourceHut instances. This month I’m focusing on accomplishing our data ownership goals, so you should see improvements like this throughout the services in the coming weeks. Ivan Habunek also implemented some nice changes, notably adding rendered markdown previews for comments.

builds.sr.ht

The alpine/3.11 image is now available, and alpine/latest now refers to it. alpine/edge has been updated to use the linux-lts kernel from edge.

Sourcehut Q4 2019 Financial report

Mon, 13 Jan 2020 00:00:00 +0000

In summary, Sourcehut is financially healthy. We have a reduced services profit from Q3 - we are currently breaking even - but this is due to a planned increase in expenses and is offset by revenue from our new consultancy. Growth has continued at a similar pace to Q3. Through the recent formalization of the SourceHut free software consultancy, we have also raised a significant amount of capital, which increases our runway substantially, and allows us to consider more hires, new hardware provisioning, and so on. 2019 was a great year for SourceHut, and I summarized our accomplishments and goals in this earlier article.

Thank you for your support in the alpha. I’m looking forward to continuing to serve you in 2020.

Disclaimer: this report is a summarized approximation of Sourcehut’s financials, and is not used for tax purposes.

Revenue sources

Sourcehut receives revenue from paid user subscriptions. During Q4, we processed 1,824 invoices. The invoices paid break down as:

803     $2  (paid monthly)
372     $5  (paid monthly)
102     $10 (paid monthly)
373     $2  (paid yearly)
128     $5  (paid yearly)
45      $10 (paid yearly)

The total gross revenue during this period was $22,856.00, which after transaction fees comes out to $21,530.35. Because of differences in the volume of yearly invoices processed during Q4, this number is not easily compared with Q3. It’s better to compare it with Q4 2018 — when compared to this, we’ve shown ~185% growth.

In Q4, 1,876 new users registered accounts, of which 267 (14%) have paid for their account. As of the end of Q4, there were 12,289 registered users in total, of which 1,410 (11.4%) have paid subscriptions.

At time of writing (2 weeks into Q1 2020), the breakdown of subscription types is as follows:

279     $2  (paid monthly)
126     $5  (paid monthly)
38      $10 (paid monthly)
660     $2  (paid yearly)
228     $5  (paid yearly)
79      $10 (paid yearly)

The monthly revenue from these subscriptions (with annual subscriptions realized over 12 months) is approximately $4,115, after transaction fees. This is a $808 increase in monthly revenue compared to Q3.

Sourcehut has $25,442 in the bank at the time of writing (2020-01-13).

Breakdown of Q4 expenses:

$4504    New server equipment (git.sr.ht)
$1950    Philadelphia datacenter lease (inc. network, power, etc)
$ 812    Travel (FOSDEM)
$ 548    Insurance renewal
$ 129    San Francisco datacenter lease (inc. network, power, etc)
$  38    Misc. equipment

Payroll:

 $5,500.00   Payroll
+$  179.07   Employer taxes & processing fees

Going into Q4, our monthly expenses are expected to be:

$3500/mo    Payroll (approximate)
$640.00/mo  Philadelphia datacenter lease
$129/qtr    San Francisco datacenter lease

At the end of Q3, we were preparing to learn how we would be impacted by the first round of annual subscription renewals, unsure of how many users would cancel. We did not see any unusual number of subscription cancellations in Q4, and it does not impact our financial planning.

At some point in early 2020, we are planning on hiring an additional engineer. We also expect to spend a bit more on FOSDEM before it’s over.

Consultancy

This is the first quarter during which our free-software consultancy has factored into our finances. Our core mission is to elevate the free software community, and part of our long-term plans have been to hire free software developers to work full-time on self-directed projects. The consultancy helps us meet this goal — 100% of the work is free software, on projects relevant to our engineers, chosen at their discretion. Additionally, we pass most of the hourly rate onto the engineer, which offsets our industry substandard salary. The profits are also reinvested into free software, helping us hire additional engineers.

However, reporting on our consulting work is not as straightforward as our other financial reporting. The income of the consultancy is not consistent and predictable like our paid services are. In terms of financial planning, the profits are treated as additional runway or capital. Runway is how we measure the longevity of the business in rainy day scenarios, and capital is sums of money we pull from to finance new hardware, new hires, and so on.

Because most of the revenue is directly passed onto the individual engineers, sharing it in too much detail raises privacy concerns. We’re working on figuring out how to be more transparent with this side of the business, but it will likely not be possible until we have more engineers on board to help anonymise the numbers.

What's cooking on Sourcehut? December 2019

Sun, 15 Dec 2019 00:00:00 +0000

This is our last “what’s cooking” of 2019! We’re seeing out the year with 11,927 registered users, 563 stronger than November. Please give our new members a warm welcome! We have some cool progress on many fronts this month.

General News

First of all, we finally got single-sign-on sorted out this month. Now, logging in once will log you in everywhere. Many other problems which stemmed from this have also been fixed - long time users will no doubt have seen the “log out and back in to enable builds.sr.ht integration” message on a git push, but thankfully that annoyance will be unknown to new users going forward. This was also a pre-requisite for dealing with user groups, which will be the next main focus of account management work.

I’ve also been experimenting with Ceph and preparing to provision our new Ceph cluster, which will give us three things: (1) object storage, (2) filesystem storage, and (3) block device storage. Each of these separately unblocks a number of features. (1) gives us things like attaching binaries to tags or publishing build artifacts from builds.sr.ht jobs, (2) gives us things like a GitHub-pages-style feature, and (3) can be leveraged for build caches. Ceph is complicated and I want this deployment to be sustainable indefinitely, so I’m going to take my time to provision it carefully and understand every piece - so please have patience while I get this set up.

git.sr.ht

The first series of performance improvements to git.sr.ht landed this month, and I wrote an article which goes into a great deal of detail for the curious. The next step is to improve the I/O throughput of the server, which can only be done with revprovisioning. Unfortunately, the hardware to do this has been a real pain in the ass to obtain - SuperMicro has been having issues with their fulfillment pipeline recently and I’m struggling to complete the build. We may be in for a bit of a wait while I try to get the parts together. If I don’t have this squared away by January, I’m going to redesign the server with different parts. I already have half of the parts, and I’d rather they don’t go to waste, so I’m going to spend a little more time trying to get this system built per the original specification.

lists.sr.ht

You can now grant a new access control privilege to users: “moderate”. This allows users to update patchset status and remove emails from the archives.

builds.sr.ht

Minor maintenance updates:

Debian packages are now available.
Fedora 31 is now available.
FreeBSD 12.1 is now available.
NixOS 19.09 is now available.
OpenBSD 6.6 is now available.

names.sr.ht

Wow, this is complicated. I think having an open-source implementation of all of these rules will be a boon to the ecosystem, though. Despite the complexity, I still have a reasonably complete image of what the system should look like in my head, and we’re slowly progressing towards realizing it. Here’s another screenshot to tease you in the meanwhile:

What happens when you push to git.sr.ht, and why was it so slow?

Fri, 22 Nov 2019 00:00:00 +0000

One of the most persistent, frustrating, and entirely valid, criticisms of Sourcehut that I’ve heard has been that git operations over SSH are too slow. The reason this is a frustrating complaint to hear is that the git.sr.ht SSH pipeline is a complicated set of many moving parts, and fixing the problem involved changes at every level. However, as many of you will (hopefully) have noticed by now, pushing to and pulling from git.sr.ht is quite snappy now! So after a huge amount of work overhauling everything to get us here, I thought it would be nice to reflect on what caused these issues, how this system is structured, and how the problem was eventually solved.

There are several major tasks that need to happen when you push or pull to git.sr.ht. In order, they are:

Dispatch Which system are you SSHing into? git.sr.ht? builds.sr.ht?
Identification Who are you?
Authorization Are you allowed to do what you’re trying to do?
Execution Hand things off to git to complete your operation.
Follow-up Do we need to submit any CI jobs? Webhooks?

During each of these steps, your terminal is blocked. You have to wait for all of them to complete. Well, most of them, at least. Let’s discuss each step in detail.

Dispatching

There are several Sourcehut services which you can log onto using SSH: git.sr.ht, hg.sr.ht, and builds.sr.ht, and perhaps more in the future. Before we overhauled it, man.sr.ht used to have a dedicated SSH service as well. In our case, we run each of these services on their own servers at their own IP addresses. However, this was not always the case, and we still support third-party installations of Sourcehut services which are all sharing a single server. Therefore, we have to have a way of identifying which service you’re trying to SSH into, and for this purpose we use the user. You use [email protected], [email protected], and [email protected], to log onto each respective service.

This phase is handled by our gitsrht-dispatch binary, whose source code you can view here. This is run by OpenSSH in order to generate an authorized_keys file, with a list of SSH keys which are allowed to log into Sourcehut. That’s beyond the scope of dispatch, however, which delegates it to the next step. Instead, it just figures out which service it needs to hand you off to for authentication.

[git.sr.ht::dispatch]
#
# The authorized keys hook uses this to dispatch to various handlers
# The format is a program to exec into as the key, and the user to match as the
# value. When someone tries to log in as this user, this program is executed
# and is expected to emit an AuthorizedKeys file.
/usr/bin/gitsrht-keys=git:git
/usr/bin/hgsrht-keys=hg:hg

When you SSH into the service as git@..., the dispatcher looks up the entry corresponding to the git user. In this case, it’s gitsrht-keys. Then, it finds the uid and gid for this account, setuid¹’s itself to that user, and exec’s² into the next step.

Identification

Each discrete service is separately responsible for its own approach to identifying and authorizing users. OpenSSH executed our dispatcher hoping to get an authorized keys file, and the next step is to prepare one for it. For git.sr.ht, this responsibility falls onto gitsrht-keys.

This is where things start to get interesting in terms of performance. Ideally, we want to get out of here with zero round-trips to remote services like SQL. For this reason, we cache your SSH keys on a Redis instance at localhost. OpenSSH handed us your base64’d SSH key and dispatch forwarded it to us, and we use this to look up your key from the cache. If we find it, we can skip directly to the final step. But, if we miss the cache, we ask meta.sr.ht if it’s seen your key before. We then cache it in Redis and make sure we have a copy in our database as well, to save you time for the next push. But even in this worst case, we’re already starting to see hand-optimized SQL queries:

-- Getting the user ID is really a separate concern, but this saves us a
-- SQL roundtrip and this is a performance-critical section
WITH key_owner AS (
  SELECT id user_id
  FROM "user"
  WHERE "user".username = $1
)
INSERT INTO sshkey (
  user_id,
  meta_id,
  key,
  fingerprint
)
SELECT user_id, $2, $3, $4
FROM key_owner
-- This no-ops on conflict, but we still need this query to complete so
-- that we can extract the user ID. DO NOTHING returns zero rows.
ON CONFLICT (meta_id) DO UPDATE SET meta_id = $2
RETURNING id, user_id;

We need to get your git.sr.ht user ID while we’re here, and normally we’d look it up in Redis. With this SQL query, we fetch the user ID at the same time as we store your key. This kind of hacky SQL packing is a common tactic we’ll see employed throughout for improved performance.

At the end of this process, we print out a generated authorized keys file and exit, then OpenSSH does the rest of the work to verify that the user actually possesses the private key they claim to and so on. Then, when they’re allowed in, we move on to the shell.

restrict,command=gitsrht-shell,environment="SRHT_PUSH=<uuid>",your ssh key here

This keys file turns off almost all SSH features (restrict), overrides the command you will run to use our shell, and adds an assigned UUID for this push to the environment.

Authorization

At this point, we’ve found your account and verified that you’re an authorized agent of the user, using your SSH key. Now we need to know if you’re allowed to do what you’re trying to do, which is where gitsrht-shell comes in. Its job is to look up the repository you’re trying to do and compare the operation you want to perform (read or write) against the access control lists. Even in the best case, we’re required to do a SQL round-trip here. This is what it looks like:

-- Fetch the necessary info from SQL. This first query fetches:
--
-- 1. Repository information, such as visibility (public|unlisted|private)
-- 2. Information about the repository owner's account
-- 3. Information about the pusher's account
-- 4. Any access control policies for that repo that apply to the pusher
SELECT
  repo.id,
  repo.name,
  repo.owner_id,
  owner.username,
  repo.visibility,
  pusher.user_type,
  pusher.suspension_notice,
  access.mode
FROM repository repo
JOIN "user" owner  ON owner.id  = repo.owner_id
JOIN "user" pusher ON pusher.id = $1
LEFT JOIN access
  ON (access.repo_id = repo.id AND access.user_id = $1)
WHERE
  repo.path = $2;

Again, we’ve jammed a lot of somewhat disjoint information into a single SQL query for efficiency’s sake. We fetch everything we’ll need to know later on: what’s this repo’s ID and name, who’s the owner, what’s the repo visibility, who’s the pusher and what kind of user are they, is their account suspended and how do we break it to them if so, and are there any access list entries for them on this repository?

There are two reasons that this could fail: the repository has been renamed, or it never existed. If this query returns no results, then we try to look for a redirect — round-trip #2. The query is as follows:

SELECT
  repo.id,
  repo.name,
  repo.owner_id,
  owner.username,
  repo.visibility,
  pusher.user_type,
  pusher.suspension_notice,
  access.mode
FROM repository repo
JOIN "user" owner  ON owner.id  = repo.owner_id
JOIN "user" pusher ON pusher.id = $1
JOIN redirect      ON redirect.new_repo_id = repo.id
LEFT JOIN access
  ON (access.repo_id = repo.id AND access.user_id = $1)
WHERE
  redirect.path = $2;

You’ll notice that we’re also fetching a whole lot of information at once here. In case we do find a redirect, we don’t want to do a third round-trip to fetch all the information we’ll need for that repo, either.

Say this fails, too. Now we definitely know that your repository doesn’t exist. This is the point at which git.sr.ht’s autocreate-repos-on-push feature kicks in, if the repo you’re pushing to is under your user namespace. That code is fairly simple and involves an additional round-trip, you may study it in your own time.

But, if we get this far, then we have everything we need to finish authorizing you. If you do have permission to complete the operation you’ve requested, we exec into the appropriate git binary.

Execution

At this point, you’re in git’s hands. We’ve handed you off to one of their binaries to complete the push or pull operation. The tools that can be run at this point are git-receive-pack, git-upload-pack, or git-upload-archive — follow the links if you want to learn more about them.

Following up: hooks

There’s one last task for us to do before we can send you on your way, and it’s split into three parts. These are our git hooks. These are only run on git pushes, not for fetching. We use these hooks to complete some housekeeping after your pushes complete, and to implement a few features. The remaining tasks for us to do at this point are:

Update the mtime of your repository
Submit any necessary builds.sr.ht jobs
Deliver webhooks

This doesn’t seem like much, but it’s in fact the most complicated part of the process so far. It’s handled by gitsrht-update-hook. These tasks correspond, though not 1:1, to three steps, each mapping to a different git hook:

hooks/update
hooks/post-update
hooks/stage-3 (this is a pseudo-hook that we made up)

The entry point is pretty simple. It just figures out what stage we’re on and changes its behavior accordingly. The first step, the update hook, is really simple; we just use it to collect data from git for use in later stages. In theory we could reject your push here, but in practice we don’t have anything set up to do so right now. git executes us once for every updated ref, with the old and new sha’s for that ref, and we just shove them into a local Redis instance for later use.

“Later use” comes with stage 2, the post-update hook. This is where the majority of the work for this stage starts to get done. We first pull the “push context” out of the environment, which is some data that gitsrht-shell prepared for us earlier, to avoid fetching the same data in two different stages. We don’t get off completely scot-free, though, and we hit the next required SQL round-trip:

-- With this query, we:
-- 1. Fetch the owner's username and OAuth token
-- 2. Fetch the repository's name and visibility
-- 3. Update the repository's mtime
-- 4. Determine how many webhooks this repo has: if there are zero sync
--    webhooks then we can defer looking them up until after we've sent the
--    user on their way.
UPDATE repository repo
SET updated = NOW() AT TIME ZONE 'UTC'
FROM (
  SELECT "user".username, "user".oauth_token
  FROM "user"
  JOIN repository r ON r.owner_id = "user".id
  WHERE r.id = $1
) AS owner, (
  SELECT
    COUNT(*) FILTER(WHERE rws.sync = true) sync_count,
    COUNT(*) FILTER(WHERE rws.sync = false) async_count
  FROM repo_webhook_subscription rws
  WHERE rws.repo_id = $1 AND rws.events LIKE '%repo:post-update%'
) AS webhooks
WHERE repo.id = $1
RETURNING
  repo.name,
  repo.visibility,
  owner.username,
  owner.oauth_token,
  webhooks.sync_count,
  webhooks.async_count;

This query helps us determine what work we’ll need to do in the rest of the process, and gets us some of the information we need to complete that work. We update the mtime of your repo, but we also do some hacky joins and subqueries to sneak more information out of SQL in one round-trip. We grab the OAuth key we’ll need to submit builds on your behalf to builds.sr.ht, and the number of synchronous and asynchronous webhooks registered for your repository. If there are any synchronous webhooks, that means (by definition) that we need to complete those in this stage, and a second query is performed:

SELECT id, url, events
FROM repo_webhook_subscription rws
WHERE rws.repo_id = $1
  AND rws.events LIKE '%repo:post-update%'
  AND rws.sync = true;

Next, we open up your git repository to have a peek inside. We iterate over each ref which you updated during your push, and pull the old and new references out of Redis from stage 1. We pull some info out of git and turn it into a webhook payload to be delivered later, and we look for builds.sr.ht manifests. If we find one, we set up a build submitter for it, and it kicks off any necessary CI jobs.

That webhook payload we’ve prepared now needs to be submitted to any synchronous webhooks before we let go of your terminal - after all, it’s a feature that these webhooks can return some text to show to the pusher before they’re off. The webhook code is pretty straightforward, with one minor exception: webhook payloads are signed.

Once that’s done, we can finally release your terminal and send you on your way. However, we’re not done: we still have to run stage 3. Actually, we only have to run stage 3 if there were any synchronous or asynchronous webhooks on your repository. In the interests of speed, we don’t record your synchronous webhook deliveries into SQL right away, and by definition we do your asynchronous deliveries asynchronously. So, we execute stage 3 as a detached child before we leave, and we send it any information we’ve already collected for it to complete these tasks.

Stage 3’s job can be undertaken at a much more leisurely pace. First, it looks up any asynchronous webhooks, then delivers the same payload to them, graciously forwarded by stage 2. It collects all of their deliveries, combines them with a list of deliveries forwarded from stage 2, and records them in SQL. Now, at long last, this lengthy process is complete.

So why was everything slow?

Well, for a few reasons. First, each step of this pipeline used to be written in Python, not Go, which matched the rest of Sourcehut’s implementation. Spinning up the Python VM and pulling in all of our imports - including, for example, all of our SQLAlchemy riggings - is not cheap. Each of these steps would separately incur a startup cost on the order of up to a second each.

Another reason is that the SQL riggings were not especially well optimized. In the worst case, it could take upwards of 20 SQL round-trips before your push operation was completed. This was mainly due to poor design choices, mostly motivated by DRY (Don’t Repeat Yourself (Don’t Repeat Yourself)), which tried to leverage more generalized and less efficient code used by the web application to perform some of these steps. In the best case, the new approach makes only one SQL round-trip for pulls, or 2 round-trips for a push.

A major misstep was, in the process of optimizing these steps, the decision to move some of this logic into the web application. Python isn’t very good at asynchronous operations, and Sourcehut’s codebase pre-dates asyncio. We use Gunicorn as our web daemon, which runs several worker processes to handle web requests. At one point, to reduce the Python startup time, I thought it would be smart to bundle up all of your information in a little HTTP request and shoot it over to the web worker, which already had a SQL connection and a warm Python VM.

However, the process was still very slow even with those bottlenecks removed - which was a huge problem. Because there are a fixed number of web workers, and they can only do one thing at a time, the addition of processing complicated git push logic to their workload would quickly pin workers, causing the web application and git pushes to slow. Before I fixed this mistake, the average time for an HTTP request and a git push were pushing 60 seconds. That was an awful, terrible, no good, very bad idea.

Now we’re fast! Or are we?

Astute observers and frequent git-pushers may notice that the ideal performance of git pushes on git.sr.ht have been vastly improved, to the point of often appearing instant, but they are not consistently that fast. There is one remaining bottleneck: I/O throughput. When git.sr.ht’s servers are busy, we can still tie up I/O serving various other users before we get to your requests. This is a limitation of our current hardware deployment, which is more I/O bound than anything else. To address this last bottleneck, new server hardware is being provisioned, with different I/O constraints, and git.sr.ht will be moving to it as soon as it arrives and is put together. And then we’ll be blazing fast, all the time, hooray! At least until we can afford better peering…

setuid changes the user identity of the current process. ↩︎
exec replaces the current process with another. ↩︎

Sourcehut's year in alpha

Fri, 15 Nov 2019 00:00:00 +0000

Today marks the 1 year anniversary of the commencement of Sourcehut’s public alpha, after 2 years in private development. I’m immensely thankful for your support during the alpha. It’s mostly been a one-man operation here, but thanks in large part to the financial support of the alpha users, I’ve been able to work on Sourcehut and free software in general full-time for a full year as of this February. With this first year of the alpha behind us, what have we accomplished, and what do we have planned for 2020?

Just looking for November’s status update? It’s at the bottom, this a-way… ↓

2019 in summary

I’m really proud of what we accomplished in 2019. Here are some of my personal favorite highlights:

Deployment of APIs and webhooks throughout
Establishing comprehensive systems monitoring
Became profitable and self-sustaining
Bringing on our first staff member
Initial development of web-based code review within our design framework
Maturation of todo.sr.ht into a great ticket tracking system
Code annotations added for git.sr.ht
SSH access to builds.sr.ht VMs for debugging and hacking
Availability of Alpine, Arch, and Debian repositories for third-party sr.ht deployments

Want to read plans for 2020? Skip the pretty pictures →

SSH access to a failed builds.sr.ht job

Browsing an annotated codebase on git.sr.ht Your web browser does not support the webm video codec. Please consider using web browsers that support free and open standards. Preparing patchsets to be emailed with git.sr.ht

Reviewing a patchset on the web with lists.sr.ht

Expectations for 2020

I believe Sourcehut will enter the beta phase in 2020, and be approaching the production phase by the end of the year. Some features I’m looking forward to are:

User groups/organizations
A centralized project hub, bringing together our disparate services
Data ownership and portability for all services
Completed web-based patch submission/review
Maturation of hg.sr.ht into a competitive Hg hosting service
Integration with more projects throughout the software project hosting ecosystem
The fruits of financially supporting free software developers — exactly what comes of this is unknown, but I’m excited as hell about it.

From developers sponsored by Sourcehut going into 2020, I’m expecting to see more developments in Wayland and Linux graphics, mail clients and servers, improvements to Alpine Linux, mrsh’s completion and initial release, miscellaneous patches to any free software project which crosses our path, and new projects which I expect will pleasantly surprise us.

Thank you to everyone for your support in our first year, and I’m looking forward to continuing to serve you into the next.

How exactly we’re going to accomplish these goals is a subject worthy of some attention, too. I frequently get questions about the stability of the alpha, when the beta will come and how things will change, and so on. So, I’ve taken this opportunity to write down The Master Plan.

Development roadmap

First, briefly, the plan for Sourcehut’s development lifecycle has been roughly:

Build several standalone services which each solve a problem for software projects.
Develop APIs and webhooks for each of these services.
Teach them how to talk to each other, creating a complete system.

We’ve mostly completed phases 1 and 2 and development on phase 3 is underway.

Phase one

Base development of git.sr.ht, builds.sr.ht, lists.sr.ht, man.sr.ht, and more. This is close to complete, insofar as all of the unknowns are known. Some services are missing a few features which arguably fit into this step, such as git blame support for git.sr.ht, but for the most part everything is complete in its own right.

Phase two

This phase is the introduction of APIs and webhooks for every service, which is a necessary step before integrating everything together. The APIs and webhooks each service will use (and now, do use) to talk to each other are no different than those offered to end-users, which helps to prove the API design is flexible enough to support a broad variety of use-cases and integrations with other parts of the ecosystem.

This phase is mostly complete — see the API documentation links here for an idea of which services have API coverage. The main exceptions are man.sr.ht and dispatch.sr.ht, which don’t have an API because they’re low-priority targets for integration with other services; and hg.sr.ht, which technically has an API that is mostly compatible with git.sr.ht’s API but is left undocumented for want of an Hg specific API design document from the community.

Building out these APIs also informed a lot of design decisions which helped to move the codebases from “experimental” into “maintainable”. In phase one, we establish the scope for each project and prove that its ideas are feasible, and in stage two we refined the prototypes into production-worthy systems.

Phase three

The final phase of development is perhaps the most exciting phase, as it’s in this step that we get to deliver on many of the usability promises of Sourcehut. Some early progress has already been made in this direction. Some examples include:

Teaching git.sr.ht & hg.sr.ht to push CI jobs to builds.sr.ht
Overhauling man.sr.ht to replace its internal git repos with git.sr.ht repos
Teaching git.sr.ht to prepare and send patchsets to mailing lists, e.g. lists.sr.ht

Future plans include tasks like:

Teach lists.sr.ht how to submit CI jobs to validate incoming patchsets
Teach lists.sr.ht how to merge patches to git/hg repos
Refactor todo to use lists in a similar way to man’s use of git

An advantage of this three-stage approach is that each of these phase three features can, in theory, be extended to include other parts of the ecosystem. lists.sr.ht can be re-purposed to manage patches and code review for a Gitlab repository, builds.sr.ht can be used to run CI for GitHub repositories, git.sr.ht can be used to send patches to the Linux kernel, and so on. This is part of our commitment to embrace existing communities rather than building a new walled garden.

Operations roadmap

In addition to being a software project, Sourcehut offers a hosted service at sr.ht. The administration of this hosted service is another factor which requires planning and plays into the stability of the service. There are three major concerns of the operations roadmap:

Security
Data integrity
Availability
Monitoring

Security is the steps we take to ensure your account and its data is safe from malicious actors. This includes careful access controls - ensuring each system has access to only the data it needs; establishing trust levels and securing each system accordingly - for example, the multiple layers of isolation used for CI jobs on builds.sr.ht; and other miscellaneous concerns like secure SSH configuration, SSL maintenance, network administration, and so on.

Data integrity was important to address early in the alpha period, and has been essentially completed. For all persistent data, extreme pains are taken to ensure that it remains consistent even in emergencies - and our systems to ensure this have been battle tested during emergencies, including disk failures on critical machines. Our multiple layers of redundancy for data preservation were laid out in detail in January and have only been improved since.

Availability is concerned with making sure that our services are always online, and during the alpha is a secondary concern. Our long-term goal is to approach zero downtime, by setting up load balancers and tiered deployments. Our software is designed with this in mind, which will ease the eventual deployment of high availability services. However, today we still have planned outages, and each deployment costs between 10 and 60 seconds of outage for the affected services.

Monitoring is a cross-cutting task which is essential to the success of the other goals. Setting up systems to observe and characterize the behavior of our services informs the planning and execution of security, data integrity, and availability goals. For example, we use monitoring to watch for suspicious activity, ensure our backups are up-to-date, and understand our system’s load to prevent outages. This is mostly complete but alarming is due for an overhaul - Grafana alarms aren’t great for serving all of our needs.

An operations matter that was settled early and no longer factors into our planning is the release and deployment process for Sourcehut services. An acceptance requirement of builds.sr.ht’s initial deployment was that it be self-hosted and self-deploying and support the deployment of *.sr.ht as well, and as such this goal was met before even the public alpha began.

Business roadmap

The health of Sourcehut as a business is also my responsibility. Providing the hosted service is not free, and getting myself paid enough to keep food on the table is important for the development and health of the project as a whole. This basically involves lots of spreadsheets, tracking the income and expenses of the business, and making projections for the future. The monitoring system maintained by the operations work is also put to use for BI (business intelligence) reporting and monitoring.

I basically maintain three projections: pessimistic, realistic, and optimistic. A rough approximation of how these are done is: pessimistic projections assume that no one signs up for new subscriptions, monthly subscriptions continue paying out, and yearly subscriptions don’t. Realistic projections amortize yearly subscriptions over a year, assume no one cancels, but also assumes no growth. Optimistic projections expect the current rates of growth to continue. Most of our long-term planning is based on the pessimistic projections. These models are going to be updated soon as we start to get data on how many users cancel their subscription when the first yearly renewals come up this month.

This work also involves dealing with taxes, regulatory compliance, making sure contractors are paid on time, and so on. Boring stuff, but it eats up a lot of time. Additionally, marketing falls under this umbrella, which mainly takes the form of posts on this blog and showing up when people talk about us online to answer questions. I’ve tried asking happy users to blog about their experiences, but most people don’t have a blog or are shy about writing, so there’s some yak shaving to be done first.

The main business goal at first was profitability, which has been achieved. Additional goals are to generate a revenue stream which can continue to support investment in new hardware, sponsoring free accounts for less fortunate users, marketing and outreach efforts, and sponsoring free software developers to work on self-directed projects. All of these goals are proceeding nicely.

Stability roadmap

How do these development phases play into the external stability presented by the service? Thus far, all of this development has been considered “alpha”. There are two additional phases: beta and production. The scope and completion criteria of each phase in the stability roadmap is informed by the progress in each other roadmap.

For example, some of the development tasks blocking the graduation from alpha to beta include:

User groups (for organizations managing many accounts or projects)
Completely web-driven patchset submission and code review
A centralized, indexed project hub which allows you to link a project’s git/hg repos, bug trackers, mailing lists, and so on under a single top-level “project”
Data ownership, i.e. self-service data import and export tools for all services

In operations, an overhauled alarming system is a blocker for the beta, and high availability is a blocker for production. Confidence in our data integrity and security systems is a beta requirement, and has already been achieved.

On the business side, a beta task was to achieve sustainability - that is, to make a profit - which was achieved in Q1 2019. A business which is losing money cannot be considered stable, after all. An outstanding business task which still holds up the beta is establishing the final pricing model for the beta and into the future.

Status update

Phew! What a long article. Thanks to everyone for helping out during the alpha, I really appreciate your support. Since the last update, we have another 825 users to welcome to the platform, bringing our grand total up to 11,364. 1,320 of you have chosen to pay for your alpha account, which I am deeply grateful for. Thank you, everyone! Let’s get to the updates for this month.

General updates

Thanks to Eli Schwartz’s help, the Arch Linux package repo saw some improvements this month, and Luca Weiss pitched in to fix some outdated database migration scripts. Thanks guys!

git.sr.ht

I’ve been working on performance improvements for git over SSH again. An earlier attempt was an abject failure - I had moved some logic into the web app and accessed it over an internal API, under the rationale that the web app had already spent the Python VM and SQLAlchemy setup time. In fact what really happened is that git operations - which happen all the time - started to pin the web workers, because Python can’t into async. The result is that the entire stack slowed down dramatically. I rolled a nat’ 1 on that one.

The new solution is way, way better. I rewrote two pieces of the pipeline in Go and overhauled the SQL queries, and now the time-to-/usr/bin/git is significantly reduced, often less than a second. Prior to the disastrous change, it could be as much as 10 seconds, and post disastrous change it was as much as 45, with the web app being slowed to a similar crawl. With this approach, git operations are fast and the web app is buttery smooth. The last piece to improve on is the post-update hook. You’ll notice the slowness there, since git will quickly show its push output, then stop again while we run the post-update. I hope to improve this soon.

I know that the git-over-SSH performance has been disappointing for a while, and I’m sorry. The pipeline here is a complex machine with lots of moving parts, and all of them are slow. A comprehensive solution requires a lot of overhauling.

Third-party contributions to git.sr.ht have also been coming in recently, with Ian Moody making improvements to the diff view (for hg.sr.ht, too), and Honza Porkorny switching our tree views to a monospaced font. I’ve also made a number of bug fixes to the new patchset preparation UI.

todo.sr.ht

I’ve implemented half of the necessary steps for todo.sr.ht to meet its data ownership goals, by implementing a data export tool. There is some interesting novelty here that merits some attention. The export prepares a gzipped JSON dump of all of your tracker’s tickets and events, and is more or less just a shortcut to the API. However, certain information - usernames, comment text, and such - are signed with SourceHut’s webhook signing key. The idea is that if you re-import these dumps later, we can verify that anything our users have said on the tracker has not been tampered with, and confidently show their username next to their comments. Additionally, third-party Sourcehut instances which trust our word upstream can check these signatures against our key and be confident that those are authentic comments authored by Sourcehut users.

However, this will only be useful when the data import side of this work is completed, which is still to come. In other news, Ivan Habunek has written a UI for editing labels, and added URL parameters for auto-filling the title and description for new tickets, which is useful, for example, to provide links to file tickets pre-populated with stack traces.

dispatch.sr.ht

dispatch.sr.ht can now build GitLab commits! I wanted to add merge requests as well, but it’s blocked by this GitLab bug. If/when this issue is addressed, running builds.sr.ht CI for GitLab merge requests will be possible as well. This is currently available for gitlab.com and gitlab.freedesktop.org - if you want me to set it up for your GitLab instance, please send me an email.

meta.sr.ht

Just some small updates: I have added URLs for obtaining a user’s SSH and PGP public keys. Just visit meta.sr.ht/~username.keys or ~username.pgp. There have also been some improvements to the billing pages, which is especially important as the first yearly payments will start renewing soon.

paste.sr.ht

Thanks to Mykyta Holubakha’s hard work, pastes on paste.sr.ht can now be deleted! This was non-trivial because we de-duplicate paste contents in the database. Thanks Mykyta! We’re hoping to see support for editing pastes coming soon as well.

builds.sr.ht

Some mundane image updates:

freebsd/12.x builds are now using FreeBSD 12.1
Fedora images have been improved and updated
NixOS images have been improved and updated

Additionally, OpenBSD 6.6 support is right around the corner. Thanks to our hardworking build image maintainers for keeping up with these releases!

names.sr.ht

Progress has been made since this was teased in October. That’s all I’ve got to say :)

Our model is customers first, investors never

Wed, 23 Oct 2019 00:00:00 +0000

SourceHut was built by veterans of the Internet, who have watched the turbulent lifecycle of Internet-based businesses coming into popularity and fading into obscurity time and again. Most large online businesses these days are for-profit companies funded with venture capital, and that leads to a certain set of incentives. It was no surprise to us, accordingly, to receive this email from GitLab today:

We have launched important updates to our Terms of Service surrounding our use of telemetry services. Starting with GitLab 12.4, existing customers who use our proprietary products (that is, GitLab.com and the Enterprise Edition of our self-managed offerings) may notice additional Javascript snippets that will interact with GitLab and/or third-party SaaS telemetry service (such as Pendo).

For GitLab.com users: as we roll out this update you will be prompted to accept our new Terms of Service. Until the new Terms are accepted access to the web interface and API will be blocked. So, for users who have integrations with our API this will cause a brief pause in service via our API until the terms have been accepted by signing in to the web interface.

For Self-managed users: GitLab Core will continue to be free software with no changes. If you want to install your own instance of GitLab without the proprietary software being introduced as a result of this change, GitLab Community Edition (CE) remains a great option. It is licensed under the MIT license and will contain no proprietary software. Many open source software projects use GitLab CE for their SCM and CI needs. Again, there will be no changes to GitLab CE.

You can read the whole email here, or in your inbox if you have a GitLab account. The summary is that GitLab will soon:

Disable web and API access until users consent to be tracked
Add third-party telemetry to their hosted and enterprise offerings
With obfuscated, third-party, proprietary JavaScript

This can naturally be frustrating to privacy-concious users of their service, and to free software enthusiasts alike. This follows closely on the news that GitLab updated official policy to state that they will do business with those who don’t share their values, which many see as a response to GitHub taking fire for accepting ICE contracts a few days prior. These kinds of changes are not implemented with the user in mind - these decisions are more easily explained by following the money. GitLab is trying to figure out how it can turn a profit that can support its $2.75B valuation. The nature of this business model leaves businesses like GitLab indebted to investors, who’ve sunk millions into the business and demand a return. An individual user’s investment is comparatively meaningless, and the incentives this creates easily leads to compromises like the ones we’re seeing in GitLab recently.

I¹ have written about the up-and-down lifecycle of Internet businesses as early as 2014, when I noticed a trend among image hosting websites to make decisions which increasingly favored their business at the expense of their users. This culminated in the failure of an early business venture of mine, MediaCrush, when we chose to close the service instead of making these compromises. In the years since I’ve found that this problem generalizes across the entire industry and beyond. When designing the business model for SourceHut, I vowed that this would never be our fate.

SourceHut has never accepted any outside funding, it’s completely bootstrapped. I started the business while working a full-time job and built it in my spare time over the course of two years. Next month will mark the third anniversary of the project and the end of the first year of public alpha. Today, it’s a profitable business and we’ve just brought on our first full-time sponsored free software developer. Some people have complained that paying for their account on SourceHut is a deal breaker. But, consider that the incentives that this approach creates hold us accountable only to the users. When you let venture capitalists foot your bill, you also give them power over you. And in any case, free SourceHut accounts are available by request to anyone with extenuating circumstances².

Thanks to this incentive model, SourceHut can easily skip anti-features:

We don’t send your data to third parties³
We have no tracking whatsoever
Everything works without JavaScript anyway
We don’t use opt-out marketing emails
There are no advertisements in the UI
The hosted code is the same as the open source code
No one can strong-arm us into taking on unethical business partners

Since we get all of our income directly from the users, we don’t have to worry about finding other ways to monetize you. To me that seems like a pretty good business model, even if it’s never going to be a “unicorn”.

UPDATE: Thanks for the feedback. There were many more concerns than we expected. We’re going to process the feedback and rethink our plan. We will not activate product usage tracking on GitLab.com or GitLab self-managed for now. We’ll make sure to communicate in advance on our blog when we do have a new plan.

via gitlab.com/www-gitlab-com!33289

Hi, I’m Drew DeVault, founder of SourceHut and your friendly neighborhood sysadmin. ↩︎
Technically, you don’t have to pay at all during the alpha. ↩︎
The exception is your billing information, but we don’t have much of a choice about that. ↩︎

Sourcehut Q3 2019 Financial report

Mon, 21 Oct 2019 00:00:00 +0000

In summary, Sourcehut is financially healthy, with an operating monthly profit of about $1,038. Slowed growth in Q2 picked back up in Q3, to levels similar to Q1. Though perpetual growth is not a goal of Sourcehut, growth does help us accomplish our goals. One of these goals is to sponsor members of the open-source community to work on self-directed projects - a goal which was met for Q4, when Simon Ser will be joining us. My goals for future growth are:

Investing in additional hardware and resources
Marketing & outreach
Investing in the broader open-source ecosystem

Thank you for your support in the alpha. I’m looking forward to continuing to serve you.

Disclaimer: this report is a summarized approximation of Sourcehut’s financials, and is not used for tax purposes.

Revenue sources

Sourcehut receives revenue from paid user subscriptions. During Q1, we processed 1,134 invoices. The invoices paid break down as:

612     $2  (paid monthly)
282     $5  (paid monthly)
65      $10 (paid monthly)
121     $2  (paid yearly)
43      $5  (paid yearly)
11      $10 (paid yearly)

The total gross revenue during this period was $9,172.00, which after transaction fees comes out to $8,511.83. This is an improvement over Q2, but falls just short of Q1 revenue. In Q3, 1,650 new users registered accounts, of which 204 (12%) have paid for their account. As of the end of Q3, there were 10,331 registered users in total, of which 1,272 (12.3%) have paid subscriptions.

At time of writing (2 weeks into Q4), the breakdown of subscription types is as follows:

223     $2  (paid monthly)
104     $5  (paid monthly)
26      $10 (paid monthly)
582     $2  (paid yearly)
174     $5  (paid yearly)
62      $10 (paid yearly)

The monthly revenue from these subscriptions (with annual subscriptions realized over 12 months) is approximately $3,307, after transaction fees. This is a $734 increase in monthly revenue compared to Q2.

Sourcehut has $10,748 in the bank at the time of writing (2019-10-21).

Breakdown of Q3 expenses:

$1950    Philadelphia datacenter lease (inc. network, power, etc)
$1693    New server equipment
$ 360    Contractor payments (artwork)
$ 129    San Francisco datacenter lease (inc. network, power, etc)
$ 107    Misc
$  95    Research expenses (names.sr.ht)

Payroll:

 $1,500.00   Drew DeVault (CEO)
+$  179.07   Employer taxes & processing fees

Going into Q4, our monthly expenses are expected to be:

$1500/mo    Payroll (approximate)
$640.00/mo  Philadelphia datacenter lease
$129/qtr    San Francisco datacenter lease

Starting in Q4, Sourcehut will be sponsoring the work of free software developer Simon Ser, which is being recorded under Payroll for these quarterly financial reports. To protect his privacy, I have adjusted my personal salary and summed payroll into a single figure. This figure will continue to be opaque until a few more hires are made, after which I will resume disclosure of my personal salary.

The closure of Q4 will mark one year of Sourcehut’s public alpha. Accordingly, yearly subscriptions will be renewing for the first time. What fraction of these subscriptions will bounce, be cancelled, or refunded will have a significant impact our financial planning, and is the biggest financial unknown we face leading into Q4. We’re planning a light marketing drive to remind our subscribers of what we’ve accomplished in 2019, and what we’re planning going into 2020.

What's cooking on Sourcehut? October 2019

Tue, 15 Oct 2019 00:00:00 +0000

I’m more excited to give today’s status update than any other so far. To get the necssary bits out of the way: this month, 368 users have joined the community, bringing our total to 10,539¹. Welcome!

General news: Simon Ser joins us

I’m very excited to announce that Simon Ser will be joining Sourcehut. His role will be interesting, and representative of my vision for Sourcehut in the long-term. Simon’s responsibilities are to simply continue working on self-directed free software projects. I chose to work with him because he’s already a talented generalist and motivated member of the free software community. He’s written more about his specific plans in his own blog post, but with our support he’ll be working on projects like mrsh, Wayland, and the maintenance of large swaths of Golang’s email landscape.

Some of Simon’s work directly benefits Sourcehut, but the goal of sponsoring his work is to help populate the open source ecosystem as a whole with motivated, financially stable developers with the freedom to work at their own direction. If you’ve been following the financial reports, this kind of work is what I plan on spending Sourcehut’s profits on: directly elevating the free and open source software community.

Additionally, thanks to Denis Laxalde’s help, Sourcehut packages are now available for Debian from our new Debian repository. Thanks Denis!

git.sr.ht: web-based patchsets

Many of you recall one of the original promises of Sourcehut: web-driven tooling built on top of an email-driven workflow. This is part of the third broad phase of Sourcehut development. These three phases are: (1) build a bunch of self-contained tools which represent the primitives of a development system; (2) extend them with comprehensive APIs and webhooks; (3) teach them to talk to each other. This design takes longer to bear fruit, but the result is worth it. However, our wait is starting to come to a close: this week, I implemented patchset preparation on the web.

Your web browser does not support the webm video codec. Please consider using web browsers that support free and open standards.

This is now live on git.sr.ht! Please give it a shot and let me know your thoughts. Progress is being made on the other end of this workflow as well - see the lists.sr.ht updates later on. In other git.sr.ht news, some long-requested features like release blobs and git-lfs are showing signs of life. This month, a shiny new 64TB NAS was provisioned and installed in our datacenter, and will be the backbone upon which many of these features are implemented.

lists.sr.ht

I have been working on lists.sr.ht indirectly recently, in the form of improvements to libgit2 and pygit2 upstream. The idea is to expose libgit2’s pluggable backends to pygit2, so that I can create a pygit2.Repository which is backed by git.sr.ht via the git.sr.ht API.

builds.sr.ht

NetBSD folks have been helping me work through the issues, but it’s slow going. If anyone wants to have a go at it, download this image and see if you can’t figure out how to make it expand the partitions and filesystem to fill available space on first boot. It’s generated with these scripts on a NetBSD host.

That same NAS I mentioned in the git.sr.ht update, by the way, is going to be used for build artifacts soon, and likely for build caches as well, which will speed up many classes of builds. Additionally, a POWER9 server has been provisioned and installed in the DC, which I’m preparing to eventually become a ppc64le builder. I’m looking forward to seeing both in action!

paste.sr.ht

paste.sr.ht received some polish this month thanks to the hard work of Mykyta Holubakha. A public index of your pastes, access controls, and soon paste deletion are all being added to the platform.

names.sr.ht

Note: I’ve changed the way the total users are calculated this month. Previously, the number included people who signed up but never confirmed their registration via email. ↩︎

What's cooking on Sourcehut? September 2019

Sun, 15 Sep 2019 00:00:00 +0000

Another month of progress in the Sourcehut alpha! This month, we reached the big 10,000 user mark, and kept going: at the time of writing, there are 10,649 users. To the 938 of you who’ve joined us since the August update, welcome! I’m sure many of you have joined us after abandoning the sinking Bitbucket ship, and if so I hope you’ve found Sourcehut to your liking. If you have any questions or feedback, I can be reached at [email protected]">[email protected]. And a special thanks to the 1,093 users who have paid for their alpha accounts, which for the first time represents more than 10% of all registered users.

General news

First, I’ve redesigned sourcehut.org to be a little bit more attractive and lend an air of professionalism to Sourcehut. I’ve also added to it a blog, which has a greater scope than sr.ht-announce. I will be cross-posting these monthly “what’s cooking” updates there as well, for anyone who prefers to read them on the web or keep updated via RSS. Though the design of sourcehut.org is a bit more flamboyant than sr.ht in general, I’ve made a few conservative improvements to the layout of sr.ht pages, notably centering everything and bumping the base font size up to the system default.

I would also like to thank Denis Laxalde for his hard work lately, thanks to which we’re starting to see a Debian repository come into being for sysadmins who want to run Sourcehut on Debian hosts. I hope we’ll see that finished up soon!

lists.sr.ht

No notable developments this month, but there has been some planning I’d like to share with you. I’ve been working this week on improvements to libgit2 and pygit2 with the aim of improving integration between lists.sr.ht and git.sr.ht (and later hg.sr.ht, but that’s a more difficult problem for technical reasons). To address goals like continuous integration for patches sent to the mailing list, expanding the diff context beyond what’s included in the patch, and a little blue “submit” button to apply patches from the web, I basically intend to create a libgit2 backend which allows you to create a Repository object which, instead of being backed by a git repo on disk, is backed by the git.sr.ht API.

I also intend to add a similar backend for Github and Gitlab, if possible, and let you use lists.sr.ht to track patches against projects which aren’t hosted on Sourcehut. There’ll probably be lots of use-cases for this that I haven’t thought of! One interesting thing you’d be able to do with this is have a working git repository on your local machine which doesn’t have a .git directory, but instead does all of its operations remotely against a git.sr.ht repository - albeit very slowly. I’m not sure why you’d ever want to do this. Interesting nonetheless!

todo.sr.ht

Some minor bug fixes and improvements landed this month, but the most interesting addition is that of participating via email without an account. You can now send an email to the tracker or to a ticket to submit or comment on tickets without a Sourcehut account. If you are the operator of a bug tracker on the service, you can configure the permissions for such users by changing the “anonymous” permissions on your tracker. In the near(ish) future, I also plan on extending ACLs so you can give individuals more or less permission by email, which will make non-account holders first-class citizens on todo.sr.ht, more or less.

builds.sr.ht

I wrote a blog post about how builds.sr.ht is being used by lots of software to improve their BSD support, give it a read if you’re interested. On a related note, I’ve received some offers for aid in finishing our NetBSD image, so hopefully we’ll see support for that soon.

git.sr.ht

Arch Linux developer Eli Schwartz has put some effort into making busybox’s gzip output deterministic and matching GNU’s gzip, which makes our source tarball downloads from git.sr.ht consistently hash with ourselves and with other git hosts. Thanks Eli!

man.sr.ht

I redesigned it, now it doesn’t look like crap. Hooray! As part of this work, you can now explicitly enable or disable the generation of a table of contents for any page by updating its frontmatter.

Sourcehut makes BSD software better

Wed, 11 Sep 2019 00:00:00 +0000

Every day, Sourcehut runs continuous integration for FreeBSD and OpenBSD for dozens of projects, and believe it or not, some of them don’t even use Sourcehut for distribution! Improving the BSD software ecosystem is important to us, and as such our platform is designed to embrace the environment around it, rather than building a new walled garden. This makes it easy for existing software projects to plug into our CI infastructure, and many BSD projects take advantage of this to improve their software.

Some of this software is foundational stuff, and their improvements trickle down to the entire BSD ecosystem. Let’s highlight a few great projects that take advantage of our BSD offerings.

Programming languages: Janet, Nim, Zig

Several programming languages use Sourcehut to run BSD testing. The Janet language uses Sourcehut to test both FreeBSD and OpenBSD, and fast - each build takes only about a minute! The newest language to join us is Nim, which now uses builds.sr.ht to make sure every Github pull request works correctly on FreeBSD and OpenBSD. The Zig project, long time friends of Sourcehut, have also been using builds.sr.ht to test their FreeBSD support, and they host their mailing lists with us, too.

The stability of programming languages is especially important, as bugs in the compiler or implementation will affect all software compiled with it. Thanks for using Sourcehut to keep your BSD support in good shape, guys!

End-user software: Neovim, mutt

End-user software gets in on the fun, too, which puts more stable BSD software closer to your fingertips. The Neovim project, a fast moving and modern fork of the venerable Vim text editor, uses builds.sr.ht for OpenBSD testing, and working on using it to help fix bugs in their FreeBSD port. The similarly venerable mutt email client has also been experimenting with Sourcehut recently. They’re using builds.sr.ht to make sure mutt compiles on FreeBSD, Alpine Linux, and Debian, and they’re trying out lists.sr.ht mailing lists, too.

Display servers: wlroots, sway

One of my own projects, wlroots, was one of the first adopters of Sourcehut, to no surprise. We were also early adopters of FreeBSD support, and now we use it to test that dozens of Wayland compositors work well on the platform. Several of these test themselves further down the stack on FreeBSD as well, such as my own sway project, and the cage project, and more projects still are using Sourcehut to test other operating systems as well.

Small projects, too!

Lots of smaller projects have been taking advantage of Sourcehut’s BSD tools, too, to make sure they’re set up right on for our Berkley friends from the start. Simon Ser’s mrsh shell tests on FreeBSD, Arch Linux, and Alpine Linux. Michael Forney’s samurai project uses Sourcehut builds to test four operating systems, including Free and OpenBSD!

And yours?

Want to try it yourself? Just set your build manifest’s image to freebsd/latest or openbsd/latest and drop it into the submit page to give it a test drive - no need to push any half-baked test commits until it’s done. Check out the compatibility page for details on builds.sr.ht support for *BSD and other operating systems, and check out dispatch.sr.ht to add BSD CI to your Github projects.

Hey! Are you a NetBSD expert? We need some help to finish up NetBSD support on builds.sr.ht. [email protected]">Get in touch?

Sourcehut welcomes Bitbucket refugees

Wed, 21 Aug 2019 00:00:00 +0000

Atlassian announced on Wednesday that Bitbucket is shuttering its Mercurial offering.

For those looking for a new host, Sourcehut may be a good choice for you. Unlike its closed-source competitors, Sourcehut is 100% open source software, and thanks to the Mercurial community (especially Ludovic Chabant), Hg support on Sourcehut is strong and will remain so for as long as the Hg community is committed to it.

Bitbucket users may find our workflow different from what they’re used to, but it’s different for good reasons. Give it an shot and ask for help if you need it. We’ve prepared a migration tool which will import your Bitbucket repos and issues.

What's cooking on Sourcehut? August 2019

Thu, 15 Aug 2019 00:00:00 +0000

Thank you for continuing to support Sourcehut during the alpha period! I have loads of exiting developments to share with you today. Let’s welcome our newest 369 users to the platform, which now totals 9,711 in number.

builds.sr.ht

I’m happy to announce lots of cool features for builds.sr.ht this month. First, the VMs for failed builds are now being kept alive for 10 minutes after the completion of your build, and you can SSH into them to examine the failure more closely (if you do so, the 10 minute deadline is extended to the original time limit for your build).

You can also SSH into build jobs just if you’d find it useful. Need an OpenBSD shell to try something out?

image: openbsd/latest
shell: true

Paste that into builds.sr.ht/submit and you’ll get your SSH connection details a few moments later. You can also still add all of your tasks, packages, repos, and such; perhaps cloning your dotfiles repo and installing your favorite text editor.

SSH access is still in its early stages, but development is well underway on features like submitting build manifests over SSH, e.g.

ssh [email protected] submit < .build.yml

as well as tail -f-ing build logs in your terminal.

Additionally, I have progress to announce on experimental multi-arch support. arm64 is now available for Debian images, and is being kept up-to-date with automated builds. I’ve also made progress on ppc64el support using a similar approach. Currently this is done with software emulation via qemu, but in the next couple of months, I expect to have ppc64le hardware builds available.

The ubuntu/latest image now points to Ubuntu Disco and freebsd/latest to FreeBSD 11.3, following their respective upstream releases.

man.sr.ht

Thanks to the hard work of Ryan Chan and after a complex migration process, a major update to man.sr.ht has been deployed. No longer are the backing git repos stored with man.sr.ht itself - they’ve now been transferred to git.sr.ht repositories and man.sr.ht uses webhooks and the git.sr.ht API to fetch content from them. This means you can now browse your wikis on git.sr.ht using the fully featured git repository browsing interface, but you can also put your wikis directly into the git repos they document under a “wiki” branch or something similar. Thanks to Ryan for all your hard work!

git.sr.ht

A problem which has plagued git.sr.ht for some time now is slow performance on git operations over SSH. Though there’s still many improvements to be made, I spent some time this month whetting down the bottlenecks and, as a result, git pull is 5x faster and git push is 2x faster. Thanks to Preston Carpenter for helping to identify some of the bottlenecks! The remaining bottlenecks are well-understood and I expect to make further improvements soon. Anyone interested in helping on this should reach out on IRC - there’s a lot of cool stuff involved.

Additionally, following the announcement of code annotations for git.sr.ht, several projects have grown which add support for their favorite programming languages:

If you’re working on one yourself, please let me know!

lists.sr.ht

Because Sourcehut requires the use of plaintext email, many email clients will run into this limitation quickly with their default configuration. Previously, you’d get a very user-unfriendly bounce message from postfix which doesn’t give you much help towards fixing this problem. Instead, the bounces are now sent from lists.sr.ht itself and replace the mail server diagnostic info with a friendlier description of the problem and resources to address it - namely, useplaintext.email.

Other error messages which can be returned by lists.sr.ht have received similar treatment.

Interesting projects using Sourcehut

The mutt email client is experimenting with a sr.ht git mirror and mailing list, as well as builds.sr.ht CI.

Gio is an immediate mode GUI toolkit for Golang.

Thanks for using Sourcehut, guys! Be sure to let me know about your new projects - post them to the sr.ht-discuss list.

What's cooking on Sourcehut? July 2019

Mon, 15 Jul 2019 00:00:00 +0000

Hello again, and thank you for your support during SourceHut’s alpha! This month, our ranks have grown by 352 users, bringing our total company to 9,342 users. The big 10K looms over the horizon! I have some very cool developments to share with you this month.

git.sr.ht

The first and perhaps most exciting development is the availability of code annotations on git.sr.ht. This feature allows you to upload a list of annotations to your repo, which can link from function references to their definition (and vice versa), or link to ticket IDs referenced in comments, or anything else you can dream up. The system is quite flexible and should grow easily to support any number of cool use-cases you can dream up.

I’ve written a longer article on my blog which goes into detail on the implementation. If you’d just like to get a quick feel for the feature, here are a few cool annotated repos for you to browse:

scdoc: a man page generator (C)
aerc: a TUI email client (Go)
cproc: a C compiler frontend (C)

Documentation for annotations can be read here. First-party annotators are available for C and Go, and the community is working on annotators for Rust and Python as well. I hope to extend this feature to hg.sr.ht soon as well. If you write an annotator for your favorite programming language, please let us know on the sr.ht-discuss mailing list!

todo.sr.ht

Ivan Habunek has come along with another great patchset this month, this time adding per-user access control lists to your todo.sr.ht trackers. This is similar in implementation to the corresponding lists.sr.ht feature, and can be used to set up private ticket trackers with your collaborators, post-only security trackers, banning troublesome users, and more. I’ve also added support for adding and removing labels over email (by replying to the notification with “!label example” on the last line), and Paul Wise has improved our notification’s email signatures to be RFC 3676-compliant. Thanks for the patches, folks!

lists.sr.ht

A number of improvements have recently landed in lists.sr.ht. You can now search through patchsets and apply bulk updates to them, and can search them via the API as well. Another little feature, the ability to search by the sender’s timestamp, is also going to be useful for discovering the email thread which corresponds to a git or hg commit. Post-only mailing lists (with restricted browse permissions) now also show a more useful summary page to users who may post but not read the emails.

builds.sr.ht

debian/stable now refers to the newly-stablized Debian Buster, and Debian Bullseye has been added as debian/unstable. There is also a patch in my inbox from Simon Ser adding FreeBSD 11.3 support, which I plan on applying later this afternoon. Luca Weiss has also made some changes adding support for cloning repos into a different name than the URL’s basename, which along with improved support for private repos has fleshed out our GitHub integration a bit more. Thanks to both!

man.sr.ht

The realization of this workstream is still a ways out, but I wanted to mention that there have been patches and discussion hanging about on the sr.ht-dev list for overhauling man.sr.ht’s repo storage mechanism. Ryan Chan has been working on moving man.sr.ht’s git repos into your git.sr.ht account, so that you can use git.sr.ht to browse their history and files like any other git repo. We also plan to make wikis available by adding them as branches of existing repositories. This workstream is looking promising and should be completed soon, and acts as exploratory work which will influence similar overhauls for other services in the future (notably dispatch.sr.ht).

meta.sr.ht

Some initial experimentation has been undergone with single-sign-on, which will address the problem of having to log into every *.sr.ht site separately. However, more problems were found than solutions, so this will take some time yet to complete. Once this is out of the way, the plan is to then add multiple email addresses per account and user groups (organizations).

Interesting projects using SourceHut

I’d also like to briefly mention a few cool projects which have moved to SourceHut. First, Alpine Linux has finished their migration to their own lists.sr.ht instance. Additionally, the well known pkgconf software is now available on SourceHut.

Welcome to SourceHut! If you’ve got a project of your own on SourceHut that you want to share, feel free to post about it on sr.ht-discuss. I want to know about the cool stuff you’re building!

What's cooking on Sourcehut? June 2019

Sat, 15 Jun 2019 00:00:00 +0000

Thanks again for following SourceHut during the alpha! I hope you’re all enjoying the service. Remember to send me your feedback, good or bad! Another month passes and another 430 users join our ranks, bringing our total to 8,990 - just shy of 9,000 strong. To our new members: welcome! And to the 874 users who’ve chosen to pay for their account despite the alpha level of quality the service presents today, a special thanks: your support is the reason I’m able to work so hard on the service and free software in general.

aerc

Direct progress on SourceHut has been somewhat slower than usual this month, because I’ve been focusing on pushing a side project to its initial pre-release: aerc. I normally don’t mention my other projects in these articles (though if you’re interested, I suggest checking out my blog), but given the deep focus on email in SourceHut’s design I find that aerc may be relevant to your interests. It’s an email client for your terminal and it offers many design improvements over existing mail solutions. Please check it out if you have the chance!

aerc-mail.org

Back to your regularly scheduled updates.

Mercurial conference update

Ludovic and I attended the Mercurial conference in Paris, and met many of the Mercurial developers. I’d like to extend my gratitude to them for putting on such a great event, and to Ludovic for coming along to make me look like less of a moron! I learned a lot about hg thanks to the patience of the Mercurial devs. We spent a lot of time talking about their plans and discussing design directions for hg.sr.ht. Our offering stands to become the premier hosting destination for the Mercurial open-source community, and I was glad we had a chance to set our course carefully towards that end.

todo.sr.ht

A big update comes today for the bug tracking service: it now has an API and webhooks! The documentation is available here.

todo.sr.ht is the last major SourceHut service to receive these features, putting a bow on this workstream and unblocking a lot of others. The first of these now-unblocked workstreams is an overhaul to dispatch.sr.ht, which will bring us lots of long-awaited features. Check out the design doc I posted to sr.ht-dev the other day.

Feedback welcome!

git.sr.ht

Just minor news on this front: with git.sr.ht now growing to over 6,000 repositories, it has outgrown its old VM. To accommodate for this, I’ve provisioned a new host in our datacenter and given half of it to git.sr.ht, giving it a lot of room for future growth. Thanks for enduring a couple hours of partial downtime this week as I migrated to the new server.

Also, thanks to ~ignaloidas for putting git.sr.ht through a stress test with a repo designed to break web interfaces - found and fixed a few bugs thanks to that!

What's cooking on Sourcehut? May 2019

Wed, 15 May 2019 00:00:00 +0000

Another month of Sourcehut development passes! I’ve got some really cool news to share today. And a big welcome to our 240 new users, who bring our total up to 8,560. Thanks again to everyone for supporting Sourcehut during the alpha! I’d like to briefly remind you that Sourcehut depends on your financial support to thrive - please consider purchasing a subscription if you haven’t yet. We just released our [email protected]%3E">Q1 financial report, which breaks down our income today and explains where your subscription fee ends up:

Please let me know if you have any questions. Now to the development news!

lists.sr.ht

Let’s start with lists.sr.ht, because I have some exciting news: the initial version of web-based code review is now available! Check out an example here.

This is generated from a normal patch review, with no additional human input. You just respond to patch emails like you always have, and lists.sr.ht automatically applies heuristics to incoming emails to generate a web view of that discussion. In the future, I intend to expand on this with support for authoring reviews on the web, as well as expanding git.sr.ht to support sending patchsets from the web.

git.sr.ht, hg.sr.ht

git.sr.ht and hg.sr.ht both now have an API! The API reference can be found here.

Docs for the hg.sr.ht API are pending a design for how to expose hg.sr.ht internals over the API, but is compatible with the git.sr.ht API for any API method not dealing with git internals.

They also support webhooks - and there’s a pretty neat feature that comes with. When you configure a post-update webhook (equivalent to git’s own post-update hook), you have the option of making it synchronous - this will submit the HTTP request during git push, and will print the response body to the console of the person executing git push. This can be used for custom integrations which behave like git.sr.ht does when submitting builds.sr.ht jobs, printing the URLs of the jobs which started as a consequence of your git push.

hg.sr.ht has also grown support for the Mercurial evolve plugin this month - thanks to Ludovic again for working on that! You can enable this support on your repo’s settings, under the “features” tab. Ludovic and I will be attending an Mercurial conference in Paris in 2 weeks, where we hope to discuss this and more features for hg.sr.ht in the future.

todo.sr.ht

You can now submit tickets and participate in discussions on todo.sr.ht by sending an email to the tracker you want to work with. For example, to file a bug for todo.sr.ht, you can email it to ~[email protected]">sircmpwn/[email protected].

You can also do things like closing tickets via email - docs here.

I’ve also started working on an API for todo.sr.ht. It’s fairly complete now, but I still haven’t written the docs. Stay tuned!

builds.sr.ht

I’m happy to announce that OpenBSD is now supported on builds.sr.ht!

Major thanks to Jarkko Oranen for putting in the work to realize this feature. Timothée Floure has also made some improvements to Fedora support, and Francesco Gazzetta added NixOS 19.03. Thanks to Andres Erbsen as well, who fixed a bug with git submodules which have been updated outside of the default branch. My friend minus has also helped me set up an experimental caching proxy for Alpine Linux packages, which shaves a few seconds off of every Alpine Linux job. I’d like to expand this to other distros and things like npm and pypi as well.

meta.sr.ht

Our friends at Gitlab were recently in the news, as several of their users had their repositories overwritten with a ransom notice. The cause of this was users with weak passwords. I’d like to briefly share how Sourcehut is not affected by the same problem, and explain improvements I made to the service anyway.

First, I take a backup of git.sr.ht every 5 minutes. I can restore the state of your repository as it appeared at any point in time in the event that your account is compromised. Additionally, login attempts to sr.ht are throttled and monitored for suspicious activity, and I get an alarm when someone is making a lot of login requests, which quickly leads to an IP block and an investigation of any successful logins.

Even so, I’ve taken this opportunity to improve Sourcehut to prevent this, by enforcing strong password requirements when you sign up and when you reset your password. No, I don’t accomplish this with silly requirements for multiple numbers and letters and symbols and such - instead, I use Dropbox’s zxcvbn library to estimate the entropy of your password and require a minimum inherit complexity. In the case where your password isn’t strong enough, you’ll be shown a message like this:

This password is too weak - it could be cracked in less than 5 minutes if our database were broken into. Try using a few words instead of random letters and symbols. A password manager is strongly recommended.

I haven’t enforced these requirements retroactively, so if you’re concerned about your account you should reset your password here.

You can also review your audit log on this page to check for any suspicious activity on your account. Shoot me an email if you have any concerns.

paste.sr.ht

Thanks to Mykyta Holubakha, you now have the option to set the visibility of your pastes on paste.sr.ht. The “public” and “unlisted” options are presently indistinguishable, though, because there’s no public profile page for listing someone else’s pastes at yet. Hopefully we’ll see that soon!

Blogs on Sourcehut

Proposed price increases for SourceHut

Moving US accounts to Europe

Why do we need to raise prices?

Share your thoughts?

What's cooking on SourceHut? Q4 2025

Drew’s update

Conrad’s update

Everyone else

What's cooking on SourceHut? Q3 2025

Drew’s update

Conrad’s update

Everyone else

SourceHut is now accepting payments in Euro

FAQ

Which payment methods are available?

Will you support more currencies?

Will you support more payment methods?

Will you accept cash or bank transfers again?

I pay in USD now, can I switch to Euro?

Why is SourceHut moving to Europe?

What are your plans for winding down the US legal entity?

What's cooking on SourceHut? Q2 2025

Drew’s updates

Conrad’s updates

Everyone else

You cannot have our user's data

Announcing docs.sourcehut.org

What's cooking on SourceHut? Q1 2025

Drew’s updates

Conrad’s updates

What's cooking on SourceHut? September 2024

Drew’s updates

Conrad’s updates

The European NGI fund must be renewed

Update on our infrastructure plans

Quick recap

The good, the bad, and the ugly

The plot twist

What’s next?

Thank you for your patience

The state of SourceHut and our plans for the future

So far in 2024

In the months to come

Thanks for sticking with us

SourceHut network outage post-mortem

Background

The incident

What’s next

Acknowledgements

Information regarding the SourceHut outage

SourceHut's 2022 Financial report

2022 revenue sources

Platform revenue

Consulting revenue

Summary of revenue

Cash on hand

2022 expenses

2022 financial summary

Expectations for 2023

Conclusion

SourceHut will (not) blacklist the Go module mirror

tl;dr

Background

Seeking other solutions

Recommendations for the Go team

Recommendations for Go users

NLnet funding comes to an end

Four years of SourceHut

The year in review

Coming soon

Coming later

jobs.sr.ht

names.sr.ht

mail.sr.ht?

In conclusion

SourceHut terms of service updates, cryptocurrency-related projects to be removed

What to do if you don’t agree to the changes

Other changes to the terms of service

Applicability of Dutch law