はてなブックマーク

Years ago I wrote about my trouble with a job queue in Postgres, in which table bloat caused by long-running queries slowed down the workers’ capacity to lock jobs as they hunted across millions of dead tuples trying to find a live one. A job queue in a database can have sharp edges, but I’d understated in that writeup the benefits that came with it. When used well, transactions and background job

Anyone who’s seen a couple different production database environments is likely familiar with the “soft deletion” pattern – instead of deleting data directly via DELETE statement, tables get an extra deleted_at timestamp and deletion is performed with an update statement instead: UPDATE foo SET deleted_at = now() WHERE id = $1; The concept behind soft deletion is to make deletion safer, and revers

It’s been a rough month for Heroku. On April 15th, they opened a security notice that’s been active ever since – the status page, designed for incidents measured in hours rather than days or weeks, dutifully reads off a duration of “478 hours, 9 minutes” 1. According to their latest update, an attacker gained access to Heroku’s main database (called core-db back in my day) and exfiltrated its cont

Last week, we deployed a refactoring patch to our Ruby codebase that was big for us 1 – 178 files changed: Changeset. If you’d asked me five years ago how to safely deploy a change of this size in Ruby, I’d have a very simple answer: you can’t. Ruby’s parser is no help in detecting regressions – it’ll notice invalid syntax, and that’s about it. A good test suite helps, but it’s never complete enou

After a few months of research and experimentation with running a heavily DB-dependent Go app, we’ve arrived at the conclusion that sqlc is the figurative Correct Answer when it comes to using Postgres (and probably other databases too) in Go code beyond trivial uses. Let me walk you through how we got there. First, let’s take a broad tour of popular options in Go’s ecosystem: database/sql: Go’s b

One of our most influential late Heroku artifacts was The Heroku HTTP API Design Guide. We’d recently revamped our API with a new V3 iteration, and the guide was a terse description of the conventions we’d developed. It was written by Mark McGranaghan, and then entered common maintenance. It blew up when it was dropped, accumulating 13.5k stars on GitHub, and translated into a number of foreign la

Blog with Markdown + Git, and degrade gracefully through time (Problem 1.) As I was reviving my RSS habit the other week, I couldn’t help but notice how many of my old subscriptions were dead. Not “doesn’t publish anymore” dead, but “gone from the internet” dead. People had created blogs (many with great content), run them for a while, let them lapse, and eventually they were swallowed by time and

Hello from a new decade! It’s already been a great reading day for retrospectives on the last ten years / predictions for the next. Here’s to hoping that everyone had a good break (or at least a few days off), set or didn’t set resolutions, and is refreshed and ready for what comes next. You’re reading the first 2020 edition of Nanoglyph, a fledgling newsletter on software, sent weekly. Theoretica

A month ago, I wrote about how I was frustrated with my progress in Rust. These days, I’m still no expert, but I’ve made progress. Programming languages get easier to learn the more of them you know. A lot of this is just pattern matching: syntax changes, but there’s a lot of overlap in features, and the similarities get more apparent. Some of my recent acquisitions have been very fast to learn –

When it comes to the question of what the right constructs for concurrency that a language should expose to developers, I’m a true believer that Go’s channels and goroutines are as good as it gets. They strike a nice balance between power and flexibility, while simultaneously avoiding the propensity for deadlocks that you’d see in a pthread model, the maintenance hell introduced by callbacks, or t

The software industry as a whole contains a lot of people doing a lot of different things, but for every developer working on new embedded firmware, there’s about ten building the linchpin of modern software – CRUD apps that serve requests over HTTP. A lot of these apps are backed by MVC frameworks like Ruby on Rails or ASP.NET, and backed by ACID-compliant relational databases like Postgres or SQ

Anyone who’s run Unicorn (or Puma, or Einhorn) may have noticed a curious phenomena. Worker processes that have been forked from a master start with low memory usage, but before too long will bloat to a similar size as their parent. In a big production installation, each worker can be 100s of MBs or more, and before long memory is far and away the most constrained resource on servers. CPUs sit idl

If you’ve ever run an app on Heroku, you may have come across log messages produced by the Heroku router and wondered about their unusual formatting. Each line consists of a single level of key/value pairs which are densely packed together compared to other well-known structured formats like JSON. For example: at=info method=GET path=/ host=mutelight.org fwd="124.133.52.161" dyno=web.2 connect=4ms

An alarm goes off and open your laptop. Your job queue has spiked to 10,000 jobs and is still growing rapidly. The bloated queue means that internal components are not receiving critical updates which will eventually compromise the health of the whole system. You start to investigate. The worker processes look healthy and jobs are being worked in a timely manner. Everything else looks normal. Afte

We’ve recently gone on record indicating our commitment to using JSON Schema as the format for describing our API’s, then even further by releasing a set of tools to improve the process of building and working with schema-based HTTP API’s. With the recent rise of great API description formats over the last few years like Swagger, Blueprint, and RAML (among others), I wanted to write a few words on