Evert Pot's blog

Putting Curveball in maintenance mode

2024-12-29T15:00:00+00:00

Curveball is a server-side webframework for Node.js. I’ve been working on it for 6 years as a side-project, but it wasn’t successful and it’s time to let it go. This is a hard thing to do, because I probably have 1000’s of hours into this project, and there’s a strong sense of missed potential.

I’m writing this article because my psyche requires closure in order for them to not take up space in my mind. This decision was probably a few years late. Hopefully this creates more space for new things.

Some quick numbers.

Number of NPM packages: 25
Number of released packages: 377
Total number of commits: 4882
Contributers: 32
Total downloads from NPM: 1,254,023
Active users: 10?

Why build another framework

In 2018, Typescript just really started popping off and at the time I was deeply interested in optimizing hypermedia-centric REST APIs, Koa was a favourite framework of mine, but it was missing some features I was particularly interested in.

Good Typescript support (This is pretty much fixed in Koa, but at the time it was buggy).
The ability to do internal “sub-requests” without going over the network. (think: hitting a different route in your framework with a local function). Koa (and express) wrap the http Node.js library, and don’t have a good internal abstraction for requests and responses. This makes sub-requests hard and deploying into non-node runtimes such as AWS Lambdas very painful and requires mocking the whole Node http library or worse: run full HTTP server inside the lambda function just so requests can be forwarded. New frameworks like Hono do this perfectly these days, but this wasn’t around at the time.
Native ‘JSX as a template engine’ support. Hono also does this.
Deeply integrated Websockets. Probably Bun does this best at the moment.
Support HTTP/2. A big motivator for me was being able to push resources down the pipe with push, and solve the N+1 problem for REST. Push is pretty much dead now, so less of a priority.
Good support for 103 Early Hints. Not seeing a lot love for this in the new batch of frameworks.

Why did Curveball fail

If I’m being generous with myself, I think the tech was decent and generally liked by those using it. It was fairly sticky, but it never really reached anything close to critical mass.

Documentation was bad, and crucially marketing was pretty much non-existant. Over the entire 6 years it was mostly me working on it (with some great smaller contributions!), but I have a very hard time with marketing and speaking about the project with some consistency. The last time I blogged about a release was in 2020. It never had a chance.

I made the same mistake I made many times before, thinking ‘If I build it, they will come’ and I guess I had a hope it would attract some early strong contributors that could help pick up the slack in areas where I was weak, but obviously this never happened.

I also don’t see how I could have gotten the time and motivation to promote the project. I find self-promotion somewhat painful to do and Curveball was always at best just incrementally better than what was out there when I started it, so not a lot of people are going to take the chance on something with low user numbers, even if they were able to find it.

I’m still interested in Open Source and love building things, but probably the biggest lesson here is that if I want it to be successful I need to do group projects with people who make up for the qualities I lack.

I’m using Curveball for my project, now what?

The project has been very stable and only had 1 major breaking change, which was the move to ESM.

Express has gotten by without a major release in over a decade, so if you’re using Curveball today I think there’s very little chance you need to do anything in a long while. I will even fix bugs as they are reported, but this blog post is effectively my statement that I’m no longer going to put much energy in it. Curveball is stable. Don’t use it for new projects.

If you need advice on a migration path, I’m happy to do a free 30 minute consultation! You probably know how to reach me.

If you like the project, why can’t you work on it despite not having many users

I get a lot of energy from working with people, getting feedback and seeing my ideas validated. If this is not happening, to me it feels a bit like writing a book that nobody reads. It’s a constant reminder of failure.

Are there any bits salvagable?

I’m going to rename @curveball/http-errors because it sucks less than some other HTTP error packages, it’s not tied to the framework and is generally useful.
I think the built-in HATEOAS API browser is great and I’m going to port it to a new framework. (most likely Hono as it’s the closest in spirit).
a12n-server, which is an authn/authz server similar to keycloak is still very interesting to me. It needs its own brand and logo and will port this project to a different framework at some future date. Despite being under the ‘curveball’ namespace I always saw this as its own project. This project does suffer from a similar problem though. Low uptake and just me maintaining it. I need to set some goals for 2025 and kill this as well if I don’t meet them.

Thank you

If you’re a current user of Curveball: thank you for giving this a shot! I hope this news is not too frustrating.

To my friends that gave me encouragement and feedback over the years: I love you and appreciate you!

In the future using top-level await might be cause a backwards compatibility break in Node

2024-10-18T01:13:00+00:00

Node 23 was released this week, and the hot ticket item probably is the fact that you can now require() files that use ESM (import/export).

This is helpful because ESM and CommonJS (require/module.exports) are kind of different worlds and before this change if you wanted to use a “module” from your CommonJS file, you would need to do something like:

const theThing = await import('some/module/file.mjs');

This is called a dynamic import and it’s extra annoying, because you can’t just put this at the top of your file. It returns a Promise, so you can only import these modules ‘asynchronously’ or in functions. The reason is that you can only await inside functions in CommonJS files. So this syntax prevents importing a “module” and immediately using it. To use a module in CommonJS, you’ll need some kind of initialization logic.

Many Javascript packages resort to shipping both a ‘CommonJS’ and a ‘ESM’ version of their code to reduce this kind of annoyance, but not everyone does.

The Node 23 change

Node 23 makes it possible to load ESM modules transparently via require(). This means that the above example can be rewritten to:

const theThing = require('some/module/file.mjs');

The important part here is not the usage of require, but the absense of await. This allows ESM modules to be loaded without this kind of initialization logic.

But there’s a big caveat:

This only works if the module you loaded in is not using top-level await. “Top level await” means awaiting things outside of (async) functions, which is possible in modules (but not CommonJS).

If a top-level await was used, a ERR_REQUIRE_ASYNC_MODULE error will be thrown. But the important thing to note is that this doesn’t just apply to the file you are directly importing/requiring. It also applies to any files loaded by that file, at any level in either your project or any dependency or sub-dependencies.

Using top-level await is now a BC break

Typically when we think of backwards compatibility breaks that require a major new version in semver, you might think of functions changing, or removing or arguments no longer being supported.

Before this change in Node, if your project was fully switched to ESM you might not think of placing a top-level await anywhere in your code as a backwards compatibility break, but if it’s the first await you might now inadvertently break Node.js users, if they used require() to bring in your module.

This means that the first top-level await in your project (or any of your dependencies) might now constitute a new major version if you follow semver.

If you don’t want to do this, here are some other things you could do:

1. Tell your users you don’t support require()

You could effectively tell them if they use require() they’re on their own. Chances are that your users won’t read this and do it anyway, but arguably they should read the readme of a package they bring in.

2. Add a dummy await

Do you think you or your dependencies might use a top-level await in the future, but you don’t yet? You could add this line to your source to reserve the right to do it later:

await "Good things come to those that support await"

3. Explictly break CommonJS

Packages can provide both CommonJS and ESM support via the exports key in package.json. It could be an option to export a single CommonJS file that just throws an error to warn CommonJS users they shouldn’t use this package.

Out of these 3 options, I kind of like 2 because if Node.js ever changes behavior and does support loading modules that use top-level await in CommonJS files it will one day just start working.

Regardless. it does feel important to me that package maintainers that only ship ESM, or ship CommonJS and intend to use this new feature have a strategy, because it’s bound to blow up if not.

How do other runtimes do it?

In Bun you can use import and require in the same file, so there’s fewer trade-offs to make. But if you do use require() to import a module with top-level await it still has the same issue. There’s just fewer reasons in Bun to even want to do this.

Deno doesn’t support CommonJS so it doesn’t have that problem.

Does the issue exists when importing CommonJS files into modules?

Importing CommonJS files into modules was always possible via the standard import syntax, and because CommonJS can’t asynchronously export this problem doesn’t exist there.

What should I do as a user?

Most of this article was about the effects for package maintainers, but not every package will do this well so as a user you’re still potentially exposed to this issue.

In order of preference, consider the following:

Stop using CommonJS. Start using ESM everywhere. This issue only exists in CommonJS files using ESM, not the reverse. ESM is the future. You’re working with dead-end technology.
Have some testing in place that at the very least loads the entirely of your code-base. If all (top-level, not-dynamic) requires() are ran/resolved, any changes in depndencies should just blow up your CI environment and application.
Find out if any of your dependencies are ESM-only, and never require() them. This protects you from issues with direct dependencies. But keep in mind that this can still be an issue in sub-dependencies even if your direct dependencies are CommonJS themselves!

My opinion

For the above reasons I don’t think this should land in a stable Node.js version. The intentions are good but it creates yet another avenue of confusion.

It effectively expands the number of Javascript flavors from 2 to 3:

CommonJS
ESM
ESM but without top-level await.

If a module uses flavor #3, it’s compatible with flavor #1, but if anywhere in the dependency tree something starts using top-level await, suddenly the entire tree ESM dependency tree cascades from flavor #3 to #2 and breaks compatibilty with flavor #1.

Given the already confusing landscape (and reputation) of Javascript modules, this feels like a step in the wrong direction, as it adds a feature to CommonJS (which in my opinion is time to freeze), and makes ESM less reliable as a result.

Discovering features using HTTP OPTIONS

2024-10-16T13:44:00+00:00

Say you have an API, and you want to communicate what sort of things a user can do on a specific endpoint. You can use external description formats like OpenAPI or JSON Schema, but sometimes it’s nice to also dynamically communicate this on the API itself.

OPTIONS is the method used for that. You may know this HTTP method from CORS, but it’s general purpose is for clients to passively find out ‘What can I do here?’.

All HTTP clients typically support making OPTIONS request. For example with fetch():

const response = await fetch(
  'https://example.org',
  {method: 'OPTIONS'}
);

A basic OPTIONS response might might look like this:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS

Based on the Allow header you can quickly tell which HTTP methods are available at a given endpoint. Many web frameworks emit this automatically and generate the list of methods dynamically per route, so chances are that you get this one for free.

To find out if your server does, try running the command below (with your URL!):

curl -X OPTIONS http://localhost:3000/some/endpoint/

One nice thing you could do with the Allow header, is that you could also communicate access-control information on a very basic level. For example, you could only include DELETE and PUT if a user has write access to a resource.

Accept and Accept-Encoding

There’s server other standard headers for discovery. Here’s an example showing a few at once:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS
Accept: application/vnd.my-company-api+json, application/json, text/html
Accept-Encoding: gzip,brotli,identity

You may already be familiar with Accept and Accept-Encoding from HTTP requests, but they can also appear in responses. Accept in a response lets you tell the client which kind of mimetypes are available at an endpoint. I like adding text/html to every JSON api endpoint and making sure that API urls can be opened in browsers and shared between devs for easy debugging.

The Accept-Encoding lets a client know in this case that they can compress their request bodies with either gzip or brotli (identity means no compression).

Patching, posting and querying

3 other headers that can be used are Accept-Patch, Accept-Post and Accept-Query. These three headers are used to tell a client what content-types are available for the PATCH, POST and QUERY http methods respectively.

For all of these headers, their values effectively dictate what valid values are for the Content-Type header when making the request.

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: OPTIONS, QUERY, POST, PATCH
Accept-Patch: application/json-patch+json, application/merge-patch+json
Accept-Query: application/graphql
Accept-Post: multipart/form-data, application/vnd.custom.rpc+json

In the above response, the server indicates it supports both JSON Patch and JSON Merge Patch content-types in PATCH requests. It also suggests that GraphQL can be used via the QUERY method, and for POST it supports both standard file uploads and some custom JSON-based format.

Typically you wouldn’t find all of these at the same endpoint, but I wanted to show a few examples together.

Where’s PUT?

Oddly, there’s no specific header for PUT requests. Arguably you could say that GET and PUT are symmetrical, so perhaps the Accept header kind of extends to both. But the spec is not clear on this.

I think the actual reality is that Accept-Patch was the first header in this category that really clearly defined this as a means of feature discovery on OPTIONS. Accept-Post and Accept-Query followed suit. I think Accept-Patch in OPTIONS was modelled after in-the-wild usage of Accept in OPTIONS, even though the HTTP specific doesn’t super clearly define this.

If I’m wrong with my interpretation here, I would love to know!

Aside: If you’re wondering about DELETE, DELETE should never have a body, so all a user would need to know is can they delete, which you can see in the Allow header. If this is new to you to, read my other article about GET request bodies. Most of the information there is applicable to DELETE as well.

Linking to documentation

The OPTIONS response is also a great place to tell users where to find additional documentation. In the below example, I included both a machine-readable link to a documentation site, a link to an OpenAPI definition, and a message intended for humans in the response body:

HTTP/1.1 200 OK
Date: Mon, 23 Sep 2024 04:45:38 GMT
Allow: GET, QUERY, OPTIONS
Link: <https://docs.example.org/api/some-endpoint>; rel="service-doc"
Link: <https://api.example.org/openapi.yml>; rel="service-desc" type="application/openapi+yaml"
Content-Type: text/plain

Hey there!

Thanks for checking out this API. You can find the docs for this
specific endpoint at: https://docs.example.org/api/some-endpoint

Cheers,
The dev team

I recommend keeping the response body as mostly informal and minimal any real information should probably just live on its own URL and be linked to.

I used the service-doc and service-desc link relationships here, but you can of course use any of the IANA link relationship types here or a custom one. Also see the Web linking spec for more info.

Obscure uses

WebDAV usage

WebDAV, CalDAV and CardDAV also use OPTIONS for feature discovery. For example:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 05:01:50 GMT
Allow: GET, PROPFIND, ACL, PROPPATCH, MKCOL, LOCK, UNLOCK
DAV: 1, 2, 3, access-control, addressbook, calendar-access

The server-wide asterisk request

Normally HTTP requests are made to a path on the server, and the first line looks a bit like the following in HTTP/1.1:

GET /path HTTP/1.1

But, there are a few other “request line” formats that are rarely used. One of them lets you discover features available on an entire server, using the asterisk:

OPTIONS * HTTP/1.1

The asterisk here is not a path. Normally asterisks aren’t even allowed in URIs. Many HTTP clients (including fetch()) don’t even support this request.

Classic webservers like Apache and Nginx should support this. To try it out, use CURL

curl -vX OPTIONS --request-target '*' http://example.org

Final notes

If you have a reason to allow clients to discover features on an endpoint, consider using OPTIONS instead of a proprietary approach! As you can see in many of these examples, it’s especially useful if you use mimetypes well.

If you have questions, other novel uses of OPTIONS or other ideas around feature discovery, you can respond via:

New Structured Fields RFC out, and so is my Javascript package

2024-10-03T15:56:31+00:00

A new RFC was released for Structured Fields: RFC9651.

What is it?

HTTP headers have been a bit of a free-for all in terms of how complex values are encoded, with many headers requiring their own mini-parser.

A while back an effort was started to fix this for headers going forward, named ‘Structured Fields’. They’re called Fields and not ‘Headers’ because HTTP has both Headers and Trailers!

Structured fields let you encode things like lists, dictionaries, strings, numbers, booleans and binary data. The original RFC from 2021 is pretty successful and although many existing headers can’t be retrofitted to this format, a lot of new standards are taking advantage.

Some examples:

# Parsed an ASCII string
Header: "foo"

# A simple string, called a 'Token' in the spec
Header: foo

# Parsed as number
Header: 5
Header: -10
Header: 5.01415

# Parsed into boolean
Header: ?1
Header: ?0

# Binaries are base64 encoded
Header: :RE0gbWUgZm9yIGEgZnJlZSBjb29raWU=:

# Items can have parameters
Header: "Hello world"; a="5"

# A simple list
Header: 5, "foo", bar, ?1

# Each element can have parameters
Header: sometoken; param1; param2=hi, 42

# A list can also contain lists itself. These are called 'inner lists' and
# use parenthesis
Header: sometoken, (innerlistitem1 innerlistitem2), (anotherlist)

# A simple dictionary
Header: fn="evert", ln="pot", coffee=?1

# Each item may have parameters too
Header: foo=123; q=1, bar=123, q=0.5

# A dictionary value may be an inner list again
Header: foo=(1 2 3)

The new RFC published last week adds 2 new data types: Dates and ‘Display strings’, which is a Unicode serialization that fits in the HTTP header (and trailer) format.

# Parsed into a Date object
Header: @1686634251

# A Unicode string, called a 'Display String' in the spec. They use
# percent encoding, but encode a different set of characters than
# URLs.
Header %"Frysl%C3%A2n"

Why should you care?

If you encounter these headers in the wild, it’s a really good idea to use a standard parser. One of the reasons is that with using structured-fields, there’s a built-in extension mechanism. You’ll want to make sure that when a new parameter appears your application doesn’t suddenly break.

You may also want to define and use your own HTTP headers. The structured fields format is a very good ‘default choice’ that removes decisions such as ‘How should I encode a key value object’ or ‘how do I encode a UTF-8 string’.

With parsers popping up for every language, you don’t have to worry about writing your own one-off formats.

Javascript package

I’m the maintainer of a Javascript library for Structured Fields, called “structured headers”, which I’ve also updated for this new RFC. I wish I picked the name “structured-fields”, but I picked the name before the original standard changed it’s name.

I’ve just released v2 of this library supporting these new types, and also added ES Modules support.

Comments?

Reply to one of these:

Hello World, meet Kian

2024-10-02T15:08:31+00:00

One week ago on September 24th my son Kian was born, after a 5 year fertility journey with my wife Roxy. Roxy and Kian are well and I’m really excited for everything that comes next!

Creating a fake download counter with Web Components

2024-07-02T16:07:33+00:00

Over the years I’ve written several open source libraries. They’re mostly unglamorous and utilitarian, but a bunch of them obtained got a decent download count, so I thought it would be fun to try and get a grand total and show a ‘live’ download counter on my blog.

This is how that looks like:

My open source packages were downloaded roughly 138945563 times.

Like most live counters, this number isn’t tracked in real-time. Instead it just uses a start number and updates the number based on an average number of downloads.

One day it might be nice to make it live, but this is a static blog and this would need proper hosting and a database.

Why is this number so high?

This data comes from NPM and Packagist, and they both count any download. So this number doesn’t represent necessarily 138 million users, but simply this many downloads by any means including bots, CI environments and so on. I think for both package managers the goal was probably not to have a realistic representation of users, but rather a number that makes developers feel good. And I like that. It’s nice to see a number go up and it’s still a nice proxy for relative popularity.

The Web Component

This seemed like a good use-case for a web component. Always wanted to build one! I was surprised how easy it was.

This is how this looks in the HTML:

<p>
  My open source packages were downloaded roughly
  <strong>
    <download-counter inc-per-day="157444" date="2024-06-29T15:34:00Z" >
      138945563
    </download-counter>
  </strong> times.
</p>

What’s nice is that if Javascript is not enabled, or Web Components are not supported, this will just fall back on showing the static number.

I included 3 parameters:

The last recorded download count (in the element value)
When that number was recorded (date)
Average number of downloads per day.

I wanted to include the date because I only intend to update these numbers rarely, so we need to know how many downloads have elapsed since the last time.

Writing the web component was surprisingly straightforward too. Here is it in its fully glory:

class DownloadCounter extends HTMLElement {

  connectedCallback() {
    this.count = +this.textContent;
    this.date = new Date(this.getAttribute('date'));
    this.inc = (+this.getAttribute('inc-per-day')) / (3600 * 24 * 1000)
    this.calculateCurrentDownloads();

  }

  calculateCurrentDownloads() {

    const currentDownloads =
      Math.floor(
      this.count +
      (Date.now()-this.date) * this.inc
      );

    // Intl.NumberFormat adds thousands separators
    this.textContent = Intl.NumberFormat().format(currentDownloads);

    setTimeout(
      () => this.calculateCurrentDownloads(),
      // Add some randomnes
      Math.floor(Math.random() * 150)+50,
    );

  }

}

customElements.define(
  'download-counter',
  DownloadCounter,
);

Now just include the file in the HTML and you’re good to go:

<script type="module" src="/assets/js/downloadcounter.mjs"></script>

Obtaining the data

I’ve published 20 PHP libraries on packagist and 30 Javascript libraries on NPM. This is too much to count, so instead I wrote some scripts that pull in the data for their respective APIs.

You cannot easily ask these APIs what the download count or download rate at a given date was, and for both the numbers might be a bit delayed. So I’ve opted to simply:

Get a grand total of all downloads up until this date.
Wait at least 24 hours or more.
Get another grand total and use the difference to caclulate average download rate.

Below is my approach for getting the numbers from NPM and Packagist. It’s a bit messy and imperative.

NPM

The NPM api gave me weird, incomplete results for getting the whole list of packages I’ve authored, so I worked around this by hardcoding some package names, and then augmenting this with the result of 3 searches.

This gives us the full list of packages I’m interested in:

// Searches

const npmSearches = [
  // by author
  'author:evrt',

  // by org name
  'scope:badgateway',
  'scope:curveball',
];

// Hardcoded extra packages that for some reason didn't get returned with
// the searches
const npmPackages = {
  'davclient.js': 0,
  'structured-headers': 0,
  'bigint-money': 0,
  'fetch-mw-oauth2': 0,
  'hal-types': 0,
  'react-ketting': 0,
  'ketting': 0,
  'html-form-enhancer': 0,
  'changelog-tool': 0,
};


async function fetchNpmPackageList() {

  for(const search of npmSearches) {

    const res = await fetch('https://registry.npmjs.org/-/v1/search?text=' + search);
    const body = await res.json();

    for(const object of body.objects) {
      npmPackages[object.package.name] = 0;
    }
  }

}

NPM doesn’t return absolute total download counts but instead we can get a count for a specific time range with a maximum range of 18 months. I’ve opted to instead to get download counts per year, counting backwards for each package until get a result of 0.

async function fetchNpmDownloadCounts(packageName) {

  let year = new Date().getFullYear();
  let count = 0;
  let yearCount;

  do {
    yearCount = await fetchNpmDownloadCountsByYear(packageName, year);
    count += yearCount;
    year--;
  } while (yearCount > 0);

  console.log('%s: %i', packageName, count);
  return count;
}

async function fetchNpmDownloadCountsByYear(packageName, year) {

  const res = await fetch(`https://api.npmjs.org/downloads/point/${year}-01-01:${year}-12-31/${packageName}`);
  const body = await res.json();
  return body.downloads;

}

Packagist

Getting totals from packagist was considerably easier:

const packagistOrgs = [
  'sabre',
  'evert',
];

const packagistPackages = {};

async function fetchPackagistPackages() {

  for(const vendor of packagistOrgs) {
    const res = await fetch(`https://packagist.org/packages/list.json?vendor=${vendor}`);
    const body = await res.json();
    for(const pkg of body.packageNames) {
      packagistPackages[pkg] = 0;
    }
  }

}

async function fetchPackagistDownloadCounts(packageName) {

  const res = await fetch(`https://packagist.org/packages/${packageName}/stats.json`);
  const json = await res.json();
  const count = json.downloads.total;

  console.log('%s: %i', packageName, count);

  return count;

}

Putting it together

async function main() {

  await fetchNpmPackageList();
  await fetchPackagistPackages();

  for(const pkg of Object.keys(npmPackages)) {
    npmPackages[pkg] = await fetchNpmDownloadCounts(pkg);
  }

  for(const pkg of Object.keys(packagistPackages)) {
    packagistPackages[pkg] = await fetchPackagistDownloadCounts(pkg);
  }

  const packagesCombined = [];
  for(const [packageName, downloads] of Object.entries(npmPackages)) {
    packagesCombined.push({
      ecosystem: 'npm',
      packageName,
      downloads
    });
  }
  for(const [packageName, downloads] of Object.entries(packagistPackages)) {
    packagesCombined.push({
      ecosystem: 'packagist',
      packageName,
      downloads
    });
  }
  console.log(packagesCombined);
  console.log('Total: %i', packagesCombined.reduce((acc, cur) => acc+cur.downloads, 0));

}
await main();

Conclusion

Whenever I need small bits of Javascript to enhance a web application (and not a full-blown framework) I tend to write code that looks for an element with a particular class, and then start my logic and hook up events.

Web Components seems like a really great replacement for that.

Got comments? Liked this aticle? You can reply to this Mastodon post to make the comments show up here.

Moving on from Mocha, Chai and nyc.

2024-04-21T03:00:00+00:00

I’m a maintainer of several small open-source libraries. It’s a fun activity. If the scope of the library is small enough, the maintenance burden is typically fairly low. They’re usually mostly ‘done’, and I occasionally just need to answer a few questions per year, and do the occasional release to bring it back up to the current ‘meta’ of the ecosystem.

Also even though it’s ‘done’, in use by a bunch of people and well tested, it’s also good to do a release from time to time to not give the impression of abandonment.

This weekend I released a 2.0 version of my bigint-money library, which is a fast library for currency math.

I originally wrote this in 2018, so the big BC break was switching everything over to ESM. For a while I tried to support both CommonJS and ESM builds for my packages, but only a year after all that effort it frankly no longer feels needed. I was worried the ecosystem was going to split, but people stuck on (unsupported) versions of Node that don’t support ESM aren’t going to proactively keep their other dependencies updated, so CommonJS is for (and many others) in the past now. (yay!)

Probably the single best way to keep maintenance burden for packages low is to have few dependencies. Many of my packages have 0 dependencies.

Reducing devDependencies also helps. If you didn’t know, node now has a built-in testrunner. I’ve been using Mocha + Chai for many many years. They were awesome and want to thank the maintainers, but node --test is pretty good now and has pretty output.

It also:

Is much faster (about twice as fast with Typescript and code coverage reporting, but I suspect the difference will grow with larger code bases).
Easier to configure (especially when you’re also using Typescript. Just use tsx --test).
It can output test coverage with (--experimental-test-coverage).

Furthermore, while node:assert doesn’t have all features of Chai, it has the important ones (deep compare) and adds better Promise support.

All in all this reduced my node_modules directory from a surprising 159M to 97M, most of which is now Typescript and ESLint, and my total dependency count from 335 to 141 (almost all of which is ESLint).

Make sure that Node’s test library, coverage and assertion library is right for you. It may not have all the features you expect, but I keep my testing setup relatively simple, so the switch was easy.

OAuth2 client updates

2024-02-05T15:00:00+00:00

I just released v2.3.0 of @badgateway/oauth2-client, which I wrote because there weren’t any lean, 0-dependency oauth2 clients with modern features such as PKCE.

This new version includes support for:

Resource Indicators for OAuth 2.0 (RFC8707).
OAuth2 Token Revocation (RFC7009).

Hope you like it!

Using JSX on the server as a template engine

2023-11-14T13:14:02+00:00

The React/Next.js ecosystem is spinning out of control in terms of magic and complexity. The stack has failed to stay focused and simple, and it’s my belief that software stacks that are too complex and magical must eventually fail, because as sensibilities around software design change they will be unable to adapt to those changes without cannibalizing their existing userbase.

So while React/Next.js may be relegated to the enterprise and legacy systems in a few years, they completely transformed front-end development and created ripple effects in many other technologies. One of many great ideas stemming from this stack is JSX. I think JSX has a chance to stay relevant and useful beyond React/Next.

One of it’s use-cases is for server-side templating. I’ve been using JSX as a template engine to replace template engines like EJS and Handlebars, and more than once people were surprised this was possible without bulky frameworks such as Next.js.

So in this article I’m digging into what JSX is, where it comes from and how one might go about using it as a simple server-side HTML template engine.

What is JSX?

JSX is an extension to the Javascript language, and was introduced with React. It usually has a .jsx extension and it needs to be compiled to Javascript. Most build tools people already use, like ESbuild, Babel, Vite, etc. all support this natively or through a plugin. Typescript also natively supports it, so if you use Typescript you can just start using it without adding another tool.

It looks like this:

const foo = <div>
  <h1>Hello world!</h1>
  <p>Sup</p>
</div>;

As you can see here, some HTML is directly embedded into Javascript, without quotes. It’s all syntax. It lets you use the full power of Javascript, such as variables and loops:

const user = 'Evert';
const todos = [
  'Wash clothes',
  'Do dishes',
];

const foo = <div>
  <h1>Hello {evert}</div>
  <ul>
    {todos.map( todo => <li>todo</li>)}
  </ul>
</div>;

It has a convention to treat tags that start with a lowercase character such as <h1> as output, but if the tag starts with an uppercase character, it’s a component, which usually is represented by a function:

function HelloWorldComponent(props) {

  return <h1>Hello <span>{props.name}</span></h1>

}

const foo = <section>
  <HelloWorldComponent name="Evert" />
</section>;

Anyway, if you’re reading this you likely knew most of this but it’s important to state that this are all JSX features, not React.

#Inspo

JSX probably has it’s roots in E4X (and more directly XHP). E4X was a way to embed XML in Javascript. E4X has been supported in Firefox for years, and was a part of ActionScript 3 onwards, but there’s a major conceptual difference between E4X and JSX.

With E4X you embed XML documents as data, similar to defining a JSON object in a Javascript file. After defining the XML document you can manipulate it. A fictional transpiler for E4X could (as far as I can tell) could just take the XML structure and turn it into a string and pass it to DOMParser.parseFromString(). In E4X there are no variables, functions, components, or logic. Similar to how a Regular expression is part of the JS language.

JSX is quite different to this. It’s fully integrated in the language and it effectively compiles down to nested function definitions. These functions are don’t get turned into HTML until they are called by some render function. Before this they are defined but not evaluated.

So while E4X and JSX share that they both make XML/HTML first-class citizens in the language, the goals and features are wildly different. JSX really is a DSL.

JSX Transpiled

Lets turn the previous example into Typescript, and see what Typescript does with it:

function HelloWorldComponent(props: { name: string }) {

  return <h1>Hello <span>{props.name}</span></h1>

}

const foo = <section>
  <HelloWorldComponent name="Evert" />
</section>;

By default, Typescript will turn it into this:

function HelloWorldComponent(props) {
    return React.createElement("h1", null,
        "Hello ",
        React.createElement("span", null, props.name));
}
const foo = React.createElement("section", null,
    React.createElement(HelloWorldComponent, { name: "Evert" }));

Now, there’s definitely a react dependency here, but in recent versions of Typescript, we can configure it to use a different ‘factory’ for JSX. By setting the jsx and jsxFactory settings in tsconfig.json, we can get Typescript to output more generic code:

function HelloWorldComponent(props) {
    return _jsxs("h1", { children: ["Hello ", _jsx("span", { children: props.name })] });
}
const foo = _jsx("section", { children: _jsx(HelloWorldComponent, { name: "Evert" }) });

So what’s _jsx? This is a ‘jsx factory’ function that can be provided by React, but also a number of other libraries such as Preact or Solid.js. It’s also relatively easy to create your own.

What’s interesting then about JSX is that while it’s syntax, it’s not always clear what this yields:

const foo = <h1>Sup!</h1>

Because what’s in foo depends on what JSX Factory was used during transpiling. This is frustrating because it requires out of band information and external configuration, but also a benefit because it opens the doors to alternative implementations.

Using JSX as a template engine

When generating HTML on the server with Node, it’s pretty common to use template engines such as EJS or Handlebars. When building a new (multi-page) web application, it occurred to me that neither are quit as good as JSX.

Some of the advantages of JSX are:

Deep IDE/Language server/intellisense integration, because it’s all syntax.
Static type checking with Typescript.
It also enforced correctly structured HTML. No way to forget to close an element.
By default dynamic data is escaped. They made it challenging to get unescaped HTML!

This had me wondering, can we use server-side microframeworks such as Koa, Fastify or Express but instead of string-based template parsers and get all the benefits of JSX?

Turns out the answer is, yes! It’s not only pretty simple to implement, it works exceedingly well.

I’ve implemented this pattern 3 times now for different frameworks, here’s some sample code that works:

Koa

Here’s an example of a small controller:

import Router from 'koa-router';

const router = new Router();

router.get('/foo.html', ctx => {

  ctx.response.body = <h1>Hello world with JSX in Koa!</h1>;

});

As you can see here we can set JSX straight on the body property. A middleware ensures this gets transformed into HTML.

Fastify

Similar to Koa, we can use JSX instead where otherwise HTML strings would be used:

import { FastifyInstance } from 'fastify'

export function routes(fastify: FastifyInstance) {

  fastify.get('/hello-world', request => <h1>Sup Fastify!</h1>);

}

How does this magic work?

For both of these I used the React library. Despite it’s ecosystem being rather bulky, standalone React is still pretty lean and highly optimized.

For both of these frameworks, I simply created a middleware that checks if the body is a React node, and if so use React’s renderToStaticMarkup to do the transformation before the response is sent.

Koa middleware

import { renderToStaticMarkup } from 'react-dom/server';
import { isValidElement } from 'react';
import { Context, Middleware, Next } from 'koa';

export const jsx: Middleware = async(ctx: Context, next: Next) => {

  await next();

  if (isValidElement(ctx.response.body)) {
    ctx.response.body = '<!DOCTYPE html>\n' + renderToStaticMarkup(
      ctx.response.body
    );
    ctx.response.type = 'text/html; charset=utf-8';
  }

};

This is how it’s used:

const application = new Koa();
application.use(jsx);

Fastify hooks

This one needed a few more hacks, but the result is worth it:

import { onSendHookHandler, preSerializationHookHandler } from 'fastify';
import { isValidElement } from 'react';
import { renderToStaticMarkup } from 'react-dom/server';

/**
 * The preserialization hook lets us transform the response body
 * before it's json-encoded.
 *
 * We use this to turn React components into an object with a ___jsx key
 * that has the serialized HTML.
 */
export const preSerialization: preSerializationHookHandler<unknown> = async (_request, reply, payload: unknown) => {

  if (isValidElement(payload)) {
    reply.header('Content-Type', 'text/html');
    return {
      ___jsx: '<!DOCTYPE html>\n' + renderToStaticMarkup(payload as any)
    };
  } else {
    return payload;
  }

};

/**
 * The onSendHookHandler lets us transform the response body (as a string)
 * We detect the ___jsx key and unwrap the HTML.
 */
export const onSend: onSendHookHandler<unknown> = async (_request, _reply, payload: unknown) => {

  if (typeof payload==='string' && payload.startsWith('{"___jsx":"')) {
    return JSON.parse(payload).___jsx;
  }
  return payload;

};

To use the hook:

const fastify = Fastify();

// These 2 hooks allow us to render React/jsx tags straight to HTML
fastify.addHook('preSerialization', jsxRender.preSerialization);
fastify.addHook('onSend', jsxRender.onSend);

Using Preact or your own factory should also totally work here.

Limitations to this approach

Users of React and other frameworks may be used to pulling in data asynchronously and updating their document. This approach only allows for a single pass, so this means that all dynamic data to your JSX templates needs to be fetched in advance and passed down as props.

This means no hooks or state.

To me this is an advantage because the paradigm it replaces is regular templates, and with those the data is typically also passed in ‘at the top’.

It would certainly be possible to implement Suspend and allow for asynchronous data fetching or wait for other signals, but I haven’t yet needed this.

Frequently asked questions

How do you handle routing?

The short answer is: you don’t. Routing is already handled by your framework, and each route/endpoint/controller is responsible for rendering it’s entire page.

To reuse things like the global layout, you use components. A slightly fictionalized example of a page for us looks like this:

ctx.body = <PublicLayout>

  <div>
    <h1>Welcome back!</h1>
  </div>
  <form method="post" action="/login">

    <label>
      <span>Email</span>
      <input type="email" name="email" required minLength={10} />
    </label>
    <label>
      <span>Password</span>
      <input type="password" name="password" required minLength={4} />
    </label>
    <button type="submit">Log In</button>

  </form>
</PublicLayout>;

This in effect ‘inherits’ from PublicLayout, which looks like this:

type Props = {
  children: JSX.Element[]|JSX.Element;
  className: string;
}

/**
 * This is the main application wrapper, shared by all HTML pages.
 */
export function PublicLayout(props: Props) {

  return <html>
    <head>
      <title>Sprout Family</title>
      <link href="/css/main.css" rel="stylesheet" type="text/css" />
      <meta name="viewport" content="width=device-width, initial-scale=1" />
      <script src="/js/utils.js"></script>
    </head>
    <body className={props.className}>
      { props.children }
    </body>

  </html>;

}

Is there still a reason to use Handlebars or EJS?

I think one benefit of these template languages are they they are their own isolated format with limited capabilities.

This is helpful when for example you build an application and let your own users edit templates. Perhaps you for example have a ‘welcome email’ and want to let your users/tenants modify it.

Giving them the full power of a Javascript + a DSL that needs to be transpiled might be a negative here, because limiting features makes it easier to evolve systems and there’s also a major security component.

My template engine of choice for these is probably handlebars or even the more limited variant mustache.

Anyway

Hope this was interesting. If there’s interest in turning my Koa and Fastify code into real NPM packages, let me know! Happy to do it if there’s a non-0 number of users.

In the future I might even be interested in developing my own JSX Factory that allows awaiting for async components.

If any of this sounds interesting, you have a scathing critique or found punctuation in the wrong place, you can leave a comment by replying to this Mastodon post.

Why aren't there more 80% jobs?

2023-10-12T20:29:12+00:00

There’s been a bit of a trend recently for some companies to move to 4-day workweeks. This is making a decent amount of noise, but the actual number of companies offering this still seems pretty few and far between. It’s not hard to imagine why CEOs might feel this is risky, considering that many don’t even trust people to work from home.

What I don’t see much are 80% jobs, which are 32 hour jobs, at 80% salary. This should be a low-risk proposition that I think a lot of people in the tech industry would take, if it were an option.

My background

I grew up in the Netherlands, but I moved to Canada for my first programmer job at age 20. I stayed for a few years, moved back and started a job there, and eventually moved back to Canada again and pretty much settled here.

I got strong roots and professional experience in both places, but there was one thing that surprised me most in the Netherlands that I haven’t really seen in North America.

Apparently Netherlands has a very high rate of part-time careers. Statistics suggest that they are the highest in the world by a large margin. My personal experience matches this too. I know a number of people in the Netherlands that don’t work full-time, and I think there are more opportunities for part-time career jobs. I don’t see them much in North America.

OECD Chart: Part-time employment rate, Total, % of employment, Annual, 2022

My first-hand experience is when I was interviewing a few years into my career at a company named Ibuildings in Utrecht, Netherlands. After the interview I was told I pretty much had the job, and then the interviewer asked me if I wanted to work 4 or 5 days per week.

I only had Canadian work experience, so I was pretty shocked being asked this. Picking 4 days seemed like a no-brainer to me. I would get paid for 32 hours per week, instead of 40 and got to pick a day of the week I wanted off (I picked Friday). At this company I believe most people took the 4-day option. So yes, I got paid 20% less, but with the tax bracket I was in this was closer to 10%.

For me it was great. 5 days actually feels like a lot, and 2 days in a weekend never feels quite enough. A sentiment so common it’s boring talk about it. I used the extra time to work on open source, errands, and picked a few small freelance gigs here and there.

I’m leaning pretty socialist, but even with a capitalist hat on, I’m surprised this doesn’t happen more. Why not give employees the option to do this? Not everyone will take the option, but for those that do here’s some advantages:

Advantages of offering 80% jobs

For the salary cost of 4 employees at 40 hours, you get 5 employees at 32 hours.
Those 5 employees are more rested, and may be generally happier.
There’s a lower risk of burn-out and attrition.
You get the combined experience of 5 people instead of 4.
Also I don’t believe for the 80% workers, you only get 80% output. Most people just aren’t productive all the time.
Advertising this as a benefit in your company may also be attractive to candidates for hiring.

There are also some drawbacks. Every employee will have some overhead such as a laptop, benefits and general admin. Probably a larger expense is the cost to recruit, although in the long run a lower attrition rate might make up for that a bit.

But my intuition tells me that this overhead is probably well worth the benefit of a smarter, larger, happier workforce.

Some notes:

I want to stress that working 4 days should be presented as a choice. Some people prefer to work more for more money, and you don’t want to scare them.
Another, even more lightweight option for irriationally risk-averse traditionalists is to offer 90% jobs, resulting in an extra day every 2 weeks.

Does OAuth2 have a usability problem? (yes!)

2023-04-27T02:18:00+00:00

I read an interesting thread on Hackernews in response to a post: “Why is OAuth still hard in 2023”. The post and comments bring up a lot of real issues with OAuth. The article ends with a pitch to use the author’s product Nango that advertises support for supporting OAuth2 flows for 90+ APIs and justifying the existence of the product.

We don’t need 90 browsers to open 90 websites, so why is this the case with OAuth2? In a similar vain, the popular passport.js project has 538(!) modules for authenticating with various services, most of which likely use OAuth2. All of these are NPM packages.

Anyway, I’ve been wanting to write this article for a while. It’s not a direct response to the Nango article, but it’s a similar take with a different solution.

My perspective

I’ve been working on an OAuth2 server for a few years now, and last year I released an open source OAuth2 client.

Since I released the client, I’ve gotten several new features and requests that were all contributed by users of the library, a few of note are:

Allowing client_id and client_secret to be sent in request bodies instead of the Authorization header.
Allow ‘extra parameters’ to be sent with some OAuth2 flows. Many servers, including Auth0 require these.
Allow users to add their own HTTP headers, for the same reason.

What these have in common is that there’s a lot of different OAuth2 servers that want things in a slightly different/specific way.

I kind of expected this. It wasn’t going to be enough to just implement OAuth2. This library will only work once people start trying it with different servers and run into mild incompatibilities that this library will have to add workarounds for.

Although I think OAuth2 is pretty well defined, the full breadth of specs and implementations makes it so that it’s not enough to (as an API developer) to just tell your users: “We use OAuth2”.

For the typical case, you might have to tell them something like this:

We use OAuth2.
We use the authorization_code flow.
Your client_id is X.
Our ‘token endpoint’ is Y.
Our ‘authorization endpoint’ is Z.
We require PKCE.
Requests to the “token” endpoint require credentials to be sent in a body.
Any custom non-standard extensions.

To some extent this is by design. The OAuth2 spec calls itself: “The OAuth 2.0 Authorization Framework”. It’s not saying it is the protocol, but rather it’s a set of really good building blocks to implement your own authentication.

But for users that want to use generic OAuth2 tooling this is not ideal. Not only because of the amount of information that needs to be shared, but also it requires users of your API to be familiar with all these terms.

A side-effect of this is that API vendors that use OAuth2 will be more likely roll their own SDKs, so they can insulate users from these implementation details. It also creates a market for products like Nango and Passport.js.

Another result is that I see many people invent their own authentication flows with JWT and refresh tokens from scratch, even though OAuth2 would be good fit. Most people only need a small part of OAuth2, but to understand which small part you need you’ll need to wade through and understand a dozen IETF RFC documents, some of wich are still drafts.

Sidenote: OpenID Connect is another dimension on top of this. OpenID Connect builds on OAuth2 and adds many features and another set of dense technical specs that are (in my opinion) even harder to read.

OAuth2 as a framework is really good and very successful. But it’s not as good at being a generic protocol that people can write generic code for.

Solving the setup issue

There’s a nice OAuth2 feature called “OAuth 2.0 Authorization Server Metadata”, defined in RFC8414. This is a JSON document sitting at a predictable URL: https://your-server/.well-known/oauth-authorization-server, and can tell clients:

Which flows and features are supported
How to pass credentials
URLs to every endpoint.

Here’s an example from my server:

{

    "issuer": "http://localhost:8531",
    "authorization_endpoint": "/authorize",
    "token_endpoint": "/token",
    "token_endpoint_auth_methods_supported": [
        "client_secret_basic"
    ],
    "token_endpoint_auth_signing_alg_values_supported": [
        "RS256"
    ],
    "jwks_uri": "http://localhost:8531/.well-known/jwks.json",
    "scopes_supported": [
        "openid"
    ],
    "response_types_supported": [
        "token",
        "code",
        "code id_token"
    ],
    "grant_types_supported": [
        "client_credentials",
        "authorization_code",
        "refresh_token"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "service_documentation": "http://localhost:8531",
    "ui_locales_supported": [
        "en"
    ],
    "introspection_endpoint": "/introspect",
    "revocation_endpoint": "/revoke",
    "revocation_endpoint_auth_methods_supported": [
        "client_secret_basic"
    ]

}

If your server and client supports this, it can simplify the setup a great deal. Here’s an example using my client:

import { OAuth2Client } from '@badgateway/oauth2-client';

const client = new OAuth2Client({
  server: 'https://my-auth-server/',
  clientId: 'my-client-id'
});

The problem is majority of OAuth2 servers and clients don’t support this, so even if your server did, your setup instructions would still need to include all the ‘setup info’ for clients that don’t support the discovery document.

And for the clients that do support it, you would need to call out that your users’ client needs support for RFC8414.

So while I think the discovery spec is great and solves real problems; it alone cannot solve the OAuth2 usability problem.

My proposal

Currently work is underway to define OAuth 2.1. OAuth 2.1 will remove features that are considered insecure or bad practices (such as implicit and password flows) and PKCE is brought in as a core feature.

If you’re a OAuth 2 client or server maintainer and kept up with the specs, you likely are already compatible with OAuth 2.1.

I don’t think OAuth 2.1 goes far enough.

I think that this proposal should require support for the discovery document and make it a required step to find features and endpoints. I also think it should have an opinion on how clients should support custom extensions and how they might work. (or forbid them).

This version of OAuth should also provide a way to discover the discovery endpoint (hear me out).

If a client makes a HTTP request to an API, and the API replies with 401, it should tell the client where to find the discovery document and which flow(s) to use.

Lastly, I think that it should be renamed to OAuth 3, so API vendors no longer have to state:

We use OAuth 2.
We use the authorization_code flow.
Your client_id is X.
Our ‘token endpoint’ is Y.
Our ‘authorization endpoint’ is Z.
We require PKCE.
Requests to the “token” endpoint require credentials to be sent in a body.
Any custom non-standard extensions.

But instead:

We use OAuth 3.
Your client_id is X.

A nice aspect of this proposal is that OAuth 2 clients can still talk to OAuth 3 servers, it also doesn’t obsolete the OAuth 2 framework.

Perhaps you could name this “OAuth 2.1: the good parts”, but I think increasing the major version number sends a clear signal to users they should be looking for a OAuth 3 library.

Then maybe, 10 years from now we no longer need 538 Passport.js modules for 538 APIs. Perhaps browsers could even facilitate authentication.

Final notes

A future OAuth version should also explicitly allow http://localhost as a redirect_url. We need to be able to test.
I’m aware that there was a OAuth 3 proposal, which is now called XYZ (or maybe GNAP?). I’m not very familiar with this proposed protocol.
XKCD 927 is funny, but ultimately a conversation stopper and a bit cynical.

Switching to Fedora from Ubuntu

2023-03-30T16:00:00+00:00

It seems like every 7-8 years I’m switching operating systems. In 2006 I first started using Apple, because it was just so damn cool to have a well working Unix-like system. (I’ll never forget you Snow Leopard).

In 2015 I switched to Ubuntu. Apple’s Software seemed to hit rock bottom at this point from a quality perspective, and hardware seemed to go obsolete at a rate I hadn’t seen before. Fed up paying a premium price for a sub-par product, it was time for a change.

Ubuntu’s fall

Ubuntu was the obvious choice. I want something that just works, and Dell’s XPS 13 Developer Edition ships with Ubuntu which means hardware support from Dell itself. Breath of fresh air and fast as hell. The experience was similar to what people have been saying about the new M1 chips. But it’s fast because of software, not hardware.

But something changed with Ubuntu in recent years. I think linux users have a thicker than usual skin when it comes to software issues and are willing to look past more things. But Ubuntu’s quality has been consistently falling. I was being worn down, and it seems to be a common sentiment in my bubble.

The best example is Ubuntu’s package manager Snap. A pretty good idea, I like the branding too but the execution hasn’t been there. Ubuntu users have been effectively beta-testing this for years. Just to give you an idea I made a giant list of bugs that I ran into when Ubuntu switched Firefox from apt to Snap.

To be honest I feel a bit bad ragging on Ubuntu, because without knowing anything about how the project and Canonical is run, my intuition is that the steam is just kind of running out for them. Ubuntu feels ‘depressed’, but maybe it’s all in my head.

Onto Fedora

So I pledged to try something new in 2022, and it took me another 14 months to find the energy and motivation to actually follow through.

I went for Fedora. Named after a fashion faux pas, it seems to be on the #2 spot for desktop linux. It’s funded by Red Hat, and seems to have a focus on keeping it’s packages very up to date, which I like and why I originally switched from Debian to Ubuntu!

I’m a week and a bit in, and here are my first impressions.

Installation

Installation was super smooth. I always forget to make a separate /home mount, so it took a while to move everything to an external disk and back.

The one thing I always forget to move is my MySQL databases, and today was no exception.

Non-free stuff

Fedora does not ship with things that aren’t open source. Nothing against that philosophy (awesome in fact), but personally I don’t mind adding some binaries for a better experience.

I miss Ubuntu’s Additional Drivers tool, because it told me what to install. I’m sure the drivers I need are available for Fedora, but I don’t know what to look for which makes me slightly worried my computer is not running optimally. Battery feels worse but that could also be my imagination.

Video in Firefox didn’t work at all in stock Fedora. I had to install ffmpeg to get it to barely function, but then I discovered RPM Fusion, where I got an even better ffmpeg, plus gstreamer and Intel drivers and I can now watch beautiful smooth 4K video, and confirmed with intel_gpu_top that I’m using hardware acceleration.

Gnome

Ubuntu used to have their own desktop environment called Unity. In 2018 they switched to Gnome, but they modified Gnome to keep their Unity look.

This felt like a good move, because it let them kept their look while taking advantage of all the Gnome plumbing.

One drawback is that Ubuntu was usually a bit behind with Gnome features.

Fedora uses stock Gnome. As a result there’s more consistency overall, and it’s nice to have the latest features. I miss the strong visual identity Ubuntu has though. It sets itself apart from Mac and Windows.

I’m sure I can customize Fedora to be more fun, but if I’m being real with myself I haven’t changed my desktop background in 10 years, and so this will never happen.

Flatpak

The only thing that Flatpak had to do to be better than Snap was to not remind me of its existence unless I’m installing something, and so far it’s done that. Flathub is nice, and I love the idea of developers not having to repackage for every distro under the sun.

Bugginess

Fedora does generally feel more stable. Fewer background processes pegged at 100%. 3rd party application support doesn’t seem as good on first sight. I had to find workarounds for KeeppassXC and Element to not crash all the time.

This is no fault of the Fedora team, but it does tell you something about how much Fedora is on other people’s radars. If developers test 1 linux distro, it will be Ubuntu.

Software Center app

I falsely assumed that the Software Center application was buggy due to Ubuntu’s Snap changes, but it also hangs all the time in Fedora.

The solution is to kill the gnome-software process if you want to use ‘Software’ more than once per session. Apparently this has been an issue for at least 12 months

Things are never perfect, but even with some of the issues I ran into I’m very happy I switched. It’s hard to describe, but things feel more solid.

To get to this state I did have to put in some work and research, but not at a level of recompiling kernels.

If you are comfortable googling, using the terminal for some commands and interpreting an occasional error message I would now recommend Fedora over Ubuntu.

If you are not a technical user or programmer-adjacent, I think Ubuntu is still going to be the choice with the least amount of friction. In large I think this is simply because it is the biggest, has the most eyes and the most support.

I’m excited though. Fresh start!

Supporting CommonJS and ESM with Typescript and Node

2023-03-21T17:11:00+00:00

I maintain a few dozen Javascript libraries, and recently updated many of them to support CommonJS and Ecmascript modules at the same time.

The first half of this article describes why and how I did it, and then all the hoops I had to jump through to make things work.

Hopefully it’s a helpful document for the next challenger.

A quick refresher, CommonJS code typically looks like this:

const MyApp = require('./my-app');
module.exports = {foo: 5};

And ESM like this:

import MyApp from './my-app.js';
export default {foo: 5};

Except if you use Typescript! Most Typescript uses ESM syntax, but actually builds to CommonJS. So if your Typescript code looks like the second and think ‘I’m good’, make sure you also take a look at the built Javascript code to see what you actually use.

Why support ESM

The general vibe is that ESM is going to be the future of Javascript code. Even though I don’t think the developer experience is quite there yet, but more people will start trying ESM.

If you decided to plunge in with ESM, I want my libraries to feel first-class.

For example, I’d want you to be able to default-import:

import Controller from '@curveball/controller';

At the same time most people are still on CommonJS, and I want to continue to support this without breaking backwards compatibility for the forseeable future.

The general approach

My goal is for my packages to ‘default’ to ESM. From the perspective of a library maintainer you will be dealing with ESM.

During the build phase steps are being made to generate a secondary CommonJS build. As a maintainer you might run into CommonJS-specific issues, but I suspect people will only see those if the CI system reports an issue.

Our published NPM packages will have a directory structure roughly like this:

- package.json
- tsconfig.json

- src/     # Typescript source
- cjs/     # CommonJS
- esm/     # ESM
- test/

We include the original typescript sources to make step-through debugging work well, and have a seperate directory for the ESM and CommonJS builds.

If you just want to skip to the end and see an example of a package that has all this work done, check out my @curveball/core package:

Github source: https://github.com/curveball/core
Published NPM package contents: https://www.npmjs.com/package/@curveball/core?activeTab=code

Typescript features we don’t use

esModuleInterop

We don’t use the esModuleInterop setting in tsconfig.json. This flag lets you default-import non-ESM packages like this:

import path from 'node:path';

instead of the more awkward:

import * as path from 'node:path';

On the surface esModuleInterop seems like it would be helpful, but it has a major problem: if our libraries use esModuleInterop, anyone who uses that library is forced to also turn it on.

So turning by turning esModuleInterop off we don’t force anyone downstream into a specific setting. I generally recommend anyone writing libraries to turn this off to reduce friction and ironically increase interopability.

Path mapping

Typescript lets you map arbitrary paths to specific directories in your source. This is popular because it lets you do things like:

import MyController from '@controllers/my';

Instead of being forced to always do relative import

import MyController from '../../../controllers/my';

This is quite a popular feature for larger code-bases because people find relative paths annoying.

I’m also not using this feature. A big issue is that only Typescript is aware of this configuration, so any other tooling that uses your files may need to be configured separately to also understand this, which doesn’t always work.

So while I don’t have an exact list of exact reasons or configurations where this fails, it’s a common enough issue that I’ve decided to just avoid this feature altogether, for the sake of reducing magic. If you do insist on using path mapping, you’re on your own.

Configuring package.json

Modern node versions now have a syntax in package.json that lets you specify both the default CommonJS and ESM files. These are the relevant lines of one of our packages:

{
  "name": "@curveball/controller",
  "version": "0.5.0",
  "description": "A simple controller pattern for Curveball.js",
  "type": "module",
  "exports": {
    "require": "./cjs/index.js",
    "import": "./esm/index.js"
  },
  "main": "cjs/index.js"
}

When specifying your package.json this way, Node will automatically load the correct (cjs/esm) directory depending on what the user needs. This all happens automatically. Neat!

main is no longer used, but I’m keeping it in case of older tooling.

Building with Typescript

Before we made this change, we just had a src/ and dist/ directory for Typescript and Javascript files respectively. To do the build, we could just run npx tsc.

This still works, except npx tsc now builds to esm for the regular developer flow.

But when we build for creating the NPM package, we need to create both. We use a Makefile for this, but these are roughly the commands to run:

npx tsc --module commonjs --outDir cjs/
echo '{"type": "commonjs"}' > cjs/package.json

npx tsc --module es2022 --outDir esm/
echo '{"type": "module"}' > esm/package.json

As you can also see that both our cjs and esm packages get a 1-line package.json file with a type property.

This is required because Node needs to know wether files with a .js extensions should be interpreted as CommonJS or ESM. It can’t figure it out after opening it.

Node will look at the nearest package.json to see if the type property was specified.

It’s also possible to use .cjs or .mjs file extensions, but this doesn’t play well with Typescript as Typescript can’t automatically adjust import paths for you.

Changes we had to make in our code

I think it’s reasonable to say that that vanilla javascript and javascript modules are slightly different programming languages

They are interopable, but not the same. They have slightly different syntax and behavior. This is why you also need to tell HTML in advance what kind of flavour of Javascript you’re going to be using:

<script type="module" src="my-module.mjs"></script>

So naturally I expected to have to make some changes to write code in a way that works identical in both targets.

An obvious example is top-level await which only works in ESM. Here are all the things I ran into:

Extensions in imports

All our (local) typescript imports had to change from:

import Foo from './foo';

to:

import Foo from './foo.js';

ESM requires extensions, whereas in CommonJS it’s not needed. Here’s a sed command I used to change all these in bulk (probably not perfect, so you’ll really want to be able to roll this back):

find . -name "*.ts" | xargs -n1 sed -i "s/from '\(.*\)';/from '\\1.js'/g"

So the strange thing with all this is that your code will have statements like:

import Foo from './foo.js';

But actually the file in that directory is called ./foo.ts (.ts not .js) yes, this is confusing and annoying.

I don’t really want a future contributor of my library to have to understand the build chain, so it’s a leaky abstraction and a limitation of Typescript.

I hope this is rectified in the future. I understand the philosophy of wanting to avoid magic as much as possible, but just give me a setting to globally rewrite a .ts to .js when generating the .js files.

I secretly suspect that despite Typescript explicitly not supporting this, this might still happen later anyway. Maybe they don’t want to commit to building this before they have a good plan how 🤞.

Directory imports

In CommonJS in node you can import a directory, and if a index.js exists in that directory, it will open that. ESM requires exact paths, so your

import * from './controllers'

Now has to be:

import * from './controllers/index.js'

`dirname` and `filename`

Node defines 2 useful very variables in every CommonJS file, probably borrowed from PHP.

__dirname is a reference to the path the current file is in, and __filename is the whole path + filename.

I used these to load in non-javascript assets from relative paths. For example, I had a snippet like this to get the version of the current package:

import * as fs from 'node:fs';

const pkg = JSON.parse(
  fs.readFileSync(
     __dirname + '../package.json'
  )
);

console.log(pkg.version);

Unfortunately, these variables no longer exist in ESM. Instead we have import.meta. In Node, this is an object looking like this:

{
  url: 'file:///home/evert/src/curveball/controller/test.mjs'
}

So instead of a filesystem path, we get a file:// url with the location. So to write the equivalent in ESM we get something like this:

import * as fs from 'node:fs';
import * as url from 'node:url';

const pkg = JSON.parse(
  await fs.readFile(
    url.fileURLToPath(url.resolve(import.meta.url, './package.json')),
    'utf-8',
  )
);

console.log(pkg.version);

The challenge is writing this code in a way that will work with both. The ‘crazy’ way to solve this is to take an Error object, and get the path from the string that appears in the stack trace. This means parsing the stacktrace :(

import * as url from 'node:url';
import * as path from 'node:path';

function getDirName() {

  const err = new Error()
  console.log(err.stack?.split('\n')[1]);
  const fileUrl = err.stack?.split('\n')[1].match(/at .* \((file:.*):\d+:\d+\)$/);

  if (!fileUrl) throw new Error('This misrable idea failed');

  return path.dirname(url.fileURLToPath(fileUrl[1]));

}

console.log(getDirName());

I don’t believe that the exact format of the stack trace is stable or even the same across different interpreters, so this seems like a bad idea.

The problem is that the more obvious approach doesn’t work:

import * as path from 'node:path';
import * as url from 'node:url';

/** @ts-ignore CommonJS/ESM fun */
const dirname = 
  typeof __dirname !== 'undefined' ?
  __dirname :
  /** @ts-ignore CommonJS/ESM fun */
  path.dirname(url.fileURLToPath(import.meta.url));

The above code creates a new dirname variable. While this works for ESM, it breaks for CommonJS on Node. The reason is that import is not a regular object, it’s syntax and even if the branch with import.meta never gets run, Node will error while parsing:

/home/evert/src/curveball/controller/cjs/dirname.js:9
    path.dirname(url.fileURLToPath(import.meta.url));
                                          ^^^^

SyntaxError: Cannot use 'import.meta' outside a module

Sadness!

I don’t have a satisfying solution for this yet. I’ve worked around this in some of my packages by avoiding this altogether, or rewriting certain files after running them. eval('import.meta.url') causes the ESM build to fail because technically things running in eval is no longer in a module scope.

Importing CommonJS modules with ‘default exports’

Some of our packages rely on 3rd party dependencies that are written in plain Javascript and use CommonJS.

In CommonJS dependencies you can ‘default’ export things like:

module.exports = function foo(a, b) { a+b; };

If we were to use this library in Typescript (CJS build) we could use it this way:

import * as WebSocket from 'ws';
const ws = new WebSocket('ws://localhost:8000');

But if we are in ESM land in Typescript, this doens’t work. We actually get an object with a property named .default. For these modules, this is the approach I’ve come up with:

import * as WebSocketImp from 'ws';

// ESM shenanigans
const WebSocket = 'default' in WebSocketImp ? (WebSocketImp.default as any) : WebSocketImp;

Yep! It’s a pain. I’ve had to do this for websocket, chai-as-promised, ajv, raw-body, accepts and http-link-header. In case you haven’t figured it out, I’m building a framework.

Testing

To have confidence in both builds, I want to write tests that run against both. My tests are written using Mocha. My tests are also written in Typescript. By default, they only run against the ESM build, and I expect myself and other developers to be in this mode during the main ‘development loop.

To make Mocha understand Typescript, we need to install the ts-node package, and add the following properties to the root package.json:

"mocha": {
  "loader": [
    "ts-node/esm"
  ],
  "recursive": true,
  "extension": [
    "ts",
    "js",
    "tsx"
  ]
}

This all works perfectly well, but what about CommonJS? For this I had 2 approaches: Either reconfigure mocha to use the CommonJS Typescript loader instead of the ESM typescript loader, but this was too much of a pain to get right with writing configuration files on the fly.

Instead, I decided to just ask Typescript to build the entire project including tests in a separate directory and then just run Mocha on the javascript files.

mkdir -p cjs-test
cd test; npx tsc --module commonjs --outdir ../cjs-test
echo '{"type": "commonjs"}' > cjs-test/package.json
cd cjs-test; npx mocha --no-package

This required a separate tsconfig.json in our test directory.

Conclusion

So this is a fairly large amount of annoyances to work through, some without a solution. Would I recommend this?

I think if you’re in my position where:

You’re doing this for a library.
You consider ESM the future, but want to continue to support CommonJS users.
The experience of users of your library is far more important than your own experience.

Then, I think this is a good idea. If you don’t want this hassle and want just one target, I think CommonJS will still give you the best experience.

ESM will probably catch up at some point, but right now I think we’re still in the awkward phase.

Winding down Bad Gateway

2023-02-20T19:18:15+00:00

In 2019 I started Bad Gateway as a software development agency. Last year we grew all the way to 7 people. It was crazy challenging, especially with Covid in the mix; but ultimately could not get the company into a good financial state to be able to carry on.

Big thanks to my co-workers and partners Ju, Becky, Phil, Michael, Siep, Richard and Syed. I’m incredibly grateful you came on this journey with me. We shared some tears, exchanged some words but mostly had lots of laughs. Despite the challenges I feel my relationship with you has only strengthened and I wish you well in the next steps of your career.

Also thank you to our customers and especially Underknown who’ve stuck with us since the start.

Despite this outcome, I have a hard time seeing the last few years as a failure. It’s been hella fun, and I feel we stuck to our values even in times of crisis. Off to the next adventure.

Building a simple CLI tool with modern Node.js

2023-02-13T19:05:14+00:00

I’m a maintainer of several dozen open source libraries. One thing I’ve always done is maintain a hand-written changelog.

Here’s an example from a12n-server

0.22.0 (2022-09-27)
-------------------

Warning note for upgraders. This release has a database migration on the
`oauth2_tokens` table. For most users this is the largest table, some
downtime may be expected while the server runs its migrations.

* #425: Using a `client_secret` is now supported with `authorization_code`,
  and it's read from either the request body or HTTP Basic Authorization
  header.
* The service now keeps track when issuing access tokens, whether those tokens
  have used a `client_secret` or not, which `grant_type` was used to issue them
  and what scopes were requested. This work is done to better support OAuth2
  scopes in the future, and eventually OpenID Connect.
* Fixed broken 'principal uri' in introspection endpoint response.
* OAuth2 service is almost entirely rewritten.
* The number of tokens issued is now displayed on the home page.
* Large numbers are now abbreviated with `K` and `M`.
* #426: Updated to Curveball 0.20.
* #427: Typescript types for the database schema are now auto-generated with
  `mysql-types-generator`.

These are all written in Markdown. You might think: isn’t Git also a log? Why bother hand-writing these?

The reason is that the audience for these is a bit different. I want to bring attention to the things that are the most important for the end-user, and focus on the impact of the change to the user.

I thought it would be handy to write a CLI tool that makes it a bit easier to maintain these. So, I did!. If you are curious what kind of technology choices went into this, read on.

Goals and features

The tool should be able to do the following:

Reformat changelogs (a bit like prettify) (changelog format)
Add an entry via the command line (changelog add --minor -m "New feature").
Automatically set the release date (changelog release)
Pipe a log of a specific version to STDOUT, so it can be used by other tools (like integrating with github releases).

I also had a bunch of a non-functional requirements:

Use the latest Node features.
Use up to date Javascript standards and features (ESM).
Avoid dependendencies unless it’s unreasonable to do so.
Make it low maintanance.

Want to find the finished tool right now? It’s open source so just go to Github.

The implementation

ESM & Typescript

Ecmascripts modules worked really well here. It’s a small change of habits, but the general recommendation I would have is to just save your files as .mjs and start using it.

Here’s the first few lines of parse.mjs:

// @ts-check
import { Changelog, VersionLog } from "./changelog.mjs";
import { readFile } from 'node:fs/promises';

/**
 * @param {string} filename
 * @returns {Promise<Changelog>}
 */
export async function parseFile(filename) {

  return parse(
    await readFile(filename, 'utf-8')
  );

}

The CommonJS -> ESM transition is not without pain, but for an new project like this it’s really the ideal choice. (top level await 🎉)

I’ve also made the choice to not write my code in Typescript, but use JSDoc annotations instead (These are the @param and @returns comments).

Not everyone knows that you don’t need to write .ts files to benefit from Typescript. Typescript can also check your Javascript files quite strictly, and there’s a lot of work still being done in adding new features to this syntax.

This has the benefit of not needing a build phase. You don’t even need Typescript during development which reduces the barrier to entry.

Here’s my minimal tsconfig.json:

{
  "compilerOptions": {
    "target": "es2022",
    "module": "esnext",
    "rootDir": "./",
    "allowJs": true,
    "checkJs": true,

    "moduleResolution": "node",
     
    "noEmit": true,
    "strict": true,
    "useUnknownInCatchVariables": false,
  
  }
}

The Typescript documentation has a page detailing the JSDoc annotations they support, if you’d like to learn more.

Command line parsing

CLI tools need to be able to parse command line options. Since Node 18.3 (backported to Node 16.17) Node comes with a built-in options parser.

Here’s a sample of the code:

import { parseArgs } from 'node:util';

const { positionals, values } = parseArgs({
  options: {
    help: {
      type: 'boolean',
      short: 'h',
      default: false,
    },
    all: {
      type: 'boolean',
      default: false,
    },
    message: {
      type: 'string',
      short: 'm'
    },
    patch: { type: 'boolean' },
    minor: { type: 'boolean' },
    major: { type: 'boolean' },
  },
  allowPositionals: true,
});

This confguration adds flags like --major, lets you specify a message with --message "hello!" or use a short-hand alternative like -m "Hi".

Does it do everything? No! There’s packages out there that are more sophisticated, use colors, automatically create help screens but they also ship with large dependency trees.

In my case, this was good enough.

Check out the Node docs for more info.

Testing

Most people probably use Jest or Mocha as their test framework, but since Node 18 (also backported to 16) Node has a built-in test-runner.

It has an API similar to Mocha and Jest with keywords like it, test, describe, before, etc.

Here’s a sample of one of my tests:

// @ts-check
import { test } from 'node:test';
import { parse } from '../parse.mjs';
import * as assert from 'node:assert';

test('Parsing changelog metadata', async () => {

  const input = `Time for a change
=========

0.2.0 (????-??-??)
------------------

* Implemented the 'list' command.
* Added testing framework.

0.1.0 (2023-02-08)
------------------

* Implemented the 'help' and 'init' commands.
*
`;

  const result = await parse(input);

  assert.equal('Time for a change', result.title);
  assert.equal(2, result.versions.length);
  
  assert.equal(null, result.versions[0].date);
  assert.equal('0.2.0', result.versions[0].version);
  assert.equal('2023-02-08', result.versions[1].date);
  assert.equal('0.1.0', result.versions[1].version);

});

To run the tests, just run node --test. No configuration needed, it will discover tests following directory and filename conventions.

The Node 18 test output is a bit rough, it’s the TAP format and looks like this:

TAP version 13
# Subtest: /home/evert/src/changelog-tool/test/parse.mjs
    # Subtest: Parsing changelog metadata
    ok 1 - Parsing changelog metadata
      ---
      duration_ms: 1.713409
      ...
    # Subtest: Parsing changelog entries
    ok 2 - Parsing changelog entries
      ---
      duration_ms: 0.2595
      ...
    # Subtest: Preface and postface
    ok 3 - Preface and postface
      ---
      duration_ms: 0.193591
      ...
    1..3
ok 1 - /home/evert/src/changelog-tool/test/parse.mjs
  ---
  duration_ms: 70.901055
  ...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 81.481441

In Node 19 new test reporters will be shipped, but I haven’t tested this yet.

Frankly, after using this I don’t know if would use Mocha anymore. I’ve been using probably over a decade, and it has some nice features and benefits, for the kinds of tests I write (and I write a lot), I think there’s anything I need beyond what Node is offering here.

Some links:

package.json, annotated

I wanted to end this article with how I’ve set up my package.json, so you can see how it all ties together. (If only npm supported JSON5 so I could keep my comments right in the package 😭).

{
  // The name of the package, and how it's published on NPM
  "name": "changelog-tool",

  // Package version!
  "version": "0.5.0",

  // This will show up in NPM searches
  "description": "A CLI tool for manipulating changelogs",

  // This tells Node to assume this is an ESM package. Not strictly needed
  // if we use the .mjs extension everywhere though.
  "type": "module",

  // If you use this package programmatically (not on CLI), this file is
  // where all 'exports' are for this package.
  "main": "index.mjs",

  "scripts": {
    // Test runner!
    "test": "node --test",

    // I like to keep Typescript running in the terminal to insta-warn me of
    // of any issues.
    "watch": "tsc --watch"
  },

  // This helps people discover the package on npmjs.org
  "keywords": [
    "changelog",
    "markdown"
  ],

  // It's me!
  "author": "Evert Pot (https://evertpot.com/)",

  // Do (almost) anything you want
  "license": "MIT",

  "engine": {
    // Warn people if they haven't upgraded yet
    "node": ">16"
  },

  "bin": {
    // When people install this package, this makes `npx changelog` work. If
    // you installed this package globally, you now have a `changelog` command
    // at your disposal.
    "changelog": "./cli.mjs"
  },
  "devDependencies": {
    // The only 2 dependencies. You don't even really need these if you just want
    // use or hack the package. Github actions will run Typescript as well.
    "@types/node": "^18.11.19",
    "typescript": "^4.9.5"
  }
}

Conclusion

I love building new things and being deliberate about every choice.

The result is that I’m more likely to end up with something minimal, low-maintance and I gain a deeper understanding of the tools I use.

In the near future I would probably make all these choices again. The Node test runner is fast and low-cruft, ESM is great when it works and not needing a build step feels right for a project this size.

I hope this inspires someone in the future to start their next project from an empty directory instead of copying a large boilerplate project.

changelog-tool project on Github.

Knex (with MySQL) had a very scary SQL injection

2023-01-12T21:31:43+00:00

Knex recently released a new version this week (2.4.0). Before this version, Knex had a pretty scary SQL injection. Knex currently has 1.3 million weekly downloads and is quite popular.

The security bug is probably one of the worst SQL injections I’ve seen in recent memory, especially considering the scope and popularity.

If you want to get straight to the details:

Check out the Github issue, which was opened 7 years ago(!)
An article from Ghostccamm explaining the vulnerability.
CVE-2016-20018.

My understanding of this bug

If I understand the vulnerability correctly, I feel this can impact a very large number of sites using Knex. Even more so if you use Express.

I’ll try to explain through a simple example. Say, you have MySQL table structured like this:

CREATE TABLE `users` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
)

And you have a query that does a SELECT using Knex:

const lookupId = 2;

const result = await knex('users')
  .select(['id', 'name'])
  .where({
    id: lookupId
  });

You’d expect the query to end up roughly like this

SELECT `id`, `name` FROM `users` WHERE `id` = 2

The issue is when the user controls the value of lookupId. If somehow they can turn this into an object like this:

const lookupId = {
  name: 'foo'
}

You might expect an error from Knex, but instead it generates the following query:

SELECT `id`, `name` FROM `users` WHERE `id` = `name` = 'foo'

This query is not invalid. I don’t fully understand fully understand MySQL’s behavior, but it causes the WHERE clause to be ignored and the result is equivalent to:

SELECT `id`, `name` FROM `users`

I think it has something to do with how MySQL casts things. In MySQL this yields true (or 1):

SELECT 1 = 'foo' = 'bar'

Query strings and Express

One place where this is especially scary, is if lookupId was provided by a user, via a JSON body or query string.

Consider this example from an Express route:

app.get('/', async (req, res) => {
  const result = await knex('users')
    .select(['id', 'name'])
     .where({id: req.query.id});

  res.send(result)
})

Here the id in the WHERE clause is provided by the URL query string:

If a the server is opened with something like:

http://localhost:3000/?id=2

It will return a single user, but if it was opened using:

http://localhost:3000/?id%5Bname%5D=foo

It will return every user. And this issue is not limited to SELECT, it could also trigger in a WHERE clause in DELETE.

The reason this works is that by crafting URL query parameters in a special way, a user can have things in req.query show up as objects or arrays.

Express calls this the ‘extended’ syntax and turns it on by default. In my opinion this is a bad default because it’s not really what users expect will happen. PHP does this as well, and I believe this may have been where Express (or specifically the qs library) got the syntax from.

This is why I think Express users are expecially likely to be vulnerable. I think that most developers in professional settings are decent at validating JSON request bodies with a variety of tools, but I’ve noticed this is often not the case for query parameters. I believe a big reason for this is that the ‘extended’ syntax is not well known and ‘surprising’ behavior. Many people assume these are just strings. This is not intended as a plug, but this is also why the default behavior in our Curveball framework is to not support this. In most cases it’s not needed, and if you do want it you can use qs and explicitly opt-in.

Other examples

But of course, this issue is not limited to query strings. If you’re not validating input somewhere and this ends up in a .where() statement there’s risk.

A specific example is a situation where part of the contents of your WHERE clause is unguessable.

For instance, say you had a database table with records that users can only see if they know a secret code, and it’s selected with:

SELECT * FROM coupons WHERE product_id = 5 AND coupon_code = 'BIG_OOF2023'

A user presumably would have to know the coupon_code to see the coupons, but if there’s no code to validate coupon_code is a string, a user would have the power to change the query to this:

SELECT * FROM coupons WHERE product_id = 5 AND coupon_code = product_id = 'bla'

Which is equivalent to:

SELECT * FROM coupons WHERE product_id = 5

And thus no longer requiring coupon codes to get ALL the discounts.

Lastly, if you can rewrite a query that expects 1 result to return every record in a database is also an easy way to cause a Denial of service attack.

This does demonstrate again how is critical validating is and throw errors whenever you get data you don’t expect. Even if under normal circumstances nothing weird can happen, a library you use might do the wrong thing with unexpected data instead of just rejecting it.

On Knex

It’s quite unfortunate to see that this went unpatched for so long. I’d invite anyone reading this to try to not pile on on the (likely volunteer) authors but think about the larger ecosystem.

This bug was hidden in plain sight. Lots of people must have randomly ran into it and a ticket was opened. Probably the most responsible thing to do would have been what @rgmz did: do their best to contact the authors, failing that contact Github Security Team. After the Github Security Team also weren’t able to connect to the authors, make a CVE which puts this on every everyone’s radar. This ultimately led to a random bystander make their first contribution and submit a fix.

Knex feels high risk now though, and I can’t help wondering what else might be unpatched. I’ve only recently made the jump from just writing my own queries for like 20 years to Knex, but I’m probably reversing that decision for my next project.

I wish JSON5 was more popular

2023-01-09T21:29:36+00:00

As developers we write a lot of code, but we also deal with a lot of configuration files.

The three major formats I tend to use day to day are:

JSON
YAML
.env

And, they all kinda suck. JSON feels like it should never have become a format that people hand-write. So many quotes, and and configuration files need comments to tell users why certain decisions were made. .env has a specific purpose (and it’s ok at that), but it’s not a great universal format, and YAML has always been difficult to read and write to me. I can somehow never retain the syntax and end up copy-pasting things from examples.

Why YAML is difficult for me

A small example from Github workflows/actions:

steps:
  - uses: actions/checkout@v2
  - uses: actions/setup-node@v2
    with:
      node-version: 14
      registry-url: https://registry.npmjs.org/
  - run: npm ci
  - run: npm publish

I couldn’t tell you why uses has a dash in front, and node-version does not. If there’s a difference in how a YAML reader outputs them, I’m not sure how I would be able to retain this while writing YAML.

I also use/love home assistant, which lets you write some pretty cool automations using YAML. I wanted to play with this but it’s been a barrier I’ve not been able to overcome. I don’t know if it’s me. I’m been working as a programmer for 22 years. I’m decent at it, but when when I chat with some of my peers (hi mhum!) they did not share my sentiment.

YAML can also have very surprising behavior, with casting types:

From the linked article, this:

- country1: ca
- country2: no

Becomes:

- country1: ca
- country2: false

It’s a bit cherry picked, and I’m sure there’s YAML linters out there that help avoid the pitfalls, but in my mind configuration files should be simple.

There’s some configuration formats I like, such as TOML and JSON5. They strike the right balance to me with being easy to read and write, unambigious, supporting comments, strictness and not being incredibly hard to write a parser for.

TOML is like ini files on steroids, and JSON5 is JSON but with fewer quotes, comments and multi-line strings.

I could write my NPM configuration file as package.json5 and automatically convert it to package.json but that feels too surprising. My projects are already kind of eclectic, so I want the ‘plumbing’ to be unsurprising. Plus there’s the whole chicken and egg thing with needing a JSON5 parser before we have dependencies.

I’d love the NPM project to adopt JSON5. It seems like a great fit. JSON and YAML can’t be the final word for human-maintained data formats. It’s so obviously sub-optimal.

If NPM adopted JSON5, I would annotate so much in my package.json. I’d document why a dependency is needed, why we are stuck using a previous major version of a dependency and what the purpose is of each script.

I wouldn’t know what format would be ideal for Github Actions. Maybe the answer is ‘nothing’ and they need a good DSL.

And while we’re at it, stop polluting my projects root directory! Can’t we all agree on a .meta directory for finding configuration files?

Neko - A brief history and porting to Javascript

2022-11-06T21:50:00+00:00

In the early 90’s, being a frisian kid obsessed with computers there weren’t a ton of ways to get access to new software or learn more about computers.

The two main ways were exchanging 3.5” diskettes with friends, or go to the library. One book I remember more than others was “Windows for Kinderen” (“Windows for Kids”) by Addo Stuur.

I must have been around 10 years old and was obsessed by this book. It covered Windows 3.0, and when you got the book from the library, it came with a diskette filled to the brim with shareware. Mostly games and useless toys, but it still baffles me thinking they were able to cram it all on a 1.44 megabyte disk. Using floppys from the libraries was even back then a risky business given that they’re writable! Luckily this mostly went ok.

One that I remembered particularly well was ‘Neko’, an application that renders a cat in a window that follows your mouse. This must have been a popular thing to make at the time, because the diskette somehow had space for 3(!) different ports of this same application.

I don’t know what reminded me of Neko last week, but I started doing some more research, and found out that the first version was written all the way back in the 1980’s by Naoshi Watanabe for the NEC PC 9801.

Neko for the NEC PC 9801 (1980's)

After that, it was ported to the Macintosh in 1989 by Kenji Gotoh, and this art style seems to be the basis of almost every future port:

Neko on Macintosh (1989)

In 1991, IBM included it in OS/2! Apparently they paid 300,000 YEN, which is about $3000 CAD in todays money. At this point it also became public domain.

Neko on OS/2 (1991)

Since then there’s been countless ports for every platform. If you’re running linux you might be able to install one by just running:

apt install oneko

I also decided to make a version. Neko is now close to 40, so my Neko is no longer interested in following the mouse, and prefers to just sleep all day unless you wake it.

If you want to know more, the Eliot Akira wrote a great article with way more information about Neko and all its ports, this is also where I took the screenshots from this page.

You can also check out the source of my port ‘jneko’. It uses the assets of oneko, which are public domain. It was a fun exercise to create a little state machine.

Creating the assets

To create the assets for jneko, I had to convert them from the .xbm format from oneko.

I did this with imagemagick on the command line using a command like this to loop through all the files:

for f in ../oneko/bitmaps/neko/*.xbm
  convert "$f" `basename "${f%.xbm}.png"`

Sadly I still had to open each one of them with Gimp and add the alpha layer.

I hadn’t really heard of the .xbm format, but when I accidentally opened one with an editor I was surprised to see that they’re actually a text format:

#define awake_width 32
#define awake_height 32
static char awake_bits[] = {
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x04,
   0x40, 0x10, 0x10, 0x02, 0x80, 0x28, 0x28, 0x01, 0x00, 0x49, 0x24, 0x00,
   0x06, 0x44, 0x44, 0x60, 0x18, 0x84, 0x42, 0x18, 0x60, 0x82, 0x83, 0x06,
   0x00, 0x02, 0x80, 0x00, 0x00, 0x22, 0x88, 0x00, 0x0f, 0x22, 0x88, 0x78,
   0x00, 0x22, 0x88, 0x00, 0x00, 0x02, 0x80, 0x00, 0x00, 0x3a, 0xb9, 0x00,
   0x00, 0x04, 0x40, 0x00, 0x00, 0x08, 0x20, 0x00, 0x00, 0x70, 0x1c, 0x02,
   0x00, 0x40, 0x04, 0x05, 0x00, 0x20, 0x88, 0x04, 0x00, 0x10, 0x50, 0x02,
   0x00, 0x08, 0x20, 0x01, 0x00, 0x0b, 0xa0, 0x01, 0x80, 0x0c, 0x61, 0x02,
   0x40, 0x18, 0x31, 0x04, 0x40, 0x10, 0x11, 0x04, 0xc0, 0x11, 0x11, 0x07,
   0x60, 0x90, 0x13, 0x0c, 0xe0, 0xff, 0xfe, 0x0f, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};

Turns out this is the X BitMap format, which has the interesting property that it aside from an image format, they’re also valid C source files. This has the advantage that they can easily be statically compiled into C files without requiring a parser.

This is really similar to the history of the JSON format, which was designed to be a subset of Javascript, allowing Javascript programs to read the format without a new parser (using eval). Ironically the current JSON format is no longer a subset of Javascript, and Javascript engines now have a JSON.parse() function built in.

A deep dive into the source

jneko uses a state machine. All its possible states and transitions are expressed with this configuration object:

const stateMachine = {

  // Name of the state
  sleep: {

    // Which images are used for the state
    image: ['sleep1', 'sleep2'],

    // How quickly we loop through these images
    imageInterval: 1,

    // What state does this go to when clicked
    click: 'awake'
  },
  awake: {
    image: 'awake',

    // We automaticall transition to this state
    nextState: 'normal',

    // How long it takes to transition
    nextStateDelay: 2.5,
  },
  normal: {
    image: 'mati2',

    // If there's multiple nextState values, a random one is picked.
    // You make make 1 state more likely by addding it multiple times
    // to the array
    nextState: ['normal', 'normal', 'normal', 'tilt', 'scratch', 'yawn'],
    nextStateDelay: 1.5,
  },
  tilt: {
    image: 'jare2',
    nextState: 'normal',
    nextStateDelay: 1,
  },
  yawn: {
    image: 'mati3',
    nextState: ['normal', 'normal', 'sleep'],
    nextStateDelay: 1,
  },
  scratch: {
    image: ['kaki1', 'kaki2'],
    imageInterval: 0.1,
    nextState: 'normal',
    nextStateDelay: 3,
  }
};

Before Neko starts, we need to preload the images. This ensures that the animations happen instantly and not after a delay.

/**
 * The filenames are a bit funny because they were taken straight from
 * the oneko project.
 */
const imageNames = [
  'awake',
  'jare2',
  'kaki1',
  'kaki2',
  'mati2',
  'mati3',
  'sleep1',
  'sleep2',
];
/**
 * Images are actually 32x32 but it's too small for modern screens.
 */
const nekoSize = 64;
const images = Object.fromEntries(imageNames.map(name => {
  const image = new Image(nekoSize, nekoSize);
  image.src = '/assets/posts/neko/bitmaps/' + name + '.png';
  return [name, image]
}));

Finally, this class does all the heavy lifting:

class Neko {

  constructor(elem) {

    // The HTML element that hosts neko
    this.elem = elem;
    this.stateMachine = stateMachine;

    // Creating a new <img> element
    this.imgElem =  new Image(nekoSize, nekoSize);

    elem.appendChild(this.imgElem);
    this.imgElem.addEventListener('click', () => this.onClick());

    this.setState('sleep');

  }

  /**
   * This property is used to keep track of states with multiple frames
   */
  #animationIndex = 0;

  renderImage() {

    let name = this.stateMachine[this.#state].image;

    this.#animationIndex++;
    if (Array.isArray(name)) {
      name = name[this.#animationIndex % name.length];
    }
    this.imgElem.src = images[name].src;
  }

  #state = null;
  #nextStateTimeout = null;
  #imageCycleInterval = null;

  setState(stateName) {

    clearTimeout(this.nextStateTimeout);
    clearInterval(this.imageCycleInterval);

    if (Array.isArray(stateName)) {
      // If stateName was supplied as an array of strings, we'll randomly
      // pick a new state.
      stateName = stateName[Math.floor(Math.random()*(stateName.length))];
    }
    if (!this.stateMachine[stateName]) {
      throw new Error('Uknown state: ' + stateName);
    }
    this.#state = stateName;
    const stateData = this.stateMachine[this.#state];

    // If there was a nextState, we automatically transition there after
    // a delay.
    if (stateData.nextState) {
      this.nextStateTimeout = setTimeout(
        () => this.setState(stateData.nextState),
        stateData.nextStateDelay * 1000
      );
    }

    // This block is responsible for cycling through multiple images
    // of the current state.
    if (stateData.imageInterval) {

      this.imageCycleInterval = setInterval(
        () => this.renderImage(),
        stateData.imageInterval*1000
      );

    }
    this.renderImage();
  }
  onClick() {
    const stateData = this.stateMachine[this.#state];
    if (stateData.click) {
      // If the current state had a 'click' property, we'll transition
      // to that state, otherwise it's ignored.
      this.setState(stateData.click);
    }

  }

}

Now if you’d like to call Neko, make a <div id="neko"></div> somewhere. and hook up Neko with:

new Neko(document.getElementById('neko'));

To make Neko not look ugly, tell the browser to not use anti-aliasing when scaling:

#neko img {
  /* Both are needed to support all browsers currently. */
  image-rendering: pixelated;
  image-rendering: crisp-edges;
}

Thanks for reading! If you have any comments you can reply to this tweet or this mastadon post to make it show up below this article automatically.

Taking a look at Mastodon

2022-11-01T19:35:00+00:00

I’ve been a Twitter user and fan since 2007. With Twitter’s future looking a bit grim, I started looking around if there’s another place to go.

Twitter can’t really be replaced with anything else, because everyone’s Twitter experience is unique to them and their community. For me, it’s the main way I stay in touch with my Open Source / HTTP / API / Hypermedia bubble and some friends. Losing that would suck! Unfortunately, there’s no way that group would all flock to another platform.

But for the ones that do try something else, the talk of the town seems to be Mastodon.

Mastodon is interesting. On the surface it might just seem like a Twitter clone, but it’s based on a federated protocol called ‘ActivityPub’. What this means in practice is that there’s no central server. There’s many instances. Each of these instances is managed by different people, and many of them focus on specific interests.

With email, it doesn’t matter which provider you go with. Thanks to universal SMTP standards that every server uses, you can exchange messages with everyone else. This is the same with Mastodon. You’re not siloed into a single instance, and you can follow people from any other instance. Unlike email, it appears that with Mastodon you can actually migrate to different instances if you don’t like your current one.

This has some interesting side effects too. I joined the IndieWeb instance, which is a community I already loved. And even though I’m not siloed in, I get access to a local feed of like-minded people from that community. Everything feels new and more intimate.

Also, instead of one central authority that you have to trust to make the right moderation decisions, you can join one of many that aligns with your values, and you can block entire instances that don’t.

So should you join? If you use Twitter to stay on top of news and follow high profile people then probably not. If you’re like me, you might be able to find a community that fits your interest.

Will I stick to this? Who knows… but Twitter, like everything before, will fall out of favor one day and I’m enjoying Mastodon’s ad-free, open source, developer-friendly experience. Reminds me of early Twitter or mid-2000’s blogging culture.

Links

If you’re just getting started, check out https://joinmastodon.org/ and find a server.
Follow me!
To find your Twitter friends on Mastodon try Twitodon and Fedifinder.
To automatically cross-post everything from Twitter to Mastodon and vice versa, use Moa.

Lastly, one of the interesting results of Mastodon building on open protocols, is that it allows alternative implementations.

The Microblog.pub project lets you self-host a single-user instance. Instead of joining some large instance, you deploy an instance on your own domain that’s just for you. Can’t get more control than that, and this might be something I’ll consider in the future.

I don’t see why this blog couldn’t one day also be a ‘microblog’ and part of the fediverse.

Porting Curveball to Bun

2022-09-13T16:30:00+00:00

Bun is the hot new server-side Javascript runtime, in the same category as Node and Deno. Bun uses the JavascriptCore engine from Webkit, unlike Node and Deno which use V8. A big selling point is that it’s coming out faster in a many benchmarks, however the things I’m personally excited about is some of it’s quality of life features:

It parses Typescript and JSX by default (but doesn’t type check), which means there’s no need for a separate ‘dist’ directory, or a separate tool like ts-node.
It loads .env files by default.
It’s compatible with NPM, package.json, and many built-in Node modules.

I also like that it’s ‘Hello world HTTP server’ is as simple as writing this file:

// http.ts
export default {
  port: 3000,
  fetch(request: Request): Promise<Response> {
    return new Response("Hello world!");
  },
};

And then running it with:

bun run http.ts

Bun will recognize that an object with a fetch function was default-exported, and start a server on port 3000. As you can see here, this uses the standard Request and Response objects you use in a browser, and can use async/await.

These are all things that didn’t exist when Node and Express were first created, but seem like pretty good ideas for something built today. I don’t think using Request and Response are good for more complex use-cases (streaming responses, 1xx responses, trailers, upgrading to other protocols, getting tcp connection metadata like remoteAddr are some that come to mind), because these objects are designed for clients first.

But in many cases people are just building simple endpoints, and for that it’s great.

Bun supports a ton of the standard Node modules, but it’s also missing some such as support for server-side websockets and the node http/https/https packages, which for now makes it incompatible with popular frameworks like Express.

Porting Curveball

Curveball is a Typescript micro-framework we’ve been developing since mid-2018 as a modern replacement for Express and Koa. A key difference between Curveball and these two frameworks is that it fully abstracts and encapsulates the core ‘Request’ and ‘Response’ objects Node provides.

This made it very easy to create a lambda integration in the past; instead of mapping to Node’s Request and Response types, All I needed was simple mapping function for Lambdas idea of what a request and response looks like.

To get Express to run on AWS Lambda the Node http stack needs to be emulated, or a full-blown HTTP/TCP server needs to be started and proxied to. Each of these workarounds require a ton of code from libraries like serverless-express.

So with Bun up and coming, either the same work would need to be done to emulate Node’s APIs, or Bun would would need to add full compability for the Node http module (which is eventually coming).

But because of Curveball’s message abstractions it was relatively easy to get up and running. Most of the work was moving the Node-specific code into a new package.

Here’s the Curveball ‘hello world’ on Bun:

import { Application } from '@curveball/kernel';

const app = new Application();

// Add all your middlewares here! This should look familiar to Koa users.
app.use( ctx => {
  ctx.response.body = {msg: 'hello world!'}; 
});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

It’s still a bit experimental, but the following middlewares are tested:

And because JSX also just works in Bun, it’s relatively easy to use it to generate HTML:

import { Application } from '@curveball/kernel';
import reactMw from '@curveball/react';
import React from 'react';

const app = new Application();

app.use(reactMw());

app.use( ctx => {

  ctx.response.type = 'text/html; charset=utf-8';
  ctx.response.body = <html>
    <body>
      <div>
        <h1>Hello world!</h1>
      </div>
    </body>
  </html>;

});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

This is not a full-blown hydration/server-rendering solution, but JSX has replaced template engines like EJS and Handlebars for us at Bad Gateway. JSX lets you do the same things, but you get the advantage of type checking, it’s harder to create XSS bugs and full Javascript access.

Final thoughts on Bun

Compared to Deno, It was considerably easier to port Curveball to Bun. Deno was a much greater challenge, due to it having such a radically different idea about packages and module loading. This was over a year ago, so it’s worth giving a shot again.

So, I’m curious where all of this goes! Perhaps Bun is the future, or perhaps we’ll see Node get parity with Bun and making it obsolete. Either way competition is good.

I feel Bun has a better chance than Deno, because it delivers many of its advantages and features while mostly staying inside with the Node ecosystem.

Although as it turns out Deno also has changed their tune and also made it a new goal to be compatible. I can’t help wondering if this was inspired by Bun’s recent success as well.

Ubuntu bungled the Firefox Snap package transition

2022-09-01T09:39:29+00:00

I’m not a Snap hater. On paper it’s a good idea, but as a user I shouldn’t really be aware that ‘snaps’ even exist. In Ubuntu 21.10, Firefox became a snap package.

Arguably the browser is the most important application in an operating system. Here’s a non-exhaustive list of issues I’ve personally ran into. I should note that some of these issues are now fixed, but I wanted to illustrate what Ubuntu launched with:

Printing is completely broken, and I can only print to PDF.
KeePassXC, an open source password managers’ browser extension no longer works.
Firefox thinks that when opening ‘localhost:8080’ should open the URI scheme ‘localhost’ and tries to find an application that supports this scheme (now fixed!)
Gnome shell integration extension, the primary way to install gnome add-ons is now broken.
‘Set image as desktop background’ is broken.
Opening applications via a custom URI scheme no longer asks for confirmation, this makes it possible to (for example) launch a bittorrent client like Transmission via a magnet: uri without asking a user.
The Mozilla VPN product has a neat feature that lets you have specific containers always use a VPN. This doesn’t work.
Firefox creates a ‘firefox.tmp’ directory in the Downloads folder (fixed!)
When there’s an update to the Firefox package, the following notification appears, once per day. Restarting Firefox does not make this go away. The official answer is to run snap refresh firefox to make it go away.
GSConnect Firefox Add-on is broken.

This is just the stuff I ran into myself, (and I have reported most of these). I imagine the total list of bugs must be way higher. I don’t usually go out and complain on the internet like this, especially when it’s about open source projects. I’m a Linux user, so I’ve kind of come to expect things to not be quite as polished as some of its commercial counterparts. They’re small trade-offs to support Open Source.

However, I’m so surprised by the lack of quality control for arguably the #1 application on the #1 linux distro I’m frankly flabbergasted, and for the first time since switching from Debian to Ubuntu 15ish years ago I’m considering jumping ship again. What happened here?

Comments? Reply to this tweet

On syntactic sugar

2022-08-13T18:12:29+00:00

Ever so often the term ‘syntactic sugar’ comes when people discuss language features, and it’s not uncommon to see the word ‘just’ right in front of it; some examples:

The ‘just’ has a lot of meaning here. To me it suggests that language features that are ‘just’ syntactic sugar, aren’t quite as important as features that aren’t. Maybe it even suggests to me that the language would be fine without.

So while the above two examples both argue that both Javascript classes and async/await aren’t syntactic sugar, they also kind of come to the defence of those features and justify their existence. In other cases when people call something syntactic sugar, it’s often in a context that’s somewhat dismissal of the feature.

I think this is a bit odd and also confuses people, especially since any actual definition I’ve found is generally positive, such as this one from Wikipedia.

In computer science, syntactic sugar is syntax within a programming language that is designed to make things easier to read or to express. It makes the language “sweeter” for human use: things can be expressed more clearly, more concisely, or in an alternative style that some may prefer.

The thing is, isn’t every language feature beyond the bare minimum of what makes a language turing-complete syntax sugar?

You don’t need classes because you can use functions and structs.
You don’t really need types because everything can fit in a string.
Functions can be implemented with goto.
Multiply can be implemented with addition.
or and and, xor can be implemented with nand.
async/await can be implemented with generators.

These are all incredibly useful features that make it easier to read and write code and express ideas. Almost every language feature could be considered syntactic sugar. That’s not a bad thing, it’s just uninteresting to point out.

A new OAuth2 client for Javascript

2022-06-20T19:40:00+00:00

Frustrated with the lack of well maintained, minimal OAuth2 libraries, I wrote my own. This new OAuth2 library is only 3KB gzipped, mainly because it has 0 dependencies and relies on modern APIs like fetch() and Web Crypto which are built in Node 18 (but it works with Polyfills on Node 14 and 16).

It has support for key features such as:

authorization_code with PKCE support.
password and client_credentials grants.
a fetch() wrapper that automatically adds Bearer tokens and refreshes them.
OAuth2 endpoint discovery via the Server metadata document (RFC8414).
OAuth2 Token Introspection (RFC7662).

If your server does support the meta-data document, here’s how simple the process can be:

client_credentials example

import { OAuth2Client } from '@badgateway/oauth2-client';

const client = new Client({
  clientId: '..',
  clientSecret: '..',
  server: 'https://my-auth-server.example'
});

const tokens = await client.clientCredentials();

Without the meta-data document, you will need to specify settings such as the tokenEndpoint and possibly the authorizationEndpoint depending on which flow you are using.

authorization_code example

The authorization_code flow is a multi-step process, so a bit more involved. The library gives you direct access to the primitives, allowing you to integrate in your own frameworks and applications.

import { OAuth2Client, generateCodeVerifier } from '@badgateway/oauth2-client';

const client = new OAuth2Client({
  server: 'https://authserver.example/',
  clientId: '...',
});

// Part of PCKE
const codeVerifier = await generateCodeVerifier();

// In a browser this might work as follows:
document.location = await client.authorizationCode.getAuthorizeUri({
  redirectUri: 'https://my-app.example/',
  state: 'some-string',
  codeVerifier,
  scope: ['scope1', 'scope2'],
});

Handling the redirect back

const oauth2Token = await client.authorizationCode.getTokenFromCodeRedirect(
  document.location,
  {
    redirectUri: 'https://my-app.example/',
    state: 'some-string',
    codeVerifier,
  }
);

Docs and download

Github
npmjs

Reasons why abolishing DST in the US will be worse for users and developers

2022-03-18T08:51:00+00:00

Daylight savings time is hated by many, and twice per year a discussion reignites to get rid of it. Lot of folks feel this is a great idea. This year this decision seems especially close in the US. If this law passes, it will probably also change where I live 🇨🇦.

No doubt there’s lots of benefits and advantages to this, I don’t want to dispute that. This is also not an endorsement against that change, but I do however want to bring light to at least one disadvantage:

The timezone change was for a lot of developers a twice-per-year reminder that we need to use timezone databases and libraries.

This is useful, because every year many countries change their timezone rules. This means that if you schedule something in the future, say.. 2pm 6 months from, you don’t know yet with absolute certainty what UTC timestamp this will be. This is especially important when scheduling between people in multiple timezones.

The right way to handle this is to store the intended local time + a location such as America/Toronto. EST is not enough, because it’s EDT half the year, and obviously neither is -0500. I grew up in the netherlands, and it was only when I got into programming I realized that our timezone is not +0100 all year round, unlike what we learned in school.

Timezones are relatively stable in North America, but the US also made a change in 2007, and it sounds like we may have another one in the future!

So this bi-annual time change was a great reminder to many developers that timezones are a thing, and you can’t just naively assume a UTC time + an offset is enough. Even more so for teams that are spread cross-continent because the DST change doesn’t fall on the same day. Currently I’m in the 3 weeks per year the time difference between me and my parents is 5 instead of 6 hours.

A lot of programming is (seems?) anglo-centric. A similar situation is that before Emoji became wide-spread it was way more common to see a lot more issues around encoding non-ascii characters 🤷 (billpg). Especially in languages that don’t have good native unicode support (looking at you PHP).

So if DST goes away in North America, I predict we’ll see more people assuming using the offset is enough, resulting in bugs related to:

Times in countries that have not yet abolished DST.
Countries that ever change timezone rules. (This happens more often than you think!)
Applications that deal with historical data.

It doesn’t help that one of the most common date formats (ISO 8601) uses an offset! (2022-03-18T17:05:30.996-0400). This is OKish for things that have already happened, but not good for anything in the future.

So when you hear developers excited about the US abolishing DST because it will make their (work) life simpler, remind them this is only true if you never intend your software to be used outside of North America, or when the entire rest of the world makes the same change and also freezes all timezone rules forever!

Evert Pot's blog

Putting Curveball in maintenance mode

Some quick numbers.

Why build another framework

Why did Curveball fail

I’m using Curveball for my project, now what?

If you like the project, why can’t you work on it despite not having many users

Are there any bits salvagable?

Thank you

In the future using top-level await might be cause a backwards compatibility break in Node

The Node 23 change

Using top-level await is now a BC break

1. Tell your users you don’t support require()

2. Add a dummy await

3. Explictly break CommonJS

How do other runtimes do it?

Does the issue exists when importing CommonJS files into modules?

What should I do as a user?

My opinion

Discovering features using HTTP OPTIONS

Accept and Accept-Encoding

Patching, posting and querying

Where’s PUT?

Linking to documentation

Obscure uses

WebDAV usage

The server-wide asterisk request

Final notes

New Structured Fields RFC out, and so is my Javascript package

What is it?

Why should you care?

Javascript package

Comments?

Hello World, meet Kian

Creating a fake download counter with Web Components

Why is this number so high?

The Web Component

Obtaining the data

NPM

Packagist

Putting it together

Conclusion

Moving on from Mocha, Chai and nyc.

OAuth2 client updates

Using JSX on the server as a template engine

What is JSX?

#Inspo

JSX Transpiled

Using JSX as a template engine

Koa

Fastify

How does this magic work?

Koa middleware

Fastify hooks

Limitations to this approach

Frequently asked questions

How do you handle routing?

Is there still a reason to use Handlebars or EJS?

Anyway

Why aren't there more 80% jobs?

My background

Advantages of offering 80% jobs

Does OAuth2 have a usability problem? (yes!)

My perspective

Solving the setup issue

My proposal

Final notes

Switching to Fedora from Ubuntu

Ubuntu’s fall

Onto Fedora

Installation

Non-free stuff

Gnome

Flatpak

Bugginess

Software Center app

Would I recommend Fedora?

Supporting CommonJS and ESM with Typescript and Node

Why support ESM

The general approach

`dirname` and `filename`