Qubyte Codes

Private data for JS classes with WeakMap

2015-12-30T14:35:00Z

Private data has always been awkward in JavaScript. It's particularly difficult when it comes to constructors, and with ES2015 recently published, classes too. Let's say we have an example class, exported by an ES2015 module:

export default class Example {
  constructor() {
    this._privateDatum = Math.random();
  }

  log() {
    console.log(this._privateDatum);
  }
}

In the constructor, a field with some private data, _privateDatum is appended (the value is a placeholder for illustration). The initial underscore in the name is a common convention and is meant to tell developers using the class that they shouldn't touch or look at that field. Why should this be private? Private stuff is subject to change without your users needing to know about it. This field could be renamed or go away completely if you refactor, without affecting the public API. So what's the problem?

You can't trust your users!

This isn't meant as an insult. Your users are cunning, and if they can solve a problem without filing an issue or raising a pull request, they probably will. They have deadlines after all... If your class gets very popular, it becomes inevitable that someone is going to use your private-by-convention field to hack together a solution to a problem they're having, and you'll break their code when you change it. Changes to your public API should be clearly indicated by changes to the version number and updated documentation. The inner workings of your code on the other hand, including private data, are subject to dramatic change at any time.

The solution is to hide the private data, removing the temptation. You can do this using a closure. Consider:

export default class Example {
  constructor() {
    const privateDatum = Math.random();

    this.log = function () {
      console.log(privateDatum);
    }
  }
}

In this example, the private data is now assigned to a variable in the constructor. Since the variable is not returned, nothing outside the constructor will have access to it. The pain now is that the log method has to be attached to the instance inside the constructor, so that it can have access to the variable. It's a real shame to lose the nice method syntax. It also make an individual log method for each instance, which means objects will each use more memory.

This is where WeakMap comes in. An instance of WeakMap has keys which are objects of some kind, and values which can be whatever you like. WeakMap instances are especially good, since if they are the last thing to hold a reference to an object (as a key), then the JS engine is allowed to garbage collect it. This means the risk of memory leaks is lessened. You could simulate most aspects of WeakMap using existing structures like arrays, but that would always result in a memory leak, since the garbage collector thinks those objects are in use and cannot clear them up. The final example below shows what this looks like:

const privateDatum = new WeakMap();

export default class Example {
  constructor() {
    privateDatum.set(this, Math.random());
  }

  log() {
    console.log(privateDatum.get(this));
  }
}

The keys of privateDatum are the instances of the example class. If nothing else holds a reference to an instance of the example class, the garbage collector doesn't count the reference in privateDatum and can clear it up! Since the instances are keys, this can be used in any method to access the private data. The privateDatum variable hidden by the module, so the user will have no access to it!

This approach can be used with constructor functions and methods appended to the prototype too. The following constructor produces objects with similar behaviour to those produced by the class in the previous example:

const privateDatum = new WeakMap();

export default function Example() {
  privateDatum.set(this, Math.random());
}

Example.prototype.log = function () {
  console.log(privateDatum.get(this));
};

The good news is that WeakMap is one of the most well supported features of ES2015. With the exception of IE Mobile and Opera Mobile, all current versions of major browsers support the functionality in this post. See the compatibility table. If you're using a maintained version of Node, you're good to go!

Addendum

It's still possible to gain access to private data stored in a WeakMap by patching WeakMap.prototype.set or WeakMap.prototype.get. This should absolutely never be done! Along with the usual reasons to not modify the prototype of a built in constructor, modifying WeakMap risks undoing the whole reason for using it in the first place. By monitoring objects used as the keys of a WeakMap, references can be created and the garbage collector may not be able to clean up after you. That said, patching can be done. If you want to avoid that risk, you can Object.freeze both WeakMap and WeakMap.prototype before any other code runs.

About this blog

2016-01-03T12:22:00Z

This blog took a long time to get started. Every time I tried to build it, I wound up focussed on some tech I wanted to use to host it. In the previous iteration, I even wrote a server framework. I took some holiday over the Christmas period, so I decided to throw everything away and make something minimal.

I chose to go with NGINX serving flat files produced by a static site generator. This is still me of course, and on my own time I like to allow myself to indulge in a little reinvention of the wheel, so I wrote the generator myself. It's about a hundred lines of code, mainly stitching together other small modules. I used Node to make the generator, with the following modules:

module	explanation
`front-matter`	For keeping YAML metadata at the top of post files.
`marked`	To compile post markdown to HTML.
`highlight.js`	To highlight code listings.
`handlebars`	To render posts into templates and an index page.
`remark`	To pluck the first paragraph from each post to render into the index page.
`slug`	To make post URLs readable.
`clean-css`	To compile CSS sources together.

Posts are markdown files committed to a git repo. I use the pre-commit hook to compile and add the rendered posts. At the moment I log into the server to pull changes down. I'll have that working on a GitHub hook soon.

I've no intention of posting the generator, since that would mean supporting it. It's strictly for my use. I've linked the modules above since they've been very useful to me. If you're thinking about setting up a blog and you're a programmer, I recommend this approach!

About this blog 2

2016-01-11T20:15:00Z

I touched briefly on the technology used in this blog in a previous post, but I didn't explain the motivation behind a lot of the choices I made when building it. I'd like to do that in this post. The design and architecture of this blog is the product of what things I like in other blogs, and also those things that I find frustrating. Where a choice was not obvious, I opted for the simplest option. The point of the exercise was to get it online. Below are a few points in no particular order.

HTTPS

SSL certificates are now free thanks to Let's Encrypt, so there is absolutely no excuse to host blog content over plain HTTP. By doing this you honour the privacy of your readers. It's a little clunky if you're using NGINX (as this blog is), but still far easier than buying a domain name and hosting, sorting DNS etc.

No tracking

I don't currently use tracking or cookies, mainly because I don't need them. Tracking for a blog is largely narcissism. There are no sessions for a blog, so no cookies are necessary. I might track some basic data like user agent later, but that's mainly so I know what browser features I can get away with using. That sort of thing can be done server side. I don't need that now though.

Render once

I really like single page apps. I work on one as part of my day job. The distinction here is that a blog is not an application (at least not in my head). A blog is really just the good old fashioned home page. Given that, I host HTML pages, and there's no JavaScript code rendering DOM. This has some nice repercussions. The browser does very little, so this is kind to batteries. Since the pages are not dynamic, the pages can be statically generated by me, and hosted as files. This avoids building HTML on the server and a database to store pre-rendered data. Just as this is kind to the browser, it's kind to the server.

Responsiveness

My CSS skills are pretty basic. I knew when building this that I wanted a classic centred column on conventional computers, and also for posts to be readable on mobile devices. The CSS is configured such that the text (including margins) will occupy the full width up to a maximum 800px, and thereafter the text column remains at 800px wide and centred. Pretty old school, but also clean.

Dated posts

It's rare, but I do come across how-to style blog posts which lack dates. This is frustrating, since it's difficult to know if the information in the post is stale.

Getting to the point

Probably my biggest gripe with modern blogs and blog platforms is the massive image which dominates the top of each post. This fad is terrible. You force the reader to scroll before they can begin reading. You're also making the browser download that image, which probably takes more data than the rest of the page combined. This blog doesn't do it. I want a reader to be able to start reading with no interaction at all. This blog has been tested with devices as small as an iPhone 4 to that end.

Mobile friendly

I've already touched on this above. A big part of mobile friendly is handling crappy networks. Crappy give each request a good chance of failing. Since the CSS is very simple, I minify and inline it in the head of each post. This means that the typical post is just a single file, so there is as good a chance as possible that the reader will receive the whole post.

No comment

Comments are complex. Rolling my own would mean a database and rendering on the server or in the browser. Using a third party would probably mean scripts. Both are fine, but I'm not sure if comments are worth the trouble. If something is really worth saying, you can always tweet to me or tweet a gist to me. If comments become desirable in the future, I can revisit them.

How I schedule posts using atd

2016-01-14T19:30:00Z

This blog is built with a static site generator. The generator, the markdown source files, and the generated HTML files are all kept together in the same git repository. Every time I commit a change, a pre-commit hook runs the generator and adds the generated HTML, so that the blog entries are always up to date. Then the changes are pushed up to GitHub.

On the server (running Linux), NGINX is hosting a folder containing the files to serve from a clone of the repo. To publish a new post, or update an old one, all I have to do is pull the changes from GitHub. Publishing via the terminal allows me to use a one liner to schedule the publication. Many know about cron for scheduling repeated tasks, but fewer are aware of atd, which is for one time scheduled tasks. The one liner I use is:

echo "git -C /absolute/path/to/repo pull origin master" | at 07:00 tomorrow

at accepts input through stdin, which is why I've echoed the command and piped it. One big gotcha is that you have to be aware of the timezone your machine is configured for. I have mine set to UTC, and schedule according to that.

There are companion utilities to manage scheduled jobs. The man page for at is extremely good, so I won't try to better it. The purpose of this post is to show you that it exists and how simple (especially when compared with cron) it is to use. You can use it to schedule pretty much anything!

Private methods for JS classes

2016-01-31T02:00:00Z

This is a short companion to an earlier article I wrote on using WeakMap for private data with JS classes. While private data belongs to instances, private methods can be shared between instances of a class (just like its regular methods). An implementation using ES2015 modules looks like:

function setContent(content) {
  const text = document.createTextNode(content);

  this.el.appendChild(text);
}

export default class Paragraph {
  constructor(content) {
    this.el = document.createElement('p');

    setContent.call(this, content);
  }

  replace(content) {
    this.el.innerHTML = '';

    setContent.call(this, content);
  }
}

This module exports a class which wraps a paragraph element (which is not a particularly useful thing to do, but hopefully illustrates what I'm about to say).

Each instance of Paragraph is constructed with some content. The one public method it has is replace, which allows the content to be replaced. Both the constructor and the replace methods set content, so rather than repeat the code that does that, the code is placed in the private method setContent. setContent operates on a Paragraph instance passed in as the context by using call. You could just as easily pass the instance in by using a parameter. I've chosen the former approach here to keep the difference between public and private methods to the minimum.

By virtue of the setContent method being declared outside of the class declaration, the class does not make it available. You control which things you export from the module, so if the class doesn't make replace available, and the module doesn't export it, then nothing outside the module has access. Thus setContent is a private method.

For clarity, replace being public means that I can do this:

var paragraph = new Paragraph('hello');

// replace is pubic, so we can use it.
paragraph.replace('goodbye');

And setContent being private means that I cannot do this:

var paragraph = new Paragraph('hello');

// setContent is not a part of the Paragraph
// definition. We cannot do this.
paragraph.setContent('goodbye');

So there you have it. There's not a lot to the pattern really. Earlier module types and patterns allow the same thing to be done. I recommend reading the section on module patterns in Addy Osmani's excellent book Learning JavaScript Design Patterns if you want to know more (and particularly the sub section on the module pattern).

Interfaces for JavaScript

2016-03-13T10:45:00Z

I use instanceof a lot in JavaScript. It's very handy when writing unit tests. It's easier to do an instanceof check than it is to exhaustively probe an object.

Unfortunately instanceof usually means an object has been constructed. If the constructed object is coming from a third party library, or there is no access to the constructor, it can become fiddly.

This is why I'm excited about ES2015s Symbol.hasInstance. It allows you to tune the behaviour of instanceof for a class. Here's a minimal example of an interface class:

class PositiveInteger {
  constructor() {
    throw new Error('PositiveInteger is an interface class.');
  }

  static [Symbol.hasInstance](value) {
    if (typeof value !== 'number') {
      return false;
    }

    if (value < 0) {
      return false;
    }

    return value % 1 === 0;
  }
}

assert.ok(10 instanceof PositiveInteger);   // does not throw
assert.ok(-10 instanceof PositiveInteger);  // throws
assert.ok('hi' instanceof PositiveInteger); // throws
var positiveInt = new PositiveInteger();    // throws

The class above exists only to provide this instanceof check. A more interesting example might be a view. I assert that a view has an element, and render and remove methods. An interface class for this might be:

class View {
  constructor() {
    throw new Error('View is an interface class.');
  }

  static [Symbol.hasInstance](value) {
    if (!value) {
      return false;
    }

    if (typeof value.render !== 'function') {
      return false;
    }

    if (typeof value.remove !== 'function') {
      return false;
    }

    return value.element instanceof HTMLElement;
  }
}

Now view objects can come from any source as long as they have render and remove methods and an element. Objects which implement various interface classes also mean that these interface classes don't have to be in the same prototype chain. In other words, it gives you a way to use instanceof without invoking inheritance.

Sadly hasInstance will be one of the last ES2015 features to make it into browsers, so we'll have to wait a while before we can use it. See the compatibility table.

A presentation on async-await and Toisu!

2016-04-06T22:00:00Z

Just before Christmas I gave a presentation on the upcoming async-await JavaScript language feature, its basis in promises and generators, and finally a tiny server framework (like Express but a lot leaner and more modular) which can make use of async functions as middleware (since an async function is indistinguishable from a normal function which returns a promise). I'll introduce Toisu! in a blog post soon, but until then here's the presentation:

Adding missing features to Set

2016-05-22T16:30:00Z

ES2015 bought a Set constructor to JavaScript. It's pretty barebones, consisting of a constructor which creates objects with a few methods for adding, removing, checking if something is a member, and iterating over the set. Instances have the essential quality of a set; an item is a member of the set or not a member. Unlike an array, an item cannot be an element more than once. In other words you can avoid using arrays and doing a lot of indexOf checking.

For example, abusing an array to act like a set is very common:

var myCollection = [];

// Check if element is in the collection.
function has(collection, element) {
  return collection.indexOf(element) !== -1;
}

// Add an element to the collection.
function add(collection, element) {
  if (!has(collection, element)) {
    collection.push(element);
  }
}

add(myCollection, 123);
has(myCollection, 123); // returns true
has(myCollection, 456); // returns false

Using a set is more straight forward:

var myCollection = new Set();

myCollection.add(123); // add an element
myCollection.has(123); // returns true
myCollection.has(456); // returns false

That's all grand, but a colleague of mine, a programmer dipping a toe into modern JS, noted that Set lacks some functions that you might expect it to have out of the box, such as union, intersection and (relative) complement.

I'd implement these as static methods of Set, and shim them in. Hopefully these methods will be added to the standard in the future. Until then, here are some possible implementations.

Union

The union of two sets is defined as the set of elements found in either or both sets.

Set.union = function (a, b) {
  const unionSet = new Set();

  for (const element of a) {
    unionSet.add(element);
  }

  for (const element of b) {
    unionSet.add(element)
  }

  return union;
};

This function iterates over both sets, adding their elements to a new set, which is then returned.

As a one liner:

Set.union = (a, b) => new Set([...a, ...b]);

Sets are iterable, so the spread operator can be used to expand them as arrays. Set can also take an array as an argument, filling the constructed set object with elements from the array. This means that both sets can be spread into an array literal and fed back into the set constructor. Since repeated elements are ignored, the resultant set is the union of the two input sets.

Intersection

The intersection of two sets is defined as the set of elements they have in common.

Set.intersection = function (a, b) {
  const [small, big] = a.size < b.size ? [a, b] : [b, a];
  const intersectionSet = new Set();

  for (const element of small) {
    if (big.has(element)) {
      intersectionSet.add(element);
    }
  }

  return intersectionSet;
};

The first line of this one is an optimisation. The largest an intersection can be is the size of the smaller set. The first line uses destructuring assignment and a ternary to assign the smaller set to small and the larger set to big. The intersectionSet is then constructed, and small looped over to fill it with elements which are also in the big.

If you want a one liner for this (and don't care about the size optimisation):

Set.intersection = (a, b) => new Set([...a].filter(element => b.has(element)));

This uses similar tricks to Set.union. One set is expanded as an array, and then filtered to an array of elements which are in the other. The filtered array is then fed to Set to get the intersection set.

Relative complement

The relative complement of A in B is the set of those elements in B which are not in A.

Set.complement = function (a, b) {
  const complementSet = new Set();

  for (const element of b) {
    if (!a.has(element)) {
      complementSet.add(element);
    }
  }

  return complementSet;
};

This function makes a new set and iterates over the second set, filling the new set with elements of the second set which are not in the first.

As a one liner:

Set.complement = (a, b) => new Set([...b].filter(element => !a.has(element)));

This uses similar tricks to the Set.intersection one liner. It expands the second set as an array, filters it down to elements not in the first, and then instantiates a new Set with the filtered array.

Tip: customizing npm version

2016-09-05T20:00:00Z

The npm CLI has a bunch of useful utilities for managing projects. The obvious one is npm test but there are others. I particularly like working with npm version (the subject of this tip).

Without customization, npm version checks that the working git directory is clean, sets the new version in the package file, and then commits it and tags the repository with the same new version.

That's great, but what if you have a bower.json file, or some other task which needs the new version before the tag is made? npm version updates the version in package.json and it misses the version in the bower.json file. The bower.json version is just an example, but one that illustrates this point.

This is where customizing npm version comes in. Continuing the bower.json example, we can write a script which performs the version update, and uses git add on it. In the package.json file:

{
  "name": "my lovely app",
  "version": 1.0.0,
  "scripts": {
    "version": "node update-bower-version.js && git add bower.json"
  }
}

To reiterate, npm version will update the version in the package file, run the version script above, and then commit the changes and tag the result. That means that the script needs to do something to the bower.json file, then git add the result.

// update-bower-version.js

var fs = require('fs');
var bowerJsonPath = require.resolve('./bower');
var bowerJson = require(bowerJsonPath);

bowerJson.version = process.env.npm_package_version; // npm injects this

fs.writeFileSync(bowerJsonPath, JSON.stringify(bowerJson, null, 2));

This script is going to be run by npm from the package.json file, which handily injects some environment variables. One of these variables is the version in the package.json file. This is great because it means that no matter what the new version will be we have access to it in this script.

Since the bower.json file contains JSON it can be read and parsed by require in one step. The version is updated, and the result stringified (I'm using two space indentation here) and written back to the bower file.

With a script in place, you can use npm version without further thought. for example, a major version bump looks like:

npm version major

Enjoy!

Progressive enhancement #1

2016-10-15T18:00:00Z

When I first put together the CSS for this blog I avoided a fixed header since the header felt a bit large, and I didn't want to take up too much space which could be used for content.

The solution is a header which shrinks as the reader scrolls down. This gives back a little space, and maintains access to the navigation bar (which is a part of the header). To make it look nice, the title animates between the smaller and larger states using a CSS transition. The CSS looks something like:

h1 {
  font-size: 2rem;
  margin: 1rem 0;
  transition: all 0.3s;
}

.smaller h1 {
  font-size: 1rem;
  margin: 0.5rem 0;
}

I'm a novice when it comes to CSS, so let me know if I'm missing a trick!

The markup for the top header (which has not changed), looks like:

<header class="top-header">
  <h1>...</h1>
  <nav>...</nav>
</header>

To trigger the transition, a little JavaScript is needed to detect when a scroll event occurs. When the page is not scrolled down the header has no additional classes added. When the page is scrolled a class is added to make the header h1 text and margin smaller. When the page is scrolled down and then back to the top, the header h1 text returns to its original size.

var $header = document.querySelector('.top-header');

function checkHeaderSmallText() {
  if (window.pageYOffset > 0) {
    $header.classList.add('smaller');
  } else {
    $header.classList.remove('smaller');
  }
}

window.addEventListener('scroll', checkHeaderSmallText, false);

checkHeaderSmallText();

So why is this progressive? Most importantly, no markup (with the exception of an added script tag in the head) has been added. Browsers with JavaScript disabled or screen readers will interpret the site as they did before. If the feature fails for any reason, or example if deferred scripts or classList are not supported, then an error will be thrown and the header will fail to shrink. Should CSS transitions not be supported, the header will immediately go from the larger to the smaller state on scrolling. These failure modes are acceptable in my opinion.

In the future I have a more elaborate header animation in mind. If you're interested in such things, watch this space!

Progressive enhancement #2

2016-11-12T03:00:00Z

I recently attended ffconf, and was introduced to position: sticky;. Support for it is patchy, but where not available the header will scroll out of view as it did in the past. Where available, the navbar will stick to the top of the window when the rest of the header is scrolled out of view.

This is a pure CSS solution, so this blog once again serves no JavaScript! the only thing which I wanted that CSS does not provide is a selector for elements in the stuck state.

Update: You can view the video of Rachel Andrew's presentation including position: sticky; here.

Promises and Node.js event emitters don't mix

2016-12-24T17:00:00Z

To many experienced Node developers, the title of this post will seem intuitively obvious. Nevertheless, it's useful to see what unexpected behaviour can occur when the two are used together. Here's an example:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

Promise.reject(new Error('Oh noes!'))
  .catch(e => emitter.emit('error', e));

If you run this code (tested in Node v7.3 from a script, not REPL), what do you expect to happen? I expected an uncaughtException event, since an event emitter is emitting an error event with no event handler, but that's not what happens at all. Instead you get an UnhandledPromiseRejectionWarning!

This is bad news. If something genuinely nasty has happened, you might want to emit an error like this which is either explicitly handled or causes an uncaughtException event. Uncaught exceptions should lead to the process exiting, and a promise has just stifled it.

So, what happened? Here's a hint:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

try {
  emitter.emit('error', new Error('Oh noes!'));
} catch (e) {
  console.log('caught');
}

What do you expect to happen here? Again, at first glance I would have expected an uncaughtException event. uncaughtException events can happen when one of two conditions is met. The first is an error is thrown but not caught (thus the name). The second is when an error event is emitted and the emitter has no handler to deal with it.

Did I say there were two conditions? I meant to say one! The second condition is really the first in disguise. When an emitter lacks an error handler and it emits an error event, the default error handler takes control and throws the error. Since event handlers are called synchronously upon event emissions, this:

const { EventEmitter } = require('events');
const emitter = new EventEmitter();

emitter.emit('error', new Error('Oh noes!'));

is equivalent to

throw new Error('Oh noes!');

So, the very first snippet is equivalent to:

Promise.reject(new Error('Oh noes!'))
  .catch(e => {
    throw e;
  });

The promise chain captures the thrown error and wraps it as a rejected promise, which leads to an UnhandledPromiseRejectionWarning since a second .catch is needed in the chain to handle it.

If you want to ensure that unhandled error events lead to uncaught exceptions which aren't captured by a catch or a promise chain, then you can wrap the emission in a setTimeout or a setImmediate.

Making arcade controls: Arduino Leonardo code

2017-05-01T00:00:00Z

I recently got it into my head that I wanted to build an arcade control panel from parts. Specifically, an 8 way digital joystick and a bunch of buttons. How it'll look when finished isn't important at the moment. It's enough now to say that there'll be a joystick, six regular buttons, and two buttons for start and select use.

I decided to use an Arduino Leonardo to accept inputs from the buttons and stick. The Leonardo presents itself as a keyboard to the machine you plug it into, which is perfect for this. The only thing I needed to do is to initialize the pins on the Arduino as inputs with inline resistance, and bind them to the desired keys.

Usually when a programmer thinks "the only thing I need to do", what follows is three or four times as much effort as predicted. I was surprised this time that the effort required was so low, particularly given that I've not written any C/C++ in years. In addition to functions for setting key states, keyboard.h provides a bunch of useful constants for special keys.

#include <Keyboard.h>

// MAMEish. An array of pin-key pairs.
struct { int pin; int key; } pinkeys[] = {
  { 2,  KEY_LEFT_ARROW  },
  { 3,  KEY_UP_ARROW    },
  { 4,  KEY_RIGHT_ARROW },
  { 5,  KEY_DOWN_ARROW  },
  { 6,  KEY_LEFT_CTRL }, // Fire 1
  { 7,  KEY_LEFT_ALT  }, // Fire 2
  { 8,  ' ' },           // Fire 3
  { 9,  'a' },           // Fire 4
  { 10, 's' },           // Fire 5
  { 11, 'q' },           // Fire 6
  { A0, '1' },           // start
  { A1, KEY_ESC }        // select
};

void setup() {
  // Set all used pins to handle input
  // from arcade buttons.
  for (auto const &pinkey : pinkeys) {
    pinMode(pinkey.pin, INPUT_PULLUP);
  }

  // Initialize keyboard.
  Keyboard.begin();
}

void loop() {
  // For each pin-key pair, check the
  // state of the pin and set the
  // associated key state to match.
  for (auto const &pinkey : pinkeys) {
    if (digitalRead(pinkey.pin) == LOW) {
      Keyboard.press(pinkey.key);
    } else {
      Keyboard.release(pinkey.key);
    }
  }
}

Since the Arduino is programmed in a dialect of recent C++, range-based for loops are available, as is type inference with auto. This meant I could use an array of instances of an anonymous structure to express the pin-key pairs. Not a sizeof in sight!

Test friendly mixins

2017-07-20T13:30:00Z

I've recently been attempting to code a clone of the classic game asteroids using canvas in the browser. Since this is me, I've been distracted by all sorts of programming detours.

This post is roughly the process I went through for one such detour.

I began planning the game by sketching out the objects it would contain. This lead to a relatively (for the subject matter) deep class hierarchy including an abstract entity as a base class, a ship class, an asteroid class, a bullet class, and so on. It quickly became obvious that it was getting convoluted, with some repetition to avoid artificial seeming relationships between these classes.

To avoid this sort of convolution, I could eschew classes in favour of plain objects and mixins, where mixins embody chunks of useful behaviour which are copied onto host objects. For example, most objects in the game can move, so I'd write a mixin function to copy a move method onto host objects.

I'm not a purist though. There's still value to a base class to encode all essential properties of all objects in the game. Extending the base class for other objects in the game makes sense for those behaviours and properties unique to the child class. For example, the ship can be controlled by the user, so such control would be defined as part of the Ship child class. Other behaviours will not be unique to child classes, so these can go into mixins.

There are various approaches to mixins in the wild, but I decided to roll my own. I wrote a function which builds and returns another function. The returned function applies a mixin. It looked like this:

function createMixin(descriptors) {
  return function mix(obj) {
    Object.defineProperties(obj, descriptors);
  };
}

The argument descriptors is an object with property descriptors. I use these rather than simple object fields since it allows me to create mixins which are extremely configurable.

Using this I can program the bulk of the behaviour of objects in the game. For example, since all objects can be assumed to have position and velocity, one such mixin could add a move method (which I suggested earlier in this post):

const movable = createMixin({
  move: {
    value(dt) {
      this.position.x += this.velocity.x * dt;
      this.position.y += this.velocity.y * dt;
    },
    configurable: true, // These three properties
    enumerable: false,  // are like how class
    writable: true      // methods are set.
  }
});

Consider the ship at the centre of the game. We could code it like this:

const ship = {
  position: { x: 50, y: 50 },
  velocity: { x: 1, y: 2 }
};

movable(ship); // Apply the mixin.

ship.move(2); // Now at x: 51, y: 52

I want the ship to be a class extending a base class though:

// Base class for all game objects.
class Entity {
  constructor({ position, velocity }) {
    this.position = { x: position.x, y: position.y };
    this.velocity = { x: velocity.x, y: velocity.y };
  }
}

class Ship extends Entity {
  // Ship specific behaviour in here.
}

// Give all Ship instances access to move.
movable(Ship.prototype);

const ship = new Ship({
  position: { x: 50, y: 50},
  velocity: { x: 1, y: 2 }
});

movable(ship); // Apply the mixin.

ship.move(2); // Now at x: 51, y: 52

All sorts of objects and classes may be composed with mixins.

I'm a professional JavaScript programmer, which means that each time I write something there's a little voice in my head asking me how hard it'll be to write tests for it. Applying the mixin to a simple object makes writing tests for the mixin in isolation possible. In mocha:

describe('movable', () => {
  it('appends a single method "move"', () => {
    const entity = {
      position: { x: 0, y: 0 },
      velocity: { x: 0, y: 0 }
    };

    movable(entity);

    assert.equal(typeof entity.move, 'function');
  });

  it('updates the position given a dt', () => {
    const entity = {
      position: { x: 5, y: 6 },
      velocity: { x: 1, y: 2 }
    };

    movable(entity);

    entity.move(3);

    assert.deepEqual(entity.position, { x: 8, y: 12 });
  });
});

I'd be more exhaustive in real world tests, but I hope you get the gist.

What about objects which have mixins applied to them though? How can we test them? Naïvely you could write the above tests for every class that uses the mixin. That's repetitive though. It would be nicer to have some mechanism to ask a mixin if an object has had the mixin applied to it. Then test for a class can use a single test for each mixin to check that the mixin has been applied, and all the repetition can be avoided.

Revisiting the mixin creating function, the first go at such a mechanism looks like:

function createMixin(descriptors) {
  // References to objects the mixin has been
  // applied to.
  const mixed = new WeakSet();

  function mix(obj) {
    Object.defineProperties(obj, descriptors);
    mixed.add(obj);
  };

  mix.isMixed = function (obj) {
    return mixed.has(obj);
  };

  return mix;
}

I've used a WeakSet here. A WeakSet contains weak references to objects. These are references which the garbage collector ignores. When no strong references to an object in a WeakSet remain, the garbage collector can clear the object. If I used a regular Set or an array here, the references contained would be strong, and references could never be cleaned up automatically by the garbage collector, resulting in a memory leak. This would be a problem in particular for asteroids, as each time one is destroyed the object itself would be held in the Set or array be a strong reference, leading it to grow as more and more asteroids are spawned.

Now we can ask the mixin if an object has had the mixin applied to it:

movable.isMixed(entity); // false

movable(entity);

movable.isMixed(entity); // true

What about classes though? This won't work because the mixin is applied to the prototype and no reference to an instance will be stored by the mixin. We can fix this by climbing up the prototype chain and checking each prototype object:

function createMixin(descriptors) {
  // References to objects the mixin has been
  // applied to.
  const mixed = new WeakSet();

  function mix(obj) {
    Object.defineProperties(obj, descriptors);
    mixed.add(obj);
  };

  mix.isMixed = function (obj) {
    // Walk up the prototype chain and check
    // if each step has had the mixin applied
    // to it.
    for (let o = obj; o; o = Object.getPrototypeOf(o)) {
      if (mixed.has(o)) {
        return true;
      }
    }

    return false;
  };

  return mix;
}

This works perfectly well now, and any child classes can inherit the method as expected and the isMixed check will still work. It is now possible to avoid duplication of tests. For example, part of a test suite for the ship class might look like:

it('is movable', () => {
  const ship = new Ship({
    position: { x: 0, y: 0 },
    velocity: { x: 0, y: 0 }
  });

  assert.ok(movable.isMixed(ship));
});

One final adjustment can be made. It's a shame that we can use instanceof for objects constructed by a class, but not objects which have had the mixin applied to them. This can be achieved by using Symbol.hasInstance. A first attempt:

function createMixin(descriptors) {
  const mixed = new WeakSet();

  function mix(obj) {
    Object.defineProperties(obj, descriptors);
    mixed.add(obj);
  };

  mix[Symbol.hasInstance] = function (obj) {
    for (let o = obj; o; o = Object.getPrototypeOf(o)) {
      if (mixed.has(o)) {
        return true;
      }
    }
  };

  return mix;
}

Unfortunately this doesn't work because Function.prototype[Symbol.hasInstance] is not writable. When appending a property to something, the field you're appending it as is checked for writability, and that check propagates up the prototype chain. Since mix is a function, it has a non-writable Symbol.hasInstance field in its prototype chain, and we need to work around that.

We can once again use Object.defineProperty to do it:

function createMixin(descriptors) {
  const mixed = new WeakSet();

  function mix(obj) {
    Object.defineProperties(obj, descriptors);
    mixed.add(obj);
  };

  Object.defineProperty(mix, Symbol.hasInstance, {
    value(obj) {
      for (let o = obj; o; o = Object.getPrototypeOf(o)) {
        if (mixed.has(o)) {
          return true;
        }
      }
    },
    configurable: false,
    enumerable: false,
    writable: false
  });

  return mix;
}

Finally we can do this:

class Ship extends Entity { /* ... */ }

movable(Ship.prototype);

const ship = new Ship({ /* ... */ });

ship instanceof Entity;  // true
ship instanceof Ship;    // true
ship instanceof movable; // true

So the test suite for a ship can include tests like this:

it('is movable', () => {
  const ship = new Ship({
    position: { x: 0, y: 0 },
    velocity: { x: 0, y: 0 }
  });

  assert.ok(ship instanceof movable);
});

So there you have it! All the test friendliness of classes with the composability of mixins.

I wrapped this code up in a module called mixomatic, which I intend to use heavily in my gaming codebases.

My first custom element

2017-11-16T01:45:00Z

After some years of browser vendors working out what web components should look like, they're almost ready for the prime time. The part which I find most intriguing (custom elements) has finally stabilised. With custom elements, you can make new HTML elements which have custom behaviour which you define using JavaScript. In this post I'll demonstrate a custom element for fuzzy counting.

Custom elements are created in two parts. Firstly we need to extend an element with a JavaScript class.

class FuzzyCount extends HTMLElement {

}

So far this describes no custom behaviour. All this is is the extended class, with identical behaviour to an HTMLElement. We can customize the constructor to add the behaviour we want. In order to give the element data to use, it must be passed in as an attribute. We'll use an attribute called count.

class FuzzyCount extends HTMLElement {
  constructor() {
    // The parent constructor must be called
    // before using `this`.
    super();

    // rawCount is a string.
    const rawCount = this.getAttribute('count');
    const count = parseInt(rawCount, 10);

    // Set text content based on the count.
    if (count === 0) {
      this.textContent = 'none';
    } else if (count === 1) {
      this.textContent = 'one';
    } else if (count === 2) {
      this.textContent = 'a couple';
    } else if (count < 5) {
      this.textContent = 'a few';
    } else if (count < 10) {
      this.textContent = 'several';
    } else {
      this.textContent = 'lots';
    }
  }
}

The element reads and parses the count attribute, giving itself a text content accordingly.

To make the element available to a page, it must be registered. One interesting restriction placed upon custom elements is that they must contain a hyphen so that they can be distinguished from built-in elements. We're going to register our element as fuzzy-count, but in the real world you should prefix it with a namespace. For example, if we're working at a place called Funky Corp, we could name the element funkycorp-fuzzy-count.

customElements.define('fuzzy-count', FuzzyCount);

Now we can use this to make elements on the page!

<fuzzy-count count="3"></fuzzy-count>

Any matching custom elements which exist before the element is defined will be upgraded. This means a page can be sent by the server with custom elements included, and everything will be rendered properly once the custom element is registered.

But what if we want to create elements in JS? We can try:

const fuzzyCount = document.createElement('fuzzy-count');

but an error will be thrown. It turns out that newly constructed elements cannot contain other stuff (the text content). Even if no error was thrown, we'd have an element with no count attribute, and so we've missed our chance since all the logic is in the constructor. It turns out that the constructor is the wrong place for this stuff.

Thankfully we can make a few changes to defer the setting of text content.

class FuzzyCount extends HTMLElement {
  // Called when the element is inserted into
  // the document or upgraded.
  connectedCallback() {
    this.setTextContent();
  }

  setTextContent() {
    const rawCount = this.getAttribute('count');

    if (!rawCount) {
      this.textContent = '';
      return;
    }

    const count = parseInt(rawCount, 10);

    if (count === 0) {
      this.textContent = 'none';
    } else if (count === 1) {
      this.textContent = 'one';
    } else if (count === 2) {
      this.textContent = 'a couple';
    } else if (count < 5) {
      this.textContent = 'a few';
    } else if (count < 10) {
      this.textContent = 'several';
    } else {
      this.textContent = 'lots';
    }
  }
}

The logic which sets the textContent has moved to its own method setTextContent, and connectedCallback calls it when the element is "connected" to avoid the error when constructing the element in JS. Connected is called when the element is inserted into the document, or the element is upgraded by custom element registration.

This is enough to get the behaviour we want, so long as the count attribute of a new fuzzy-count element is set before it is appended to the document.

const fuzzy = document.createElement('fuzzy-count');
fuzzy.setAttribute('count', 10);
document.body.appendChild(fuzzy);

We can take one more step to make the element react to changes to the count attribute...

class FuzzyCount extends HTMLElement {
  // Called for each watched attribute when
  // the element is added to the document or
  // upgraded. Also fired for a watched
  // attribute when it is added, updated, or
  // removed.
  attributeChangedCallback() {
    this.setTextContent();
  }

  // A static getter because we can't add a
  // static value within a class declaration
  // directly (yet).
  static get observedAttributes() {
    return ['count'];
  }

  setTextContent() {
    // Same as before.
  }
}

connectedCallback has been replaced with attributeChangedCallback, and a static getter defines the list of attributes to watch. For each watched attribute, when an element is inserted, upgraded, or the attribute is added, updated, or removed, this attributeChangedCallback is called. These changes allow us to update elements before appending as before, and also after appending.

const fuzzy = document.createElement('fuzzy-count');
document.body.appendChild(fuzzy);
fuzzy.setAttribute('count', 10);

Overall I'm impressed. I've only covered some of the API available here. It is a little fiddly, but upon consideration it all seems to make sense so far. It strikes me that a good way to use it would be for small components, or as a low level primitive for a front end framework.

Finally, a note on compatibility. At the time of writing Chrome and Safari support custom elements with no additional effort. Other browsers must be polyfilled for support.

Tip: Array.from

2017-12-01T20:00:00Z

SPOILER ALERT: If you're doing the advent of code this year, you may not want to read onward. This post does not give any solutions away, but does contain information about how I approached a part of the first challenge.

Onward!

Today I started doing the 2017 advent of code challenges (don't worry, I won't give solutions away). The first challenge involves summing numbers in a sequence. The sequence itself is too long to express as an integer, and that wouldn't be particularly useful for the task anyway. The most convenient form is probably an array of integers. The sequence is given as raw text, and the quickest way to get it into JS is as a string.

There are a few ways to do this which spring to mind. One might argue that you can loop directly over a string anyway, but getting a list of integers avoids parsing each character in the string more than once for these tasks, so let's ignore that option.

The first thing to come to mind was a loop:

const numbers = [];

for (let i = 0; i < sequence.length; i++) {
  numbers.push(parseInt(sequence[i], 10));
}

There's a slightly nicer to read version of this introduced by ES2015:

const numbers = [];

for (const character of sequence) {
  numbers.push(parseInt(character, 10));
}

Both suffer from being a bit bulky though. ES2015 also introduced Array.from and the spread operator (...), which offer a similar way of creating an array when used in combination with Array.prototype.map:

const numbers = [...sequence].map(char => parseInt(char, 10));
// or
const numbers = Array.from(sequence).map(char => parseInt(char, 10));

The first spreads characters from the sequence into an array, and maps these characters to a fresh array of numbers. The second uses Array.from to get an array of characters from a string, and uses the same mapping to get an array of integers.

I prefer the latter of the two (it's a little easier for humans to parse). It would be nice to avoid the explicit mapping though, right? The solution I settled upon was to use the little-known second parameter to Array.from. It turns out you can include a function to map elements to be inserted into the returned array:

const numbers = Array.from(sequence, char => parseInt(char, 10));

Which is my tip! In essence, don't do this:

[...stuff].map(func);

Or this:

Array.from(stuff).map(func);

Do this:

Array.from(stuff, func);

Advent of Code 2017 day 20 task 2

2018-01-02T20:50:00Z

SPOILER ALERT: If you're doing the 2017 Advent of Code, you may not want to read onward.

Over the holiday period I found a little time to work on the 2017 advent of code challenges. The later ones get quite involved, and I particularly enjoyed day 20. The second part of the task involves a 3D grid containing accelerating particles. Particles which collide at a grid point (occupy that point at the same time) are removed, and we're asked to determine how many particles remain after all collisions have occurred. This one was fun for me because the solution I came up with involved a little mathematics to avoid brute forcing the answer.

At this point in the challenges, writing a parser for an input file is something I'll skip over. The parser gives back an array of objects, each representing a particle, with $p$ , $v$ , and $a$ fields to represent position, velocity, and acceleration respectively. These quantities are all represented as length three arrays for the $x$ , $y$ , and $z$ axes. The input looks like:

[
  {
    "p": [-4897, 3080, 2133],
    "v": [-58, -15, -78],
    "a": [17, -7, 0]
  },
  {
    "p": [395, -997, 4914],
    "v": [-30, 66, -69],
    "a": [1,-2,-8]
  },
  {
    "p": [-334, -754, -567],
    "v": [-31, 15, -34],
    "a": [3,1,4]
  },
  // ...
]

The general algorithm I used was to calculate a collision times for each pair of particles, order them by time ascending, and then remove all pairs for each time step when both colliding particles are still in the set of all particles up to that time (earlier collisions can invalidate later collisions since collisions remove colliding particles).

To calculate potential collisions, I used the following:

const collisions = [];

for (let i = 0; i < input.length - 1; i++) {
    for (let j = i + 1; j < input[i].length; j++) {
        const t = collisionTime(input[i], input[j]);

        if (t !== null) {
            if (!collisions[t]) {
                collisions[t] = [];
            }

            collisions[t].push([i, j]);
        }
    }
}

I've used the index of each particle in the input array as its ID for convenience. The snippet grabs all pairs and uses the function collisionTime, which returns either an integer (time of their first collision from $t = 0$ ), or null if they never collide. The collisions array is indexed by time step, and populated with arrays of collisions. I'll come back to collisionTime later, because that's where interesting things happen.

Next I create a set containing the IDs of all particles.

const particles = new Set(Array.from({ length: input.length }, (v, i) => i));

See my Array.from tip for a partial explanation of this, and the MDN article on Set for the rest.

The rest is looping over collision times to discover which collisions are valid (both particles have not yet collided) and remove the particles which collide from the particles set. The size of the set after all collisions have been checked is the number of remaining particles, which is the goal of this problem.

for (const collisionsAtTime of collisions) {
    if (!collisionsAtTime) {
        continue;
    }

    const toDelete = new Set();

    for (const [a, b] of collisionsAtTime) {
        if (particles.has(a) && particles.has(b)) {
            toDelete.add(a);
            toDelete.add(b);
        }
    }

    for (const index of toDelete) {
        particles.delete(index);
    }
}

console.log(particles.size);

I don't care about the time of each set of collisions; I only care about the order. Since collisions is a sparse array (indexed by time step), I do have to check for undefined elements and continue to skip those.

I've modelled all collisions as pairs, so a collision between three or more particles will look like multiple collisions of pairs at the same time. In order to properly count these, particle IDs to be removed are held in the toDelete set and removed after all collisions at a given time step have been checked.

Let's go back to the function collisionTime. For a given pair of particle objects, it determines when their first collision is beginning at $t = 0$ . Here's a naïve implementation:

// Calculates the Manhatten distance between two particles.
function manhatten(a, b) {
    return Math.abs(a[0] - b[0]) + Math.abs(a[1] - b[1]) + Math.abs(a[2] - b[2]);
}

function vectorAdd(a, b) {
    return [a[0] + b[0], a[1] + b[1], a[2] + b[2]];
}

// Returns a new particle which represents the evolution of a given particle by
// one time step.
function tick(particle) {
    const a = particle.a.slice();
    const v = vectorAdd(particle.v, a);
    const p = vectorAdd(particle.p, v);

    return { p, v, a };
}

function collisionTime(a, b) {
    let nextD = manhatten(a.p, b.p);

    // Trivial case in which particles are initially in the same place.
    if (nextD === 0) {
        return 0;
    }

    let t = 0;
    let d;
    let v;
    let nextV = manhatten(a.v, b.v);
    let particle0 = a;
    let particle1 = b;

    do {
        d = nextD;
        v = nextV;
        particle0 = tick(particle0);
        particle1 = tick(particle1);
        t++;
        nextD = manhatten(particle0.p, particle1.p);

        if (nextD === 0) {
            return t;
        }

        nextV = manhatten(particle0.v, particle1.v);

        // Continue while the particles are moving together, or failing that,
        // the rate at which they're moving apart is slowing (indicating that
        // they will move toward each other in the future).
    } while (nextD < d || nextV < v);

    return null;
}

This implementation takes a pair of particles, and evolves both while either they are approaching each other, or accelerating toward each other. If the two particles are at zero distance during the evolution, they have collided. It works, but it's slow.

All I really need is the location of each particle at a given time. Since the particles move under a very simple set of rules, I should be able to predict where a particle will be at any time without needing to resort to loops. If I can predict the location of two particles, I can calculate the relative location of one particle with respect to another, and ultimately I can solve an equation for which that location is the same (zero distance).

First I have to predict the location of a particle though. We are told that at each time, the acceleration vector is added to the velocity vector, and then the updated velocity vector is added to the position vector. The position of a particle is thus (in one dimension, where zeros denote initial values):

p_t = p_0 + v_0 t + a\sum_{n=0}^t n

That sum can be swapped out for an expression (high schoolers will know this but I, with the full weight of two degrees in physics, had forgotten and had to look it up). This is equivalent:

p_t = p_0 + v_0 t + a\frac{(t + 1)t}{2}

Armed with this equation, I need to calculate the difference in position between two particles:

\Delta p_t = \Delta p_0 + \Delta v_0 t + \Delta a\frac{(t + 1)t}{2}

Where the delta signifies a difference (the above is the equation of one particle subtracted from the equation for a another particle). The solutions for $t$ I want are when the left hand side (difference in position at time step $t$ ) is zero. To avoid a division and retain integers as long as possible, I multiply both sides by two and rewrite slightly to collect powers of $t$ .

0 = \Delta a t^2 + \left(2\Delta v_0 + \Delta a\right) t + 2\Delta p_0

This is a quadratic equation! When the acceleration is zero this collapses down to a linear equation with the solution:

t = -\frac{\Delta p_0}{\Delta v_0}

When acceleration is non-zero, I have to solve a quadratic equation:

t = \frac{-b\pm\sqrt{b^2-4ac}}{2a}

where

\begin{align*} a &= \Delta a \\ b &= 2\Delta v_0 + \Delta a \\ c &= 2\Delta p_0 \end{align*}

I'm interested in solutions which are natural numbers (real positive integers). To get natural number solutions to quadratic equations, I wrote these functions:

function isNaturalNumber(num) {
    return num >>> 0 === num;
}

// Provides only solutions which are natural numbers.
function solveQuadratic(a, b, c) {
  if (a === 0) {
    const solution = -c / b;

    return isNaturalNumber(solution) ? [solution] : [];
  }

  const rootDiscriminant = Math.sqrt(b * b - 4 * a * c);

  return [
    (-b + rootDiscriminant) / (2 * a),
    (-b - rootDiscriminant) / (2 * a)
  ].filter(isNaturalNumber);
}

The function isNaturalNumber uses the right bit shift operator. This is a trick to get an integer out of a number. Positive numbers will be floored, negative numbers will overflow (become large positive integers), and NaN will become zero. Only positive integers will pass the strict equality check. solveQuadratic itself encodes the solution to both the linear case (acceleration of zero) and the quadratic case. JavaScript is on our side here with how it represents numbers.

This solution is in one dimension. For collisions in three dimensions, this equation must be solved for each, and only when all three dimensions have a solution which is the same can I say that the two particles are projected to collide. At last, here is the improved implementation of collisionTime:

function collisionTime(a, b) {
    let commonSolutions;

    // Calculate the solutions for each axis and initialise or whittle down
    // commonSolutions.
    for (let i = 0; i < 3; i++) {
        const dp = a.p[i] - b.p[i];
        const dv = a.v[i] - b.v[i];
        const da = a.a[i] - b.a[i];

        const axisSolutions = solveQuadratic(da, 2 * dv + da, 2 * dp);

        // Most of the time there are no solutions, so this is a good
        // optimisation.
        if (axisSolutions.length === 0) {
          return null;
        }

        if (!commonSolutions) {
            // Initialise the set with solutions for the first axis.
            commonSolutions = new Set(axisSolutions);
        } else {
            // Remove solutions for other axes which this axis does't have.
            for (const solution of commonSolutions) {
                if (!axisSolutions.includes(solution)) {
                    commonSolutions.delete(solution);
                }
            }
        }

        // If the set of common solutions is ever empty, there will be no
        // collision.
        if (commonSolutions.size === 0) {
            return null;
        }
    }

    // I only want the earliest collision.
    return Math.min(...commonSolutions);
}

This function first calculates solutions for the x-axis. Solutions which are not in common with both the y-axis and z-axis are subsequently removed. If the set of common solutions is ever empty, the function returns null. If any axis has no solutions I can take a shortcut and immediately return null. If the set has remaining entries, these represent times when the particles will occupy the same coordinates in all three dimensions (a collision). The lowest (earliest) is plucked out as the solution!

On my machine, the naïve implementation takes ~3.2 seconds with the input I'm given. With the improved implementation, it takes less than 0.1 seconds. The full solution can be found in this gist.

Essential tools for JavaScript beginners

2018-02-11T17:00:00Z

I’ve noticed when helping people to learn JS is that I’m happy to let them learn without any tools. In hindsight this is very strange. I wouldn’t dream of programming like this! I make mistakes all the time, and tools help me to catch them early. Tools also help me to streamline repetitive tasks.

In this post I'll talk a minimal set of tools common to all projects I use. This consists of Visual Studio Code, Node.js, ESLint, and Editorconfig files.

Visual Studio Code

There are many editors out there, and which one you use is a personal preference. Consider this section optional, but recommended.

VS Code is my current editor of choice. It integrates well with tools, feels uncluttered, and doesn’t lag. It's useful out of the box. Many editors can integrate with tools, but none quite as nicely. I recommend installing it and using its extensions system to install the ESLint extension and the Editorconfig extension (more on this in the sections below). The installation of ESLint into your project itself is covered in a later section.

Tip: Open the project directory in VS Code, not just a single file. It’ll present you with a file explorer on the left so you can quickly open other files in your project without leaving the editor. You’ll also be able to see files which are usually hidden.

Tip: You can use VS Code to view git diffs, compose git commits, and push and pull from a repository. If you prefer not to use git from the terminal, you may find VS Code good enough most of the time.

Tip: If you do want to use git (or npm etc.) from the terminal, you can open a terminal a panel in VS Code. From the view menu, select "Integrated Terminal".

Node.js

Node is a JavaScript environment without the browser. It is often used to write servers (most of my day job), but that’s not what we need it for here.

All modern JavaScript tools are built to be executed by Node and installable with npm, the JavaScript package manager, and executable with npx, the JavaScript package runner, which are both installed along with Node.

With Node installed, when you are starting a new project or want to add packages or tooling to an existing one, open a terminal (on a Mac or Linux machine, or on windows the Node.js prompt) and navigate to the new project directory (make a new directory if you need to). When you're in there, run

npm init

and answer the questions. You can use the defaults for most of the fields. This initialises your project with a package.json file. This file is very powerful because it can be used to describe your project and its dependencies (other libraries and tools it requires). If you're unsure, run

npm help init

for some helpful information.

ESLint

The most important tool (after the editor) I use is ESLint. It scans your code, and tries to tell you when you’ve made a mistake or done something you probably didn’t intend. ESLint can also be used to enforce style. Enforcing style might seem dictatorial, but consistent code is beautiful, and it helps a great deal when collaborating. Install with ESLint and a configuration with:

npm install —-save-dev eslint eslint-config-qubyte

I'm suggesting my own ESLint config here, but there are lots of them to be found on npm. You can create your own too!

When the command has finished, take a look in package.json. You’ll see a devDependencies section with the installed packages and version numbers. You’ll also see a new folder called node_modules. If you delete node_modules, you can run:

npm install

and npm will read package.json to know what to install. There may also be a package-lock.json. This lock file makes sure that exactly the same version of dependencies are installed each time. This is useful for sharing your project without having to include all the dependencies (which can easy amount to megabytes). If you're using git, add node_modules to your .gitignore file so that git ignores that directory.

The devDependencies section is for packages used in development, so it's the natural place for tools like ESLint. the --save-dev flag of npm install tells npm to install as a development dependency.

With ESLint and an ESLint config installed, ESLint it must be told to use the config. Create a file in your project directory called .eslintrc.json (the leading . is important) and put the following into it:

{
  "extends": "qubyte",
  "env": {
    "browser": true
  }
}

This minimal configuration tells ESLint that you're "extending" the config from eslint-config-qubyte. You can find out more about configuring ESLint here. If you're using newer JavaScript features, try setting extends to "qubyte/ES2017".

VSCode has an "extension" for ESLint. On the left of a VS code window there are several icons. Click on the one for extensions (hover to see) and search for "ESLint" to find it, and install. From the view menu, click on "Problems" to open a panel which shows you problems ESLint finds.

If you want to run ESLint from the terminal, navigate to your project directory and run

npx eslint .

Note the trailing dot, and that the command here is npx not npm.

Editorconfig

Editorconfig is not really a tool. It’s a standard for a file which describes how code should look in each code file. Make a file called .editorconfig (all lower case with a leading . as the first character in the name). Put the following text into it and save:

# Apply settings to all text file
# types.
[*]

# Windows likes to choose weird
# encodings sometimes. Use this to make
# sure that wherever the file is saved,
# it's in the expected encoding.
charset = utf-8

# Use unix line endings everywhere for
# similar reasons as the charset above.
end_of_line = lf

# Wise to end on a new line character
# for unix.
insert_final_newline = true

# Indent using two spaces at a time.
indent_style = space
indent_size = 2

# Trims redundant spaces from the end
# of lines.
trim_trailing_whitespace = true

Tip: If you prefer another editor, Editorconfig has broad support. It's always worth having this file in your projects.

VS Code has an extension for Editorconfig files. To install it, open "Extensions" via the icon on the left or via the view menu, and search for "Editorconfig". The most downloaded (probably the top hit) is the one you want. Once this is installed, there's nothing left to do. VSCode will honour Editorconfig files whenever it finds them. It does this by applying Editorconfig rules to a file when it is saved.

Finally

There are lots of tools out there. If you have a mentor they may have some suggestions! These are the essential tools I use as a career programmer though.

About this blog 3

2018-02-17T02:40:00Z

It's been a while since the last entry about how I've built this blog. Since it is constantly evolving, now seems as good a time as ever to write about some of the changes I've made.

Netlify

The biggest shift has been the move from a DigitalOcean droplet, with a linux install managed by me, to Netlify. The service they provides hooks into GitHub, and builds when new things are pushed. It also lets me try new things out on hard-to-guess subdomains by opening pull requests (useful for trying things out which might otherwise be risky, like updated security headers).

Webmentions

I've said before that I'm not a fan of comments. They're complex to manage and open to abuse. However, I do like the IndieWeb convention of using webmentions from site a to site b to notify b that a links to it. I use a Netlify deploy hook to call a glitch. This little service scans the updated blog for new links to other sites, and sends those sites webmentions. This is currently disabled and I'm monitoring the logs to make sure it doesn't do anything unintentional. It'll probably go active this weekend once I've enabled signature checking.

On the receiving side, my blog has a webmention form which Netlify provides a simple backend to. I can act upon each web mention from there. This form also allows me to include a manual webmention form at the bottom of each blog post. Other blogs can discover the form by looking for a link tag with rel="webmention", which is found in the head:

<link href="/webmention" rel="webmention">

Since I don't expect a large volume of mentions, I'll add each to the front matter of the blog post being mentioned. When a post has mentions, a section listing them is added at the end of the post.

Payments

A nice CLI utility called thanks was recently published. It scans your project for all the modules it uses and lists, in order of most relied upon, collectives and authors along with their payment links. At the time of writing, it contains a little database of module authors and collectives, and links to ways to pay them.

There's an obvious scaling problem which is addressed in this issue. The issue suggests a new field in the package.json file of a module which thanks can discover, shifting the burden onto the module authors.

Since I'm keen on IndieWeb stuff, the issue led me to wonder if there is an IndieWeb rel attribute for payments (much like that used for discovering webmention endpoints). It turns out that there is and it's rel="payment". It doesn't see too much use in the wild, and it should not be the primary means of discovering payment methods by thanks, but as a fallback it could be useful. thanks would check for payment links on the project home page (from the homepage property of the package.json file), followed by author URLs if provided.

I've added a link tag with this attribute to the blog as well. I don't expect it to get used, but one can always hope! Perhaps I'll become a professional blogger.

Better links

I write my posts in markdown, and use marked to process them. Marked doesn't know which anchors will link out to external domains, so it compiles then to <a> tags with only an href attribute. When linking to external sources, the current best practice is to add a rel="noopener" attribute. To do this, I've slightly customised the anchor renderer of marked to add these in:

const marked = require('marked');
const urlResolve = require('url').resolve;
const renderer = new marked.Renderer();
const oldLinkRenderer = renderer.link;

// In the actual code, this is fed in to a function, not a module-scoped const.
const baseUrl = 'https://qubyte.codes';

renderer.link = (href, title, text) => {
  const fullyQualified = urlResolve(baseUrl, href);
  const rendered = oldLinkRenderer.call(renderer, href, title, text);

  if (fullyQualified.startsWith(baseUrl)) {
    return rendered;
  }

  return rendered.replace('<a ', '<a target="_blank" rel="noopener" ');
};

marked.setOptions({
  // Some unrelated stuff skipped.
  renderer
});

It's a hack for sure, but it works!

Mathematics

Toward the end of last year, I wrote a post that required some typeset mathematics. I used to write a lot in LaTeX, and so I already knew of MathJax (and was saddened to find that MathML in the browser failed to catch on). I knew I needed to convince marked to use it, but marked currently lacks extensibility so I couldn't define new blocks. Instead I adjusted the behaviour of code blocks to treat blocks labelled as mathematics differently:

const mathjax = require('mathjax-node');
const xml2js = require('xml2js');
const marked = require('marked');
const highlight = require('highlight.js');

const renderer = new marked.Renderer();
const codeRenderer = renderer.code;

// This function is synchronous, so I couldn't call MathJax
// in here (it's async), and must do in the highlight method.
renderer.code = function (code, lang, escaped) {
  if (lang === 'mathematics') {
    return code;
  }

  return codeRenderer.call(this, code, lang, escaped);
};

function highlight(code, language, callback) {
  // Non-mathematics code is syntax highlighted.
  if (language !== 'mathematics') {
    return callback(null, highlight.highlight(language, code).value);
  }

  // Render with MathJax.
  mathjax.typeset({ math: code, format: 'TeX', svg: true }, ({ errors, svg }) => {
    if (errors) {
      return callback(errors);
    }

    // MathJax puts some unnecessary inline style in the output
    // which my server response headers disagree with. The below
    // strips those out, and adds a "mathematics" classname to
    // the resultant SVG for styling.
    xml2js.parseString(rendered, (err, xmlobj) => {
      if (err) {
        return callback(err);
      }

      delete xmlobj.svg.$.style;

      xmlobj.svg.$.class = 'mathematics';

      const builder = new xml2js.Builder({ headless: true });

      callback(null, builder.buildObject(xmlobj));
    });
  });
},

marked.setOptions({
  highlight,
  renderer
});

I'm very happy with how it turned out, but I do look forward to a more direct means to add functionality to marked.

Not a progressive web app

For a while this blog had a service worker and an app manifest. You could even "install" it on android phones. The caching strategy I was using was a bit wonky though. Any time hashes changed (common when I updated CSS, since each update to the CSS gets a new filename for caching reasons), a visit to any page would lead to the entire blog being downloaded. In the end I came to the conclusion that being a progressive web app was not good for readers bandwidth in this case and the only reason for me to have it was vanity.

A gutted service worker remains purely to clear the cache, so that a new script can be loaded which will uninstall the service worker for good.

Refactoring

Since the blog software (the static site generator) was built, async-await became a feature of JavaScript. The generator has been substantially revised to make use of promises and async-await, and is far more compact and readable as a result. The code comes in at a little under 300 source lines, not counting templates.

Putting back the service worker

2018-03-08T01:20:00Z

In the last post about this blog I wrote about why I removed the service worker which made this blog a progressive web application.

The way my blog handles CSS predates the wide availability of service workers. Since CSS link tags are blocking, it's good to give CSS a long cache time. In order to do this, but still deliver fresh CSS without readers having to wait for it, I give the CSS file a URI including a hash of its content. If the CSS is updated, its URI changes, as does the href of the CSS link tag in each page of this blog.

The server sends this header along with CSS it serves:

Cache-Control: max-age=315360000, public, immutable

The immutable tells the browser the file will never change while within the max-age (so I make it very long). Chrome doesn't support immutable, but does exhibit similar behaviour. The public allows proxies to similarly cache the response.

I instructed the browser not to cache HTML at all. Since the HTML was always fresh, updated CSS would be loaded at most once on each change. The server sends this header along with HTML:

Cache-Control: no-cache

Which forces it to make a fresh request for HTML each time. These headers are still in place now (even since the move to Netlify).

Unfortunately this didn't mesh well with the caching strategy of the service worker I added. The worker would download the entire blog, HTML and CSS, the first time a user navigated to it. The service worker was generated by sw-precache, which I had set up to be generated every time the blog was updated. It worked out what to add and remove from its cache based on file hashes. Since an update to CSS^† meant changing the address in a link header in every HTML page, it removed not only the CSS, but all the HTML any time the CSS changed. Worse, it proactively downloaded all the updated files.

The net effect was minor CSS changes triggering a mass download of my blog. This wasn't a good use of server or browser resources, so I removed the worker.

I recently attended a Homebrew Website Club held by Jeremy Keith. By chance he'd written a blog post on this issue the day before, in which he provides a minimum viable service worker.

This service worker caches everything lazily (no precache). When an HTML page is requested, it always tries the network first for a fresh copy, and falls back to the cache when necessary. For everything else it hits the cache first, but gets an update via the network in the background.

This resolves the CSS issue very nicely. Old CSS and HTML will be cached, so if your Southern Rail train is stuck in a tunnel, you can still read that blog entry about mixins you saw earlier and are now bored enough to take another look at. If you're stuck outside of a tunnel and I've updated the CSS^‡, the request for fresh HTML will succeed, which will also bring in and cache the fresh CSS! For things like images, which will probably never change, this also works well. If a change is urgent to any file, it can still be given a fresh URL to cache bust it (though I doubt this'll ever be necessary).

I'll still need to do some work though. As mentioned in that post, cleanup isn't addressed. I'm happy for HTML to be cached indefinitely, but for the CSS one way to clean it up might be to remove it once it is older than every HTML entry in the cache. For particularly large resources such as images, a relatively short cache time and good alt-text might be a good approach...

In the meantime, I've borrowed the service worker code from that post mostly unchanged (most changes are just to align it with the style enforced by my ESLint config). The one small addition is to allow server sent events (EventSource) connections. I use these in development to hot-reload my blog when changes are made. The original script ignores all but GET requests:

if (request.method !== 'GET') {
  return;
}

I've extended this to also ignore server sent event connections:

const acceptHeader = request.headers.get('Accept');

if (request.method !== 'GET' || acceptHeader.includes('text/event-stream')) {
  return;
}

You can see the current service worker code I'm using here.

^† I had to add some CSS to handle superscripts, such as the one used for this footnote.

^‡ Perhaps the blog will be a different shade of beige...

A brighter shade of beige

2018-04-02T22:00:00Z

When I originally built this blog, I gave it a very simple no nonsense theme. One colour (beige) for the background and black for text and the odd horizontal rule. After a couple of minor iterations I added a sticky navigation bar (in CSS, no JS).

To be frank though, it looked dated from the start. The rules I used were mostly borrowed from my days as an academic. It lacked playfulness, so I decided to revisit it. While the style is still a work in progress, I've published some changes, and this post is about those.

I took this as an opportunity to overhaul my CSS tooling along with the style. I switched from clean-css to PostCSS with import, cssnext, and cssnano plugins. These do the import inlining and minification I got from clean-css, but also allowed me to compile out some calc custom properties the style overhaul introduces. To make full use of CSS custom properties as variables, I turned off CSS compilation while in development mode.

For the palette, I opted to use HSL with CSS custom properties for the the individual parameters, and calc to apply offsets to these base value for derived colours. PostCSS compiles all this out for production, but during development they can be tweaked in the browser with no compilation. This approach was inspired by a recent post by Rob Weychert on colours for his blog, but I decided to stick with CSS rather than move to Sass.

The result doesn't appear to be a large departure from what was there before. I've brightened the background colour, added distinct colour to the navigation bar and made it span the width of the page, removed text justification, and lightened text to a blue complimentary to the background. The look as been described as Battenberg chic, which I rather like! If you'd like to compare, the previous style can be seen on the Wayback Machine.

Here are the custom properties at the time of writing:

:root {
  --base-background-hue: 45;
  --base-background-sat: 100%;
  --base-background-lum: 90%;

  --base-foreground-hue: calc(var(--base-background-hue) + 180);
  --base-foreground-sat: 100%;
  --base-foreground-lum: 30%;

  --background-color-main: hsl(
    var(--base-background-hue),
    var(--base-background-sat),
    var(--base-background-lum)
  );

  --background-color-alt: hsl(
    calc(var(--base-background-hue) - 30),
    var(--base-background-sat),
    var(--base-background-lum)
  );

  --standout-color-main: hsl(
    var(--base-foreground-hue),
    var(--base-foreground-sat),
    var(--base-foreground-lum)
  );
}

Where --background-color-main is the background colour of the body, --background-color-alt is the background colour of the navigation bar (and probably other things later), and --standout-color-main is the colour of the text.

I'm looking forward to CSS color module level 4, which will introduce color-mod to obtain one colour based on another (I found out about this via Charlotte Jackson's blog). For example, the background colours could be defined much more tersely as

:root {
  --background-color-main: hsl(45, 100%, 90%);
  --background-color-alt: color-mod(var(--background-color-main) hue(-30));
}

so there would be less need for separate values for hue, saturation, and lightness. I could use it now since a PostCSS plugin can compile it out, but then I lose the ability to adjust the theme in development without compilation. I prefer to avoid using syntax which isn't ratified yet anyway. I'm using PostCSS solely to make the CSS of my blog available as a single, minified, and backwards-compatible file.

Update on webmentions

2018-04-04T19:10:00Z

In a recent post I wrote that I had integrated webmentions, and some of that has since changed. Time for an update.

I was using Netlify form handling as an easy way to handle webmentions, but unfortunately they inject an additional input into forms. This input is required by Netlify to name the form, and POST requests which lack it get a 404 response. This in turn meant that the manual mention form at the bottom of each post worked, but regular webmentions (which only have source and target parameters) did not.

I replaced this functionality by pointing the form and webmention link to webmention.io, which was simple since I already have IndieAuth working with this blog. I may make a custom receiver for webmentions later, but this is an excellent stopgap.

On to more exciting stuff... I've been enjoying Mastodon lately. An issue was recently opened about having Mastodon dispatch webmentions for links in status updates. Unfortunately (but I think for good reasons), it didn't work out and the issue is now closed. I'm happy for my own status updates to dispatch webmentions though, so I put together a glitch to do this for me! Take a look, and remix if you like the idea.

Content-Security-Policy and service workers

2018-04-23T23:00:00Z

I was recently tripped over by a subtlety in how service worker fetch events and fetch works in conjunction with content security policy (CSP). This happened while adding an image to the about page. This post is the result of a conversation I had with Jake Archibald on twitter (with thanks for helping me to understand what was going on).

When I originally built this blog I set the content security policy (omitting policies which aren't pertinent):

Content-Security-Policy: default-src 'self'; img-src *;

This sets the policy for all requests to be limited to the same domain, except for images which may come from anywhere. I set it like this since images may come from a content delivery network (CDN). This means other domain names could be used, even for my own images.

Until now this blog has had no images at all, so no issues with this content security policy with respect to images were obvious.

When I added an image the browser refused to load it. Firefox wasn't much help here, but Chrome gave me a useful error message in the console (the URL is omitted for brevity):

Refused to connect to '<URL>' because it violates the following Content Security Policy directive: "default-src 'self'". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

This error was being thrown from a fetch performed by the service worker. With the worker bypassed, the image loaded as expected. The error above was trying to tell me that the request for the image within the worker was happening under a different security policy to that expected. Specifically, the worker is using the connect-src policy when performing the request for the image, and not the image-src policy I expected. connect-src is the policy used by scripts making requests. Since I don't define a connect-src policy, the fallback is default-src, which is limited to the domain of the site, and does not allow an image to be downloaded from a CDN.

There is a quick solution, which is to add a connect-src *; policy. By limiting this policy to the service worker, no other scripts will get to make requests to anywhere. The Netlify config for this looks like:

[[headers]]
  for = "/sw.js"
  [headers.values]
    Content-Security-Policy = "connect-src *;"

But what's actually going on here? I was confused because I had expected the fetch performed inside the worker to be subject to the image-src policy. I even checked that the initiator and destination of the request in the fetch event handler were for an image.

The fetch event and handler looks something like the following:

addEventListener('fetch', fetchEvent => {
  const responseFromFetch = fetch(fetchEvent.request);

  // Other stuff omitted...
});

I'm effectively proxying the request. I've omitted a bunch of stuff to do with caching.

When fetch is called with a request object, the URL of the request is used to make an entirely new request before processing it. This new request lacks information about the initiator and destination, and so it is subject to the connect-src policy.

This seemed bad to me at first. CSP being different with and without a service worker would be bad because you'd have to test both each time a new resource type is added.

Fortunately, it turns out that the browser also performs CSP checks on the request before the service worker receives the fetch event and on the response it receives from the service worker (important if the worker changes a URL, which I'm not doing). Restating for the example of an image, these three checks are made:

The initial request for is checked for image-src violation.
A request with the same URL is checked in the service worker for connect-src violation.
The response from the service worker is checked for image-src violation.

This means that I'm safe with the connect-src *; policy for the service worker mentioned above, since the browser was already applying the image-src policy to image requests before the service worker saw them!

A Battenberg in SVG

2018-06-10T18:14:20Z

In celebration of the Battenberg theme, here is an animated Battenberg! It's made of two SVG paths composed of lines and arcs. These are calculated using three angles and a bucket load of trigonometry (I'm not as good as I used to be at trig). A requestAnimationFrame loop updates these angles and the paths. Click on start to begin the animation. You can tweak the angular speeds using the three number inputs.

There are better ways to do this. I chose SVG to challenge myself!

Automatic announcement of new blog entries

2018-08-01T18:40:00Z

It occurred to me a couple of days ago that it'd be neat to build a glitch to announce new blog posts. Since I deploy this blog by pushing to a master branch on GitHub, creation of a blog post is somewhat less obvious than when publishing on a platform like wordpress or medium, so I needed to figure out another approach.

As part of the build process this blog generates a sitemap. All blog entries hang off of the /blog path, so it's not difficult to filter the sitemap down to only blog entires. By comparing a sitemap before and after deployment, it's possible to know when one or more entries have been added, and should be tweeted!

Glitch is a natural fit for this. It gives you a little persistent storage (a directory called .data), which can be used to stash the sitemap after each time it gets triggered.

In depth...

Step 1: Commits are pushed

As was the case before this enhancement, I push to deploy. This process starts with me creating or editing a post, committing it, and then pushing it to the master branch on GitHub.

GitHub then dispatches a notification to Netlify.

Step 2: Netlify receives a notification

Netlify is configured to build and deploy the blog each time it gets a notification from GitHub that the master branch has changed. It builds and deploys the blog.

The new part is that a notification is configured to send a POST to glitch when a deploy succeeds.

Step 3: A glitch app receives a notification

I built this glitch app to receive and verify POSTs from Netlify. Netlify uses a JSON web token, and validation is done by shared secret. When the request is validated, its context (a field on the JSON body of the request) is checked. This is so that only the production deploy is acted upon, and not branch deploys.

For valid POSTs with a production context, the app makes a request for the sitemap of this blog, and loads the previous sitemap from its .data directory. Both are filtered down to only blog posts, and compared for new entries. The new sitemap is saved in place of the old one.

Finally, each new blog post is formatted as a tweet, and posted to twitter!

Lessons

While it's pretty cool to automate things using webhooks (glitch in particular shines for this use-case), the thing which really stood out was how the sitemap made it all fairly straight forward. Originally I added the sitemap almost as an afterthought, since it was cheap to do with a template and is good for search engines. I'm glad I made it a newline separated list rather than an XML monstrosity because it's so easy to parse this way.

Routes of RESTful services with nested resources

2018-10-19T19:00:30Z

Nested resources are common in real life, and when you're a programmer working on RESTful APIs you may have noticed that there are often trade-offs when it comes to formatting the routes of nested resources.

Let's say we're building an API for carpenters. Their guild wants a way to help the carpenters manage their clients, and the items which they're creating for their clients (jobs). A set of routes for this might look like

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /clients
POST /clients
GET  /clients/:clientId

GET  /jobs
POST /jobs
GET  /jobs/:jobId

where segments preceded by a colon are placeholders for a UUID. The first route of each group gets a collection, the second creates a new member, and the third gets a member by its ID. There may well be other operations like the third which act on individual members of a collection, such as PUT for replacing and DELETE for removal.

The problem with this layout is that, as a carpenter, you're rarely going to want to see clients which aren't your clients. There may also be lots of jobs, and making a request for a list of jobs could select a huge number of them. Usually you'll want to filter it by the client. We can use query parameters to supply additional context, but this seems a bit clumsy. Let's have another go

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /carpenters/:carpenterId/clients
POST /carpenters/:carpenterId/clients
GET  /carpenters/:carpenterId/clients/:clientId

GET  /carpenters/:carpenterId/clients/:clientId/jobs
POST /carpenters/:carpenterId/clients/:clientId/jobs
GET  /carpenters/:carpenterId/clients/:clientId/jobs/:jobId

It's pretty clear now that these are nested resources. It's no longer possible to select a collection without refining it by its parent resources.

The problem now is that it's a bit annoying to have to use all the parent IDs of a list or a member. I recently realised (and I'm probably late to the party) that a good solution is a compromise between the two

GET  /carpenters
POST /carpenters
GET  /carpenters/:carpenterId

GET  /carpenters/:carpenterId/clients
POST /carpenters/:carpenterId/clients
GET  /clients/:clientId

GET  /clients/:clientId/jobs
POST /clients/:clientId/jobs
GET  /jobs/:jobId

In this arrangement, we need to use a route with a parent ID when listing a resource or creating a new member. When we already have an ID of a resource we don't need to include its parent in the path. This includes other singular operations such as PUT and DELETE.

Finally, a word on nesting. The implicit assertion made above is that a child resource has a single parent, which may not be the case. Think of the solution above as a filtering mechanism. i.e. a client may have contracted more than one carpenter. You may also want to list jobs for a carpenter, rather than for a client. If you find yourself writing too many routes, using query parameters may be the lesser evil. Be pragmatic!

Parsing input from stdin to structures in Rust

2019-01-01T23:30:00Z

I've been working on this post for a while, but about a month ago my firstborn arrived, and he's been getting the lion's share of my attention. I start work again tomorrow, so I decided to just publish this post and be done with it. Apologies if it's a little rough around the edges!

In a post a while back I described in depth my solution to an Advent of Code challenge. This year I took part again, but rather than using Node.js to write programs to solve challenges I decided to use Rust.

The input for these challenges fell into one of three broad categories; input short enough to provide as an argument to the solver program, longer input as a single line in a file, and input over many lines in a file. In the last case, each line usually represents an individual structure of some kind in a collection. In all cases, I chose to keep the input separate. I'll be focussing on how I approached the last case for this article.

Rust has excellent IO support, but parsing of strings into structures is more difficult than for a language like JavaScript. As a meta-challenge, I found parsing input interesting, and I learnt a lot about iterators, the Option and Result types, and type casting in the process. I'll use the challenge of day 10 for illustrative purposes.

To be unix-y, I decided to read input from stdin rather than reading from a file.

cargo run < input.txt

On day 10 the data looked like

position=<-10427, -42253> velocity=< 1,  4>
position=< 21343,  42515> velocity=<-2, -4>
position=<-10417, -52846> velocity=< 1,  5>

Each line represents what I will call a particle, including its initial position and velocity.

Rust ships with some nice functionality for working with input from stdin. The main function for my solution to Day 10 looks like

use std::io::{stdin, BufRead};

// ...

fn main() {
  let mut particles: Vec<Particle> = stdin()
    .lock()
    .lines()
    .filter_map(|line_result| line_result.ok()) // #1
    .filter_map(|line| line.parse().ok())       // #2
    .collect();                                 // #3

  // use particles
}

I like the readability of this chain. stdin().lock().lines() returns an iterator which yields lines, each of which is a Result containing either a string or an error. The rest of this chain filters out errors (#1), parses line strings into a custom Particle type (#2), and finally collects elements into a vector of particles (#3)`.

Steps #1 and #2 both make use of filter_map. This is a method of iterators which takes a closure. The closure takes an element and returns an Option. When the option is None the element is filtered out. When it returns Some(value) the element is mapped to value in the ongoing stream. Since elements of the line iterator are Results, I call ok on them, which returns an Option type which discards an Err(error) for a None. filter_map then unwraps the value for the next step in the chain, or filters out the None (so line errors are effectively ignored).

Early on in my journey into Rust I found Results and Options quite annoying. I wrote a lot of ifs and unwraps to force values out of them. After I while I came across things like filter_map, if let, and match for working with them more fluently, I realised that they're really neat.

The next step (#2) is to parse the line string into a Particle structure wrapped in a result, and again discard errors and unwrap. Finally (#3) the results are collected into a vector for easy use in the rest of the program.

You might have noticed that I've skipped a big chunk of what's going on in #2. How do I parse lines into a structure which I've defined? By implementing std::str::FromStr for the Particle structure, parse gains the ability to parse to Particle. In the chain above, the type of particles is Vec<Particle>, which is enough for Rust to infer that the call to parse means to parse to Particle. This type annotation is also used by collect to determine that we want Particles to be collected into a vector.

Here's the Particle definition:

struct Particle {
  position_x: isize,
  position_y: isize,
  velocity_x: isize,
  velocity_y: isize
}

There's not a lot to see here. It's really just a wrapper for x and y positions and velocities as read from the input. I assumed that isize will be sufficient to contain any values read (though this probably isn't the correct integer type to use).

It's possible for a string to not parse to a Particle. For this, a new type of error needs to be defined. I'm just going to put it below here since there's a lot I don't understand about defining errors in Rust yet.

use std::{
  fmt::{self, Display, Formatter},
  error::Error,
  num::ParseIntError
};

#[derive(Debug, Clone)]
struct ParseParticleError;

impl Display for ParseParticleError {
  fn fmt(&self, f: &mut Formatter) -> fmt::Result {
    write!(f, "Unable to parse to a Particle.")
  }
}

impl Error for ParseParticleError {
  fn description(&self) -> &str {
    "Unable to parse to a Particle."
  }

  fn cause(&self) -> Option<&Error> {
    None
  }
}

impl From<ParseIntError> for ParseParticleError {
  fn from(_error: ParseIntError) -> Self {
    ParseParticleError
  }
}

The one part I will dwell on is the final implementation. It allows ParseIntError errors to be cast to ParseParticleError. This becomes handy in the FromStr implementation:

use regex::Regex;
use lazy_static::lazy_static;
use std::str::FromStr;

const PARTICLE_REGEX: &str = r"<\s*(-?\d+),\s*(-?\d+)>.*<\s*(-?\d+),\s*(-?\d)>";

impl FromStr for Particle {
  type Err = ParseParticleError;

  fn from_str(particle_str: &str) -> Result<Self, Self::Err> {
    lazy_static! {
      static ref regex: Regex = Regex::new(PARTICLE_REGEX).unwrap();
    }

    regex.captures(particle_str)
      .ok_or(ParseParticleError)
      .and_then(|cap| Ok(Particle {
        position_x: cap[1].parse()?,
        position_y: cap[2].parse()?,
        velocity_x: cap[3].parse()?,
        velocity_y: cap[4].parse()?
      }))
  }
}

While Rust doesn't come with a library for working with regular expressions, there is the excellent regex crate on crates.io. I used this and lazy_static to match and extract numbers (as strings) from each line of the input. This is a quick-and-dirty solution to parsing input. Rust has other crates for parsing input which may be a better fit for other purposes. I found their APIs difficult to comprehend and laborious though and given that the source of the input here is low risk and the regular expression unsophisticated I decided it was a reasonable course.

The creation of the regular expression is wrapped in lazy_static, which means that this only occurs once, and subsequent calls to from_str will reuse it. The documentation for the regex library suggests the use of lazy_static, so there was no ingenuity on my part.

The regular expression matches lines which look like lines of the above input sample, capturing the numbers (including a leading - if there is one).

Onto the chain! The regular expression is applied to particle_str. The call to captures returns an option. The ok_or step in the chain turns a None into a ParseParticleError. When the regular expression finds no match in a string then it cannot be parsed to a particle!

The next step is an and_then. This unwraps Some(value) to value and passes the value to its closure. Errors are waved past by this step. The return value of the closure is a Result. There are four integers to parse. The regular expression already filters out most issues, but an integer might still be too large to be represented as an isize. The ? operator unwraps result so that a Particle structure can be composed. If any isize parse results in an error, the error is immediately returned (and cast to a ParseParticleError). Since the return value of this closure must be a Result, the particle is wrapped in an Ok.

All of this effort to implement string parsing for Particle might seem redundant given that errors are filtered out by the main function anyway, but I enjoyed exploring this functionality of Rust. If you're interested in the full code of my solution to day 10, you can find it here.

Update 2019-04-22

I was told today about an interesting crate which automates a lot of the parsing work in my code through use of a macro. The recap crate neatly avoids the need to manually apply a regular expression and extract elements from it, parsing each individually.

Adding parsing of lines to particle structures now looks like:

use recap::Recap;
use serde::Deserialize;

#[derive(Debug, Clone, Deserialize, Recap)]
#[recap(regex = r"<\s*(?P<position_x>-?\d+),\s*(?P<position_y>-?\d+)>.*<\s*(?P<velocity_x>-?\d+),\s*(?P<velocity_y>-?\d)>")]
struct Particle {
    position_x: isize,
    position_y: isize,
    velocity_x: isize,
    velocity_y: isize
}

That's it! Errors produced (or lack thereof) might be quite different, I've not checked it out in detail yet. For the purposes of the task though, it works well. Named capture groups isolate elements by name, and the corresponding type of the field in the particle struct is used to parse each. A gist containing the complete updated solution can be found here.

My new year resolution

2019-01-16T23:27:12Z

My resolution this year was to work on my Japanese speaking ability.

I've already failed.

The keys to making a good resolution are to make it measurable, and incremental. To achieve it you need to define what "progress" and "done" mean for it. A vague promise to get better at something does neither.

So, how do I rescue it?

It's never too late! I'm only 16 days into the year. Lots of time to achieve my goal, as long as I can define it! I also have to be realistic about my spare time. My son was born at the end of last year, and while he's the biggest reason for me to improve my Japanese (we want him to be bilingual), he's also a time sink (in a good way!)

First I must ask myself what's been holding me back. I've been learning Japanese on and off since around 2001, and I still can't hold a conversation. My listening ability by far outstrips my ability to speak, and that imbalance is revealing. I think the problem stems from three things; perfectionism, laziness, and role-play.

Perfectionism is easy to explain. I'm a lapsed physicist and a programmer. Both fields emphasise precision. I dislike making mistakes, even though I know that making them is the faster path to learning.

Laziness is also easy. As a programmer, I automate things. With a little code, I can optimise away monotonous or repetitive tasks. A language is a bulky thing though. I can't write some code to learn it for me. Not yet anyway.

Role-play is more difficult to explain. A language isn't just words. It has a personality, and you have to act it as much as you speak it. I'm bad at role-play. It terrifies me.

These three issues are holding me back, and I don't think there's a solution to that. I'm going to have to push through. I think that defining what progress looks like for learning to speak more Japanese will help with these issues, and also rescue my resolution.

So, how should I define progress? What does measurable look like for the ability to speak a language? I'd love to know your thoughts!

A recent contribution I made for Node.js

2019-02-27T23:05:00Z

I've made a couple of small contributions to Node.js in the past. These were quite esoteric, and unlikely to be discovered or noticed by most developers. Recently I made a contribution which might be noticed though.

Most Node developers are familiar with Express. When responding to a client request in Express, chaining makes setting the status and sending a body fluent. It is possible to chain the call to send after the call to status because status returns res.

// Express version.
function requestHandler(req, res) {
  res.status(200).send('Hello, world!');
}

The vanilla Node analogue is a little more verbose because writeHead had no explicit return (which means it returned undefined).

// Vanilla Node.
function requestHandler(req, res) {
  res.writeHead(200);
  res.end('Hello, world!');
}

I find the repetition a little annoying, and frequently tried to chain the call to end onto the call to writeHead by mistake.

My contribution was to make writeHead return the response object res, to enable method chaining.

// Node 11.10.0+
function requestHandler(req, res) {
  res.writeHead(200).end('Hello, world!');
}

This makes the vanilla Node API for server responses just a little more friendly.

Weeknotes #1

2019-03-03T14:00:00Z

This is my first go at writing a weeknotes entry, and this week has been a busy one!

On Monday the Brighton Rust User Group held it's first meeting since November (it's a small group and two of us had babies arrive a day apart), and the first held in the evening. We were shown some advanced code for handling game assets, and some hardware which can run embedded Rust. I'm looking forward to seeing more!

On the topic of Rust, I discovered this week that glitch can deploy Rust! If I find the time, I'll reimplement one or two of my personal bots (which are hosted by glitch in Node.js) in Rust.

On Thursday Codebar Brighton held their first edition of Codebar Talks. Jeremy Keith gave a presentation on metaphores we use as programmers both for ourselves and the tools we use, and the implications of them (I'm keen on this topic). Kate Beard gave a presentation Morse Code and the Web Audio API. I love this approach to using browser technologies such as Web Audio API to reproduce old tech like Morse code or the ZX Spectrum loading screen.

In non-web news, I've been thinking about books and records. I have lots of books. I'm thinking of donating the fiction ones, but some of the text books I'd like to keep around and have handy when the whim to pick one up takes me. There's a perfect little space in my home office to place a small book case, but I don't want the usual sort of bookcase, I want one with angled shelves so you can actually see what's on them while still being a short piece of furniture.

There's a mid-century modern design that caught my eye, but they're rare in the UK (they appear to have been more popular in the US). Since I can't find one, I figured I'd come up with a flat-pack version which slots together made of birch plywood, and outsource the machining. If I decide to do it I'll post pictures here!

Weeknotes #2

2019-03-17T23:45:11Z

This edition really covers the last two weeks.

The highlight of last week was Bytes Conf. It was held on the Thursday evening, with four excellent talks. I was also one of the raffle winners! There were lots of familiar faces attending, so I took the opportunity to catch up with folk. These last few months I’ve not been doing much extracurricular stuff because of the little one.

I redeemed my winning raffle ticket for £50 off a print from The Private Press. I put it towards a beautiful print of the Welbeck Street Car Park (a soon to be demolished brutalist multi-storey car park) by Paul Catherall. I can’t wait to get it framed.

The final two talks of the evening were particularly interesting to me. They were both themed on limitation promoting creativity. The first talk was from a technical perspective, with a focus on CSS and working within the confines of advertisements and of daily challenges. I’ve often found that imposing limitations on myself leads to my most creative work. The latter talk took a historical angle, and covered everything from French protest posters to the Futura typeface. I’ll link the talks once the videos have been uploaded.

Last week I began to design the bookshelves I mentioned in Weeknotes #1. When I was in secondary school I was pretty handy with CAD software, so I decided to try to make the schematic in FreeCAD. Unfortunately it seems my understanding has entirely elapsed. While I could spend some time learning how to use it, for a single piece of furniture it seemed like overkill. I decided to use SVG instead (via Boxy SVG with some manual tweaks). I spoke to a local wood store and they stock sheets of birch plywood and can machine it to shape for me. This week I decided to make two versions of the shelves. One will be for books and the other for CDs and LPs. I’ll take the designs to them next week to see what they think.

Finally, we booked tickets to Japan this weekend. It’ll be the little one’s first time to meet his grandparents, and I can’t wait!

Weeknotes #3

2019-03-24T23:10:00Z

This week started off a little boring. On the Thursday though, the rebooted Brighton Homebrew Website Club met at Clearleft. After catching up with the folk there, I worked on an experiment to theme this blog, with the theme determined by a toggle in localStorage. By placing an inline script in the head right after the stylesheet link tag, I demonstrated that it's possible to select a theme before a paint, avoiding an awkward flash of the wrong theme. This may not news to anyone but me. I've not implemented anything just yet.

On the topic of themes, I wanted to do this because I want to add a dark theme. To celebrate it (when it’s finished), I will make a dark mode Battenberg cake. At the moment the plan is to make it out of chocolate and red velvet cakes for the pattern, and cover it in chocolate for the outside. I've baked a red velvet cake in a tray, but it came out as a regular chocolate cake so I'm going to use more red colouring for the next try. I'll post a picture if I ever actually make it.

I've noticed folks I know are starting up, or returning to, their personal sites. Just like this blog, the majority of them are being reborn as static sites deployed by Netlify. It's good to know that this solution appeals to more folk than just me. It's even better to see more unique stuff on the web, away from the silos owned by giants like Facebook.

On Sunday evening I enhanced my new blog post tweet bot to include tags. Each post in this blog includes tags, and each of these is a link to a tag page with a rel="tag" attribute. I updated the bot to download the new entry and select rel="tag" elements for their text content. The selected text is then appended to the tweet as hash-tags.

On the home life side of things, the little one had his third set of vaccinations on Friday. A cold delayed the vaccines from the week before (which he weathered far better than my partner and I). The third set of vaccines like the first set. They're painful, and can lead to a fever. We've spent most of this weekend trying to keep him comfortable, the fever low, and the injection sites as pain free as possible. He's almost back to his usual cheerful self now. It should go without saying that these vaccinations are well worth a short period of discomfort.

The code for this blog is now public

2019-03-28T19:26:08Z

I made the source code for this site public! You can find it here. I've written at length about how I've built this, and having the code makes it easier to point to particular lines. I hope it'll inspire you to do the same!

How I schedule posts using GitHub Actions

2019-06-15T22:00:00Z

In the past I used atd to schedule the publication of my blog posts. When I moved to Netlify I lost the ability to schedule posts, and didn't think about it until a recent conversation on twitter with Remy Sharp. Remy asked how to schedule blog posts for static sites and it got me thinking.

I've been using GitHub actions recently, and one workflow trigger it provides is a cron-like schedule. I wrote a little npm script to read a directory full of scheduled posts. Each post has its publication date checked, and if it's in the past, the script moves the post to the published posts directory (at which point Netlify kicks in). The script takes advantage of the token provided by the actions environment to make two requests, one to recreate the file in the posts directory, and one to delete the scheduled file using the GitHub Contents API.

Sadly, there's no way to move a file using the Contents API, so the create-then-delete is necessary. This leads to two commits rather than just one. It may be possible to do it in a single commit using the Trees API, but that's a good deal more involved and I don't mind a little noise in the commit history.

~~You can see the workflow using this script here, and the publisher here~~. These scripts have been replaced since GitHub actions have changed! See the later post for updated specifics.

Of course, the scheduler published this post. ;)

A new service to handle webmention dispatch for you

2019-06-18T23:45:00Z

I really like webmentions. They provide a way to let folk know that you're writing about their blog posts. I see them as an alternative to comments which encourages better discourse.

Webmentions can be a pain to manage though, especially if you have a statically generated site like this. For receiving mentions there's the venerable webmention.io service. I use this to collect mentions and then I check periodically and copy mentions over to the metadata of the targeted post.

To dispatch webmentions I made a glitch, which I've written about before. It's closely coupled to the URL structure of my blog and how it compiles webmentions into pages, so I'll stick with it but it's not so useful for others except as a reference.

If you've just decided to include webmentions on your blog, then Remy Sharp created webmention.app to automate searching for links and dispatching webmentions for a page. If I were starting now I'd use it! You can read more about it in Remy's own words.

Cleaner scheduled posts publication

2019-06-27T17:00:00Z

In a previous post I talked about an npm script I had written to be executed by a GitHub action:

Sadly, there's no way to move a file using the Contents API, so the create-then-delete is necessary. This leads to two commits rather than just one. It may be possible to do it in a single commit using the Trees API, but that's a good deal more involved and I don't mind a little noise in the commit history.

I was fibbing! The additional commit does bother me, so I decided to rewrite the publisher to avoid noise in the git history.

My next move was to actually try using the Trees API. It was mostly working before I read Remy Sharp's post on the same topic. Remy uses the metadata of a post to determine if it should be published or not, rather than the directory it resides in.

Using the same idea, the only thing which differentiates a published post and a scheduled post now is that the former has a timestamp in the past. By putting both kinds of post in the posts directory and tweaking the static site builder to filter out future posts, the generator takes care of what to render.

The scheduler now does its thing just by checking the published sitemap against what it expects to see published. When the sitemap is outdated, the scheduler sends a build hook request to Netlify to refresh the rendered content. This means that the scheduler entirely avoids interacting with the GitHub API! No commits necessary to publish. No trees to map through.

Update 2019-08-11

GitHub actions have been revised and are now written in YAML. The workflow which handles publications has been updated accordingly, and the publisher script moved to a scripts folder to help keep the project directory clean.

Updating webmention dispatch

2019-07-04T18:00:00Z

I mentioned a new service which handles webmentions in a previous post. I decided to replace the glitch I've been using for one which is much leaner. It uses the library which powers webmention.app to handle webmention (and also older technologies like pingback) endpoint discovery and mention dispatch so I took this opportunity to ditch my own discovery code.

I also decided to leave out some features. The earlier glitch would check old posts for new mentions, filter out certain URLs like rendered webmentions from others and so on. This all required a database and meant that posts got checked more than once, which took a long time (beyond the timeout of a glitch run.

The new glitch keeps a list of posts it's already sent mentions for. This means that each post is only scanned once after publication. That avoids the issue of filtering out certain URLs (the rendered webmentions of others for example) since a newly published post won't have any mentions yet. It's also a little more portable. If you like the idea of using a sitemap.txt to drive a record of which posts have been handled, then take a look!

P.S. All the glitches I use to add functionality to my blog can be found here.

Dark mode

2019-10-12T01:30:00Z

That's right! After more than a year of talking about adding a dark mode I finally did it. The wider support for prefers-color-scheme is what pushed me over the edge. I'm also a slave to fashion.

Initially I wanted to create a sort of dark-mode-battenberg theme, but after a while using dark mode site I came to realise that my favourites are the ones which use as much black as possible. It's becoming more common to see OLED screens in the wild, and these use less energy for black pixels since OLED pixels are self illuminating. I settled on using black for the background, and borrowing my light mode background colour for text and borders.

The additions to the CSS aren't too dramatic, but I learnt a few things along the way

@media (prefers-color-scheme: dark) {
  :root {
    --background-color-main: #000;
    --background-color-alt: #000;
    --standout-color-main: hsl(
      var(--base-background-hue),
      var(--base-background-sat),
      var(--base-background-lum)
    );
    --standout-color-alt: hsl(
      calc(var(--base-background-hue) - 30),
      var(--base-background-sat),
      var(--base-background-lum)
    );
  }

  img:not([src*=".svg"]) {
    filter: brightness(.8) contrast(1.2);
  }

  pre, input, button {
    background-color: #000;
  }

  code {
    filter: invert();
  }
}

body > header {
  z-index: 1; /* works around stacking context issue introduced by filters */
}

The media query applies the content only when the OS is in dark mode and the browser supports the media query. The :root swaps the background colors to standout colours, and makes the background black.

Non-SVG images have their brightness and contrast adjusted with a filter. I borrowed a dark mode filter from Melanie Richards as suggested by Jeremy Keith.

Code listings are more difficult. I plan to do a bit more on these later, but for now, since the regular code is on a white background, I give dark mode code containers a black background and apply an invert filter to the code. I also set the background color of inputs and buttons to black to fit.

Finally, the filters had an unintended side effect. Filters place the elements they apply to above positioned elements in the stacking context, which led to images and code listings scrolling above the nav bar (which is part of a sticky positioned header). To address this issue I resorted to using z-index. While the rule of thumb seems to be to avoid use of z-index, I believe it's appropriate in this case.

IndieWebCamp Brighton 2019

2019-10-21T16:20:00Z

I had a great time last weekend at IndieWebCamp (IWC) Brighton. The first day was filled with discussions on various IndieWeb related topics. I attended discussions on:

How IndieWebCamp should respond to the climate crisis.
Storage and display of personal tracking data.
Local first web and how to define the term.
License detection.
Integrating Apple shortcuts (scripts) into your IndieWeb workflows.

Outcomes were interesting for some of these. For the first there was discussion around downplaying more carbon intensive modes of transport, and attending IWC before or after another event to make journeys more worthwhile (this is already often the case). Another idea was to estimate the greenhouse gas output of travel to and from IWC for all attendees, but there was disagreement over how to measure this when attending other events too.

Storage and display of tracking data was interesting, but perhaps less relevant to me. I'm not so interested in tracking where I go, my weight, and so forth, though that might change in the future. The discussion eventually focussed on how data should be summarized.

The local first discussion was one I was note taking for. I'm using my old MacBook Air since I just finished a job and haven't started my new one yet, so I'm between work laptops. The MBA is nearly 10 years old and still going strong, but it has a Japanese keyboard layout, and after being on a British layout for the last few years my ability to touch type is a little hampered. I mostly managed to keep up but missed the discussion since my focus was on taking notes. Where local first seems to shine is in content creation. For example, I could write a post or a note on a train with bad mobile coverage, and my device would stash the content and POST it whenever it next connects. Jeremy Keith mentioned background sync as an emerging option to do this (I must look into this).

License detection was interesting. There was some discussion over whether a microformat would be the right option, or a rel-bookmark style rel, since in that unusual case the rel isn't singular in a page. The latter has a potentially confusing resolution algorithm though. The issue of actually parsing licenses was avoided. We were mostly concerned with how to use multiple licenses within a page.

The final session on the day was about integrating micropub with Apple shortcuts. I found this extremely interesting. I'd considered it in passing, but Rosemary Orchard had a comprehensive set of actions to automate IndieAuth and posting to micropub endpoints. This would be the inspiration for my hack day work.

The second day was for hacking on your own stuff, and helping each other out when possible. My Node.js knowledge came in handy a few times so I was glad to be helpful!

I decided to update this blog to handle likes and replies. At the time of writing you can see these as their own sections in the navigation bar. I am considering collecting everything into a single filterable stream though, so these may eventually go away. Likes and replies are created by my micropub endpoint, so the first part of this task was to update that glitch to understand and handle payloads of those kinds. As with notes and links/bookmarks, it uses the GitHub contents API to create JSON files which are compiled to pages of my site. My static site generator gained a couple more templates to render these new types of content.

This made it possible to post likes and replies using existing micropub clients. I use Omnibear as a Firefox extension in my laptop browser. This client uses the current page to fill in details to make creating likes and replies much simpler. On mobile, I decided that a new set of shortcuts to be used from the sharing panel in iOS/iPadOS. Like Omnibear, these benefit from the context they're used in. What I implemented is fairly primitive. The shortcuts only use the URL of the current page. There's no elegant way to get hold of the title of a page like Omnibear can, but I'll work on this later as an enhancement.

I implemented shortcuts for liking, replying, and bookmarking. The latter two prompt me for additional text for my part of the reply or bookmark. I decided not to make these shortcuts use IndieAuth. Instead I updated my glitch to accept authentication using a shared secret when a special short-circuit-auth header is truthy. This means that existing clients continue to use IndieAuth, while my own software can take a shortcut if I so choose. I might revisit this in the future.

I'm enjoying the fruits of my work on the second day already. Likes and replies are more social than the notes and bookmarks I had previously, so I hope this leads to more interaction with other IndieWeb folk.

With micropub fresh in my mind, I may enhance my micropub endpoint to accept multipart bodies. This allows POSTS with media in to be sent all in one request which shortcuts can make use of. Then I'll be able to post pictures and not just walls of text!

ffconf 2019

2019-11-15T13:16:33Z

The day was packed with interesting and diverse talks, and there's just too much to talk about in a single post. It's definitely worth searching around for other blog posts to see other's takes on the day!

The talk which I find myself thinking about the most was Charlotte Dann's. The journey from exploration in generative art to producing custom real-world jewelery was inspiring! It seems to have inspired others too, since it precipitated the creation of a creative coding meetup here in brighton, which I hope to find some time to get involved with.

In the spirit of that talk, here's a quick Conway's Game of Life demo (requires JavaScript to be enabled). It loops around (so the world looks square but it's really a torus). It's pretty basic! In the spirit of the talk, I hope to apply similar rules to different tilings. Refresh to restart.

Generative art piece: domains of points with spokes

2019-11-30T21:15:00Z

The graphic below is generated randomly and rendered as an SVG. Occasionally it glitches, but that's all part of the fun! It was inspired by a graphic seen in Charlotte Dann's ffconf 2019 talk at about 6:25 in. See the next post for a description of the maths I used to do this.

The maths of Domains of points with spokes

2019-12-02T00:00:00Z

It's been a while since my last maths heavy article. I enjoy writing these but struggle to find the time to write many.

This article is on the generative art piece in my last post. I based this on my memory of an image of a piece in a talk by Charlotte Dann. It was a piece of art actually drawn using a pen and paper plotter (which I love), but I lack these tools at present, so I chose to use the browser instead (lucky you).

In hindsight, I should have realized that the domains around points are like a Voronoi diagram. I forged ahead having forgotten all about them, so please excuse my unorthodox terminology below. Looking back on the talk that inspired this, my memory of the drawing wasn't quite correct. See the real deal at about 6:25 in.

Based on my memory, the image was of a number of points. From each point lines (I'll call these spokes) radiated outwards until they reached the edge of the domain of a point. The "domain" as I call it is an area around a point bounded by the half way lines between it and neighbouring points, or the bounding box of the image.

The case of neighbouring points looks like:

I've drawn spokes radiating from the point on the left up to the domain edge shared by both points. A way of looking at this is to draw the domain boundary.

What I need is, given a point and an angle of a spoke radiating from it, how long will that spoke be to reach the closest boundary? Remember, points are randomly located!

The equation of the boundary line is related to the equation of a line between the two points, which is:

y(x) - y_0 = \frac{x_0 - x_1}{y_0 - y_1} (x - x_0)

Where the numeric subscripts $0$ and $1$ are for the two points, and we're interested in the spokes radiating from point $0$ .

Any two fixed points on the line can be used to place the line. For example, it's just as valid to use the coordinates of the second point (note the subscripts):

y(x) - y_1 = \frac{x_0 - x_1}{y_0 - y_1} (x - x_1)

The equation for the boundary line is perpendicular to the point-to-point line and halfway along it. Since the lines cross half-way along the point-to-point line, we can use that location to fix the perpendicular line:

\left(\frac{x_0 + x_1}{2}, \frac{y_0 + y_1}{2}\right)

The slope is perpendicular to the slope of the point-to-point line:

\frac{x_1 - x_0}{y_0 - y_1}

Putting these together and rearranging gave me the equation for a boundary between two points:

y(x) = \frac{x_1 - x_0}{y_0 - y_1}\left(x - \frac{x_0 + x_1}{2}\right) + \frac{y_0 + y_1}{2}

With the equation for a boundary, I needed an equation for each spoke. My intent was to use these simultaneous equations to find the solution for the length of the spoke.

The equation for the boundary line is in Cartesian coordinates $(x, y)$ , but given that we have an angle, and we're looking for a length, it's more natural to think of a spoke in polar coordinates $(r, Θ)$ . In fact, taking the point as the origin, $r$ is the length we're actually looking for.

A line in polar coordinates can be connected to Cartesian coordinates using the following equations:

\begin{align*} x - x_0 &= r \cos \theta\\ y - y_0 &= r \sin \theta \end{align*}

Substituting these into the boundary equation gave:

r \sin \theta + y_0 = \frac{x_1 - x_0}{y_0 - y_1}\left(r \cos \theta + \frac{x_0 - x_1}{2}\right) + \frac{y_0 + y_1}{2}

After some rearranging, the solution for $r$ looks like:

r = \frac{1}{2} \cdot \frac{(y_1 - y_0)^2 + (x_1 - x_0)^2}{(y_1 - y_0)\sin \theta + (x_1 - x_0) \cos \theta}

Since the equations for the bounding box edges are more simple:

\begin{align*} x &= 0\\ x &= x_\text{max}\\ y &= 0\\ y &= y_\text{max} \end{align*}

The solutions for r with these bounding box equations are a little simpler too:

\begin{align*} r &= \frac{-x_0}{\cos\theta}\\ r &= \frac{x_\text{max} - x_0}{\cos\theta}\\ r &= \frac{-y_0}{\sin\theta}\\ r &= \frac{y_\text{max} - y_0}{\sin\theta} \end{align*}

Given all these equations, for each spoke radiating from each point we need to calculate $r$ for boundaries with every other point and also the bounding box. Spokes radiate away from a point, so all negative values of $r$ can be discarded. Of the remaining possible values of $r$ , the shortest wins!

I settled on using 10 randomly placed points, each with 48 evenly spaced spokes. I've added a gap between spoke ends (both with their point and the boundary) to make the graphic look as I remember it. Points are not drawn.

Finally, I used JavaScript to create an SVG and used line elements to draw each spoke. A little inline CSS in the head applies the foreground colour of the page (the colour of the text) to the lines. This is neat because it means the graphic will look natural on the page in both light and dark modes (although I think it has a certain art deco look to it in dark mode). This ease of theming is in contrast to the effort it took to theme the Game of Life demo in a previous post.

Generative art piece: bubbles

2020-06-17T22:00:00Z

An experiment into generating SVG circles which don't overlap. It was timeboxed to an hour, so it's a little rough around the edges (but I think that adds to its charm).

superimposed triangular grids

2020-08-21T16:30:00Z

This experiment was inspired by field of twistronics, the study of the intersesting properties of misaligned graphene sheets. The misalignment produces a moiré pattern which echoes the underlying structure. I'm too lazy to do hexagonal grids in an afternoon of tinkering so I used triangular grids instead.

superimposed hexagonal grids

2020-08-22T13:30:00Z

Based on the last experiment, this one uses hexagonal grids rather than triangular ones. The way the SVG is constructed isn't pretty (a mish-mash of paths) but it gets the job done. I've given the two grids red and blue lines and a black background to make the moiré pattern stand out.

Custom markdown blocks with marked

2020-10-09T21:45:00Z

I use marked to do the markdown rendering for this blog. A recent feature makes it possible to create custom block types with a little hacking. In this post I show you how!

I'll be using mathematics (TeX) blocks for my example. The marked setup code looks like this:

marked.use({
  walkTokens(token) {
    const { type, raw } = token;

    // Modify paragraph blocks beginning and ending with $$.
    if (type === 'paragraph' && raw.startsWith('$$\n') && raw.endsWith('\n$$')) {
      token.type = 'code';
      token.lang = 'mathematics';
      token.text = token.raw.slice(3, -3); // Remove the $$ boundaries.
      token.tokens.length = 0; // Remove child tokens.
    }
  },
  renderer: {
    code(code, language) {
      // Use custom mathematics renderer.
      if (language === 'mathematics') {
        return renderMathematics(code);
      }

      // Use default code renderer.
      return false;
    }
  }
});

I'm passing two things to marked to configure it here. The first is a token walker function, which is the recent feature which makes this all possible. It is called for each token, traversing the children of a token before it progresses to its siblings (so it's sort of depth-first).

The idea is for blocks of text with a $$ above and below them to be handled as mathematics. To a person this looks like a fenced code block with two dollar symbols in the place of the three backticks. For example:

Some example text with some mathematics to render below:

$$
a^2 = b^2 + c^2
$$

Some example text below.

It's a common extension to place LaTeX code inside $$ delimited blocks. Even if you're not familiar, the dollar symbols above and below are a little like a fenced code block. To marked the block looks like a paragraph. This means that the token walker will receive some paragraph tokens which need to be modified.

To know the difference, paragraph tokens are checked by the token walker to see if they begin with $$\n and end with \n$$. When they do, the block is modified to look like a code block with a special 'markdown' language. Child tokens are removed because the content shouldn't be treated as markdown, and the text property is set by snipping the leading and trailing dollar symbols and newlines off.

The second part of this trick is in the renderer option. The renderer for code blocks is modified with special handling for the 'mathematics' language. The code parameter received by it is the text we set on the token, so it's ready to be rendered to mathematics. The rendering itself is beyond the scope of this post, but I use MathJax. When code is of any other language the custom code renderer returns false to instruct marked to use the default code rendering behaviour.

Dispatching Webmentions with a Netlify build plugin

2021-02-27T15:45:00Z

This site uses a static site generator to build plain HTML pages. Since there's no database to add, update, or delete pages from, determining when to dispatch mentions can be challenging! Here's how I use a Netlify build plugin and an atom feed to manage it.

The Webmention spec requires that a mention should be sent whenever a link is added or removed from a page, or the page one is on is updated or deleted. The recipient will receive an HTTP post which looks the same whatever happened. It's up to them to determine what changed.

Netlify provides hooks for various stages of the build, and a plugin may use as many as it needs. I use the onPostBuild and onSuccess hooks. The former is called when the build is complete, but before it is deployed. This gives me access to the old atom feed and pages over the network (the old version of the site is still deployed), and new versions in the build directory. Mentions are gathered and kept until the onSuccess hook. This hook fires when the site is deployed. It's important to wait for the deployment because the receiver of a mention may automatically check the content of the source URL to know what happened.

If you prefer to read code, check it out here.

This is just the beginning! I plan to port other capabilities over to Netlify build plugins in time.

Tip: Connecting to localstack S3 using the JavaScript AWS SDK v3

2021-03-26T18:30:51Z

I had some issues getting the v3 AWS SDK for JavaScript to communicate with localstack S3, but I found a solution! With the V2 JS SDK, the configuration object for the S3 client looks like:

{
  "region": "eu-west-1",
  "endpoint": "http://localhost:4566",
  "s3ForcePathStyle": true
}

The last field tells the sdk not to use <bucket>.hostname style connections, and instead puts the bucket in the path. For local dev this is important because otherwise the SDK tries to make connections to <bucket>.localhost. This'll work when managing buckets, but not when dealing with their contents (unless you add an entry to your /etc/hosts file, which often isn't practical).

Unfortunately this field doesn't exist for v3, and after searching I didn't find all that much except for a couple of mentions of the v3 JS SDK no longer supporting it.

Fortunately it's not actually the case. It was just renamed to forcePathStyle!

const { S3Client } = require('@aws-sdk/client-s3');

const s3 = new S3Client({
  region: 'eu-west-1', // The value here doesn't matter.
  endpoint: 'http://localhost:4566', // This is the localstack EDGE_PORT
  forcePathStyle: true
});

For reference, I boot localstack with this docker-compose configuration:

version: "3.9"
services:
  mock-s3:
    image: localstack/localstack:latest
    ports:
      - "4566-4583:4566-4583"
    environment:
      - AWS_DEFAULT_REGION=eu-west-1
      - EDGE_PORT=4566
      - SERVICES=s3
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
    volumes:
      - "./.localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Pastel migraine auras

2021-10-14T08:10:00Z

A generative patchwork of pastel colours. The colours begin with a randomly picked colour in LCH. Other colours are hue rotations in LCH space so that they're perceptually nice together. Feature detection is used to render using LCH colours when the browser supports them (Safari only at the time of writing), or to pick a close colour in RGB space when LCH is not supported (the code for this is based on code in d3-color). Refreshing generates a new patchwork.

Procedural Christmas cards

2021-12-07T16:01:05Z

Below is a procedural snowflake. I'm using a little code to create Christmas cards this year, and I wanted each to be unique! If you received a card from me, you may see something like ?seed=1234567890 in the URL bar. That will be the random seed which generated your snowflake (and it's yours to keep). To see a random snowflake, remove everything after the question mark and hit enter. Refresh the page to see a fresh random snowflake!

The red colour is a convenience to make it show up in the software I use to cut card.

Marqdown

2022-05-15T09:55:00Z

Markdown is the standard for writing in techie circles these days, but it's pretty minimal. For a readme it's all you need, but if you create a site around Markdown like I have then you pretty quickly bump into its limitations. Markdown is deliberately limited, so it's no fault of the language or its creator!

Nevertheless, over time I've added my own tweaks and extensions upon Markdown, so I've decided to document them, and name the dialect Marqdown. Naming may seem a little arrogant, but it's mostly to disambiguate what I'm writing with more common Markdown variants.

My variant is based on the default configuration provided by marked, with additions layered on top. This is mostly the original flavour of Markdown with a few deviations to fix broken behaviour. As I add features I'll document them in this post.

Footnotes

I use footnotes^[1] from time to time. The way I've implemented them makes the superscript a link to the footnote text, and the footnote text itself has a backlink to the superscript, so you can jump back to where you were.

The footnote in the previous paragraph is encoded like this:

I use footnotes[^][sparingly] from time to time.

This was an interesting feature to implement because it produces content out of the regular flow of the document. The markdown engine had to be abused a bit to create the superscript links first and keep a list of their footnote texts. Once the document is rendered, a post-render routine checks for any footnote texts, and when there's at least one it appends a section with an ordered list of footnotes. Another complication is index pages. For the blog posts index page only the first paragraph of each post is used, and footnote superscripts have to be removed from those.

Languages

HTML supports language attributes. Most of the time a (well-built) page will have a single language attribute on the opening <html> tag itself, setting the language for the entire document.

I write notes in mixed English and Japanese as I learn the latter. When working with CJK text it's particularly important to give elements containing such text the appropriate language tag so that Chinese characters are properly rendered (there are divergences which are important).

I wrote a Markdown syntax extension to add these tags. Since my documents are mostly in English, this remains as the language attribute of each page as a whole. For snippets of Japanese I use the syntax extension, which looks like:

The text between here {ja:今日は} and here is in Japanese.

This snippet renders to:

<p>
  The text between here <span lang="ja">今日は</span> and here is in Japanese.
</p>

Simple enough. The span is unavoidable because there is only text within it and text surrounding it. Where the renderer gets smart is in eliminating the span! If the span is the only child of its parent, the renderer eliminates the span by moving the language attribute to the parent. For example:

- English
- {ja:日本語}
- English

migrates the language attribute to the parent <li> to eliminate a span:

<ul>
  <li>English</li>
  <li lang="ja">日本語</li>
  <li>English</li>
</ul>

Similarly, the renderer is smart enough to see when the span has only one child, and can move the language attribute to the child to eliminate the span. Example:

{ja:_すごい_}

migrates the language attribute to the spans only child, an <em>:

<em lang="ja">すごい</em>

This becomes particularly important in the case of my notes, where it's common to nest ruby elements inside these language wrappers. There's a ruby annotation in the next section, and you'll see the language attributes appear directly on the ruby element if you inspect it.

As with footnotes, the language attribute migration and span elimination is handled using JSDOM after a markdown document is rendered as part of a post-render routine. In the future I may look into adapting marked to render directly to JSDOM rather than to a string.

Ruby annotations

I'm studying Japanese. It's pretty common to see annotations to help with the pronunciation of words containing Chinese characters. This could be because the text is intended for learners like me, but it's also common to see it for less common words, or where the reading of a word may be ambiguous.

These annotations typically look like kana rendered above or below the word (when Japanese is written left-to-right), or to one side (when Japanese is written from top to bottom). Ruby annotations are not unique to Japanese, but in the Japanese context they're called 振(ふ)り仮名(がな) (furigana), and you can see them right here in the Japanese text in this sentence! The code for it looks like this:

^振,ふ,り,,仮名,がな^

The delimiters are the carets, odd elements are regular text, and even elements are their annotations. So, ふ goes above 振, nothing goes above り (it's already a kana character), and がな goes above 仮名.

There are actually specific elements for handling ruby annotations, so what you see rendered is only from HTML and CSS! They're pretty fiddly to work with manually though, so this extension saves me a lot of time and saves me from lots of broken markup.

Highlighted text

The syntax of this extension is borrowed from elsewhere (I didn't invent it). This addition allows me to wrap stuff in <mark> elements. By default, this is a bit like using a highlighter pen on text. The syntax looks like:

Boring ==important== boring again.

which renders to:

<p>
  Boring <mark>important</mark> boring again.
</p>

which looks like:

Boring important boring again.

This is another extension I use heavily in my language notes to emphasize the important parts of grammar notes.

Fancy maths

Now and then I do a post with some equations in. I could render these elsewhere, but I like to keep everything together for source control. Add to that, I want to render the maths statically to avoid client side rendering (and all the JS I'd have to include to do that).

I settled on another common markdown extension to do this, which is to embed LaTeX code. The previous extensions are all inline, whereas maths comes both in inline and block contexts. I use temml to render LaTeX directly to presentational MathML. Inline mode uses single $ symbols as delimiters, and blocks use double $$ on their own lines above and below the content.

In the past I used MathJax. Visually the SVG MathJax produces is superior to MathML, but the latter is now acceptable, and <math> elements are more appropriate than <svg>, and much less bulky.

This:

$$
x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}
$$

Results in:

x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}

It's time to build a study habit

2022-05-28T12:00:00Z

I started a new role recently, and the company is large enough that there are a number of folk learning or proficient in Japanese as a second language.

I joined a chat session with two other people whose Japanese was much further along than mine. It was great listening practice (I could mostly follow the conversation) but I was unable to contribute, which I found disappointing.

I've said it so many times in the past, but I really need to build a study habit which is substantial enough to improve my proficiency, but light enough that I can sustain it. Not just flash cards too.

All this is complicated by the little one. Evenings are complicated by his dinner/bath/bedtime routine, which can be exhausting. Mornings are more open, but that time needs to be carved out of my existing routine... I could make that work.

Sometimes it should be book work. I stalled on Genki II because there are holes in my basic Japanese from stop-start learning over such a long time. I might pick up Genki I and do one or two morning sessions a week with it.

Other times I think I should read some manga. Two I've been meaning to start reading which I think may work are よつばと! and しろくまカフェ. They're gentle and humorous stories which I think will work well as spring becomes summer here. I also have some graded readers on the shelf I can start on.

Of course, these are still not speaking, but I hope that if I can improve on reading and consuming a wider variety of discourse than just set phrases I stand a better chance of finding the words and phrases I need to contribute in the Japanese chat sessions.

Update (2022-07-09)

To prove that I'm not just talking crap, it's time for an update!

Since I wrote the above post, I've subscribed to WaniKani, and bought three volumes of each of the manga I mentioned above. I'm reading よつばと! first because it's a little more accessible at my level (and I'm really enjoying it, even if I am looking forward to しろくまカフェ). I spend 30-60 minutes each week on a Saturday or Sunday reading, and I intend to build that up to more.

I've found WaniKani to be especially effective. Over the years (decades?) I've tried a lot of flash card apps and approaches, but avoided WaniKani since my kanji wasn't too bad and I didn't want to spend a disproportionate amount of effort on it at the cost of time I could have put into other study. In hindsight, this was a mistake. The approach WaniKani takes is designed specifically for Japanese, and also includes relevant vocabulary. I can already tell it's helping me as I read manga.

To hold myself to account I'm logging my study sessions. I'm only logging sessions over about 10 minutes. Lots of WaniKani review sessions come in under that bar.

Controlling ruby annotation positioning and appearance with pure CSS and a select box

2022-11-07T12:05:00Z

The :has() CSS pseudo-class opens up all sorts of possibilities. I wanted to see if it could simplify how I handle the ruby text (annotations above or below text to help with reading) in my Japanese notes. It works (in Safari and Chrome at least, and hopefully Firefox soon)!

Demonstration time. Below the date stamp in the header above you'll see "ふりがな". To the right is a word. Click or tap on the word to select the positioning of the annotations in the Japanese text below, or hide them. This works even with JS turned off.

この文(ぶん)にはふりがながあります。

Previously I had to use some JS to achieve this. Pretty simple CSS can be used to target and style ruby text (although Safari lags behind the standard). I used JS to append and update a class to the body to set the mode desired. The markup for ruby text is managed using my own extensions to Markdown.

The replacement CSS looks like this (there's a little more to handle the quirks of <mark> elements which I omit here):

/* Select <body> when an element with class furigana-position has an <option>
 * with value "over" and that option is in the checked state.
 */
body:has(.furigana-position option[value="over"]:checked) {
  ruby-position: over;
  -webkit-ruby-position: before;
}

/* Select <body> when an element with class furigana-position has an <option>
 * with value "under" and that option is in the checked state.
 */
body:has(.furigana-position option[value="under"]:checked) {
  ruby-position: under;
  -webkit-ruby-position: after;
}

/* Select <rt> and <rp> elements in the <body> when an element with class
 * furigana-position has an <option> with value "off" and that option is in the
 * checked state.
 */
body:has(.furigana-position option[value="off"]:checked) :is(rt, rp) {
  display: none;
}

I have kept a little JS to persist this state to local storage, as it was in the earlier version of this feature. Now that only one place in the DOM needs to be synchronized (it was two before; the option and the class on the body element) the JS has become much simpler. Most importantly, it's a progressive enhancement of sorts. It's not required for the positioning feature to work, just for the state to be saved.

const select = document.querySelector('.furigana-position');

select.value = localStorage.getItem('ruby-position') || 'over';
select.onchange = e => localStorage.setItem('ruby-position', e.target.value);

Finally, a note on why I have this feature. For learners, it can be helpful to show and hide the annotations depending on what you're reading and why. It can also be helpful, especially on a tablet, to position furigana below the text it annotates, so it can be hidden with a sheet of paper as you read.

Progressively enhanced caching of JavaScript modules without bundling using import maps

2022-11-23T09:00:00Z

I went to ffconf 2022 a couple of weeks ago, and two of the talks in particular resonated with me... (more actually, but these felt actionable):

I really like having my own place on the web, and I've already put a fairly substantial amount of effort into making it as gentle on the environment as possible:

Semantic markup without lots of nesting reduces the size of HTML over the wire (and probably makes it slightly less demanding to parse for the browser).
Few images, served in the most efficient formats supported by the browser making the request.
Lazy loaded images. This is important for my notes stream.
CSS is minified, hashed, and has immutable cache headers. This means that your browser will make a single network request for it and reuse it every time you visit, unless your browser evicts it from the cache, or the CSS changes (which is reflected by a change of the hash in the CSS file name).
The site uses a static site generator, so requests for pages resolve to files which are easily cached. There's no database serving queries behind the scenes.
As little JS as possible. Most pages have only enough JS to load a service worker as a progressive enhancement to allow offline access. These pages work with JS disabled (you're reading one now).

I was inspired to put some time into reducing my footprint even further!

Caching JavaScript

The small elephant in the room are my JavaScript experiments (for example, generative art stuff). I like to build with small, reusable modules, and until now I haven't bothered bundling. The most common solution is to bundle the JS for each page, put a hash of the content in the file name, and serves it with immutable^[1] cache headers with a long max-age, like the CSS mentioned above.

Why though? Resources which are immutably cached need no more network requests once cached. This means fewer requests (especially on subsequent page views). Less energy used to transmit files. Less opportunity for network latency and failure to delay or break features using JS. It's also kinder to mobile users and their plans.

For this little site the savings are going to be pretty modest, but it proves that the approach works. Sites which use a lot more JS with frequent small changes stand to benefit a lot more.

Bundling versus small modules

I'd still rather not bundle though. The pool of small modules I've written for my experiments are intended for reuse. The bundle for each page would significantly overlap with the bundles of other pages, but would be cached independently. A single character change to any module will invalidate all bundles which include it.

It would be far better to cache these modules independently. Consider this tree of dependencies (imagine that /a.js is the entry point):

This is more or less how all browsers saw JS on this site until recently. It suffers from some problems... You can cache using ETags, but that's about as far as you can get without transforming the JS in the modules. It means that the browser only knows about each dependency as it encounters it in import statements. We could do some preloading, but whatever we do it's still a request per module, even when the module has been downloaded before and there are no changes.

We could try to hash the content of each file, and put it in the filenames (the cache-busting pattern):

This lets us cache each file immutably. If there is a change to a file, then the hash changes and the new file is downloaded.

But there's a problem with this... when a file changes, its name will change. When it's name changes, any files which depend on it will change since the imports need to be updated! For example, if /e.js changes, the imports in /c.js and /d.js must be updated, which leads to a change in /a.js too. The changes go all the way from the updated module to the top!

This is a problem. I don't want to invalidate caching for modules which have no meaningful code changes.

Import maps provide a way out of this quandary. They let us tell the browser how to resolve imports. This means that instead of rewriting the imports of each module, we keep them as they originally were. The engine looks in the import map and resolves accordingly. Changing the content of a module will mean only its entry in the import map gets updated.

{
  "imports": {
    "/a.js": "/a-5b5be8.js",
    "/b.js": "/b-913d04.js",
    "/c.js": "/c-7534a8.js",
    "/d.js": "/d-25255e.js",
    "/e.js": "/e-bb438d.js"
  }
}

For caching this is really nice. Your whole JS application can be cached, and whenever a module is updated the browser only needs to make a request for the updated content for that one module.

Implementation

For a static site generator the setup is:

Copy the files with their original names into the target directory.
Copy each file a second time to the same target, but include a prefix and the hash in the file name.
Add an import map to each page which uses JS.
Update the script entry point to use the hashed file.

The first point allows browsers which don't understand import maps yet to work as they did before. Temporary redirects would work too, but mean more requests. The prefixes of the files with content hashed in their names allows immutable cache headers to be added when they're served (the second point).

The last point was unexpected, but to spec (it's not just a Chrome quirk). Browsers which don't understand import maps actually benefit from this change, because they can at least cache the entry point.

The third point hides a lot of pain. As I add more experiments, I don't want the import maps of other pages to grow, so I don't want just one import map for all pages, but rather an import map for each page.

When the files are copied and hashed, a big import map is generated. The code I've written also parses (but does not modify) each module to determine its dependencies.

{
  "/a.js": {
    "hashedFileName": "/hashed-a-5b5be8.js",
    "dependencies": ["/b.js", "/c.js"]
  },
  "/b.js": {
    "hashedFileName": "/hashed-b-913d04.js",
    "dependencies": []
  },
  "/c.js": {
    "hashedFileName": "/hashed-c-7534a8.js",
    "dependencies": ["/d.js", "e.js"]
  },
  "/d.js": {
    "hashedFileName": "/hashed-d-25255e.js",
    "dependencies": ["e.js"]
  },
  "/e.js": {
    "hashedFileName": "/hashed-e-bb438d.js",
    "dependencies": []
  }
}

When the map of each page is generated, the entry point is looked up in the big import map, and its dependencies recursively added too. The hashed entry point is used in the entry script tag.

Content Security Policy

While the import map specification allows for them to be external to the HTML of a page, no browsers currently implement this^[2]. The HTML pages of my static site have quite strict Content Security Policy (CSP) headers. These headers prevent the execution of inline scripts and inline CSS, and I prefer not to relax them.

The way to resolve the conflict is to add the hash of the import map to the CSP header of a page. This is like the server telling the browser you can't run scripts inlined in the page, except for those with this hash.

I once used Netlify edge functions to accomplish this^[3], but now I template a headers file with an entry for each HTML page which needs a custom CSP header for an import map hash.

Preloading

One problem with lots of small modules I've not addressed yet is that on first load, the browser only discovers what it needs to fetch as it reads the imports of each module.

Since the import maps are created specifically for each page, their values are a list of URLs for the hashed files which will be needed. It's possible to preload the modules by adding module preload links in the header. Like the entry point, the browser won't use the import map to resolve JS modules. For stuff in HTML you have to do that.

<link rel="modulepreload" href="/hashed-every-90ec60.js">
<link rel="modulepreload" href="/hashed-javascript-dc77dc.js">
<link rel="modulepreload" href="/hashed-module-8d541a.js">
<link rel="modulepreload" href="/hashed-needed-54bb3e.js">
<link rel="modulepreload" href="/hashed-later-05fbf2.js">

The solution isn't perfect though. A hypothetical problem (there are no browsers like this at the time of writing) is when module preload is supported, but import maps are not. This would lead to modules being fetched twice (hashed and original file names). One way out of that would be to have the original filenames respond with a temporary redirect to the file names with the hashes in.

One potential problem is the head of a document growing very large due to lots of modules mapping to lots of links. In my case each experiment uses a handful of modules, so the increase in the size of the head is not problematic. The module preloads spec allows (but does not require) the browser to resolve the child imports of preloaded modules, so a middle ground may be to preload just some modules.

One small saving I make is to exclude preload links for scripts which will appear in script tags. This includes the index.js file which applies to all pages (mostly there to install a service worker), the entry point(s) of any experiments, and the ruby state persistence script on any page with ruby annotations. This means that most pages don't have an import map or any preload links.

Interactions with the service worker

When a file is served with the immutable cache-control header, it'll either be in the browser cache or it won't be (and will trigger a request to populate the cache). The service worker isn't needed for such files. It's only there to make decisions about more weakly cached files when browsing offline.

The service worker intercepts all requests, but we don't want it to handle ones cached immutably. By returning early from a fetch event handler the handler delegates the request back to the browser and the browser cache will be used as if the service worker isn't there at all.

Since I'm prefixing immutable resources, with hashed-, I can check the URL of the request in the fetch event, and use the prefix to know not to continue.

addEventListener('fetch', fetchEvent => {
  'use strict';

  const isHashed = fetchEvent.request.url
    .split('/')
    .pop()
    .startsWith('hashed-');

  if (isHashed) {
    return; // Delegate to the browser.
  }

  // ...
});

I like this. My site only uses a service worker to handle caching, and I'd rather not have one at all. The less it does the better! In an ideal world, all resources would be immutable, and only the HTML document would not be. In such a world a request for a page resulting in a 304 response (not changed) would also mean all resources are already cached, and no further requests are needed.

Conclusion and future work

When I first published this article, the only browsers to support import maps were those based on Chrome. Firefox shipped support about a month after, and finally Safari in March 2023. The big browsers now all support import maps!

This work so far handles all JavaScript files I deploy except for the service worker. The service worker may be tricky to handle (the URL of the service worker script is important for the running worker and its cache). I'll update this post as I improve coverage.

For smaller applications which differ in composition from page to page (while sharing many source modules) I think this solution has advantages over bundling. There's less to cache (eliminates bundle overlap) and less to fetch when there are changes (only fetch changed modules, not a whole bundle).

Procedural Christmas cards 2022

2022-12-17T15:03:00Z

Below is a procedural snowman. I'm using a little code to create Christmas cards again this year, and as before I wanted each to be unique! If you received a card from me, you may see something like ?seed=1234567890 in the URL bar. That will be the random seed which generated your snowflake (and it's yours to keep). To see a random snowflake, remove everything after the question mark and hit enter. Refresh the page to see a fresh random snowman!

The red colour is a convenience to make it show up in the software I use to cut card.

Full disclosure; I managed the time badly this year. If you got a card after Christmas then it's on me (not the mail strikes).

My 2022 Japanese language study habits

2023-01-03T09:10:00Z

This time last year I put together some custom scripts to send a record of each study session to my personal site using the micropub endpoint. You can see them here. Each entry says what I did, how long I did it for, and when I started doing it. On their own these aren't particularly interesting or useful. They mostly serve to hold me accountable. However, now that I have a full year of data, it seems like a good time to see if there are any trends!

I also took lessons during 2022, but I'm not including those here. Study sessions as I define them here are solo study time, but that does include homework. While I have categorized each session, the vast majority of time was spent on WaniKani and Bunpro, both of which I categorize as flashcards.

The most time I spent studying on a single day was 88 minutes.
The least time I spent studying on a single day was (unsurprisingly) 0 minutes.
I averaged about 17 minutes a day.
The most time I spent studying in a calendar week was 5 hours and 47 minutes.
The least time I spent studying in a calendar week was (regrettably, but we'll get to that) 0 hours.
I studied the least on Saturdays (averaging about 11 minutes), and the most on Sundays (averaging about 24 minutes).

There were days and weeks in which I apparently did no study at all. Missing days may not be surprising (family, holidays, illness, etc.), but whole weeks surely are. The study minutes for each calendar week plot below helps to explain what happened.

Week 1 aside (January 1st 2022 was a Saturday) you can see that I started out with good intentions^[1]. At the time I didn't have much structure to my study, and the tools I used were not working well for me. I was using flash card applications like Memrise and Busuu which are general purpose. Japanese (and probably many or most other languages) is a unique beast, and these general purpose flash card apps didn't do a great job in my case.

At the end of May I resolved to build a study habit. That's where you can see my study minutes really take off. A mixture of finding interesting reading material within my comprehension and a couple of learning applications (I mentioned WaniKani and Bunpro above) helped a lot. These applications are designed specifically for Japanese, and I'm getting much better results from them. I also changed my morning routine to include waking up a little earlier, heading to a local coffee shop, and clearing my review backlog for both each weekday morning before work.

I appear to have peaked in the Autumn. At the time I was doing a lot of reading and I think I need to push in that direction a bit more rather than leaning on flashcards alone. I have plenty to read to keep me going for a while!

My resolution for this year is to regain all the ground I've lost since I passed the JLPT N4 exam a decade ago, and sit the N3 exam in December. I'm going to need to increase my study minutes (I'm still considering where to place this but a minimum of 30 minutes a day seems like a good goal) and sign up for more lessons.

Tip: Type narrowing arrays for sorbet in ruby

2023-01-28T15:25:00Z

When working with type systems like TypeScript or Sorbet, type narrowing patterns are a way to handle different types a variable may contain in different branches. For example:

sig do
  params(
    n: T.nilable(Integer)
  ).returns(
    T.nilable(Integer)
  )
end
def double(n)
  if n
    # n can't be nil in this branch, so sorbet
    # knows the type is narrowed to Integer.
    n * 2
  else
    # This branch could be implied, but is
    # here for clarity.
    nil
  end
end

Type narrowing applies to more than just T.nilable(X) to X. You can use it to refine a type to a subset of some types allowed by a union, or from a value of some class to a value of a child class.

This works well on singular values, but when filtering an array sorbet asserts that the resultant array has the same type. My guess is that this is to ensure consistency between select and select! (the latter modifies in place).

sig do
  params(
    array: T::Array[T.nilable(Integer)]
  ).returns(
    T::Array[Integer]
  )
end
def filter_and_double_array(array)
  # Sorbet thinks that integers is
  # T::Array[T.nilable(Integer)] :(
  integers = array.select { |n| n }

  # So it thinks the next line is broken,
  # although we know it's safe.
  integers.map { |n| n * 2 }
end

The brute force option here is to T.cast(x, Integer) in the map, but escape hatches like T.cast and T.must are a last resort. When the type system is telling us the type we're safer. There's also a runtime overhead to using T.cast or T.must.

The solution is to use filter_map. It can be used to filter out the type(s) you don't want (by returning false or nil), and return those you do:

sig do
  params(
    array: T::Array[T.nilable(Integer)]
  ).returns(
    T::Array[Integer]
  )
end
def filter_and_double_array(array)
  integers = array.filter_map { |n| n }
  integers.map { |n| n * 2 }
end

Or in this very simple case, the two array operations can be put together:

sig do
  params(
    array: T::Array[T.nilable(Integer)]
  ).returns(
    T::Array[Integer]
  )
end
def filter_and_double(array)
  array.filter_map { |n| n * 2 if n }
end

Just like in the single value case, this works on more than just T.nilable(X) to X. Filter an array of Numeric to an array of Integer (a child type of Numeric):

sig do
  params(
    array: T::Array[Numeric]
  ).returns(
    T::Array[Integer]
  )
end
def filter_non_integers(array)
  array.filter_map { |n| n if n.is_a?(Integer) }
end

Filter an array of some union or types to a subset of the union, in this case T.any(String, Integer, Boolean) to T.any(String, Integer):

sig do
  params(
    array: T::Array[T.any(String, Integer, T::Boolean)]
  ).returns(
    T::Array[T.any(String, Integer)]
  )
end
def filter_non_integers(array)
  array.filter_map do |x|
    x if x != true && x != false
  end
end

Tip: Find and type narrow an element from an array in ruby and sorbet

2023-01-30T09:05:00Z

Recently I had a problem where I had to find the first matching element of an array by type. Ruby provides a method to return the first matching (or nil) element in an array, but sorbet isn't smart enough to type narrow it when the match is related to the type of the element.

For illustrative purposes, here's a function which takes an array of strings and integers, and returns the first string element lowercased, or nil when no strings are in the array. My first try looked like this:

sig do
  params(
    maybe_strings: T::Array[T.any(String, Integer)]
  ).returns(
    T.nilable(String)
  )
end
def find_first_and_lower(maybe_strings)
  first = maybe_strings.find { |x| x.is_a?(String) }

  # Sorbet thinks that first is 
  # `T.nilable(T.any(String, Integer))`,
  # so this doesn't work because Integer
  # has no `downcase` method.
  first&.downcase
end

This unfortunately doesn't work. The code is fine, but sorbet doesn't (at the time of writing) understand that the type of first should be T.nilable(String), so it thinks the last line of the function is incorrect.

My next attempt was to manually iterate through the array:

sig do
  params(
    maybe_strings: T::Array[T.any(String, Integer)]
  ).returns(
    T.nilable(String)
  )
end
def find_first_and_lower(maybe_strings)
  maybe_strings.each do |s|
    return s.downcase if s.is_a?(String)
  end
  nil
end

This works! It's pretty ugly though. The only good thing going for it is that it stops iterating through the array once the string is found (like find). To folk new to ruby (me) the return applying to the function as a whole and not just the block was jarring too...

In the end I used filter_map again, which encodes most of the behaviour I want:

sig do
  params(
    maybe_strings: T::Array[T.any(String, Integer)]
  ).returns(
    T.nilable(String)
  )
end
def find_first_and_lower(maybe_strings)
  first = maybe_strings
    .filter_map { |x| x if x.is_a?(String) }
    .first

  first&.downcase
end

This works too! It's not quite ideal though, because the filter_map will build us a whole new array when we only want the first (if any) element. That's why I've kept the downcase outside the filter_map. The solution was to make the filter_map lazy:

sig do
  params(
    maybe_strings: T::Array[T.any(String, Integer)]
  ).returns(
    T.nilable(String)
  )
end
def find_first_and_lower(maybe_strings)
  first = maybe_strings
    .lazy
    .filter_map { |x| x.downcase if x.is_a?(String) }
    .first
end

I've moved the downcase call into the filter_map now because it looks a little cleaner, and it will only apply to the first found element.

Bonus: How about getting the last matching element? While there's a .last method I could use, I don't want to iterate over the whole array to get to it. It turns out that reverse_each returns an enumerator when it's called without a block:

sig do
  params(
    maybe_strings: T::Array[T.any(String, Integer)]
  ).returns(
    T.nilable(String)
  )
end
def find_first_and_lower(maybe_strings)
  first = maybe_strings
    .reverse_each
    .lazy
    .filter_map { |x| x.downcase if x.is_a?(String) }
    .first
end

Otherwise, the solution looks the same, and like the one for the first element it only looks at as many entries (from the end of the array this time) as it needs.

Finally, a note on performance. If you plan to use lazy iteration in a performance sensitive code path, you should benchmark it with realistic data to see how it performs for you. It may be that an eager filter_map performs better on average for your use case, even if it does process an entire array.

My 2023 Japanese language study habits

2024-01-07T11:20:00Z

I've been tracking my Japanese language study session since 2022. You can see the first year report for comparison.

This year my regular Japanese tutor has been taking maternity leave, so I've been doing conversation lessons with another tutor. The format is a little different, and I think this change helped to spark some improvements in my development. Before a typical lesson I'm given an article to read in Japanese. I then spend part of the lesson chatting with my tutor, and the rest re-reading the article, answering comprehension questions, and discussing it. These lessons aren't included in the data, but I do include the pre-lesson read of an article as reading time. As for last time year, while I have categorized each session, the vast majority of time was spent on WaniKani and Bunpro, both of which I categorize as flashcards.

The most time I spent studying on a single day was 80 minutes.
The least time I spent studying on a single day was (unsurprisingly) 0 minutes.
I averaged about 16 minutes a day.
The most time I spent studying in a calendar week was 2 hours and 56 minutes.
The least time I spent studying in a calendar week was (regrettably) 0 hours.
I studied the least on Fridays (averaging about 10 minutes), and the most on Sundays (averaging about 25 minutes).

The gap you see was when I was in Japan! I think I can give myself a pass for that.

One vague trend I see is a decline in minutes studied over the year. I also studied slightly less over this year than last year. Within the two flashcard apps I lean most heavily on, my progress has slowed a lot too. Partly I think this is because I've built up a collection of cards I find difficult to memorize, and that's both hurting my motivation and constricting the flow of new cards. I'll make a more focussed effort to memorize these difficult cards, even if it means I have to write them out by hand a lot or something.

Another thing I've noticed is that with the grammar focussed app (Bunpro) is that it's way more than just a flashcard app, and I'm underutilizing it. This is not just to get the most value out of it; I've found that simple repetition isn't enough to learn many grammar points (what I think of as natural acquisition, like a child might learn). I'm going to have to sit down and start writing proper notes out. Of course, I have the notes section of this site, but nothing beats doing it by hand with glittery gel pens.

My resolutions from last year:

...regain all the ground I've lost since I passed the JLPT N4 exam a decade ago, and sit the N3 exam in December. I'm going to need to increase my study minutes (I'm still considering where to place this but a minimum of 30 minutes a day seems like a good goal) and sign up for more lessons.

I'm almost back to where I was when I passed the N4, and in terms of listening and speaking I'm more advanced thanks to signing up for more tuition. As for sitting the N3... not even close. That's a good goal to have, but in hindsight it was a high target given the free time I have around work and family.

This year? More of the same! Regain the remaining lost ground with respect to N4, improve my conversation skills, and work toward sitting the N3. I don't know if I'll do that this year (I'd say probably not given my trajectory). Goals should be measurable though, so here's what I'm promising myself:

Properly write out 5 grammar points a week, guided by Bunpro.
Revise one grammar point per day.
Read one article on NHK News Web Easy a day.
Continue with flashcards. At least fifteen minutes per day.

Those don't represent a lot of time, but they do need organization and discipline, which I'm not so great at. Wish me luck!

Why did I create a keyboard?

2024-02-29T21:00:00Z

In 2021 I found myself falling down a mechanical keyboard rabbit hole. It started off with two ortholinear (keys laid out in a grid) keyboards requiring minimal assembly (mostly adding my own switches and key caps). From there I moved onto a split keyboard (one unit per hand, connected together by a cable) with a column staggered layout (keys are aligned in columns, rather than rows like a conventional keyboard). In the end it was inevitable that I would try to design my own, but what did I want?

The wish list

Constraints breed creativity, so I decided to be restrictive:

It should be a split keyboard, like a Corne or a Ferris. Allowing the hands to be further apart helps to avoid some fatigue, and means the position of each hand can be adjusted. It also enforces which hand owns which keys (no wandering to the wrong side of the keyboard).
It should use only one controller, and no IO extenders. Most split keyboards use one of these approaches, but it seems wasteful to me just to save on a few wires bridging the two halves of the keyboard. Usually a TTRS cable is used (a headphone cable). The number of wires depends on a few things, but the lower bound using a matrix (there are other approaches which can use fewer wires) is twice the square root of the number of keys, rounded up.
It should be a 36 key build, with Miryoku bindings. The idea here is to minimize how far each digit needs to move, again to reduce fatigue.
It should be built for my hands. I have long fingers and a particular way of positioning my hands which feels comfortable for me. I've noticed that most split keyboards tuck the thumb keys as a short row or arc below the finger columns. My thumbs want to splay out more than that.

PCB (printed circuit board) design

While it's possible (and fairly common) to hand-wire keyboards, the constraints above led to a rough idea of what I wanted, which included some visible PCB. PCB design is new to me, so I had to figure out how to do that. Thankfully there are guides (I used this one). Since I was going with a controller, I didn't need to worry about much of this guide. The important bit for me was how to do a key matrix. The software I used is KiCad, which is free, and libraries can be installed using plain old git submodules (ordinarily I'd balk at submodules, but this is a rare use case where they make sense). I ignored the guide and borrowed the libraries used by other custom keyboard kits since many of them are in GitHub.

PCB design in KiCad is split into different parts. First you have to lay out a schematic, and then you use than to send parts to a PCB for layout and to join parts together with tracks. The nice thing about doing it this way is that the schematic is abstract. You can wire things together without considering layout, and in the PCB design view you arrange parts (particularly switches for a keyboard) and you're given guides to show which parts still need to be connected. Here's what my schematic looked like:

At the top centre is a pro-micro, with all its pins labelled. This is straight out of a component library. The two grids on the left and right are the key switch matrix. While there is a group of switches per hand, the matrix is joined through the IDC connectors in the middle. Labels are used to show how things are connected without traces cluttering up the diagram. For things like switches you can choose and change footprints, so if you want to replace MX switches with choc switches (which is something I did half-way through), the schematic doesn't change, just the footprints used on the PCB.

TODO: PCB design and layout.

The result is the FG-11 Rev1:

Retrospective

After a few months using the device I've had time to understand what worked well and what did not.

Firstly, Miryoku wasn't for me. I like the idea, but it's just too much of a jump from conventional typing for me. I ended up creating a custom layered layout, which covers all the keys I actually use (no mouse keys).

Next, the column stagger is good, but the wrong columns are staggered. If you think of the little notches or nubbins on the f and j keys, most people (myself included) seek these with their index fingers. The column stagger I designed for the FG-11 Rev1 effectively wanted makes me place my middle fingers on these keys, so it's like learning to type again because everything is shifted one column over.

On the stagger, this had the intended effect of having my hands in a sort of cupped position, with the wrists at 45 degrees to the ground (halfway between flat against the desk and vertical). This is a comfortable typing position for me.

The ribbon to connect the two sides is very flexible. I had no trouble positioning each half wherever I wanted with it.

One weird thing that occasionally happens is that a modifier key (typically shift) will stop working. I think this is a software issue since other keys in the row and column of the affected key are fine, and unplugging the keyboard for a bit seems to fix the issue. I may try updating QMK and re-flashing the controller to see if it helps.

On the build, the laser cut steel plates have a very pleasing look and feel. The give a lot of weight to the keyboard too, which I really like. I've found that split keyboards often feel too light.

The future

The keyboard is good, but flawed. The next revision will have these changes:

Column stagger moved outwards one column on each side.
Per-key backlighting. From the schematics of similar split keyboards I'm pretty sure the controller can support enough LEDs to cover 36 keys.
Brass plates. I like the steel plates, but I think brass plates will work well with the Work Louder choc key caps.
I'm considering a different connector standard to join the two halves. Maybe something silly like a pair of PS/2 connectors per side, or a D-Sub 15 connector per side. I may keep the ribbon for flexibility though. It'd be nice to source something which isn't a bland grey.
If I can make the power budget stretch, I may add a couple of extra pinky keys per side. The 5x4 matrix can support this with no need for additional wires between the two halves. I'd definitely like to have escape and tab keys on the left, and enter and backspace on the right would be nice too. Except for backspace the FG-11 Rev1 uses chording to achieve these, and it's a bit of a faff.

IndieWebCamp Brighton 2024

2024-03-16T23:00:00Z

Last weekend we held IndieWebCamp Brighton 2024, the first in Brighton since 2019. I thought I'd collect my thoughts both as a host, and as an attendee.

The first signs something like this might happen (only from my perspective) were back in November, when I tried to gather people for an impromptu Homebrew Website Club while people were in town for ffconf 2023. I had [commented to Ana] on Mastodon that I was missing the Homebrew Website Club that used to run here in Brighton, and she suggested something close to such an event. That met with limited success because of the short notice, but Paul came, and I think we both realized that there may be enough interest. I didn't know it at the time, but Paul had attended IndieWebCamp Nuremberg 2023 just a few weeks before (which is where the seeds for Brighton 2024 were actually planted).

So we (mostly Paul) got on with organizing things. I'm a member of The Skiff, which we chose for the venue, so I took charge of that bit. On the weekend it meant being first there to open up and last out to lock up. I also took on the pastries for the first day^[1], and good local coffee, teas, and milk and milk alternatives. On the weekend I also did my best to make folk feel at home and to tell people about the place and a little local history^[2].

Saturday: Introductions, planning, and discussions

After refreshments and some introductions, we started the planning session. IndieWebCamps follow the BarCamp model of collectively planning session. Jeremy officiated the process, and once we'd found a good schedule of discussions, spaces, and times.

There was a session I wanted to participate in for each time slot.

Energy Efficiency

I wanted to be a part of this one because I've put so much effort into getting the footprint of my own site as low as possible. We discussed efficiency server-side, client-side, in transmission, and the costs and benefits of single servers vs large data centres.

We explored static sites versus dynamically served sites. Static sites are a good way to serve efficiently, but harder to integrate IndieWeb standards like Webmentions into.

We also discussed how our efforts were a drop in the ocean compared with the vast energy overheads of the elephants like Facebook. I argued that my time with the IndieWeb helped me to advocate for more efficient means of creating and deploying sites in the companies I work for.

Pictures

This site uses a static site build, and the content is committed to a git repository. At the moment, so are the images. This bothers me because git works best with text files, and images will slow down clones over time. I've been wondering about storing images separately. Fortunately for me, others share this pain point, and that's what we talked about in this session.

Various bucket-like solutions were discussed, and Paul Watson shared that a combination of S3 and CloudFront cost him on the order of cents per month for a site which is image heavy (making it, and similar combinations through other vendors a good solution).

Hosting

This was a discussion about the different kinds of hosting. For example, static sites hosted on GitHub, Netlify, Vercel, etc. on one end of the spectrum, virtual private servers (or even real personal servers) on the other end, and middle ground solutions like DreamHost which help to administer server applications in the middle.

We discussed the pros and cons of them all, including avoiding hosting traps which make it hard to move your site. There's a clear need for a decision tree to help folk starting out to make a good choice without overwhelming them with lots of information. There's lots more in the linked discussion.

NFC

This was a fun session. Terence bought some NFC tags for us to encode and play with. I configured mine to trigger my study-session shortcut (which sends an h-event to my site with the category and duration of a session) when I wave my phone over it. I successfully did not accidentally try to hijack the session by talking about an ingenious spying device called The Thing, which is an ancestor technology.

Personal website pain points

The final session of the day was a sort of support group, where we could talk about the various problems with our sites as they are now that prevent us from posting to them more, or which just bother us.

A theme which emerged over Saturday was URL regret. Earlier decisions about the structure of a site determine URLs of pages in a way which no longer seems correct. For example, this URL for this page is under /blog/, but I don't really think of many posts I've written on this site as blog entries. In fact, internally they've always been managed as posts. URLs link the web together, so we don't want to break them! The options are:

Keep pages where they are.
Move pages and add permanent redirects to them so old links continue to work.
Move the pages, and accept that links will break. Perhaps give just the most important pages redirects.

Digital gardening was another theme of the day. A traditional blog lends itself to the idea of a feed. A linear collection of entries, usually in chronological order of publication. Chronology is much less important in a digital garden, since the intent is to update posts over time^[3]. The move from linear feeds to gardening was the root of some URL regret in this session.

Sunday: Hacking, writing, and demos

After a day of discussion it feels fantastic to build something! The day was split into about three hours in the morning and three hours after lunch for hacking and writing, and demos after.

In the morning, Scout and I paired to buy a domain name, and wire DNS up to GitHub Pages. Scout is an organizer at the CodeBar Brighton chapter, so she's no stranger to GitHub (making pages an ideal starting place)! Of course, being DNS, it took us a while to figure out its interaction with GitHub's funky way of doing things, but we got there. This was the highlight of the weekend for me. Another independent, personal site on the web!

I used the afternoon to scratch some itches. I recently wrote a colophon, but it needed some editing and nothing linked to it, so I gave it a bit of polish and added a link to it from my about page.

Next I worked on CSS. The main CSS file (the one used by every page here) was at risk of becoming a bit of a dumping ground. Most pages don't use syntax highlighting or MathML styles, so I put those in their own files. Both are third party anyway, so it makes sense to keep them separate and versioned with their associated dependencies. This was mostly plumbing work to configure my static site generator.

Finally, I tweaked a little of the code I use for ruby annotations. Each page with Japanese text on it has a little select dropdown which can be used to position the annotation above or below the main text, or hide it entirely. Thanks to the :has() CSS pseudo-class no JavaScript is needed to do this. Remembering the preference, however, does need JS. So these pages have a tiny JS script which stores the setting in localStorage. I tweaked the script so that when the user sets it to "above" (the default), the script deletes the setting, rather than storing "above".

To wrap up the day we demoed what we'd written and made. Click through to explore those (there's too much good stuff to do it justice here).

Wrapping up

The other attendees kindly helped me to restore The Skiff to its original layout and clean up. Once we were done, I locked up, and a few of us went for a well deserved drink.

Favourite music

2024-08-15T02:45:00Z

Here are a selection of albums which are all-time favourites of mine. They stand up as a whole (not just as single tracks). I don’t claim to have good taste in music, or that my taste is better than anyone else’s! With each album I’ve written a little about times and places I associate with it. The albums are presented in no particular order. I’ll garden this post over time and add new material. I've linked to music services where I can find each album available, but with minimal effort. There are other services, but they're either behind paywalls or incomprehensible to me.

Devin Townsend – Terria

This was the first music from Devin Townsend which I was exposed to. It was in 2002 I think, and I was listening more or less exclusively to bad metal at the time. The weight of the sound of this album was what lured me in (particularly Earth Day), but the progressive rock feeling to it led to a diversification of my taste. This album evokes a feeling of vast landscapes and greenery. It, and Ocean Machine remind me of my undergraduate years at Sussex University.

To this day I’m a big fan of Townsend. No two offerings are the same, and some of his stuff is really out there.

Strapping Young Lad – City

I sneaked in another Devin Townsend album. Until about the mid 2000s Devin Townsend also headed the metal band Strapping Young Lad (SYL). I’m less fond of the output of SYL these days, but I still come back to City. It’s probably the finest industrial metal album out there. This album also reminds me of my time at Sussex University. That, and my questionable taste in clothing at the time (very baggy, not a lot of colour, tassels).

Portishead – Dummy

While at university I had a complicated relationship with a woman who introduced me to Portishead. Things between us ended poorly, and this album reminds me of a time when my head was in bad shape. I can listen to it now with more objectivity, even if it makes me uncomfortable. The music should require no introduction. The hauntingly beautiful vocals of Beth Gibbons and trip-hop sound are timeless.

ISIS (the band) – Panopticon

While a postgrad at Leeds University my like of metal and progressive rock mutated into a like of post-rock. ISIS (long before the terrorist group sometimes affiliated with that initialism came to be) was a heavy post-rock / metal group which produced some extraordinarily heavy music. Panopticon is my favourite of their albums, to the point that I named my first open source offering after it. The stand-out track for me is In Fiction. It' takes an age to build, but when it finally gets there it's the deepest, heaviest, most crushing sound. Amazing pay-off.

Opeth – Blackwater Park

While the first Opeth album I got into was Still Life (and Deliverance & Damnation come close), my favourite remains Blackwater Park. Most of the music I listen to avoids guitar solos and such. Opeth manage to do the reverse, and have moments when every instrument seems to be executing a solo at the same time and maintain coherence. I was originally hooked by The Drapery Falls, but the entire album is exquisite. Give it a try, even if the thought of death-growl vocals is off putting to you. It’s worth it.

Múm – Finally We Are No One

I was introduced to this me by an Italian I met in Tokyo (she introduced me to a lot of really great music, I should thank her again). This album should sound light and airy, but comes across and close and comforting, like being indoors by a fire during a coastal storm. While I first listened to it in Tokyo I don’t associate it with there. Its aesthetic is so strong that it tries to invoke memories of a particular balance that I’ve not actually experienced beyond listening to this album.

The Avalanches – Since I Left You

This album is famed for the extreme lengths the creators went to make it out of samples. I was aware of it long before I listened to it (it was on in various student pubs while I was doing my undergrad). I first really listened to it one Summer I was living in Tokyo. It reminds me of the heat and humidity of the summer there, but also the detachment I felt while I lived there. I moved to Japan at relatively short notice after a difficult break up, and struggled to find my footing for a long time after I arrived. This album reminds me of the joy of summer and feeling at ease with myself and my freedom from my former life and my culture. It probably marks the point at which I stopped identifying myself as a nationality or a job, and started thinking of myself as a standard, awkward mess of a person.

This album reminds me that while we’re all a mishmash of different things, it’s important to remember the difference between what we are, and what our environments try to pigeonhole us as. Even more importantly, a mishmash can be a beautiful and coherent whole.

Portico Quartet – Memory Streams

This is a fairly recent edition, and I find it difficult to categorize. The band makes instrumental music with heavy use of the Hang (a form of steel drum), saxophones, and synths. I guess it's jazz, but I don't know enough about jazz to really say. I bought the record as an impulse purchase shortly before the pandemic lockdowns started. My son was still a baby then, so I was looking for something to relax to. Due to the timing, it became the sound of the early pandemic for me. Back then, it felt like the world was just the three of us.

Boards of Canada – Tomorrow's Harvest

This was an impulse buy, pretty much the day it came out in Japan. I used to spend a lot of time trying to read the staff recommendations in the Tower Records in Yodobashi Akihabara. Unfortunately that Tower Records has been gone for a few years now. I had listened to Boards of Canada's earlier work, but Tomorrow's Harvest blew my mind. The vintage sound and sparseness gives it a brooding and foreboding feeling. This album is excellent to work to.

I bought this album shortly before moving back to the UK from Japan, so it used to remind me of that time. I've played it so often since that any nostalgia for that time has long since gone, but I do still associate it with a change in mindset that came with changing career from a physics academic to a software engineer.

The Devin Townsend Project - Addicted

Yes, another Devin Townsend album. This album was released just days before I moved to Tokyo from the UK, and I listened to it a lot. I strongly associate it with my first year in Japan. In particular with difficulty settling into the time zone in the first few weeks, but also the fun little dance parties I had with housemates and friends in the share house I was renting a room in. The track Bend it like Bender! was a favourite of one of my Italian friends. That friendship ultimately led to me meeting my partner.

Life is full of these incredibly unlikely little sequences of events with important outcomes. What if that album came out a day later? I'd have been too busy to buy it. What if I had chosen a different share house? What if the aforementioned friend hadn't started a business with a friend of my (now) partner and set up a Roman Holiday themed club night beneath a Fiat showroom which we were both "encouraged" to attend to recoup some losses? What if my partner and I hadn't been snacking on arancini next to each other and struck up a conversation? What if...

Marconi Union - Signals

Another relatively recent addition. I'd listened to some of their music before and found it good to work to, so when Signals was released I bought it straight away. It's an exceptional album. Tracks are repetitive, build slowly, and at times feel meditative. There's a thrumming, almost dangerous quality to it. Stand out tracks for me are Strata, and A Citizen's Dream. The latter I find very affecting. It feels like longing, sadness, and hope, sometimes to the point of tears when I'm actively listening to it. I have no idea why that is though! Sometimes my reaction to music can't be explained. It's like it can be a key to a random emotional door in my mind.

I tend to react differently to others when I listen to music. I find quite heavy, negative, loud music to be uplifting, and more pop-like music intended to be uplifting will make me feel quite upset and angry.

How I syndicate links and notes to Bluesky with GitHub Actions

2024-11-25T02:15:00Z

Inspired by Jeremy's post on how he syndicates to Bluesky, I thought I'd follow suit (many examples are useful when it comes to API integration work). A disclaimer though... I'm dubious of the long term prospects of Bluesky for reasons I won't go into here. That being said, it's currently a vibrant place, and syndicating from my site to other places keeps my content^[1] in my hands.

My setup is a bit of a Rube Goldberg machine:

I publish notes (optionally with a photo) and bookmarks using Micropub endpoints.
The endpoints add the note or bookmark as a JF2 JSON file to the git repository of my personal site on GitHub.
A GitHub Actions workflow is triggered by the addition of a note or bookmark.
The workflow calls a Node.js script. It calls the script using my Bluesky handle, which is qubyte.codes for me, and an app password. Do not use your actual password! Bluesky makes it pretty easy to create an app password, and when you have one you can add it as a secret for Actions to use at the ./settings/secrets/actions path of your repo site. I've called my secret BLUESKY_APP_PASSWORD.

The rest of this post is about the script itself. As a rule of thumb, I keep bits of shell glue in Actions workflows, and store the bulk of any logic in a discrete script. That makes it easy to call the script by hand for testing.

I publish notes (optionally with an image) and bookmarks using Micropub endpoints. The note or bookmark is added as a JF2 JSON file to the git repository of my personal site on GitHub. When such a file is created there, a GitHub Actions workflow is triggered, and this workflow in turn calls a Node.js script.

The script is a vanilla Node.js script. While Bluesky does provide API client libraries, I didn't find it necessary to use one. I managed to write this without any third party libraries. API work is all achieved with plain old fetch.

There are a couple of local libraries which I've extracted into their own modules. The first is blueskyAuth, which looks like this:

const createSessionUrl = 'https://bsky.social/xrpc/com.atproto.server.createSession';

export default async function blueskyAuth(handle, appPassword) {
  const res = await fetch(createSessionUrl, {
    headers: { 'content-type': 'application/json' },
    method: 'POST',
    body: JSON.stringify({ identifier: handle, password: appPassword })
  });

  if (!res.ok) {
    throw new Error(
      `Bluesky responded with an unexpected status: ${res.status} ${await res.text()}`
    );
  }

  return res.json(); // { accessJwt, refreshJwt, did }
}

Bluesky mostly speaks JSON, which is convenient when working in Node.js. To create a session with Bluesky, you JSON encode an object containing your handle and an app password. If successful, the response contains three things of interest:

accessJwt: A short-lived token used for API requests.
refreshJwt: A token which can be exchanged for a new accessJwt and refreshJwt when the accessJwt expires.
did: An identifier. The initial "d"^[2] stands for "distributed", but I recently discovered that the DID plc method used by Bluesky is not actually distributed at all, thanks to Christine Lemmer-Webber's excellent article on Bluesky and decentralization. I highly recommend reading it to dispel some myths around how decentralized Bluesky is, by design.

My script uses the accessJwt as a token and the did as an identifier in API requests. Since the session is only needed to post one document (and possibly an image), there's no need to think about refreshing the token, so I don't use the refreshJwt.

The rest of the script is about one or two API requests. When an image is to be uploaded, that happens first. There's not much to say here beyond the size restrictions being quite low, and being constrained to JPEGs and PNGs. I don't think they're doing much processing of images, so I recommend removing metadata from image uploads. The response of the image upload contains a blob field, which is used as a reference to the image in the second API request in the form of an embedding.

The final request creates the "record" (the note or bookmark) on Bluesky. There's a lot to unpack here. The creating a post is full of useful samples to help figure out the anatomy of a record creation body.

One fascinating aspect of Bluesky is how text is composed. Rather than some sort of markup language, it uses a more limited concept called rich text facets. Text is linear, and facets (for example, links) are attached to UTF-8 byte ranges of the text. This is awkward for many languages! For example, JavaScript uses UTF-16 to represent string internally (as was the trend in the mid-'90s), so the range of characters you get with naïve string work in JS will give you incorrect offsets. Thankfully Node.js has the venerable Buffer class, which can be used to represent strings as arrays of UTF-8 bytes. A Buffer.byteLength is all it takes to get the UTF-8 size of a string.

Anyway, I've put lots of annotations in the syndication script, so hopefully it serves as a useful example!

What happened to 2025?

2026-02-06T11:45:00Z

I've not posted in over a year (the whole of 2025). I don't have a readership (there are good odds that nobody besides me will read this), so nobody has missed my writing and I've felt no pressure to force a post.

It is unusual though. A handful of times in a typical year something will grab my attention to the point that I take the time to write about it. So what was different about 2025? And why write about it now?

In short, I was very busy:

I started coming out as trans in late November 2024. Three people close to me by the end of the year.
Over 2025 I came out to everyone else, and by the middle of April I was Aura (hello!) full time in all contexts.
In late January 2025 I started on hormone replacement therapy (HRT) which means...
I just passed the first full year on HRT (the anniversary which led me to write this post)!
Lots, lots more related stuff (laser hair removal, electrolysis, clothes (I haven't boymoded^[1] since April) legal stuff and ID, etc.).

The year was a lot^[2]. Each trans person has their own path, and not all have access to or want HRT. For me though, it was the most important step so far, to the point that I'm viewing it like a birthday. In the years between realizing I'm trans and making the decision to come out, I'd gone from the persistent feeling of wrongness and disconnection I'd always had, to being in a very bad place. The decision to transition was a necessary one, but the rewards astonishing. I'm healthier and happier than I've ever been before. I'm still early in my journey, but I've come so far.

So that's what happened in 2025, and why there's a gap in the blog posts. I might expand on individual parts of my experience in future posts. I'm not certain yet. While it's an extremely personal subject, every trans person's path is different, and perhaps someone reading this will see something of themselves which helps them. I wouldn't change my life so far (except skipping the years between figuring it out and coming out), but I often wonder how things might have been if I'd have read something like this and connected the dots earlier.

If you're reading this, and your gender is on your mind, read this^[3]. If we're acquainted, then you can talk to me, in confidence.