Forbes Lindesay

I’ve spoken about package management in the client side JavaScript sphere before. I’ve extoled the virtues of component and broadly avoid mentioning browserify. A lot has happened since then though. Both have shiny new websites and browserify has hit version 2.

Why didn’t I talk about Browserify v1?

Browserify v1 has been around for a long time. It has an almost magical ability to take JavaScript code that was written to run on a server using node.js and make it work on the client. That’s exactly the issue though. It was a magical black box that seemed to do more and be more complex each time you looked at it. For a long time it was plagued with bugs when running on windows. It attempted to do all the things that should be seperate extensions internally in the one module.

Doing too much is a problem. If I rely on something I have no understanding of and it ceaces to exist, I’m completely stuck. Component, by contrast, is fantastically simple. You list where all the JavaScript files are and then it concatenates them into one long file with the standard require logic wrapped around them. It couldn’t be simpler.

This simplicity made it easy to extend. I was able to build a streaming version and a website which serves up fully compiled modules with ease. It was always easy to see what was going on and to debug.

How does Browserify v2 fix these issues?

Browserify v2 is made up of a collection of small modules. Each module is inherently streaming, which is perfect for building things like a website to let people download pre-compiled modules. All browserify does is handle the require side of things. It detects calls to require so it knows which files are needed, and then it concatenates them into one long file with the appropriate thin require logic wrappped around them.

It has clever fallback logic when it doesn’t find a module so that you can include multiple bundles where some bundles can depend on things in bundles which were loaded earlier. This means that browserify can make excellent use of CDNs and caching.

Creating a module

When creating a module, the main downside to component is fairly obvious. For every JavaScript file you create you have to add an entry to scripts in component.json. The other downside is that if you’re writing for both server side and client side you have to add both component.json and package.json with broadly very similar contents.

Both component and browserify allow you to support different code for client and server. Component does this by having an entirely seperate module definition file with its own main field. Browserify does it by adding a simple "browser" field wich overrides "main" when in the browser.

Publishing and Installing a Module

When publishing a module the main issue is finding a name for it. Component uses GitHub as it’s registry while browserify uses npm. npm has a flat namespace and you can name modules anything you want.

To publish you run:

npm publish

To install you run:

npm install my-module

And then in your code you load it using:

var mod = require('my-module');

This is great, providing nobody already has ‘my-module’ as a name. Having said that, if they do there are two ways it can still end well:

1. They might have already built a module that does what you want done
2. They might have built a module but then decided they don’t want it, and they may be willing to unpublish their module to make room for yours (this happened to me).

By contrast, GitHub enforces a strict two-tier namespace. The first level is your username and the second is your module name. You’re guaranteed to be able to get any module name you want, but you have to prefix it with your username.

To publish, just push to GitHub

To install you run:

component install ForbesLindesay/my-module

And then in your code you load it using:

var mod = require('my-module');

Notice that the username prefix is dropped once it’s installed. This works really well. It’s worth noting that you can still get this behaviour using npm. You can use npm to install anything from a GitHub repository providing it has a package.json file. If the package.json file has a name property set to my-module you can still do:

npm install ForbesLindesay/my-module

And then in your code you load it using:

var mod = require('my-module');

So in essence, npm is just as magic, but has the advantage of working for server side code as well

Finding Modules

One of the hardest thing with any new module system is finding the modules that do what you need them to. Ultimately many people just end up searching GitHub using Google. Once most people converge on one module system, this will likely become an acceptable solution, but at the moment it’s completely inadequate. Most of the libraries you will find will not work with your chosen package manager.

Both browserify and component do come with a dedicated search system. Neither come up to scratch though. Component’s is a new website, it’s very shiny, but doesn’t give you any information about the browser compatibility or quality of the components. It has a search in the bottom left corner, which is just a basic text based search. It’s difficult to browse through and for the moment I still prefer the old site at https://component.jit.su/ which at least makes the licence very clear and categorises the modules.

Browserify’s is initially just a blank page with a search box, somewhat un-inspiring as a starting point. when you start typing you get a simple list of modules and descriptions. This works fairly well but again lacks info on licenses. It does show test results for browser compatability when available from http://ci.testling.com and I’d really like to talk to them at some point about adding support for https://jepso-ci.com

What both of them currently seem to lack is any sense of the fun that can be had browsing really cool modules by category. There are awesome things out there, and sometimes seeing something you hadn’t thought of can spark and idea for something else that would be cool to build.

Testing

Disclaimer I built https://jepso-ci.com so I’m biased

It’s curicial when your writing code for the browser that you have unit tests which can be automatically run across all the browsers you want to support. https://jepso-ci.com can run any tests that will run in a real web-browser so it’s great for testing both component and browserify based code. At the moment jepso-ci does not have support for automating the build step for browserify, but this will be comming soon. You can track the progress in this issue

http://ci.testling.com is currently the only other major cross browser continuous integration platform. It’s built by the same people who built browserify, so naturally the two work well together. It has no support for easilly running the same tests in a real local web browser though, and doesn’t seem to provide any information about the html of the page the tests will be run on.

Style Sheets etc.

Component aims to view modules as more than just JavaScript (hence the name) and includes CSS and other files. In the long term this is definitely something the world will need. There’s a difficulty for now though in that CSS doesn’t have any nice systems for modularity. It’s one global namespace and it’s really ugly. It has no variables, no parameters, no opportunities for configuration. As such I would suggest that component’s CSS support is essentially degenerate and insufficient.

Browserify views modules exactly like server side JavaScript does. They’re some JavaScript code, joined together with require calls. You can easilly add a build step that converts a text file into JavaScript, and CSS just isn’t ready for a package manager yet anyway.

This process sucks there should be a better way of doing it, but here are the steps to permanently add an environment variable on Windows.

Step 1

Right click on your computer in the file browser and select properties:

Step 2

Click “Advanced system settings” in the left hand side bar

Step 3

Click “Environment Variables”

Step 4

Click new. For most things you’ll want “User variables for Username” and not “System variables”

Step 5

Enter a name and value and click OK

Step 6

Are you prepared for the worst part? Now you have to restart your computer…

Temporarily Setting Environment Variables

Sometimes you don’t need the value to persist:

If you’re in powershell

SET:

$env:VARIABLE_NAME = 'variable value'

GET:

$env:VARIABLE_NAME

If you’re in a standard command prompt

SET:

SET VARIABLE_NAME = variable value

GET:

SET VARIABLE_NAME

When it comes to asyncronous operations in JavaScript there’s only one sane way to handle them, and that’s promises. The reason for this is that they’re composible:

all(asyncThingA(), asyncThingB())
  .spread(function (resultA, resultB) {

  });

The all function and the spread function are both really easy to build, and the above code simply says wait for both async operations to complete, then give me them. Writing the same thing using code based off event emitters or callbacks is a complete nightmare (I’m not even going to go into how you’d code that).

Promises are great for lots of things then, but there’s oen thing they haven’t yet got sorted out…progress. How on the earth should we handle progress?

Progress in event emitters

Event emitters have the upper hand when it comes to dealing with progress. They can just emit a progress event which you can handle.

function doAsyncOperation() {
  var emitter = new Emitter();
  var progress = 0;
  setInterval(function () {
    if (progress < 1)
      emitter.emit('progress', progress);
    progress += 0.1
  }, 100);
  setTimeout(function () {
    emitter.emit('end');
  }, 1000)
}

var op = doAsyncOperation();
op.on('progress', functoon (p) {
  console.log('Operation is ' + (p * 100) + '% done.');
});
op.on('end', function () {
  console.log('Operation complete');
});

This actually works great. It’s simple, easy to understand, and does what you need. There are two problems with adopting the same approach in promises:

• Errors
• Composition

Errors

What happens when an exception is thrown in a progress handler?

Just throw the error

With event emitters it will simply throw and bring down your application (browsers will carry on despite the error, but node.js apps won’t, and neither will windows 8 apps). That seems fine if you come from a world of event emitters, but jarring if your used to the promises behaviour of forwarding the error:

makePromise()
  .then(funciton success(res) {
    throw new Error('This error gets caught');
  },
  null,
  function progress(p) {
    throw new Error('This error crashes your app');
  })
  .then(null, function errorHandler(err) {
    //handle error
  })
  .done();

The thing here is that it’s not as simple as saying “Never throw in progress handlers” because progress handlers aren’t really critical bits of code. You don’t really need them for your application. They might never get fired because your async operation might not support progress. Then you re-factor and decide to make the async operation support progress and your application crashes.

Ignore the error

You could just ignore the error. We’ve already said that progress handlers aren’t that critical to your application, so why bother notifying anyone of the exception? To answer that, imagine you’re developing an application for sharing images you have the perfect UX except for the fact that the progress bar doesn’t move. You spend hours trying to figure out what’s going wrong, only to find an error being silently ignored coming from your progress handler that would’ve made the problem obvious and easy to fix.

Never silently ignore an error unless it’s deliberate and acompanied by a note along the lines of “This error doesn’t matter because …”

Remaining options

So the remaining options are:

1. Log the error, without throwing an exception
2. Make the exception handleable as part of the promise
3. Accept the problems with just throwing and just throw

None of these are completely un-acceptable, but none of them are ideal.

Composition

Parallel composition

What happens to progress in the example at the start of this post? Imagine the following:

AsyncThingA emits:

10ms Progress: 25%
20ms Progress: 50%
30ms Progress: 75%
40ms Progress: 100%

AsyncThingB emits:

30ms  Progress: 25%
60ms  Progress: 50%
90ms  Progress: 75%
120ms Progress: 100%

If we just emit both sets of events we might get:

10ms  Progress: 25%
20ms  Progress: 50%
30ms  Progress: 75%
30ms  Progress: 25%
40ms  Progress: 100%
60ms  Progress: 50%
90ms  Progress: 75%
120ms Progress: 100%

That’s not going to lead to a nice smooth progress bar. What if we take the average?

10ms  Progress: 12.5%
20ms  Progress: 25%
30ms  Progress: 37.5%
30ms  Progress: 50%
40ms  Progress: 62.5%
60ms  Progress: 75%
90ms  Progress: 87.5%
120ms Progress: 100%

That’s more like it, it seems we have a winner. We have a problem if our progress isn’t represented as a percentage. Consider what would happen if we had:

10ms Progress: Stage A complete
20ms Progress: Stage B complete

Clearly the previous approach won’t work. So from that we need to either:

1. Restrict progress values to a number between 0 and 1
2. Support customised progress composition

Serial composition

What about when our promises are in serial rather than parallel:

doAsyncThingA()
  .then(function (res) {
    return doAsyncThingB(res);
  })

Consider the functions are as they were in the parallel execution:

AsyncThingA emits:

10ms Progress: 25%
20ms Progress: 50%
30ms Progress: 75%
40ms Progress: 100%

AsyncThingB emits:

30ms  Progress: 25%
60ms  Progress: 50%
90ms  Progress: 75%
120ms Progress: 100%

If we just pass all progress events straight through we might get:

10ms  Progress: 25%
20ms  Progress: 50%
30ms  Progress: 75%
40ms  Progress: 100%
70ms  Progress: 25%
100ms Progress: 50%
130ms Progress: 75%
160ms Progress: 100%

That’s better than the parallel progress we had, but still pretty useless. In the same veign as before, we could assume they take roughly equal time and get the following:

10ms  Progress: 12.5%
20ms  Progress: 25%
30ms  Progress: 37.5%
40ms  Progress: 50%
70ms  Progress: 62.5%
100ms Progress: 75%
130ms Progress: 87.5%
160ms Progress: 100%

It’s not going to be completely smooth, but it’s pretty good as an aproximation. Imagine we replace AsyncThingB with as syncronous operation of virtually 0 time. Our previous aproximation now yields:

10ms  Progress: 12.5%
20ms  Progress: 25%
30ms  Progress: 37.5%
40ms  Progress: 50%
40ms  Progress: 100%

This is actually typically true of quite a lot of the operations in a promise chain.

A solution

This problem really needs solving, fast. Parallel composition is not too urgent, because there’s nothing else that doesn’t completely suck for parallel composition anyway. Serial composition needs to get fixed though. One option:

Take a progress-transformer as the third argument to .then(cb, eb, progressTransformer). progress-transformer can be one of three things:

• false - the value false indicates that progress propogation should stop at this point.
• a function - The progress values from the previous promise are passed to this function, propogation can be stopped by returning false, making false an invalid value for progress.
• Anything else - Anything else is also acceptable, and results in progress being propogated (effectively assuming that the current operation will be syncronous).

doAsyncOperationA()
  .then(function (res) {
    return doAsyncOperationB(res);
  }, null, divideByTwo);
function divideByTwo(v) {
  return v / 2;
}

Only the progress from operationA will be passed through the transformation, so the resulting promise yields:

10ms  Progress: 12.5%
20ms  Progress: 25%
30ms  Progress: 37.5%
40ms  Progress: 50%
70ms  Progress: 62.5%
100ms Progress: 75%
130ms Progress: 87.5%
160ms Progress: 100%

Which is the best result we came up with for serial composition.

doAsyncOperationA()
  .then(function (res) {
    return doSyncOperationB(res);
  });

Yields:

10ms Progress: 25%
20ms Progress: 50%
30ms Progress: 75%
40ms Progress: 100%

Which is perfect because it’s just the progress of A which is the same as the progress of the whole because B is sycronous.

A Solution to Parallel Composition

Parallel composition is a tricky one, I’d suggest that the all function will have to do some extra work. If we let progress be any value, then the extra work should probably just be agregating all the progress into a big array. If we force progress to be a number, we should probably just take the average of that number.

API

The behaviour we have specified above should help to inform our decisions regarding the progress API. Having a progress handler as the third argument of .then strongly implies that errors should be forwarded down the promise chain. Having it as a separate ‘event’ like structure would suggest it should just throw.

Option A

Make the third argument of .then be the progress handler. Forward exceptions down the promise chain. Propogate all progress events if the argument is not a function and not false. If the argument is false, don’t propogate anything.

Option B

Separate out propagation compltely from handling. Essentially this lets us take the event-emitter aproach to handling progress.

A promise supports progress if it has a .progress function.

If .progress is called with an argument of type function then it registers that function to recieve progress updates.

If .progress is called with anything other than a function then all registered functions are called with that value.

If .progress(fn) is called after .progress(val) then fn is called in the next tick with the most recent value.

Passing true as the third argument of .then triggers basic propagation, but all other propagation must be done manually. The following two bits of code would therefore be equivalent:

var p2 = p1.then(cb, eb, true);

var p2 = p1.then(cb, eb);
p1.progress(p2.progress.bind(p2));

Conclusion

I’m actually in favour of Option B. It’s simple to impliment, and I feel that it probably covers the 80% case. By letting people manually write to the progress of a promise, they can artificially do whatever they want with progress, but we cover the case of one async operation that’s followed by a sync operation by just passing true as the third parameter.

I’ve had a need to create in-line editing components for a project I’ve been working on lately. The required behaviour is simple. There are areas of the page that are editable. When you click on them they turn into a control that lets you edit them, and when you click away, the results are saved.

I came across jEditable which seems to do the trick, but I really don’t like the style it’s built in. It’s tightly integrated with jQuery in terms of its api. That could easily be fixed with a simple wrapper though. The more problematic aspect of it though is that it seems highly opinionated about how you update the server etc. I set out to create something simpler and smaller.

I’ve created a github organisation called componitable to store these little inline editable widgets. If you want to help the development effort, talk to me about getting involved.

Once installed with component, you can use them simply:

var text = require('editable-text');
text('[data-editable="text"]', {maintainSize: true});

Would take all elements that have a data-attribute of data-editable="text" and make them click-to edit text-boxes. The maintain-size option makes the text box the same size as the text that is being edited. That’s worth doing if you’re editing text in a table and need it to not re-size the table all over the place.

Editable Dates

Because I’ve tried to keep the library un-opinionated, the default formatting of date is just one of the toString methods, but I let you specify your own formatting however you want, e.g.

var date = require('editable-date');
date('[data-editable="date"]', {
  maintainSize: true,
  displayFormat: function (date) {
    var months = ['January', 'February', 'March', 'April', 'May', 
      'June', 'July', 'August', 'September', 'October', 'November', 'December'];
    var special = {
      1: 'st',
      2: 'nd',
      3: 'rd',
      21: 'st',
      22: 'nd',
      23: 'rd',
      31: 'st'
    };
    if (date) {
      return date.getDate() + (special[date.getDate()] || 'th') + ' ' + months[date.getMonth()] + ' ' + date.getFullYear();
    } else {
      return '';
    }
  },
  parseDate: function (dateStr) {
    if (!dateStr) return null;
    else return new Date(dateStr.replace(/(\d)[a-z]{2}/g, '$1'));
  }
});

Show me the money

Formatting for number fields is a little complex, so you might want to use a library like format-number which formats them neatly:

var formatNo = require('format-number');
var number = require('editable-number');
number('[data-editable="money2"]', {maintainSize: true, format: formatNo({prefix: '£', padRight: 2, truncate: 2})});

Updating the server

I’ve tried to stay un-opinionated about how you update the server, but I’ve given you some tools to help you do it in a sane way. The editable component will emit an update event with a value whenever a field is updated (regardless of what type the field is). Here’s an example of how I handle updates in my app:

var editable = require('editable');

editable.on('update', function (element, value) {
  var path = editable.attribute(element, 'path');
  var field = editable.attribute(element, 'field');
  var data = {field: field, value: value};
  post(path, data, function (err) {
    if (err) {
      alert('Error updating database.');
      throw err;
    }
    if (editable.attribute(element, 'requires-refresh') == 'true') {
      location.reload();
    }
  });
});

That lets me set data-attributes for path, field and requires-refresh anywhere up the tree of parent items and have that information used when updating the server.

Conclusion

This set of libraries are very much in a state of flux at the moment (none of them even have proper version numbers yet). If you use them, it’s up to you to test them thoroughly, but I thought I’d give the world a peek, so hopefully anyone else trying to solve the same problems can see how I’ve done it.

Embedded JavaScript (hereafter referred to simply as EJS) is a templating engine, much like moustache, which I covered in an earlier post. At first glance, this might seem like a painfully verbose syntax by comparison to the exceptionally terse moustache, but I’d like to argue that it more than makes up for it in power and ease of use. I’d then like to conclude by talking about asynchronous templates in their various forms.

So what is it?

Broadly speaking, it does exactly what it says on the tin. Take an html document which has some JavaScript embedded in it and then turn it into a template. Unlike moustache, EJS is a two stage process. First, you compile your template, then you provide it with some local variables and execute it. Where the power comes, is that this isn’t limited to some simple constrained sub-set of JavaScript, it’s literally any (synchronous) JavaScript. That means you can write a for loop as follows:

<ul>
<% items.forEach(function(item){ %>
    <li><%= item.name %></li>
<% } %>
</ul>

If that looked strange and unfamiliar to you, fear not. EJS lets you use ANY JavaScript, so you can do your loop in any way you feel comfortable with, perhaps you’d prefer:

<ul>
<% for(var i = 0; i<items.length; i++){ %>
    <li><%= items[i].name %></li>
<% } %>
</ul>

Both those templates would print out the exact same html, and have very similar performance characteristics, so you really can use whatever you’re most comfortable with.

If Then Else

Just in case you’re not getting it yet, here’s an example using if and else to display a login/logout button.

<% /* Comments work as you would expect */ %>
<button id="toggleLogin">
<% if(user.isLoggedIn) { %>
  Log Out
<% } else { %>
  Log In
<% } %>
</button>

But I don’t trust users!!

Hmm, if you’ve been thinking really carefully, you’ll have noticed that I outputted text from a javascript value directly into my html in my for loops. Depending on just where that value came from, I might want to be real careful doing that. To help us, EJS offers two ways of outputting a value from javascript.

Unsanatised, Dangerous, only do this if you know where this variable has come from and you’re CERTAIN it’s safe:

<%- text %>

If you don’t need to be able to output html, you should always sanitise your outputs. This also has the advantage that symbols like greater than or less than will render as expected:

<%= text %>

A word about semi-colons and line comments

Because of the way EJS is compiled, you’ll have a much more reliable system if you use block comments /* comment */ instead of line comments //comment.

EJS may also do weird stuff with semi-colons, so it’s best to omit them where they’re optional (which is most places in JavaScript).

You should always be explicit about blocks in EJS too.

Good:

<% if (true) { %>foo<% } %>

Bad:

<% if (true)  %>foo

Go make cool stuff

So, that’s pretty much all there is to it. It’s a fantastic bit of kit that I’m sure you’ll absolutely love and it’s available both for the client side http://embeddedjs.com/ and as an npm module for the server (npm install ejs) if you’re using node.js (which you should be as it’s great).

Templating is what makes writing large applications for the web viable.  Server side templating has been around for a while and typically allows you to have small snippets of code executed inside a text file to fill in parts of the text file that are blank.  JavaScript now offers many templating libraries which vary from sections of fully fledged JavaScript to the extremely compact yet very powerful syntax of Mustache.

Mustache can be found at https://github.com/janl/mustache.js The language is fairly well specified so there are other implementations around.

When rendering content using mustache you have 2 parts, the view and the template.  I’m going to walk you through a real world example of displaying the navigation structure of a web-site using mustache.

The View

Let’s create a view to model a list of links.  This code should be fairly simple to follow as it’s just like what you’ve been seeing up to this point.

//define a data model for the link data-type
function link(id, name) {
    //Use prototype in order to get the selected method.
    var that = Object.create(link.prototype);
    that.url = "#/" + id + "/";
    that.name = name;
    return that;
}
link.prototype.selected = function () {
    return this.url === location.hash;
}
//build an array of links
var links = [link("home", "Home"),
             link("edit", "Edit Page"),
             link("view", "View Stuff")];
//define a mustache view
var view = {links:links};

This produces a view which you can continually re-use each time you need to render the navigation.  The key thing to note about this is that it everything in it is about the actual state of things as data, and not about their visual appearance.

Template

Now let’s create a template to display the links.

{{#links}}
    <a href="{{url}}"
       {{#selected}}class="Selected"{{/selected}}>
           {{{name}}}
    </a>
{{/links}}

OK, loads of stuff happened in that last code snippet.  Let’s run through it one stage at a time.  The <a> tag should be familiar as an html tag that will display a link, but the curly braces are totally new and weird.

Anything wrapped in {{ and }} will be interpreted as being for Mustache to render.  The two simplest tags are {{url}} and {{name}}.  These are the bread and butter of Mustache and will simply be replaced directly with the appropriate text.

The tags beginning with a # character start blocks and the tags beginning / close the block of the same name.  The links block represents an enumerable block, while the selected block represents a conditional block.

The enumerable block takes an array from the view and enumerates each element in that array.  It then uses the block it contains as the template for each element in the array.  The conditional block prints out whatever’s within it if and only if the value is “truthy”.  Mustache is clever enough to execute functions and use the returned value.

Rendering a template

The template will be rendered something like this:

First consider the {{#links}} block.  Find the links property on the view and discover it contains an array.  Therefore repeat the following template for each element in the array.

<a href="{{url}}"
   {{#selected}}class="Selected"{{/selected}}>
       {{{name}}}
</a>

The context for this template will be each element in the array. So for the first element it will replace {{url}} with #/home/ and {{name}} with Home then it will consider the block {{#selected}} Since this is returns function it will execute the function and because the function does not return an array it will be treated as a conditional block. If the link is selected it will return true.

Putting it all together

The code below uses what we’ve looked at so far on templating to render a simple web site with links you can click on to navigate between pages. You can view it live here. Note that the html page references mustache.

index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Articles</title>
  <link href="style.css" rel="stylesheet" />
</head>
<body>
  <nav id="navigation">
  </nav>
  <article id="content">
  </article>
  <script type="text/template" id="navTemplate">
  {{#links}}
    <a href="{{url}}"
       {{#selected}}class="Selected"{{/selected}}>
      {{{name}}}
    </a>
  {{/links}}
  </script>
  <script src="mustache.min.js"></script>
  <script src="script.js"></script>
</body>
</html>

style.css:

body {
  font-family: arial;
}

nav {
  background-color: #FDC107;
  height: 50px;
}

nav a {
  text-align: center;
  height: 35px;
  display: block;
  float: left;
  text-decoration: none;
  padding-top: 15px;
  padding-left: 50px;
  padding-right: 50px;
  color: #FFFFFF;
}

nav a:hover, nav a.Selected {
  background-color: #6699FF;
  color: #000000;
}

script.js:

(function () {

  var nav = document.getElementById('navigation');
  var article = document.getElementById('content');
  function template(id, data) {
    var text = document.getElementById(id).innerHTML;
    return Mustache.to_html(text, data);
  }


  var navModel = (function () {
    //define a data model for the link data-type
    function link(id, name) {
      return {
        url: "#/" + id + "/",
        name: name,
        selected: selected
      };
    }
    function selected() {
      return this.url === location.hash;
    }
    //build an array of links
    var links = [
      link("home", "Home"),
      link("hello", "Hello World"),
      link("article", "A really interesting article")];
    return { links: links };
  }());

  var content = {
    home: 'Welcome to the home page,' +
          ' click the links to see' +
          ' templating in action.',
    hello: 'Hello World is traditionally' +
           ' displayed as text for ' +
           'people\'s first programmes' +
           ' in a new language',
    article: 'This article is fascinating,' +
             ' honestly. In fact, it\'s an' +
             ' example of meta-humour.'
  }

  function onChange() {
    nav.innerHTML = template('navTemplate', navModel);
    var id = location.hash
      .replace('#/', '')
      .replace('/', '');

    article.innerHTML = content[id] ||
                'The page you looked for was not found';
  }

  location.hash = location.hash || "#/home/";
  window.addEventListener('hashchange', onChange, false);
  onChange();

}())

The main thing to note here is that we put the template inside a script tag. The type of the script tag is important but can be anything the browser won’t interpret as code. Often you might want to set this to text/html as then more browsers will attempt to do syntax highlighting and indentation for you etc. However sometimes they will just report your template as invalid html (because it isn’t really html yet). The browser will then ignore this text and we can access it using document.getElementById(id).innerHTML. We could store the template in any number of other ways. For example, we could have chosen to have it in a separate file and use ajax to download it.

Important things not used in the example

HTML Escaping and Negative Conditional Blocks are worth familiarising yourself with. If you find however that you need more power, mustache.js also has support for comments, partials, functions (which can perform advanced stuff on blocks which is really cool but fairly complex), {{.}} for accessing the current item in an array of strings.

HTML Escaping

By default all variables are html escaped. That means that if the user types in </br> it will be converted to &lt;/br&gt; so that when rendered in the browser it looks like </br> rather than creating a new line break. You can however avoid this escaping by using the triple mustache tag. It looks like this:

<div>
{{{content}}}
</div>

Normally you want the html escaping for user entered input as it prevents them entering something unexpected like scripts.

Negative Conditional Blocks

These are called inverted sections in the Mustache documentation. They allow you to have something happen when a condition is false. So we could make our template look like:

{{#links}}
    <a href="{{url}}"
       class="{{^selected}}un{{/selected}}selected">
       {{{name}}}
    </a>
{{/links}}
{{^links}}
    <p>There are no links to display :(</p>
{{/links}}

Which would give a helpful message when there were no links and give each link either a class of selected or unselected depending on whether or not they were currently selected.

JavaScript has made an unusual choice (at least, unusual for modern languages) and remarkably, has largely stuck with it. JavaScript has only one thread. What that means, in layman’s terms, is that it can only do one thing at a time. If it’s working out the 56^th Fibonacci number, it’s not going to be doing anything else until it’s done.

How does it get away with it

When most of the work your computer is doing is computational, such as calculating Fibonacci numbers, you have a problem if you only have one thread, but when most of what you’re doing is I/O (downloading web-pages) you don’t.

It’s very common in languages that do have threads to see code that looks like this:

function downloadAndProcessPage(url) {
  var page = downloadPage(url); //blocking
  process(page);
}

Note the blocking call there. It uses up that entire thread until the web-page has finished downloading. You can get round that by just calling downloadAndProcessPage on another thread though. You might be thinking there’s no problem with that. You’re right, in principal, it works. In the past, it used to be relatively expensive to create a new thread, but it’s not really that expensive these days, so I wouldn’t worry about that. It is a horrible model of execution though.

Consider the following two threads running in parallel

var counter = 0;
spawnThread(function () {
  for (var i = 0; i < 10; i++) {
    counter = counter + 1;
  }
});
spawnThread(function () {
  for (var i = 0; i < 10; i++) {
    counter = counter + 1;
  }
});

You might think that counter would be equal to 20, and you might be wrong. Notice though that the counter is read and written in separate parts of the operation. It would be possible for both threads to read, and then both threads to write, resulting in only one increment instead of two. That means that the counter could be anything between 10 and 20. This lack of determinism is terrifying.

If you know about concurrency you might say I could lock around the counter variable, or increment it atomically. But locks come with their own problems (Dead Lock) and not every operation can be performed atomically.

OK, so if I can’t spawn a thread, what can I do? Well, you can pass a callback. Essentially you’re saying, call me back when you’re done. Once you’ve done that, you can carry on doing other things.

function downloadAndProcessPage(url, cb) {
  downloadPage(url, function (error, page) {
    if (error) return cb(error);
    process(page);
    cb();
  });
}

Note that because our function contains an async call, it must be async itself. It takes a callback as its last argument, and if something goes wrong downloading the page, it passes that error to the callback to be handled. This code assumes process never throws an exception (you probably don’t want to do that in production code).

This may look more complex than our first example, but it’s much more efficient than blocking, and it gets us out of having to worry about the enormous complexity of threading.

But the code is so ugly

Yes, the code gets a bit ugly when you use callbacks. What you need is a better version of something known as co-routines. Specifically you need an equivalent to C#’s await keyword.

public async Task downloadAndProcessPageAsync(url) {
  var page = await downloadPageAsync(url);
  process(page);
}

As you can see, we get rid of all the ugly stuff about error handling and callbacks. We just have to mark our method async, and use the await keyword whenever we call another async function.

Although this looks like synchronous blocking code, it’s not really. When you say await, you free up the thread to go and do something else. Once the download is complete, it allows your method to continue.

This looks a bit like threading, but it’s not, because you’re being explicit about when the context switch can occur. It would be a mistake to write our earlier example as:

int counter = 0;
async void run() {
  for (var i = 0; i < 10; i++) {
    var temp = counter;
    await delay(10);
    counter = temp + 1;
  }
}
void main() {
  run();
  run();
}

because the context switch occurs when you call await, leading to non-determinism. You can be sure though that without threads, context switches only occur where there’s an await, making our initial example fine.

But I’m a JavaScript Programmer

You can still have some of that magic, you just can’t have all of it yet.

The underlying thing that makes await work is tasks. In JavaScript, we call these promises. There is a basic spec to which all promises should conform so that we can start using the JavaScript equivalent of await (called yield) as soon as it comes out.

This means that in the next version of JavaScript we could write:

var downloadAndProcessPage = async(function* (url) {
  var page = yield downloadPageAsync(url);
  process(page);
})

Which looks perfect to me.

Consuming promises today

Without a library, consuming promises is still fairly simple:

var downloadAndProcessPage = async(function* (url) {
  return downloadPageAsync(url)
    .then(function (page) {
      process(page);
    });
})

A good promise implementation will then pass through any exceptions and make everything work perfectly. With a library such as Q you can do all sorts of exciting things like wait for a group of promises to all be completed.

The fact that exceptions are already passed through is reason enough to use promises today. In addition to that, it’ll make your code very easy to re-write into the form that makes use of yield

Producing promises

I’ve created a very simple promises library to help you create promises: https://github.com/ForbesLindesay/promises-a

It exports a single function promise. Say you have a low level, node.js style async API and you want to provide a nice promises abstraction:

function callAsync(arg1, arg2) {
  var def = promise();
  callNodeStyle(arg1, arg2, function (error, result) {
    if (error) def.reject(error);
    else def.fulfill(result);
  });
  return def.promise;
}

If you don’t mind the heavier weight dependencies, Q has this sort of thing built right in:

var callAsync = Q.nbind(callNodeStyle);

It’s also not difficult to write a standalone nbind function on top of promises-a:

function nbind(fn, self) {
  return function () {
    var args = Array.prototype.slice(arguments);
    var def = promise();
    args.push(function (error, result) {
      if (error) def.reject(error);
      else def.fulfill(result);
    });
    fn.apply(self || this, args);
    return def.promise;
  };
}

Conclusion

Promises are a nice way to tidy up your asynchronous code. With ES-next, it’s only going to get better and there are an increasing number of libraries out there to help you use promises.

Following on from creating a reusable component I wanted to talk about how to publish a component to the world.

A simple script

The simplest way of publishing a component, when you’re new to development, is probably just as a simple file. There are lots of places to get files hosted online either for free or paid for. staticloud makes for an excellent free option that lets you upload a complete website as a zip file, and serves it for you on the domain name of your choice. People can then download your package, host it locally and reference it as:

<script src="foo.js"></script>

Or not bother downloading it and reference it as:

<script src="http://www.yourwebsite.com/foo.js"></script>

The problem with the later approach is that if you ever stop publishing your component, they’re done for, they have no guarantee that their software will carry on working.

Package Managers

Package managers are a great idea, they essentially aim to automate the process of downloading a dependency and referencing the script. They provide one global namespace so that you wont find yourself with two conflicting packages, and because they can resolve nested dependencies, they can work with much smaller packages.

Smaller packages are a good thing, because the smaller the packages are, the less duplicated effort there is likely to be. If all I want to do is add and remove classes from an HTML element, I wouldn’t want to depend on the entirety of jQuery, so I might be tempted to write it myself. Whenever code is copied and pasted, or re-written from scratch, mistakes are made. I could instead depend on something like classes which is just 133 lines of code in total (and most of that is comments).

The big problem with package managers is that they’re rarely compatible with other package managers, and they’re really easy to write. There’s rarely much to tell between the best package manager and the worst package manager.

npm

npm is the package manager for server side JavaScript. It won the battle to be the one and only, so please don’t create another one. Server side JavaScript is nicely un-fragmented with virtually all packages in npm. If you want to create a new package to publish to npm, first install node.js then run:

npm init

You’ll get asked a series of questions about your new module, then you can run:

npm publish

and it’s that simple (if it’s your first module, you’ll also need to log in).

Packages for the client

So far, nobody’s won the client-side package manager war, so this next bit is just my recommendation. Go with component. I’m not saying that because it’s technologically superior. I’m saying that because it has the best ethos. TJ Holowaychuk says it better than me so I’ll leave it up to him to describe the details of what makes this different. I’ll just say that it comes with a focus on making the smallest possible units of functionality, and it’s got a good understanding of the need to work with many different languages. One day it will hopefully have Visual Studio plugins, an integration with Drupal, etc. etc.

For now, it just has a command line client that depends on node.js to install it, open a command window and run:

npm install -g component

You can then set up a new, empty, folder as a component by running:

component create .

To publish, you’ll need to create an account on github (which is free) and create a fresh repository for your component, so that it’s visible to the world.

Once you’ve created your component in this way, you can use component install to install any dependencies the component has and in your code, you can refer to the dependencies with require('dependency name').

You can also use component build with various options to build it for different environments, such as building a standalone script, or building a version to work with requirejs etc.

To get an idea of how the end result should look you can see the clock component published at: https://github.com/ForbesLindesay/clock

This post is really intended to follow on from Map, Filter and Sort in order to cover some more useful methods.

Reduce

Reduce is approximately equivalent to the Aggregate operation in LINQ.  It is also often called Fold in other languages.  It is useful when producing things like Sums or when concatenating a collections of strings to give you a comma separated list.

What makes the reduce function exceptionally powerful is that it can take an initial value.  If you don’t give it an initial value then it will just use the first element as the initial value and start counting  at the second.

A simplified implementation would look something like:

Array.prototype.reduce = function reduce(accumulator, start) {
  var curr, i;

  if (arguments.length == 1) {
    curr = this[0];
    i = 1; // start accumulating at the second element
  }
  else {
    curr = start;
    i = 0;
  }

  for (i = i || 0 ; i 

Although the actual implementation has some advantages such as error checking. For a demonstration of just how powerful reduce can be, consider the implementations of every and some on this page.

Every

Every is approximately equivalent to the All operation in LINQ

It checks that the condition is true for every element in a list.  If it is false for any of the elements, then every will return false.  It could be implemented using reduce as follows:

Array.prototype.every = function (fn) {
  return this.reduce(function (a, b){ return a && fn(b); },
    true);
}

The actual implementation is more efficient than that though.

Some

Some is approximately equivalent to the Any operation in LINQ.  You could implement it as follows:

Array.prototype.some = function (fn) {
  return this.filter(fn).length > 0;
}

Alternatively you could use Reduce and impliment it as:

Array.prototype.some = function (fn) {
  return this.reduce(function (a, b){ return a || fn(b); },
    false);
}

In essence it checks that at least one element in an array matches the given condition.  The actual implementation is much more efficient as it won’t copy the array and it will stop checking elements as soon as it finds one that matches the condition.

If you’ve been reading any of my tutorial series, you’ll have noted that I use the example of a map function and a filter function rather a lot.  Thus far I’ve been talking about implementing them in our custom Stream class.  The Stream class is really just a toy though, today I want to talk about the far more commonly used arrays.  Arrays represent collections of objects in JavaScript.

var numbers = [1, 3, 6, 3, 6, 7, 4, 1, 11, 13];

var items = [{ name: "bob", age: 15 },
             { name: "bill", age: 25 },
             { name: "jane", age: 16 }];

We can use for loops to easilly do something with each element in an array:

for (var i = 0; i < items.length; i++) {
 alert(items[i].name + " is " + items[i].age + " years old");
}

As a basic starting point that’s fine, and makes perfect sense, we can see what’s going on so nothing to worry about.  Next lets imagine we want a list of all the people over 18 (the legal drinking age in the UK)

var drinkers = [];
for (var i = 0; i < items.length; i++) {
 if (items[i].age >= 18) {
  drinkers.push(items[i]);
 }
}

Filter

You can still see what’s going on, but you can begin to see how the complexity is going to mount up.  Now consider that using filter:

function canDrink(item) {
 return item.age >= 18;
}
var drinkers = items.filter(canDrink);

It’s actaully a very similar amount of code, but much clearer what the intended purpose of each part is. This gives us each person who can drink as an object.

Map

Now lets imagine we just want their name. Here we use the map function.

function canDrink(item) {
 return item.age >= 18;
}
function getName(item) {
 return item.name;
}
var drinkers = items.filter(canDrink).map(getName);

Sort

Now consider the list of numbers we had. What if we wanted to get all the numbers in order. Well this time we need the sort function.

var ordered = numbers.sort();

Unfortunately, by default, sort uses alphabetical sorting, even for numbers. So this won’t give the ordering you might expect. The sort method accepts a function though for comparing two objects. To do numerical ordering, just do the following:

/**
 * Compares two numbers to produce numerical ordering not
 * lexiographical.
 * 
 * @param {number} a The first number to compare
 * @param {number} b The second number to compare
 * @return {number}  Less than 0 if a<b, equal to 0 if a=b,
 *                   greater than 0 if a>b
 */
function numericalOrdering(a, b) {
    return (a - b);
}
var ordered = numbers.sort(numericalOrdering);

That gives us our numbers in ascending order, just swap a and b if you want descending order. The comment describes how this comparison function should work in general. For example consider the following which sorts people by age.

function ageOrdering(a, b) {
    return (a.age - b.age);
}
var people = items.sort(ageOrdering);

Array has a number of other methods which I will come to in a future post on Some Every and Reduce. One final thing to mention is that thes methods wont work in older browsers (anything pre Internet Explorer 9). My recomendation though is not to support such older browsers. They have security flaws and performance issues anyway, so they really aren’t worth the trouble.

ECMA-next

The next version of JavaScript has some very exciting enhancements. You may have noticed that I created separate named functions for each of those. I did that because when you use the inline function syntax, it looks pretty ugly, and can be hard to follow. In the next version of JavaScript though, I could write:

var drinkers = items.filter(item => item.age >= 18);
var drinkers = items.filter(item => item.age >= 18)
                    .map(item => item.name);
var ordered = numbers.sort((a, b) => a - b);
var people = items.sort((a, b) => a.age - b.age);

Pretty neat!

If you enjoyed these, you might want to read about some, every and reduce - even more awesome array methods.