A toolkit of utilities used across all the AdonisJS, Edge, and Japa packages

Many of my open source projects (including AdonisJS) use many single-purpose utility packages from npm. Over the years, I have faced the following challenges when using these packages.

• It takes a lot of time to find a perfect package for the use case. The package should be well maintained, have good test coverage, and not accumulate debt by supporting some old versions of Node.js.
• Some packages are great, but they end up pulling a lot of unnecessary dependencies like (requiring TypeScript as a prod dependency)
• Sometimes I end up using different packages for the same utility (because, I cannot remember what I used last time in that other package). So I want to spend time once choosing the one I need and then bundle it inside @poppinss/utils.
• Some authors introduce breaking changes too often (not a criticism). Therefore, I prefer wrapping their packages with my external API only to absorb breaking changes in one place.
• Rest are some handwritten utilities to fit my needs

Note: If you are creating an AdonisJS package, I highly recommend using this package since it is already part of the user's project dependencies.

Warning: This package is not for general use (outside the AdonisJS ecosystem). I will not add new helpers or remove any to cater to a broader audience.

A note to self and others to consider the following packages.

Even though I do not care much about the package size (most of work is consumed on server side), I am mindful around the utilities and ensure not end up using really big packages for smaller use-cases.

Here's the last checked install size of this package.

Install the package from the npm registry as follows:

npm i @poppinss/utils

# Yarn lovers
yarn add @poppinss/utils

Following are the exported modules. Only the generic helpers are shipped from the main path. The rest of the helpers are grouped inside sub-modules.

// string sub-module
import string from '@poppinss/utils/string'

// string builder
import string from '@poppinss/utils/string_builder'

// json sub-module
import json from '@poppinss/utils/json'

// lodash sub-module
import lodash from '@poppinss/utils/lodash'

// assert sub-module
import assert from '@poppinss/utils/assert'

// main module
import { base64, fsReadAll } from '@poppinss/utils'

// types sub-module
import { ReadAllFilesOptions } from '@poppinss/utils/types'

A collection of helpers to perform operations on/related to a string value.

import string from '@poppinss/utils/string'

Generate an excerpt from a string value. If the input value contains HTML tags, we will remove them from the excerpt.

const html = `<p>AdonisJS is a Node.js framework, and hence it requires Node.js to be installed on your computer. To be precise, we need at least the latest release of <code>Node.js v14</code>.</p>`

console.log(string.excerpt(html, 70))
// AdonisJS is a Node.js framework, and hence it requires Node.js to be i...

Truncate a string value to a certain length. The method is the same as the excerpt method but does not remove any HTML tags. It is a great fit when you are truncating a non-HTML string.

const text = `AdonisJS is a Node.js framework, and hence it requires Node.js to be installed on your computer. To be precise, we need at least the latest release of Node.js 14.`

console.log(string.truncate(text, 70))
// AdonisJS is a Node.js framework, and hence it requires Node.js to be i...

Generate slug for a string value. The method is exported directly from the slugify package.

Please check the package documentation for available options.

console.log(string.slug('hello ♥ world'))
// hello-love-world

You can add custom replacements for Unicode values as follows.

string.slug.extend({ '☢': 'radioactive' })

console.log(string.slug('unicode ♥ is ☢'))
// unicode-love-is-radioactive

Interpolate variables inside a string. The variables must be inside double curly braces.

string.interpolate('hello {{ user.username }}', { user: { username: 'virk' } })

// hello virk

You can also replace array values by mentioning the array index.

string.interpolate('hello {{ users.0 }}', { users: ['virk'] })

// hello virk

You can escape the curly braces by prefixing them with \\.

string.interpolate('hello \\{{ users.0 }}', {})

// hello {{ users.0 }}

Convert a word to its plural form. The method is exported directly from the pluralize package.

string.plural('test')
// tests

Convert a word to its singular form. The method is exported directly from the pluralize package.

string.singular('tests')
// test

This method combines the singular and plural methods and uses one or the other based on the count. For example:

string.pluralize('box', 1) // box
string.pluralize('box', 2) // boxes
string.pluralize('box', 0) // boxes

string.pluralize('boxes', 1) // box
string.pluralize('boxes', 2) // boxes
string.pluralize('boxes', 0) // boxes

The addPluralRule, addSingularRule, addIrregularRule, and addUncountableRule methods exposed by the pluralize package can be called as follows.

string.pluralize.addUncountableRule('paper')
string.pluralize.addSingularRule(/singles$/i, 'singular')

Find if a word is already in plural form. The method is exported directly from the pluralize package.

string.isPlural('tests') // true

Find if a word is already in a singular form. The method is exported directly from the pluralize package.

string.isSingular('test') // true

Convert a string value to camelcase.

string.camelCase('user_name') // userName

Following are some of the conversion examples.

Convert a string value to a capital case.

string.capitalCase('helloWorld') // Hello World

Following are some of the conversion examples.

Convert a string value to a dash case.

string.dashCase('helloWorld') // hello-world

Optionally, you can capitalize the first letter of each word.

string.dashCase('helloWorld', { capitalize: true }) // Hello-World

Following are some of the conversion examples.

Convert a string value to a dot case.

string.dotCase('helloWorld') // hello.World

Optionally, you can also convert the first letter of all the words to lowercase.

string.dotCase('helloWorld', { lowerCase: true }) // hello.world

Following are some of the conversion examples.

Remove all sorts of casing from a string value.

string.noCase('helloWorld') // hello world

Following are some of the conversion examples.

Convert a string value to pascal case. Great for generating JavaScript class names.

string.pascalCase('user team') // UserTeam

Following are some of the conversion examples.

Convert a value to a sentence.

string.sentenceCase('getting-started-with-adonisjs')
// Getting started with adonisjs

Following are some of the conversion examples.

Convert value to snake case.

string.snakeCase('user team') // user_team

Following are some of the conversion examples.

Convert a string value to title case.

string.titleCase('small word ends on')
// Small Word Ends On

Following are some of the conversion examples.

Generate a cryptographically secure random string of a given length. The output value is URL safe base64 encoded string.

string.random(32)
// 8mejfWWbXbry8Rh7u8MW3o-6dxd80Thk

Convert an array of words to a comma-separated sentence.

string.toSentence(['routes', 'controllers', 'middleware'])
// routes, controllers, and middleware

You can replace the and with an or by specifying the options.lastSeparator property.

string.toSentence(['routes', 'controllers', 'middleware'], {
  lastSeparator: ', or ',
})

In the following example, the two words are combined using the and separator, not the comma (usually advocated in English). However, you can use a custom separator for a pair of words.

string.toSentence(['routes', 'controllers'])
// routes and controllers

string.toSentence(['routes', 'controllers'], {
  pairSeparator: ', and ',
})
// routes, and controllers

Remove multiple whitespaces from a string to a single whitespace.

string.condenseWhitespace('hello  world')
// hello world

string.condenseWhitespace('  hello  world  ')
// hello world

Get the ordinal letter for a given number.

string.ordinal(1) // 1st
string.ordinal(2) // '2nd'
string.ordinal(3) // '3rd'
string.ordinal(4) // '4th'

string.ordinal(23) // '23rd'
string.ordinal(24) // '24th'

Parse a string-based time expression to seconds.

string.seconds.parse('10h') // 36000
string.seconds.parse('1 day') // 86400

Passing a numeric value to the parse method is returned as it is, assuming the value is already in seconds.

string.seconds.parse(180) // 180

You can format seconds to a pretty string using the format method.

string.seconds.format(36000) // 10h
string.seconds.format(36000, true) // 10 hours

Parse a string-based time expression to milliseconds.

string.milliseconds.parse('1 h') // 3.6e6
string.milliseconds.parse('1 day') // 8.64e7

Passing a numeric value to the parse method is returned as it is, assuming the value is already in milliseconds.

string.milliseconds.parse(180) // 180

Using the format method, you can format milliseconds to a pretty string.

string.seconds.format(3.6e6) // 1h
string.seconds.format(3.6e6, true) // 1 hour

Parse a string-based unit expression to bytes.

string.bytes.parse('1KB') // 1024
string.bytes.parse('1MB') // 1048576

Passing a numeric value to the parse method is returned as it is, assuming the value is already in bytes.

string.bytes.parse(1024) // 1024

Using the format method, you can format bytes to a pretty string. The method is exported directly from the bytes package. Please reference the package README for available options.

string.bytes.format(1048576) // 1MB
string.bytes.format(1024 * 1024 * 1000) // 1000MB
string.bytes.format(1024 * 1024 * 1000, { thousandsSeparator: ',' }) // 1,000MB

The string builder offers a fluent API for applying a set of transforms on a string value. You can create an instance of the string builder as follows.

import StringBuilder from '@poppinss/utils/string_builder'
const builder = new StringBuilder('hello world')

const value = builder.snakeCase().suffix('_controller').toString()
assert(value === 'hello_world_controller')

Following are the helpers we use to stringify and parse JSON.

The native implementation of JSON.parse opens up the possibility for prototype poisoning. The safeParse method removes the __proto__ and the constructor.prototype properties from the JSON string at the time of parsing it.

The method is a wrapper over secure-json-parse package.

The native implementation of JSON.stringify cannot handle circular references or language-specific data types like BigInt.

Therefore, we use the safe-stable-stringify package under the hood to overcome the limitations of native implementation.

import { safeStringify } from '@poppinss/utils/json'

const value = {
  b: 2,
  c: BigInt(10),
}

// Circular reference
value.a = value

safeStringify(value)
// '{"b":2,"c":10}'

• The circular references are removed from the final JSON string.
• The BigInt values are converted to a string.

The safeStringify API is the same as the JSON.stringify method.

• You can pass a replacer function as the second parameter.
• And number of spaces as the third parameter.

Lodash is quite a big library, and we do not use all its helper methods. Therefore we create a custom build using the lodash CLI and bundle only the once we need.

Why not use something else: All other helpers I have used are not as accurate or well implemented as lodash.

• pick
• omit
• has
• get
• set
• unset
• mergeWith
• merge
• size
• clone
• cloneDeep
• toPath

You can use the methods as follows.

import lodash from '@poppinss/utils/lodash'

lodash.pick(collection, keys)

The following assertion methods offers type-safe approach for writing conditionals and throwing error when the variable has unexpected values.

Throws AssertionError when the value is false, null, or undefined.

import { assertExists } from '@poppinss/utils/assert'

const value = false as string | false
assertExists(value)

// value is string

Throws AssertionError when the value is null.

import { assertNotNull } from '@poppinss/utils/assert'

const value = null as string | null
assertNotNull(value)

// value is string

Throws AssertionError when the value is undefined.

import { assertIsDefined } from '@poppinss/utils/assert'

const value = undefined as string | undefined
assertIsDefined(value)

// value is string

Throws AssertionError when the method is invoked. In other words, this method always throws an exception.

import { assertUnreachable } from '@poppinss/utils/assert'
assertUnreachable()

The following helpers are exported from the package main module.

import { base64, compose } from '@poppinss/utils'

Utility methods to base64 encode and decode values.

import { base64 } from '@poppinss/utils'

base64.encode('hello world')
// aGVsbG8gd29ybGQ=

Similar to the encode method, you can use the urlEncode to generate a base64 string safe to pass in a URL.

The urlEncode method performs the following replacements.

• Replace + with -.
• Replace / with _.
• And remove the = sign from the end of the string.

base64.urlEncode('hello world')
// aGVsbG8gd29ybGQ

You can use the decode and the urlDecode methods to decode a previously encoded base64 string.

base64.decode(base64.encode('hello world'))
// hello world

base64.urlDecode(base64.urlEncode('hello world'))
// hello world

The decode and the urlDecode methods return null when the input value is an invalid base64 string. You can turn on the strict mode to raise an exception instead.

base64.decode('hello world') // null
base64.decode('hello world', 'utf-8', true) // raises exception

The compose helper allows you to use TypeScript class mixins with a cleaner API. Following is an example of mixins usage without the compose helper.

class User extends UserWithAttributes(UserWithAge(UserWithPassword(UserWithEmail(BaseModel)))) {}

Following is an example with the compose helper.

• There is no nesting.
• The order of mixins is from left to right. Whereas earlier, it was inside out.

import { compose } from '@poppinss/utils'

class User extends compose(
  BaseModel,
  UserWithEmail,
  UserWithPassword,
  UserWithAge,
  UserWithAttributes
) {}

The defineStaticProperty method allows you to define static properties on a class with different reference strategies.

If you use class inheritance alongside static properties, then either, you will share properties by reference, or you will define them directly on the parent class.

In the following example, we are not inherting columns from the AppModel. Instead, we define a new set of columns on the UserModel.

class AppModel {
  static columns = ['id']
}

class UserModel extends AppModel {
  static columns = ['username']
}

In the following example, we are inherting columns from the AppModel. However, the mutations (array.push) from the UserModel will reflect on the AppModel as well.

class AppModel {
  static columns = ['id']
}

class UserModel extends AppModel {}
UserModel.columns.push('username')

The ideal behavior is to deep clone the columns array and then push new values to it.

import lodash from '@poppinss/utils/lodash'

class AppModel {
  static columns = ['id']
}

const inheritedColumns = lodash.cloneDeep(AppModel.columns)
class UserModel extends AppModel {
  static columns = inheritedColumns.push('username')
}

The defineStaticProperty method abstracts the logic to clone and also performs some interal checks to see if the value is already defined as an ownProperty or not.

class UserModel extends AppModel {}

defineStaticProperty(UserModel, 'columns', {
  strategy: 'inherit',
  initialValue: [],
})

• The inherit strategy clones the value from the parent class.
• The define strategy always re-defines the property, discarding any values on the parent class.
• The strategy value can be function to perform a custom clone operations.

A custom exception class with support for defining the error status, error code, and help description. This class aims to standardize exceptions within your projects.

import { Exception } from '@poppinss/utils/exception'

class ResourceNotFound extends Exception {
  static code = 'E_RESOURCE_NOT_FOUND'
  static status = 404
  static message = 'Unable to find resource'
}

throw new ResourceNotFound()

You can also create an anonymous exception class using the createError method. The return value is a class constructor that accepts an array of values to use for interpolation.

The interpolation of error message is performed using the util.format message.

import { createError } from '@poppinss/utils/exception'
const E_RESOURCE_NOT_FOUND = createError(
  'Unable to find resource with id %d',
  'E_RESOURCE_NOT_FOUND'
)

const id = 1
throw new E_RESOURCE_NOT_FOUND([id])

Create a flat object from a nested object/array. The nested keys are combined with a dot-notation (.). The method is exported from the flattie package.

import { flatten } from '@poppinss/utils'

flatten({
  a: 'hi',
  b: {
    a: null,
    b: ['foo', '', null, 'bar'],
    d: 'hello',
    e: {
      a: 'yo',
      b: undefined,
      c: 'sup',
      d: 0,
      f: [
        { foo: 123, bar: 123 },
        { foo: 465, bar: 456 },
      ],
    },
  },
  c: 'world',
})

// {
//   'a': 'hi',
//   'b.b.0': 'foo',
//   'b.b.1': '',
//   'b.b.3': 'bar',
//   'b.d': 'hello',
//   'b.e.a': 'yo',
//   'b.e.c': 'sup',
//   'b.e.d': 0,
//   'b.e.f.0.foo': 123,
//   'b.e.f.0.bar': 123,
//   'b.e.f.1.foo': 465,
//   'b.e.f.1.bar': 456,
//   'c': 'world'
// }

Get a list of all the files from a directory. The method recursively fetches files from the main and the sub-folders. The dotfiles are ignored implicitly.

import { fsReadAll } from '@poppinss/utils'

const files = await fsReadAll(new URL('./config', import.meta.url), { pathType: 'url' })
await Promise.all(files.map((file) => import(file)))

You can also pass the options along with the directory path as the second argument.

type Options = {
  ignoreMissingRoot?: boolean
  filter?: (filePath: string, index: number) => boolean
  sort?: (current: string, next: string) => number
  pathType?: 'relative' | 'unixRelative' | 'absolute' | 'unixAbsolute' | 'url'
}

const options: Partial<Options> = {}
await fsReadAll(location, options)

The fsImportAll method imports all the files recursively from a given directory and set the exported value from each module on an object.

import { fsImportAll } from '@poppinss/utils'

const collection = await fsImportAll(new URL('./config', import.meta.url))
console.log(collection)

• Collection is an object with a tree of key-value pair.
• The key is the nested object created from the file path.
• Value is the exported values from the module. If a module exports both the default and named values, then only the default values are used.

The second param is the options to customize the import behavior.

A filter to know if the file path ends with .js, .json, .cjs, .mjs or .ts. In the case of .ts files, the .d.ts returns false.

import { isScriptFile } from '@poppinss/utils'

isScriptFile('foo.js') // true
isScriptFile('foo/bar.cjs') // true
isScriptFile('foo/bar.mjs') // true
isScriptFile('foo.json') // true

isScriptFile('foo/bar.ts') // true
isScriptFile('foo/bar.d.ts') // false

The goal of this method is to use it as a filter with the fsReadAll method.

import { fsReadAll, isScriptFile } from '@poppinss/utils'

const dir = new URL('./config', import.meta.url)
const options = { pathType: 'url', filter: isScriptFile }

const files = await fsReadAll(dir, options)

await Promise.all(
  files.map((file) => {
    if (file.endsWith('.json')) {
      return import(file, { with: { type: 'json' } })
    }

    return import(file)
  })
)

A helper function that assert a lazy import function output to have a default export, otherwise raises an exception.

We use dynamic default exports a lot in AdonisJS apps, so extracting the check to a helper function.

import { importDefault } from '@poppinss/utils'
const defaultVal = await importDefault(() => import('./some_module.js'))

A sorting function to use natural sort for ordering an array.

import { naturalSort } from '@poppinss/utils'

const values = ['1_foo_bar', '12_foo_bar'].sort()
// Default sorting: ['12_foo_bar', '1_foo_bar']

const values = ['1_foo_bar', '12_foo_bar'].sort(naturalSort)
// Default sorting: ['1_foo_bar', '12_foo_bar']

Check if two buffer or string values are the same. This method does not leak any timing information and prevents timing attack.

Under the hood, this method uses Node.js crypto.timeSafeEqual method, with support for comparing string values. (crypto.timeSafeEqual does not support string comparison)

import { safeEqual } from '@poppinss/utils'

/**
 * The trusted value, it might be saved inside the db
 */
const trustedValue = 'hello world'

/**
 * Untrusted user input
 */
const userInput = 'hello'

if (safeEqual(trustedValue, userInput)) {
  // both are same
} else {
  // value mis-match
}

Convert OS-specific file paths to Unix file paths. The method is exported directly from the slash package.

import { slash } from '@poppinss/utils/slash'
slash('foo\\bar') // foo/bar

Message builder is a convenience layer to stringify JavaScript values with an expiry date and a purpose. For example:

import { MessageBuilder } from '@poppinss/utils'

const builder = new MessageBuilder()
const encoded = builder.build(
  {
    token: string.random(32),
  },
  '1 hour',
  'email_verification'
)

/**
 * {
 *   "message": {
 *    "token":"GZhbeG5TvgA-7JCg5y4wOBB1qHIRtX6q"
 *   },
 *   "purpose":"email_verification",
 *   "expiryDate":"2022-10-03T04:07:13.860Z"
 * }
 */

Once you have the JSON string with the expiry and the purpose, you can encrypt it (to prevent tampering) and share it with the client.

During the email verification, you can decrypt the key and then ask the MessageBuilder to verify the payload.

const decoded = builder.verify(value, 'email_verification')
if (!decoded) {
  return 'Invalid token'
}

console.log(decoded.token)

Now let's imagine someone presents the same token to reset their account password. In the following example, the validation will fail since the purpose of the original token is not the same as the purpose set during the verify method call.

const decoded = builder.verify(value, 'reset_password')

The ObjectBuilder is a convenience class to create an object with dynamic properties. Consider the following example, where we wrap our code inside conditionals before adding the property b to the startingObject.

const startingObject = {
  a: 1
  // Add "b", if it exists
  ...(b ? { b } : {})
}

// OR
if (b) {
  startingObject.b = b
}

Instead of writing conditionals, you can consider using the Object builder fluent API.

const builder = new ObjectBuilder({ a: 1 })

const plainObject = builder.add('b', b).toObject()

By default, only the undefined values are ignored. However, you can also ignore null values.

const ignoreNullValues = true
const builder = new ObjectBuilder({ a: 1 }, ignoreNullValues)

Following are the available methods on the ObjectBuilder class.

builder.remove(key)
builder.has(key)
builder.get(key)
builder.add(key)

builder.toObject() // get plain object

Creates a secret value that prevents itself from getting logged inside console.log statements, during JSON serialization, and string concatenation.

To understand why you need a special Secret object, you need to understand the root of the problem. Let's start with an example.

Given that you have a Token class that generates an opaque token for a user and persists its hash inside the database. The plain token (aka raw value) is shared with the user and it should only be visible once (for security reasons). Here is a dummy implementation of the same.

class Token {
  generate() {
    return {
      value: 'opaque_raw_token',
      hash: 'hash_of_raw_token_inside_db',
    }
  }
}

const token = new Token().generate()
return response.send(token)

At the same time, you want to drop a log statement inside your application that you can later use to debug the flow of the application, and this is how you log the token.

const token = new Token().generate()

logger.log('token generated %O', token)
// token generated {"value":"opaque_raw_token","hash":"hash_of_raw_token_inside_db"}

return response.send(token)

BOOM! You have weakened the security of your app. Now, anyone monitoring the logs can grab raw token values from the log and use them to perform the actions on behalf of the user.

Now, to prevent this from happening, you should work with a branded data type. Our old friend PHP has it, so we need it as well.

This is what exactly the Secret utility class does for you. Create values that prevent themselves from leaking inside logs or during JSON serialization.

import { Secret } from '@poppinss/utils'

class Token {
  generate() {
    return {
      // THIS LINE 👇
      value: new Secret('opaque_raw_token'),
      hash: 'hash_of_raw_token_inside_db',
    }
  }
}

const token = new Token().generate()

logger.log('token generated %O', token)
// AND THIS LOG 👇
// token generated {"value":"[redacted]","hash":"hash_of_raw_token_inside_db"}

return response.send(token)

Need the original value back? You can call the release method to get the secret value back. Again, the idea is not to prevent your code from accessing the raw value. It's to stop the logging and serialization layer from reading it.

const secret = new Secret('opaque_raw_token')
const rawValue = secret.release()

rawValue === opaque_raw_token // true

Shoutout to [https://transcend.io/blog/keep-sensitive-values-out-of-your-logs-with-types](transcend.io's article) to helping me design the API. In fact, I have ripped their implementation for my personal use.

ES modules does not have magic variables __filename and __dirname. You can use these helpers to get the current directory and filenames as follows.

import { getDirname, getFilename } from '@poppinss/utils'

const dirname = getDirname(import.meta.url)
const filename = getFilename(import.meta.url)

Similar to the Node.js path.join, but instead expects the first parameter to be a URL instance or a string with the file:/// protocol.

The return value is an absolute file system path without the file:/// protocol.

import { joinToURL } from '@poppinss/utils'

// With URL as a string
const APP_PATH = joinToURL(import.meta.url, 'app')

// With URL instance
const APP_PATH = joinToURL(new URL('./', import.meta.url), 'app')