Daniel Wakefield

Metabase tips and tricks

2024-10-05T00:00:00+00:00

flexible date bin'ed dashboards

You want to avoid duplicating questions when your data consumers want different timeframe views over the data.

E.g Total Payments by Day, Total Payments by Month, etc

We handle this with a few tricks that we can apply easily that the data consumers understand without issue.

Helper Model

We defined a Helper Model with the time periods we offer for analytics. We're lucky that we don't need super granular stats and can just offer down to hour but you could use date_bin to support things like 15 minutes if needed.

  SELECT
    time_period
  FROM (
  VALUES
    ('hour'),
    ('day'),
    ('week'),
    ('month'),
    ('quarter'),
    ('year'),
  ) t(time_period)

Filtering the result set

In any query that requires the ability to date bin we use one of the following as part of the query definition

Use alongside date field filter

This works great for simple queries where you are filtering on a single date field as the date field filter has a bunch of useful settings like inclusive/exclusive end, ability to exclude days of the week etc that are often requested by our consumers

SELECT
  date_trunc({{time_period}}, created_at)::date AS time_period,
  SUM(amount)
FROM payments
WHERE
  1=1
  [[AND {{payment_created_at_field_filter}}]]
GROUP BY time_period

When you can't use the field filter

Metabase doesnt support date field filters on columns inside CTE's, and there are times where you want to filter multiple tables to the same time range.

In those cases we add a number of periods filter that defines the how far to look back over instead

SELECT
  date_trunc({{time_period}}, created_at)::date as time_period,
  sum(amount)
FROM payments
JOIN users on payments.user_id = users.id
WHERE
  payments.created_at >= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
  AND
  users.last_sign_in_at >= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
GROUP BY time_period

Caveat 1: Quarter

date_trunc works with quarter but interval doesnt. It would be a lovely addition¹ but for now we add a case statement that we're happy with if the question actually requires this size of window. This is how you would handle using date_bin('15 minutes', ...) as well, you'd have to do the math in order to move back over the right number of the periods.

payments.created_at >= date_trunc(
  {{time_period}},
  CASE {{time_period}}
    WHEN 'quarter' THEN
      date_trunc('quarter', NOW()) - ((({{num_periods}})::integer * 3) || ' month')::interval
    ELSE
      NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval
  END
)

Caveat 2: Incomplete periods

We include data from the incomplete current period but we can change this by adding a less than version to complement each filter

SELECT
  date_trunc({{time_period}}, created_at)::date AS time_period,
  SUM(amount)
FROM payments
  payments.created_at >= date_trunc({{time_period}}, NOW() - ({{num_periods}} || ' ' || {{time_period}})::interval)
  AND payments.created_at < date_trunc({{time_period}}, NOW())

The majority of our data analysis is done on historic data, but of course you can flip the signs, or add both complements so the query both before and after NOW() if your data has a more leading than lagging profile.

Optional targets

You'll probably have a tracking board, daily or monthly that the leadership team love to watch. They will also want to compare past periods the current one without going to a new dashboard. Combining COALESCE with optional filters and you can inject SQL defaults.

This gets around the fact that raw date filters can't be set to today.

Something like this works

with filtered_payments as (/* some query */)

SELECT
  COUNT(*)
FROM filtered_payments
WHERE
  date_trunc('day', filtered_payments.created_at) = date_trunc('day', COALESCE( [[{{date_to_look_at}} ,]] NOW() ))

Percent function

To remove some error prone casting and coalesce'ing related to calculating percentages, especially over counts, we added a function to make it more readable

CREATE OR REPLACE FUNCTION public.percent(portion numeric, total numeric, round_level integer DEFAULT 1)
  RETURNS numeric
  LANGUAGE sql
  IMMUTABLE PARALLEL SAFE STRICT
AS $function$
    select
        CASE
        WHEN total = 0 THEN 0
        ELSE ROUND(100 * (portion / total), round_level)
        END
$function$

It replaces the following, while also handling counts that return 0

SELECT
  ROUND(100 * cast(count(*) FILTER (where amount > 100) as float)/cast(count(*) as float)) AS old_big_payment_rate,
  percent(
    COUNT(*) FILTER (where amount > 100),
    COUNT(*)
  ) AS new_big_payment_rate
FROM payments

Commonly accessed questions

We do periodic cleaning of the available dashboards; we'll archive things that havent been accessed in the last 6-12 months to keep the number of questions and dashboards down. The below will give you the commonly accessed queries, flipping or changing the order to last_execution will show you the ones that aren't often acessed. Use an Anti-Join to find the reports that haven't been run at all in the period you're checking.

with card_data as (
  select
    card_id,
    COUNT(*) as execution_count,
    MAX(started_at) as last_execution
  from query_execution
  where
    started_at > NOW() - '12 months'::interval
  GROUP BY 1
)

select
  report_card.id,
  report_card.name,
  CONCAT('https://data.example.com/questions/', report_card.id),
  card_data.execution_count,
  card_data.last_execution
from report_card
join card_data on card_data.card_id = report_card.id
where
  report_card.archived = false
ORDER BY card_data.execution_count DESC

Wish: support "quarter" in Interval

Using certificate based ssh authentication

2014-09-20T00:00:00+00:00

What is a Certificate?

A certificate is a form of Public Key Cryptography that allows you to trust someone.

The certificate authority's (CA) private key is used to sign the public key of anyone or anything that is trusted by the CA.

Anybody can then decide to trust the same people as the CA by adding the CA's public key to their keyring.

Why should I use one?

One of the main reasons is that CA signing works in both directions.

A user can check that a machine is trusted instead of relying on adding individual hosts to ~/.ssh/known_hosts.

This is useful if you are using the same IP address but you are recreating a machine for testing purposes and you don't want to be bugged by SSH key checking messages.

A machine can check that a user is trusted instead of relying on adding user's public key's to ~/.ssh/authorized_keys for each user that should be allowed to login.

Signing keys also allows very fine grained control over many aspects that key based authentication doesn't.

The reason I have switched from key based to cert based authentication is that it is much easier to handle access controls access your personal machines when you are regularly creating new VM's and VPS's.

You only need to add the CA's public key to a server rather than all of your user's public keys to the correct ~/.ssh/authorized_keys file.

Setting it up.

For this you will need:

2+ machines running your preferred form of Linux (I used Debian 7)
Openssh 5.4+
Basic knowledge of the command line

In this example I will be using the machine I am writing this on as both the CA and the user I want to trust. This isn't totally secure but will work perfectly fine with a small number of users.

First we will create a user CA which will allow the users to sign into servers without a prompt.

The first step is to create your CA's keypair.

$ mkdir -p ~/.ssh/ca
$ cd ~/.ssh/ca
$ ssh-keygen -f ca -C "Comment on the keys"
    Generating public/private rsa key pair.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in ca.
    Your public key has been saved in ca.pub.
$ ls
    ca  ca.pub

You should provide a strong passphrase when asked as this will mean that if attacker gains access to your keys your security is not compromised.

The next step is to place the public key on the machine(s) that should trust the users.

$ scp ca.pub [email protected]:/home/user
$ ssh [email protected] "sudo cp ~/ca.pub /etc/ssh"
$ ssh [email protected] "sudo sed -i '/TrustedUserCAKeys/d' /etc/ssh/sshd_config"
$ ssh [email protected] "echo 'TrustedUserCAKeys /etc/ssh/ca.pub' | sudo tee -a /etc/ssh/sshd_config"
$ ssh [email protected] "sudo /etc/init.d/ssh restart"
 # The above will only work on a machine set up with passwordless sudo
 # Otherwise ssh to the machine and run the commands manually

This sets up the server to trust signed keys from the CA, so the next step is to sign our users public key.

If you already have a set of user keys you should skip the first step.

$ ssh-keygen
    Generating public/private rsa key pair.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in ~/.ssh/id_rsa.
    Your public key has been saved in ~/.ssh/id_rsa.pub
    # This generates a keypair with default settings
    # Like with the CA keypair you should use a strong password for
    # added security.
$ mkdir signed_users
$ cd signed_users
$ cp ~/.ssh/id_rsa.pub user.pub
$ ssh-keygen -s ~/.ssh/ca/ca -I "Comment" -n "user" user.pub
    Enter passphrase:
    # Enter the passphrase you used to create the ca keypair
    # You can add more than one user by comma separating them
    # You will only be allowed to login as the users specified
    # in this list and it is required
$ chmod 600 user-cert.pub
    # Set the correct permissions on your signed key
$ ls
    user.pub    user-cert.pub
$ cp user-cert.pub ~/.ssh/id_rsa-cert.pub

You should keep a copy of user.pub safe because if you ever want to revoke a users trusted status you need to know their public key.

You can easily repeat this for each machine you use regularly.

If you use an automation tool to set up new machines it is simple to set up a task that will create and sign a keypair allowing you to log in to any of your machines from it straight away.

There are many other options you can add to the signed key to restrict and secure the user further.

Some examples are:

Setting an expiration on the signed key
Only allowing the key to be used from certain address's
Forcing a specific command to be run instead of a shell when using the certificate

Revoking users

If you lose access to a set of user keys it is fairly easy to revoke their access to your machines.

$ ssh [email protected] "[ -f /etc/ssh/ssh_revoked_keys ] || sudo install -m 644 -o root -g root /etc/ssh/ssh_revoked_keys"
$ cat user_to_revoke.pub | ssh [email protected] "sudo tee -a /etc/ssh/ssh_revoked_keys"
$ ssh [email protected] "sudo sed -i '/RevokedKeys/d' /etc/ssh/sshd_config"
$ ssh [email protected] "echo 'RevokedKeys /etc/ssh/ssh_revoked_keys' | sudo tee -a /etc/ssh/sshd_config"
$ ssh [email protected] "sudo /etc/init.d/ssh restart"
    # Again this requires passwordless sudo

This does have to be run on each of your machines but it is still less work than combing through authorized_keys files for each user account on every machine you have.

Final notes.

Keep you CA private key safe.
This can be used to sign new keys that can then access your machines.
Use passphrase's and a SSH agent.
These improve security so that an attacker must have access to a key and know the passphrase before using it. Using an SSH agent means you only have to input your passphrase once across machine reboots.
You should try to follow the principle of least privilege for user accounts and give each one their own signed key.